Next Implicit Token Prediction

🔗 Source: arXiv

NITP: Next Implicit Token Prediction for LLM Pre-training

🚀 Technical Novelty

• Mechanism: Predicts the implicit semantic content of the next token by aligning deep-layer hidden states with temporally shifted, stop-gradient shallow-layer representations via a cosine similarity loss.
• Nuance: Unlike standard NTP or static layer-wise distillation, NITP enforces autoregressive temporal prediction in continuous space, explicitly regularizing the optimization landscape to counteract anisotropic geometric collapse without external encoders or heavy compute.

💡 Yield

• Theoretically proves NTP leaves latent degrees of freedom under-constrained, causing representation degeneration; empirically shows consistent downstream gains (e.g., +5.7% on MMLU-Pro for 9B MoE) with only ~2% additional training FLOPs and zero inference cost.

⚠️ Limitations

• Relies heavily on shallow-layer representations preserving lexical/local semantics, which may degrade for tasks requiring deep contextual abstraction; performance is sensitive to the auxiliary loss weight λ across different model scales.