🔗 Source: arXiv

Skip the Good Part: Representation Structure & Inference-Time Layer Skipping in Diffusion vs. Autoregressive LLMs

🚀 Technical Novelty

  • Mechanism: Static, task-agnostic inference-time layer skipping that bypasses high-similarity early layers in native diffusion models, leveraging coarse-to-fine abstraction redundancy without KV-cache sharing or architectural modifications.
  • Nuance: Unlike cache-centric or parameter-tied efficiency methods, this exploits objective-induced representational structure unique to native dLLMs, revealing that AR-initialized models retain brittle, non-redundant dynamics despite diffusion training.

💡 Yield

  • Native dLLMs tolerate skipping up to 6 layers (18.75% FLOPs reduction) while retaining >90% performance on reasoning and code benchmarks, whereas native AR and AR-initialized models degrade sharply with minimal skipping.
  • First systematic layer/token-wise representational analysis demonstrating diffusion objectives induce hierarchical abstraction with early-layer redundancy and reduced recency bias compared to AR’s incremental refinement.

⚠️ Limitations

  • Static skip policy lacks task-awareness or dynamic adaptation, potentially underperforming on out-of-distribution or safety-critical applications where skipped computations might affect nuanced behaviors.
  • Evaluation limited to text-only reasoning and code synthesis; extension to multimodal diffusion architectures remains unexplored.