🔗 Source: arXiv

Latent-Condensed Transformer for Efficient Long Context Modeling

🚀 Technical Novelty

  • Mechanism: Directly condenses redundant context within MLA’s low-dimensional latent space using query-aware weighted pooling for semantic vectors and hard anchor selection for positional keys.
  • Nuance: Unlike prior sparse attention methods that require expensive full-dimensional KV reconstruction before sparsification, LCA operates natively on compressed latent codes to jointly optimize memory and compute without additional parameters.

💡 Yield

  • Theoretically proves a length-independent error bound for the approximation; empirically delivers up to 2.5× prefilling speedup and 90% KV cache reduction at 128K context while maintaining competitive accuracy across long-context benchmarks.

⚠️ Limitations

  • Requires custom Triton kernel implementation for optimal efficiency, limiting out-of-the-box framework compatibility; exhibits modest accuracy degradation in tasks demanding precise token-level retrieval under aggressive condensation and lacks extensive testing on lower-precision formats (e.g., int8).