🔗 Source: arXiv

Latent-Condensed Transformer for Efficient Long Context Modeling

🚀 Technical Novelty

  • Mechanism: Directly condenses redundant context within MLA’s disentangled latent space using query-aware weighted pooling for semantic vectors and hard anchor selection for positional keys.
  • Nuance: Unlike prior sparse methods that require costly full-dimensional KV reconstruction, LCA operates natively on compressed latents to jointly optimize memory and compute while remaining architecture-agnostic (extends seamlessly to GQA).

💡 Yield

  • Proves a length-independent theoretical error bound for the approximation; achieves up to 2.5× prefilling speedup and 90% KV cache reduction at 128K context with negligible performance degradation on standard long/short-context benchmarks.

⚠️ Limitations

  • Requires custom Triton kernels to realize full efficiency gains, limiting out-of-the-box framework compatibility; exhibits modest accuracy drops in tasks demanding precise token-level retrieval under aggressive condensation; not extensively validated on lower-precision formats (e.g., int8).