🔗 Source: arXiv

Functional Attention: From Pairwise Affinities to Functional Correspondences

🚀 Technical Novelty

  • Mechanism: Replaces quadratic softmax pairwise affinities with optimal linear solves in a learned spectral space, treating attention as a functional correspondence between adaptive bases.
  • Nuance: Differs from prior SOTA by explicitly decoupling function representation from basis learning via feed-forward networks and solving for coefficients rather than approximating dense token-wise attention matrices.

💡 Yield

  • Achieves state-of-the-art accuracy on PDE solvers, 3D segmentation, and regression tasks while demonstrating strong zero-shot super-resolution and out-of-distribution generalization on Airfoil RANS data.
  • Proves Lipschitz continuity of the operator to establish theoretical stability under input perturbations.

⚠️ Limitations

  • Relies on a simple softmax projection for basis learning, limiting expressiveness compared to more structured designs.
  • Lacks rigorous approximation guarantees and formal generalization bounds; the relationship between compression ratio and error remains unproven.
  • Computational overhead increases with larger basis counts, requiring careful hyperparameter tuning for high-frequency fields.