🔗 Source: arXiv

Functional Attention: From Pairwise Affinities to Functional Correspondences

🚀 Technical Novelty

  • Mechanism: Reinterprets attention as a functional correspondence between learned adaptive bases, replacing softmax affinities with structured linear operators solved via least-squares regression in the spectral domain.
  • Nuance: Unlike token-centric or fixed-basis methods (e.g., FNO, Galerkin), it dynamically learns query/key-value bases via lightweight feed-forward networks and decouples function representation from discretization resolution, avoiding quadratic scaling while preserving global structural dependencies.

💡 Yield

  • Achieves state-of-the-art accuracy across PDE solving (Burgers’, Darcy, Elasticity), 3D point cloud segmentation, and aerodynamic regression tasks.
  • Demonstrates robust zero-shot super-resolution generalization (training on 2048 grid points, testing on 8192) and superior out-of-distribution performance under varying Reynolds numbers and geometric angles.
  • Proves Lipschitz continuity with respect to input functions, establishing mathematical stability for continuous field mappings.

⚠️ Limitations

  • Relies on a simple softmax projection for basis learning, which may limit expressiveness compared to more structured or orthogonal designs.
  • Lacks rigorous approximation guarantees or generalization bounds; the formal relationship between compression ratio and approximation error remains unproven.
  • Currently validated only on geometric/physical domains; theoretical and empirical extension to discrete sequence modeling (e.g., NLP) is left for future work.