Direct Feedback Alignment Training

🔗 Source: arXiv

Direct Feedback Alignment Provides Learning in Deep Neural Networks

🚀 Technical Novelty

• Mechanism: Propagates error signals directly from the output layer to all hidden layers via fixed, randomly initialized feedback matrices, eliminating iterative backward passes and symmetric weight requirements.
• Nuance: Unlike backpropagation (which requires exact weight transposes) or Feedback Alignment (which chains feedback sequentially), DFA decouples feedback paths entirely from forward activations, enabling simultaneous, independent layer updates from zero initialization without vanishing gradient issues.

💡 Yield

• Achieves near-backpropagation accuracy on MNIST and CIFAR benchmarks while successfully training ultra-deep networks (100+ layers) without careful weight initialization; theoretically proves gradient alignment under linear assumptions.

⚠️ Limitations

• Test performance slightly trails backpropagation, particularly on convolutional architectures and complex datasets like CIFAR-100; theoretical guarantees are limited to linear single-hidden-layer networks, leaving deep nonlinear convergence unproven.