🔗 Source: arXiv

Text-to-LoRA: Instant Transformer Adaption

🚀 Technical Novelty

  • Mechanism: A lightweight hypernetwork encodes a natural language task description into low-rank weight matrices (A and B), which are injected into a frozen base LLM via a single forward pass to dynamically modulate its behavior.
  • Nuance: Unlike prior routing or compression methods that rely on learned task IDs or explicit structural constraints, T2L directly maps text prompts to adapter weights, enabling true zero-shot generalization to unseen tasks without gradient updates during adaptation.

💡 Yield

  • Matches the performance of individually fine-tuned LoRA adapters across 9 benchmarks (e.g., GSM8K, Arc) while compressing hundreds of adapters into a single model.
  • Achieves strong zero-shot generalization to entirely unseen tasks using only user-provided text descriptions, outperforming multi-task LoRA and Arrow Routing baselines.
  • Demonstrates semantically meaningful clustering in generated adapter space, confirming task-specific steering capabilities.

⚠️ Limitations

  • Performance is highly dependent on the quality of the input text description; low-quality prompts degrade adapter effectiveness.
  • Does not fully match the peak performance of task-specific fine-tuned adapters in a zero-shot setting.
  • Currently limited to LoRA as the output space and primarily evaluated on LLMs, leaving transfer to larger architectures or vision-language models unexplored.