Text-to-LoRA: Instant Transformer Adaption

Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Tjarko Lange

TL;DR: T2L is a hypernetwork that generated task-specific LoRA given a short task description

Abstract:

While Foundation Models provide a general tool for rapid content creation, they regularly require task-specific adaptation. Traditionally, this exercise involves careful curation of datasets and repeated fine-tuning of the underlying model.
Fine-tuning techniques enable practitioners to adapt foundation models for many new applications but require expensive and lengthy training while being notably sensitive to hyperparameter choices.
To overcome these limitations, we introduce Text-to-LoRA (T2L), a model capable of adapting large language models (LLMs) on the fly solely based on a natural language description of the target task. T2L is a hypernetwork trained to construct LoRAs in a single inexpensive forward pass.
After training T2L on a suite of 9 pre-trained LoRA adapters (GSM8K, Arc, etc.), we show that the ad-hoc reconstructed LoRA instances match the performance of task-specific adapters across the corresponding test sets.
Furthermore, T2L can compress hundreds of LoRA instances and zero-shot generalize to entirely unseen tasks.
This approach provides a significant step towards democratizing the specialization of foundation models and enables language-based adaptation with minimal compute requirements.

Our code is available at https://github.com/SakanaAI/text-to-lora

Lay Summary: This paper introduces Text-to-LoRA (T2L), a method to make adapting Language Models for specific tasks much easier and more accessible. Traditionally, customizing these models requires gathering large datasets and performing expensive, time-consuming fine-tuning for specific applications. T2L bypasses this by training a special model (a hypernetwork) that can instantly generate the necessary task-specific adaptations (called LoRAs) in a single, inexpensive step, based solely on a simple textual description of the task. This approach is a step towards dramatically lowering the technical and computational barriers, allowing non-technical users to specialize foundation models using plain language, rather than needing deep technical expertise or large compute resources.

Link To Code: https://github.com/SakanaAI/text-to-lora

Primary Area: General Machine Learning->Transfer, Multitask and Meta-learning

Keywords: hypernetworks, test-time adaptation, low-rank adaptation, language models, transformers, meta-learning

Submission Number: 6183