Instruction tuning: teaching language models to follow requests
Instruction tuning is a supervised fine-tuning technique that trains a language model on datasets of instruction prompts and desired outputs. It helps bridge the gap between next-word prediction and user intent, improving zero-shot performance on unseen tasks. The approach was popularized by models like FLAN and InstructGPT.
Definition and role of instruction tuning
Instruction tuning is a supervised fine-tuning technique applied to large language models. According to IBM's overview of instruction tuning, it uses a labeled dataset of instructional prompts and corresponding outputs to improve the model's ability to follow instructions in general, not just on specific tasks.
This technique is part of the broader fine-tuning category that adapts pre-trained foundation models. It can be combined with other methods such as reinforcement learning from human feedback, and is often used alongside parameter-efficient fine-tuning methods like low-rank adaptation (LoRA) to reduce computational demands.
Why pre-trained models need instruction tuning
Pre-trained language models are optimized to predict the next token in a sequence, which is not aligned with user instructions. Google's research blog on Google's FLAN blog notes that fine-tuning a model for a specific task requires many training examples and separate weights for each task, which is impractical for large models.
Instruction tuning aims to make the model more amenable to solving tasks in general. The blog explains that the instruction tuning phase takes only a small number of updates compared to the massive pre-training phase, making it a relatively inexpensive adaptation step.
The FLAN approach: instruction tuning at scale
The FLAN (Fine-tuned LAnguage Net) model, presented in the paper Finetuned Language Models Are Zero-Shot Learners, demonstrates the power of instruction tuning. The researchers took a 137B parameter pretrained model and instruction-tuned it on over 60 NLP tasks verbalized via natural language instruction templates.
The resulting model showed substantial improvements on unseen task types, outperforming zero-shot GPT-3 on 20 of 25 tasks. Ablation studies revealed that the number of fine-tuning datasets, model scale, and natural language instructions are key to success.
Other research: multitask prompting and human feedback
Another line of work, Multitask Prompted Training Enables Zero-Shot Task Generalization from BigScience, converted many supervised datasets into prompted form with multiple prompts per task. This multitask mixture allowed a fine-tuned encoder-decoder model to outperform models up to 16 times its size on certain benchmarks.
OpenAI's InstructGPT project, described in OpenAI's InstructGPT paper, used supervised learning on labeler demonstrations, followed by reinforcement learning from human feedback. The 1.3B parameter InstructGPT model was preferred to the 175B GPT-3 in human evaluations, despite having 100 times fewer parameters.
Implementing instruction tuning with chat templates
For practical use, chat models often employ chat templates that define how message roles are converted into token sequences. Hugging Face's Hugging Face's chat templates explain that control tokens like <|user|> and <|assistant|> allow the model to see the chat structure, and different models may use different formats even if they are fine-tuned from the same base model.
The apply_chat_template method in the Transformers library formats a list of role-content dictionaries into the expected token sequence. Choosing the correct template is critical because wrong control tokens can drastically degrade performance.
| Approach | Training signal | Example from sources |
|---|---|---|
| Instruction tuning | Labeled instruction-response pairs | FLAN fine-tuned on 60+ NLP tasks |
| Reinforcement learning from human feedback | Rankings of model outputs | InstructGPT improved truthfulness |
| Multitask prompted training | Prompted datasets with multiple templates | BigScience model outperformed larger models |
| Parameter-efficient fine-tuning (LoRA) | Low-rank updates to weights | Reduces computational demands for adaptation |
Frequently asked questions
What is instruction tuning?
Instruction tuning is a supervised fine-tuning technique that trains a language model on a dataset of instruction prompts and corresponding outputs, improving its ability to follow instructions on both seen and unseen tasks.
How does instruction tuning differ from standard fine-tuning?
Standard fine-tuning adapts a model to a specific task with labeled examples. Instruction tuning uses a variety of task descriptions and desired responses, teaching the model general instruction-following behavior.
Why is instruction tuning important for language models?
Pre-trained models only predict next text. Instruction tuning bridges the gap to user intent, making models more useful for practical applications.
What role do chat templates play?
Chat templates convert a list of messages with roles into a token sequence with control tokens, ensuring the model interprets the chat correctly. Different models require different templates.