Reinforcement learning from human feedback: learning human preferences

Updated October 21, 2026

Reinforcement learning from human feedback (RLHF) trains a reward model from human comparisons, then fine-tunes a language model with reinforcement learning. It aligns models with subjective goals such as helpfulness and truthfulness. Direct Preference Optimization offers a simpler alternative to the RLHF pipeline.

The role of human preferences in AI training

Reinforcement learning from human feedback (RLHF) is a machine learning technique that uses direct human feedback to train a reward model, which then guides the optimization of an AI agent through reinforcement learning, as described by IBM's RLHF overview. It is also called reinforcement learning from human preferences. The method is suited for tasks where goals are complex or difficult to specify, such as defining humor or helpfulness.

Standard language models are trained with a next token prediction loss that does not capture subjective qualities like truthfulness or creativity. Metrics such as BLEU and ROUGE compare generated text to references but remain limited. Human feedback provides a more direct signal, as described in the Hugging Face RLHF illustration.

The RLHF training process

RLHF typically begins with a pretrained language model. In the InstructGPT paper, the model was fine-tuned on labeler demonstrations using supervised learning, and then further fine-tuned with reinforcement learning from human rankings, as described in the arXiv InstructGPT paper. This approach pairs a demonstration dataset with a ranking dataset.

The second stage is reward model training. Human annotators rank or compare outputs from the model. These comparisons become training data for a reward model that returns a scalar score representing human preference. The AWS overview of RLHF describes this as part of a four-stage pipeline that starts with data collection and supervised fine-tuning.

The final stage applies reinforcement learning to maximize the reward model's score. Early work by OpenAI and DeepMind used this method to solve Atari games and simulated robot locomotion, requiring feedback on less than one percent of agent interactions, as shown in the arXiv deep RL from human preferences paper. That work established the practical feasibility of the approach.

Applications and evidence

RLHF has improved summarization. Research from OpenAI collected a large dataset of human comparisons between summaries, trained a reward model to predict the preferred summary, and used it to fine-tune a summarization policy. The resulting model outperformed human reference summaries and transferred to new datasets, according to the arXiv summarization paper.

For instruction following, RLHF aligns language models with user intent. The InstructGPT model, trained with human demonstrations and rankings, was preferred by human evaluators over a much larger GPT-3 model, with improvements in truthfulness and reduced toxic outputs. This demonstrates that smaller models can be made more useful through human feedback.

An alternative: Direct Preference Optimization

Direct Preference Optimization (DPO) offers a simpler alternative to RLHF. The DPO paper introduces a parameterization that extracts the optimal policy in closed form, solving the RLHF problem with a classification loss instead of reinforcement learning. This avoids training a separate reward model and eliminates sampling from the language model during fine-tuning, according to the arXiv DPO paper.

DPO is stable and computationally lightweight, avoiding the complexity and instability often seen in RLHF. It matches or exceeds RLHF in controlling sentiment and in response quality for summarization and single-turn dialogue. The table below summarizes the key differences.

Frequently asked questions

What is reinforcement learning from human feedback?

RLHF is a machine learning technique that uses human feedback to train a reward model, then fine-tunes an AI model with reinforcement learning to align it with human preferences.

Why is human feedback used instead of a predefined metric?

Many tasks have subjective goals that are difficult to specify algorithmically, such as humor or helpfulness. Human ratings capture these nuances better than metrics like BLEU or ROUGE.

What are the main stages of RLHF?

The process usually starts with a pretrained language model, then a reward model is trained from human comparisons, and finally the model is fine-tuned with reinforcement learning to maximize the reward.

How does Direct Preference Optimization differ from RLHF?

DPO eliminates the separate reward model and the reinforcement learning step by using a classification loss to optimize the policy directly, making it simpler and more stable.

What applications benefit from RLHF?

RLHF has improved summarization, instruction following, and chatbot behavior, and has been used for game playing and simulated robotics.