Definition
Reinforcement Learning from Human Feedback, commonly shortened to RLHF, is a family of post-training methods in which human feedback provides a reward signal and reinforcement learning optimizes a policy against it. In a common language-model pipeline, reviewers compare candidate responses. A reward model learns from those comparisons, then reinforcement learning updates the policy to raise its reward without drifting too far from a reference model.
Supervised fine-tuning often precedes the reinforcement-learning stage to establish useful instruction-following behavior.
Simple example
For each support prompt, reviewers rank candidate answers based on correctness, clarity, and policy compliance. A reward model learns to score answers to each prompt in line with those rankings. Reinforcement learning updates the response model to score higher while limiting changes to its existing behavior.
Separate evaluations check whether the adapted model improves the target criteria without introducing unacceptable regressions.
Why it matters
Many desired behaviors are easier to express through comparisons than through a complete hand-written rule or one perfect response. RLHF uses those preferences to shape how a model responds to new prompts.
RLHF produces feedback data, policy checkpoints, and evaluation results, often alongside a reward model. Keeping versions and provenance for these artifacts helps teams reproduce training runs and audit changes in behavior.
One important nuance
Implementations differ in how they collect feedback, derive rewards, update the policy, and control policy drift. Optimizing a learned reward can exploit its blind spots rather than improve real quality. A response can win a preference comparison and still be wrong or unsafe. Teams need independent evaluation and ongoing review of model behavior.