Definition
A reward model is a learned model that assigns a score to an output, usually given the input that produced it. In preference-based RLHF, the model is trained to give higher scores to responses favored by the recorded preference judgments. Other reward models learn from correctness labels, observed outcomes, or feedback on intermediate steps.
Reviewers might compare two responses to the same prompt. The model learns to score the chosen response above the rejected one and can then score responses the reviewers have not seen. With pairwise training, the numbers express a learned ranking rather than a calibrated measure of response quality.
There is no hand-written scoring rule for each prompt. When reviewers supply the preferences, their judgments can also carry inconsistent criteria and blind spots.
Simple example
A support team has a runbook that says an operator must retry a failed import. For the same customer question, one candidate reply claims the retry happens automatically. Another correctly explains that a support operator must initiate the retry manually. Reviewers prefer the second reply.
After training on many comparisons, a reward model should assign the second reply a higher score for that question. A training run or evaluation can then rank new replies without asking a reviewer to judge every pair.
Why it matters
In a common RLHF pipeline, reinforcement learning uses reward-model scores to update the response model. A team can also use a reward model to rank candidates during evaluation or data selection. Reviewers label a sample of response pairs. The model can score many more outputs.
One important nuance
A higher reward is not proof that an answer is correct. The model may learn to favor confident wording or long replies if those features correlate with preferred examples. An optimized response model can exploit such patterns and raise its score without improving the underlying answer. Check reward-model rankings against held-out human judgments, and use separate checks for requirements such as factual accuracy or policy compliance.