Definition
Preference fine-tuning adapts a model using judgments about which response to an input is better. A common training record contains a prompt, a chosen response, and a rejected response. Training uses these comparisons to change the model’s behavior.
The term describes the training signal, not the update algorithm. Human reviewers or automated evaluators can supply comparisons. Record who or what made each choice and which criteria they were asked to apply.
Simple example
A support team generates two replies to each of many customer questions. Reviewers choose the reply that answers the question accurately and follows the escalation policy. Those choices become preference data for adapting the model. The team then tests the updated model on questions that were kept out of training.
Why it matters
Reviewers can compare two concrete answers without writing an ideal answer from scratch. This helps when several answers are acceptable but differ in accuracy, policy fit, or style.
The result depends on the criteria behind each choice. If some reviewers favor brevity while others favor detail, the labels can conflict. If examples cover only routine questions, the adapted model may still struggle with escalations.
One important nuance
A reviewer can choose the answer that appears more accurate, even if both contain errors. The label records a judgment under the chosen criteria. It does not establish that either answer is correct. Check factual accuracy and policy compliance on held-out cases alongside preference win rates.