Definition
Proximal Policy Optimization (PPO) is a family of policy-gradient algorithms for reinforcement learning. In a common language-model RLHF setup, the model generates replies to sampled prompts. A reward model scores those replies, and PPO uses reward-derived signals to update the response policy.
PPO-Clip is one variant. Its objective compares the probabilities of sampled tokens under the policy that generated the batch and the policy being updated. Clipping their probability ratios reduces the incentive to make large changes in the direction favored by the advantage estimates. PPO can use the same batch for several updates.
Simple example
A support assistant answers a question about a failed import. It says the import will retry automatically, although an operator must start the retry. A reward model trained on comparisons of support replies gives that answer a low score. PPO uses an advantage estimate derived from the reward and a baseline. If the estimate is negative, an update can reduce the probability of the sampled tokens in that context.
Reviewers did not write a replacement answer for this prompt. Their earlier comparisons trained the reward model.
Why it matters
In PPO-based RLHF, the policy often starts from a model trained through supervised fine-tuning. PPO then updates it using reward-derived advantage estimates. The training run can use feedback from generated replies without requiring a single ideal answer for each prompt.
Track reward scores alongside evaluations on prompts held out from training. A rising training reward may mean the model has learned a quirk of the scorer rather than improved its answers.
One important nuance
Clipping reduces the incentive for large changes relative to the policy that generated the current batch. It imposes no hard limit on an individual update or on cumulative drift from the starting model. Some RLHF training setups also penalize drift from a fixed reference model. That reference penalty is separate from clipping. It cannot correct a reward model that scores the wrong behavior highly.