Home › Claude › aifoundations › instructgpt_rlhf › The PPO objective includes a KL penalty against …
The PPO objective includes a KL penalty against the SFT model to
AReduce the memory footprint of the RL step
BEncourage longer responses to user prompts
CMitigate over-optimisation of the reward model
DSpeed up convergence of the policy gradient
Answer & Solution
Correct answer: C. Mitigate over-optimisation of the reward model
Without the KL term, the policy can find degenerate outputs that game the RM (reward hacking). Pulling the policy toward the SFT distribution at every token keeps generations sane.
Related questions
The FLAN and T0 datasets are mentioned as baselines because theyWhy is honesty measured via 'truthfulness' rather than directly?Prioritising helpfulness during training while prioritising harmlessness at evaluation is RM models in InstructGPT are sized atInter-annotator agreement among training labelers is roughlyRM training uses all C(K,2) comparisons from a single prompt asThe RM training pairwise loss function uses'PPO-ptx' differs from plain PPO by