Home › Claude › aifoundations › instructgpt_rlhf › The final RL step uses which algorithm to optimi…
The final RL step uses which algorithm to optimise the policy?
AREINFORCE with no baseline correction
BQ-learning with experience replay buffers
CAlphaZero with Monte Carlo tree search
DPPO with a per-token KL penalty
Answer & Solution
Correct answer: D. PPO with a per-token KL penalty
Step 3 is PPO. A per-token KL penalty from the SFT model prevents the policy from drifting too far and over-optimising the reward model. PPO is standard for RLHF.
Related questions
The FLAN and T0 datasets are mentioned as baselines because theyWhy is honesty measured via 'truthfulness' rather than directly?Prioritising helpfulness during training while prioritising harmlessness at evaluation is RM models in InstructGPT are sized atInter-annotator agreement among training labelers is roughlyRM training uses all C(K,2) comparisons from a single prompt asThe RM training pairwise loss function uses'PPO-ptx' differs from plain PPO by