alleryStart learning →'PPO-ptx' differs from plain PPO by
ADoubling the policy network parameter count
BRemoving the per-token KL penalty entirely
CReplacing PPO with a value-function loss
DMixing pretraining gradients into the PPO update
Answer & Solution
Correct answer: D. Mixing pretraining gradients into the PPO update
PPO-ptx adds a pretraining-distribution log-likelihood term (weight gamma) to the RL objective.
This patches performance regressions on public NLP datasets that plain PPO causes.
Keep practising InstructGPT RLHF
Browse all InstructGPT RLHF questions for Claude, or explore more Aifoundations topics.
Related questions
Knock out the false alignment claimsRM models in InstructGPT are sized atInter-annotator agreement among training labelers is roughlyRM training uses all C(K,2) comparisons from a single prompt asThe RM training pairwise loss function usesThe alignment goal is summarised as the model beingThe final RL step uses which algorithm to optimise the policy?The reward model is trained on