'PPO-ptx' differs from plain PPO by
ADoubling the policy network parameter count
BRemoving the per-token KL penalty entirely
CReplacing PPO with a value-function loss
DMixing pretraining gradients into the PPO update
Answer & Solution
Correct answer: D. Mixing pretraining gradients into the PPO update
PPO-ptx adds a pretraining-distribution log-likelihood term (weight gamma) to the RL objective. This patches performance regressions on public NLP datasets that plain PPO causes.
Related questions
The FLAN and T0 datasets are mentioned as baselines because theyWhy is honesty measured via 'truthfulness' rather than directly?Prioritising helpfulness during training while prioritising harmlessness at evaluation is RM models in InstructGPT are sized atInter-annotator agreement among training labelers is roughlyRM training uses all C(K,2) comparisons from a single prompt asThe RM training pairwise loss function usesThe PPO objective includes a KL penalty against the SFT model to