Home › Claude › aifoundations › instructgpt_rlhf › RM training uses all C(K,2) comparisons from a s…
RM training uses all C(K,2) comparisons from a single prompt as
AShuffled into one large dataset always
BK separate batch elements one per response
CDiscarded after a single epoch of training
DA single batch element together
Answer & Solution
Correct answer: D. A single batch element together
Shuffling the C(K,2) pairs across the dataset caused the RM to overfit (highly correlated within a prompt). Treating all C(K,2) pairs as one batch element fixes this and is much more compute-efficient.
Related questions
The FLAN and T0 datasets are mentioned as baselines because theyWhy is honesty measured via 'truthfulness' rather than directly?Prioritising helpfulness during training while prioritising harmlessness at evaluation is RM models in InstructGPT are sized atInter-annotator agreement among training labelers is roughlyThe RM training pairwise loss function uses'PPO-ptx' differs from plain PPO byThe PPO objective includes a KL penalty against the SFT model to