alleryStart learning →RM training uses all C(K,2) comparisons from a single prompt as
AShuffled into one large dataset always
BK separate batch elements one per response
CDiscarded after a single epoch of training
DA single batch element together
Answer & Solution
Correct answer: D. A single batch element together
Shuffling the C(K,2) pairs across the dataset caused the RM to overfit (highly correlated within a prompt).
Treating all C(K,2) pairs as one batch element fixes this and is much more compute-efficient.
Keep practising InstructGPT RLHF
Browse all InstructGPT RLHF questions for Claude, or explore more Aifoundations topics.
Related questions
Knock out the false alignment claimsRM models in InstructGPT are sized atInter-annotator agreement among training labelers is roughlyThe RM training pairwise loss function uses'PPO-ptx' differs from plain PPO byThe alignment goal is summarised as the model beingThe final RL step uses which algorithm to optimise the policy?The reward model is trained on