alleryStart learning →The reward model is trained on
APairwise comparisons of model outputs
BSingle absolute scores per output
CRule-based outputs from a regex set
DHuman-written critiques of each token
Answer & Solution
Correct answer: A. Pairwise comparisons of model outputs
Labelers rank K outputs to a prompt (K is 4 to 9).
All C(K,2) pairs become preference labels.
The RM learns to assign higher scalar rewards to the preferred completions.
Keep practising InstructGPT RLHF
Browse all InstructGPT RLHF questions for Claude, or explore more Aifoundations topics.
Related questions
Knock out the false alignment claimsRM models in InstructGPT are sized atInter-annotator agreement among training labelers is roughlyRM training uses all C(K,2) comparisons from a single prompt asThe RM training pairwise loss function uses'PPO-ptx' differs from plain PPO byThe alignment goal is summarised as the model beingThe final RL step uses which algorithm to optimise the policy?