alleryStart learning →RM models in InstructGPT are sized at
A1 trillion parameters mostly
B1.3 billion parameters mostly
C6 billion parameters mostly
D175 billion parameters mostly
Answer & Solution
Correct answer: C. 6 billion parameters mostly
Only 6B RMs are used in the paper.
175B RMs were unstable as the value function during RL training
6B saves substantial compute without losing alignment performance.
Keep practising InstructGPT RLHF
Browse all InstructGPT RLHF questions for Claude, or explore more Aifoundations topics.
Related questions
Knock out the false alignment claimsInter-annotator agreement among training labelers is roughlyRM training uses all C(K,2) comparisons from a single prompt asThe RM training pairwise loss function uses'PPO-ptx' differs from plain PPO byThe alignment goal is summarised as the model beingThe final RL step uses which algorithm to optimise the policy?The reward model is trained on