RM models in InstructGPT are sized at
A1 trillion parameters mostly
B1.3 billion parameters mostly
C6 billion parameters mostly
D175 billion parameters mostly
Answer & Solution
Correct answer: C. 6 billion parameters mostly
Only 6B RMs are used in the paper. 175B RMs were unstable as the value function during RL training; 6B saves substantial compute without losing alignment performance.
Related questions
The FLAN and T0 datasets are mentioned as baselines because theyWhy is honesty measured via 'truthfulness' rather than directly?Prioritising helpfulness during training while prioritising harmlessness at evaluation is Inter-annotator agreement among training labelers is roughlyRM training uses all C(K,2) comparisons from a single prompt asThe RM training pairwise loss function uses'PPO-ptx' differs from plain PPO byThe PPO objective includes a KL penalty against the SFT model to