alleryStart learning →InstructGPT is trained by which three high-level steps?
APretrain, distil, ship to product
BSupervised fine-tune, train RM, PPO
CQuantise, prune, deploy on the edge
DTokenise, embed, classify per token
Answer & Solution
Correct answer: B. Supervised fine-tune, train RM, PPO
The InstructGPT recipe:
(1) collect labeler demonstrations and fine-tune GPT-3 (SFT),
(2) collect preference comparisons and train a reward model (RM),
(3) fine-tune the SFT policy against the RM with PPO.
Keep practising InstructGPT RLHF
Browse all InstructGPT RLHF questions for Claude, or explore more Aifoundations topics.
Related questions
Knock out the false alignment claimsRM models in InstructGPT are sized atInter-annotator agreement among training labelers is roughlyRM training uses all C(K,2) comparisons from a single prompt asThe RM training pairwise loss function uses'PPO-ptx' differs from plain PPO byThe alignment goal is summarised as the model beingThe final RL step uses which algorithm to optimise the policy?