Start learning →

Claude InstructGPT RLHF — practice questions

Explore 11 free InstructGPT RLHF questions in Claude Aifoundations. Open each question or exercise to read its content, reference answers and available explanation.

Practise InstructGPT RLHF with Quill →

InstructGPT RLHF questions and exercises with answers

Showing questions 1–11 of 11.

InstructGPT is trained by which three high-level steps?Read the answer and worked solution →The base pretrained model fine-tuned for InstructGPT isRead the answer and worked solution →The reward model is trained onRead the answer and worked solution →The final RL step uses which algorithm to optimise the policy?Read the answer and worked solution →The alignment goal is summarised as the model beingRead the answer and worked solution →'PPO-ptx' differs from plain PPO byRead the answer and worked solution →The RM training pairwise loss function usesRead the answer and worked solution →RM training uses all C(K,2) comparisons from a single prompt asRead the answer and worked solution →Inter-annotator agreement among training labelers is roughlyRead the answer and worked solution →RM models in InstructGPT are sized atRead the answer and worked solution →Knock out the false alignment claimsRead the answer and worked solution →

Worked practice questions

InstructGPT is trained by which three high-level steps?

  1. Pretrain, distil, ship to product
  2. Supervised fine-tune, train RM, PPO
  3. Quantise, prune, deploy on the edge
  4. Tokenise, embed, classify per token
Show answer and solution

Correct answer: B. Supervised fine-tune, train RM, PPO

The InstructGPT recipe: (1) collect labeler demonstrations and fine-tune GPT-3 (SFT), (2) collect preference comparisons and train a reward model (RM), (3) fine-tune the SFT policy against the RM with PPO.

Open this question →

The base pretrained model fine-tuned for InstructGPT is

  1. GPT-2 with 1.5 billion parameters
  2. GPT-3 with up to 175B parameters
  3. T5 with 11 billion parameters total
  4. LLaMA with 70 billion parameters
Show answer and solution

Correct answer: B. GPT-3 with up to 175B parameters

The starting checkpoint is GPT-3 (up to 175B). InstructGPT is fine-tuned variants of these checkpoints — not pretrained from scratch.

Open this question →

The reward model is trained on

  1. Pairwise comparisons of model outputs
  2. Single absolute scores per output
  3. Rule-based outputs from a regex set
  4. Human-written critiques of each token
Show answer and solution

Correct answer: A. Pairwise comparisons of model outputs

Labelers rank K outputs to a prompt (K is 4 to 9). All C(K,2) pairs become preference labels. The RM learns to assign higher scalar rewards to the preferred completions.

Open this question →

How to practise

  1. Choose a InstructGPT RLHF question and try to work out your answer.
  2. Open the question to check the correct answer and read its worked solution.
  3. Continue in the app for bite-sized lessons, explanations and review with Quill. You can start without creating an account.

More Aifoundations topics

AlexNet CNN ImageNetMany Shot in Context LearningLinear Regression and Supervised LearningMath for AI LA ProbabilityMLP and ActivationsTransformer AttentionWhat Is AI Intro