Knock out the false alignment claims

Exercise and reference answers

Questions

1. Question

   Three of these statements about the InstructGPT recipe are false.

   Choices

   1. Item

      PPO fine tunes the policy in step three

      Correct

      Yes

   2. Explanation

      Demonstrations and supervised fine tuning come first.

      Item

      The reward model is trained first of all

   3. Explanation

      The KL penalty pulls the policy toward the SFT model.

      Item

      The KL term pulls the policy away from SFT

   4. Explanation

      Labelers rank several outputs, and the pairs become the labels.

      Item

      Labelers score each output on its own

2. Question

   Three of these statements about the reward model are false.

   Choices

   1. Item

      Reward models in the paper are 6B in size

      Correct

      Yes

   2. Explanation

      175B reward models were unstable as the value function.

      Item

      175B reward models were the stable choice

   3. Explanation

      Shuffling the pairs across the data made the model overfit.

      Item

      Pairs from one prompt are shuffled apart

   4. Explanation

      PPO-ptx adds a pretraining log likelihood term to the objective.

      Item

      PPO-ptx drops the pretraining term

Instructions

Tap away the three false lines in each set about how InstructGPT was trained.

Explanation