alleryStart learning →The RM training pairwise loss function uses
AMean squared error on absolute reward scores
BCross-entropy hinge over pairwise margins
CSigmoid of the reward difference
DTriplet loss on response embeddings
Answer & Solution
Correct answer: C. Sigmoid of the reward difference
Loss = -log(sigmoid(r(x, y_w) - r(x, y_l))) where y_w is preferred over y_l.
This makes the RM's reward gap reflect the human preference probability.
Keep practising InstructGPT RLHF
Browse all InstructGPT RLHF questions for Claude, or explore more Aifoundations topics.
Related questions
Knock out the false alignment claimsRM models in InstructGPT are sized atInter-annotator agreement among training labelers is roughlyRM training uses all C(K,2) comparisons from a single prompt as'PPO-ptx' differs from plain PPO byThe alignment goal is summarised as the model beingThe final RL step uses which algorithm to optimise the policy?The reward model is trained on