Home › Claude › aifoundations › instructgpt_rlhf › Prioritising helpfulness during training while p…
Prioritising helpfulness during training while prioritising harmlessness at evaluation is justified because
AProduction deployments are evaluated by separate labelers
BHarmlessness data is impossible to collect from labelers
CIt tracks what users say versus what we care about
DHelpfulness and harmlessness never actually conflict
Answer & Solution
Correct answer: C. It tracks what users say versus what we care about
The paper acknowledges that during training, optimising harmlessness too aggressively would block helpful responses to ambiguous prompts. Final evaluations weight what we actually want (truth + safety), exposing the gap.
Related questions
The FLAN and T0 datasets are mentioned as baselines because theyWhy is honesty measured via 'truthfulness' rather than directly?RM models in InstructGPT are sized atInter-annotator agreement among training labelers is roughlyRM training uses all C(K,2) comparisons from a single prompt asThe RM training pairwise loss function uses'PPO-ptx' differs from plain PPO byThe PPO objective includes a KL penalty against the SFT model to