Home › Claude › aifoundations › instructgpt_rlhf › Inter-annotator agreement among training labeler…
Inter-annotator agreement among training labelers is roughly
AAbout 95% pairwise agreement
BAbout 30% pairwise agreement
CAbout 50% pairwise agreement
DAbout 73% pairwise agreement
Answer & Solution
Correct answer: D. About 73% pairwise agreement
Training labelers agree about 72.6%; held-out labelers agree about 77.3%. Comparable to the 73% researcher-researcher agreement in the Stiennon et al. summarization work.
Related questions
The FLAN and T0 datasets are mentioned as baselines because theyWhy is honesty measured via 'truthfulness' rather than directly?Prioritising helpfulness during training while prioritising harmlessness at evaluation is RM models in InstructGPT are sized atRM training uses all C(K,2) comparisons from a single prompt asThe RM training pairwise loss function uses'PPO-ptx' differs from plain PPO byThe PPO objective includes a KL penalty against the SFT model to