Home › Claude › aifoundations › many_shot_in_context_learning › Long-context scaling laws (next-token loss)
Long-context scaling laws (next-token loss)
AReliably predict downstream ICL performance
BMay NOT reliably predict ICL performance
COnly apply to text generation tasks specifically
DPredict performance on math but not on translation
Answer & Solution
Correct answer: B. May NOT reliably predict ICL performance
Section 4.8 shows next-token-loss scaling laws do not reliably predict ICL performance on problem-solving and reasoning. Lower loss does not always mean better downstream ICL.
Related questions
The implication for prompt design when shots are abundant isAn architectural risk of many-shot ICL exposed by the XSum result isWhy is many-shot ICL practically more attractive than fine-tuning?Many-shot ICL can overcomeMany-shot ICL can perform comparably toThe order of examples in many-shot promptsAbstractive summarization on XSumMachine translation many-shot results