The central finding is that
AFew-shot ICL beats many-shot whenever shots exceed 16
BMany-shot only helps when full fine-tuning is also done
CMany-shot performance is flat regardless of shot count
DMany-shot ICL beats few-shot on a wide range of tasks
Answer & Solution
Correct answer: D. Many-shot ICL beats few-shot on a wide range of tasks
Many-shot (hundreds to thousands of shots) consistently outperforms few-shot across translation, summarization, planning, reasoning, and reward modelling. The gap is largest on non-natural-language tasks.
Related questions
The implication for prompt design when shots are abundant isAn architectural risk of many-shot ICL exposed by the XSum result isWhy is many-shot ICL practically more attractive than fine-tuning?Long-context scaling laws (next-token loss)Many-shot ICL can overcomeMany-shot ICL can perform comparably toThe order of examples in many-shot promptsAbstractive summarization on XSum