Many-shot ICL can overcome
AThe need for any prompt template at all
BPretraining biases that block few-shot adaptation
CHardware limitations of small consumer GPUs
DTokenizer limitations of subword vocabularies
Answer & Solution
Correct answer: B. Pretraining biases that block few-shot adaptation
Section 4.1: many-shot ICL can overcome pretraining biases that few-shot cannot. For example, on sentiment with semantically-unrelated labels, many-shot adapts where few-shot stays anchored to the pretrained labelling.
Related questions
The implication for prompt design when shots are abundant isAn architectural risk of many-shot ICL exposed by the XSum result isWhy is many-shot ICL practically more attractive than fine-tuning?Long-context scaling laws (next-token loss)Many-shot ICL can perform comparably toThe order of examples in many-shot promptsAbstractive summarization on XSumMachine translation many-shot results