Abstractive summarization on XSum
ACannot be evaluated with many-shot ICL approach
BContinues improving monotonically with more shots
CPeaks around 50 shots then performance deteriorates
DImproves only when fine-tuning is also applied
Answer & Solution
Correct answer: C. Peaks around 50 shots then performance deteriorates
XSum performance peaks around 50 shots and declines beyond. The model also occasionally hallucinates dates and times, which were absent from the in-context examples. Many-shot is not always monotonic.
Related questions
The implication for prompt design when shots are abundant isAn architectural risk of many-shot ICL exposed by the XSum result isWhy is many-shot ICL practically more attractive than fine-tuning?Long-context scaling laws (next-token loss)Many-shot ICL can overcomeMany-shot ICL can perform comparably toThe order of examples in many-shot promptsMachine translation many-shot results