Home › Claude › aifoundations › many_shot_in_context_learning › KV caching is used in many-shot experiments to
KV caching is used in many-shot experiments to
AConvert exemplars into a vector database
BReduce the inference cost per prompt
CIncrease the maximum context window size
DAvoid retraining the model between rounds
Answer & Solution
Correct answer: B. Reduce the inference cost per prompt
KV caching reuses key/value tensors from prior tokens so a long shared prefix (the many-shot exemplars) is computed once. This makes the inference cost of many-shot regimes manageable.
Related questions
The implication for prompt design when shots are abundant isAn architectural risk of many-shot ICL exposed by the XSum result isWhy is many-shot ICL practically more attractive than fine-tuning?Long-context scaling laws (next-token loss)Many-shot ICL can overcomeMany-shot ICL can perform comparably toThe order of examples in many-shot promptsAbstractive summarization on XSum