Home › Claude › aifoundations › transformer_attention › Multi-head attention is preferred over a single …
Multi-head attention is preferred over a single d_model-dimensional head because
AMulti-head reduces total parameter count substantially
BHeads attend to different representation subspaces
CSingle-head attention requires much more GPU memory
DSingle-head attention has higher arithmetic complexity
Answer & Solution
Correct answer: B. Heads attend to different representation subspaces
With a single head, averaging blurs information across positions. Multi-head lets the model jointly attend to different representation subspaces — distinct heads can learn distinct relations (syntax, coreference, etc.).
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isResidual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?The THREE distinct ways attention is used in the Transformer areThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyWhat is the maximum path length in self-attention layers?The per-layer complexity of self-attention is