Home › Claude › aifoundations › transformer_attention › Multi-head attention uses how many heads in the …
Multi-head attention uses how many heads in the base model?
AEight heads in parallel
BSixteen heads in parallel
CTwo heads in parallel
DFour heads in parallel
Answer & Solution
Correct answer: A. Eight heads in parallel
The paper uses h = 8 parallel heads. Each head operates on d_k = d_v = d_model / h = 64 dimensions, so total compute is similar to one full-dimensional head.
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isResidual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe THREE distinct ways attention is used in the Transformer areThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyWhat is the maximum path length in self-attention layers?