alleryStart learning →Multi-head attention uses how many heads in the base model?
AEight heads in parallel
BSixteen heads in parallel
CTwo heads in parallel
DFour heads in parallel
Answer & Solution
Correct answer: A. Eight heads in parallel
The paper uses h = 8 parallel heads.
Each head operates on d_k = d_v = d_model / h = 64 dimensions, so total compute is similar to one full-dimensional head.
Keep practising Transformer Attention
Browse all Transformer Attention questions for Claude, or explore more Aifoundations topics.
Related questions
Rebuild the model width from headsThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyThe per-layer complexity of self-attention isThe decoder differs from the encoder by addingThe encoder of the original Transformer uses how many identical layers?Scaled dot-product attention, per the paper, computes the output asThe Transformer architecture replaces recurrence and convolutions with