alleryStart learning →The per-layer complexity of self-attention is
AO(k·n·d²) where k is the convolutional kernel size
BO(r·n·d) for a restricted-neighbourhood variant
CO(n²·d) in n the sequence length and d dimension
DO(n·d²) which equals the complexity for recurrent layers
Answer & Solution
Correct answer: C. O(n²·d) in n the sequence length and d dimension
Self-attention is O(n²·d): all pairs of positions multiplied by representation size.
Recurrent layers are O(n·d²) — better when n < d but cannot be parallelised.
Keep practising Transformer Attention
Browse all Transformer Attention questions for Claude, or explore more Aifoundations topics.
Related questions
Rebuild the model width from headsThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyThe decoder differs from the encoder by addingMulti-head attention uses how many heads in the base model?The encoder of the original Transformer uses how many identical layers?Scaled dot-product attention, per the paper, computes the output asThe Transformer architecture replaces recurrence and convolutions with