Home › Claude › aifoundations › transformer_attention › The per-layer complexity of self-attention is
The per-layer complexity of self-attention is
AO(k·n·d²) where k is the convolutional kernel size
BO(r·n·d) for a restricted-neighbourhood variant
CO(n²·d) in n the sequence length and d dimension
DO(n·d²) which equals the complexity for recurrent layers
Answer & Solution
Correct answer: C. O(n²·d) in n the sequence length and d dimension
Self-attention is O(n²·d): all pairs of positions multiplied by representation size. Recurrent layers are O(n·d²) — better when n < d but cannot be parallelised.
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isResidual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe THREE distinct ways attention is used in the Transformer areThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyWhat is the maximum path length in self-attention layers?