Home › Claude › aifoundations › transformer_attention › The architectural property that lets self-attent…
The architectural property that lets self-attention parallelise where RNNs cannot is
ASmaller embedding dimension than a recurrent state
BUse of feed-forward layers between attention blocks
CConstant minimum number of sequential operations
DLower asymptotic complexity than recurrent networks
Answer & Solution
Correct answer: C. Constant minimum number of sequential operations
Self-attention is O(1) in sequential operations — all positions computed in parallel within a layer. Recurrent layers are O(n) sequential by definition (each step waits for the previous), which is what blocks GPU parallelism.
Related questions
Residual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe THREE distinct ways attention is used in the Transformer areThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyWhat is the maximum path length in self-attention layers?The per-layer complexity of self-attention is