Home › Claude › aifoundations › transformer_attention › What is the maximum path length in self-attentio…
What is the maximum path length in self-attention layers?
AO(n), linear in the length of the sequence
BO(log_k n), logarithmic via convolutions of size k
CO(n / r), in restricted-neighbourhood attention
DO(1), constant regardless of sequence length
Answer & Solution
Correct answer: D. O(1), constant regardless of sequence length
Any two positions are O(1) apart in self-attention — direct attention edges. Recurrent layers have O(n) maximum path length, hindering long-range learning.
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isResidual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe THREE distinct ways attention is used in the Transformer areThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyThe per-layer complexity of self-attention is