Home › Claude › aifoundations › transformer_attention › Residual connections and layer normalisation are…
Residual connections and layer normalisation are applied as
AAround the entire encoder but not its sub-layers
BLayerNorm(x + Sublayer(x)) around each sub-layer
COnly at the final output of each encoder stack
DBetween heads inside one multi-head attention block
Answer & Solution
Correct answer: B. LayerNorm(x + Sublayer(x)) around each sub-layer
Every sub-layer (attention or FFN) wraps with residual + layer norm: LayerNorm(x + Sublayer(x)). This is what makes deep stacks trainable without vanishing gradients.
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe THREE distinct ways attention is used in the Transformer areThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyWhat is the maximum path length in self-attention layers?The per-layer complexity of self-attention is