Home › Claude › aifoundations › transformer_attention › The decoder differs from the encoder by adding
The decoder differs from the encoder by adding
AJust one more identical encoder-style attention layer
BConvolutional layers between self-attention blocks
CRecurrent units for autoregressive generation steps
DA masked self-attention plus encoder-decoder attention
Answer & Solution
Correct answer: D. A masked self-attention plus encoder-decoder attention
The decoder adds masked self-attention (so position i cannot attend to j > i) and a third sub-layer that does cross-attention over the encoder's output stack.
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isResidual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe THREE distinct ways attention is used in the Transformer areThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyWhat is the maximum path length in self-attention layers?