Home › Claude › aifoundations › transformer_attention › The THREE distinct ways attention is used in the…
The THREE distinct ways attention is used in the Transformer are
ASparse attention, dense attention, learned attention
BEncoder self-attention, decoder self-attention, cross-attention
CEncoder-only, decoder-only, and bidirectional attention
DPre-norm attention, post-norm attention, residual attention
Answer & Solution
Correct answer: B. Encoder self-attention, decoder self-attention, cross-attention
Encoder self-attention (K, V, Q from previous encoder layer); decoder masked self-attention (decoder past only); cross-attention (Q from decoder, K + V from encoder output).
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isResidual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyWhat is the maximum path length in self-attention layers?The per-layer complexity of self-attention is