Home › Claude › aifoundations › transformer_attention › Scaled dot-product attention, per the paper, com…
Scaled dot-product attention, per the paper, computes the output as
Asoftmax(QKᵀ / √d_k) · V over the value tokens
BA simple element-wise product of Q and V tensors
CAn average of all value vectors with equal weights
DA direct lookup in the embedding table by token id
Answer & Solution
Correct answer: A. softmax(QKᵀ / √d_k) · V over the value tokens
Attention(Q, K, V) = softmax(QKᵀ/√d_k) · V. The √d_k denominator scales dot products to keep softmax in its sensitive region.
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isResidual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe THREE distinct ways attention is used in the Transformer areThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyWhat is the maximum path length in self-attention layers?