alleryStart learning →Scaled dot-product attention, per the paper, computes the output as
Asoftmax(QKᵀ / √d_k) · V over the value tokens
BA simple element-wise product of Q and V tensors
CAn average of all value vectors with equal weights
DA direct lookup in the embedding table by token id
Answer & Solution
Correct answer: A. softmax(QKᵀ / √d_k) · V over the value tokens
Attention(Q, K, V) = softmax(QKᵀ/√d_k) · V. The √d_k denominator scales dot products to keep softmax in its sensitive region.
Keep practising Transformer Attention
Browse all Transformer Attention questions for Claude, or explore more Aifoundations topics.
Related questions
Rebuild the model width from headsThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyThe per-layer complexity of self-attention isThe decoder differs from the encoder by addingMulti-head attention uses how many heads in the base model?The encoder of the original Transformer uses how many identical layers?The Transformer architecture replaces recurrence and convolutions with