Home › Claude › aifoundations › transformer_attention › The √d_k scaling in scaled dot-product attention…
The √d_k scaling in scaled dot-product attention exists to
AMatch the dimensionality of the value vector input
BSpeed up the matrix multiplication on TPU hardware
CPrevent dot products growing large and starving softmax
DAvoid integer overflow on GPU floating-point math
Answer & Solution
Correct answer: C. Prevent dot products growing large and starving softmax
Without scaling, dot products grow proportional to d_k, pushing softmax into saturation where gradients vanish. Dividing by √d_k keeps the softmax in its sensitive region.
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isResidual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe THREE distinct ways attention is used in the Transformer areThe chosen positional encoding scheme usesPosition-wise feed-forward networks in the Transformer are essentiallyWhat is the maximum path length in self-attention layers?