alleryStart learning →The chosen positional encoding scheme uses
AA separate learned embedding per absolute index
BA relative position bias in attention weights
CRotary half-angle phase shifts on each key vector
DSinusoidal functions of different frequencies
Answer & Solution
Correct answer: D. Sinusoidal functions of different frequencies
PE_(pos, 2i) = sin(pos / 10000^(2i/d_model)); odd dimensions use cos.
Wavelengths form a geometric progression from 2π to 10000·2π.
Chosen partly so the model can extrapolate to longer sequences than seen in training.
Keep practising Transformer Attention
Browse all Transformer Attention questions for Claude, or explore more Aifoundations topics.
Related questions
Rebuild the model width from headsPosition-wise feed-forward networks in the Transformer are essentiallyThe per-layer complexity of self-attention isThe decoder differs from the encoder by addingMulti-head attention uses how many heads in the base model?The encoder of the original Transformer uses how many identical layers?Scaled dot-product attention, per the paper, computes the output asThe Transformer architecture replaces recurrence and convolutions with