The chosen positional encoding scheme uses
AA separate learned embedding per absolute index
BA relative position bias in attention weights
CRotary half-angle phase shifts on each key vector
DSinusoidal functions of different frequencies
Answer & Solution
Correct answer: D. Sinusoidal functions of different frequencies
PE_(pos, 2i) = sin(pos / 10000^(2i/d_model)); odd dimensions use cos. Wavelengths form a geometric progression from 2π to 10000·2π. Chosen partly so the model can extrapolate to longer sequences than seen in training.
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isResidual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe THREE distinct ways attention is used in the Transformer arePosition-wise feed-forward networks in the Transformer are essentiallyWhat is the maximum path length in self-attention layers?The per-layer complexity of self-attention is