Home › Claude › aifoundations › transformer_attention › Position-wise feed-forward networks in the Trans…
Position-wise feed-forward networks in the Transformer are essentially
ATwo linear layers with ReLU between them
BA single bilinear layer over the entire batch
CA recurrent gated cell over neighbouring tokens
DA pure attention head with no parameters at all
Answer & Solution
Correct answer: A. Two linear layers with ReLU between them
FFN(x) = max(0, xW₁ + b₁)W₂ + b₂. The same weights apply at every position but differ across layers. d_model = 512, d_ff = 2048 in the base.
Related questions
The architectural property that lets self-attention parallelise where RNNs cannot isResidual connections and layer normalisation are applied asWhy does the decoder need to mask future positions in self-attention?Multi-head attention is preferred over a single d_model-dimensional head becauseThe THREE distinct ways attention is used in the Transformer areThe chosen positional encoding scheme usesWhat is the maximum path length in self-attention layers?The per-layer complexity of self-attention is