alleryStart learning →Position-wise feed-forward networks in the Transformer are essentially
ATwo linear layers with ReLU between them
BA single bilinear layer over the entire batch
CA recurrent gated cell over neighbouring tokens
DA pure attention head with no parameters at all
Answer & Solution
Correct answer: A. Two linear layers with ReLU between them
FFN(x) = max(0, xW₁ + b₁)W₂ + b₂.
The same weights apply at every position but differ across layers.
d_model = 512, d_ff = 2048 in the base.
Keep practising Transformer Attention
Browse all Transformer Attention questions for Claude, or explore more Aifoundations topics.
Related questions
Rebuild the model width from headsThe chosen positional encoding scheme usesThe per-layer complexity of self-attention isThe decoder differs from the encoder by addingMulti-head attention uses how many heads in the base model?The encoder of the original Transformer uses how many identical layers?Scaled dot-product attention, per the paper, computes the output asThe Transformer architecture replaces recurrence and convolutions with