Home › Claude › aifoundations › mlp_and_activations › Why does a stack of affine layers collapse witho…
Why does a stack of affine layers collapse without an activation function?
AStacking them runs out of memory on the GPU
BAffine of affine is affine, equivalent to one layer
CThe optimiser refuses to update multiple layers
DStacking is only allowed with skip connections
Answer & Solution
Correct answer: B. Affine of affine is affine, equivalent to one layer
If H = XW¹ + b¹ and O = HW² + b², then O = X(W¹W²) + (b¹W² + b²), a single linear layer. Nonlinearity is what gives depth its power.
Related questions
Why are sigmoids still used in recurrent networks (e.g., LSTMs) even after ReLU's rise?A 'thresholding' activation taking 0 or 1 cleanly is impractical for gradient-based learniDepth often beats width for the same expressive power becauseWhy is the sigmoid prone to the vanishing-gradient problem?For an MLP layer with input X ∈ ℝⁿˣᵈ, hidden width h and output q, the weight shapes areParametrised ReLU (pReLU) adds which capability over plain ReLU?Why has ReLU largely replaced sigmoid in hidden layers?The universal approximation theorem (Cybenko 1989) for MLPs states that