Home › Claude › aifoundations › mlp_and_activations › Depth often beats width for the same expressive …
Depth often beats width for the same expressive power because
AWider networks cannot be optimised on a GPU
BDeeper networks always have lower parameter count
CDeeper networks compose functions more compactly
DWider networks always overfit on tabular data
Answer & Solution
Correct answer: C. Deeper networks compose functions more compactly
A given function may need exponentially many units in a shallow network but only polynomially many in a deeper one. Composition gives depth its modelling efficiency.
Related questions
Why are sigmoids still used in recurrent networks (e.g., LSTMs) even after ReLU's rise?A 'thresholding' activation taking 0 or 1 cleanly is impractical for gradient-based learniWhy is the sigmoid prone to the vanishing-gradient problem?For an MLP layer with input X ∈ ℝⁿˣᵈ, hidden width h and output q, the weight shapes areParametrised ReLU (pReLU) adds which capability over plain ReLU?Why has ReLU largely replaced sigmoid in hidden layers?The universal approximation theorem (Cybenko 1989) for MLPs states thatThe derivative of sigmoid(x) equals