Home › Claude › aifoundations › mlp_and_activations › Why has ReLU largely replaced sigmoid in hidden …
Why has ReLU largely replaced sigmoid in hidden layers?
AReLU runs faster on specialised TPU hardware
BReLU has better behaved gradients during training
CReLU is provably mathematically more expressive
DReLU is required by most modern frameworks
Answer & Solution
Correct answer: B. ReLU has better behaved gradients during training
ReLU's derivative is 0 or 1, mitigating the vanishing gradients that plagued sigmoid networks. Sigmoid's gradient peaks at 0.25 and dies as inputs move from zero.
Related questions
Why are sigmoids still used in recurrent networks (e.g., LSTMs) even after ReLU's rise?A 'thresholding' activation taking 0 or 1 cleanly is impractical for gradient-based learniDepth often beats width for the same expressive power becauseWhy is the sigmoid prone to the vanishing-gradient problem?For an MLP layer with input X ∈ ℝⁿˣᵈ, hidden width h and output q, the weight shapes areParametrised ReLU (pReLU) adds which capability over plain ReLU?The universal approximation theorem (Cybenko 1989) for MLPs states thatThe derivative of sigmoid(x) equals