Home › Claude › aifoundations › mlp_and_activations › Why is the sigmoid prone to the vanishing-gradie…
Why is the sigmoid prone to the vanishing-gradient problem?
AIts gradient depends on the batch size used
BIts gradient saturates near zero for |x| large
CIts gradient is identically zero across all inputs
DIts gradient grows without bound as x grows
Answer & Solution
Correct answer: B. Its gradient saturates near zero for |x| large
The sigmoid derivative s(x)(1 − s(x)) approaches zero whenever |x| is large (the function saturates near 0 or 1). Stacking many sigmoid layers multiplies these tiny derivatives, vanishing the gradient.
Related questions
Why are sigmoids still used in recurrent networks (e.g., LSTMs) even after ReLU's rise?A 'thresholding' activation taking 0 or 1 cleanly is impractical for gradient-based learniDepth often beats width for the same expressive power becauseFor an MLP layer with input X ∈ ℝⁿˣᵈ, hidden width h and output q, the weight shapes areParametrised ReLU (pReLU) adds which capability over plain ReLU?Why has ReLU largely replaced sigmoid in hidden layers?The universal approximation theorem (Cybenko 1989) for MLPs states thatThe derivative of sigmoid(x) equals