Home › Claude › aifoundations › mlp_and_activations › Why are sigmoids still used in recurrent network…
Why are sigmoids still used in recurrent networks (e.g., LSTMs) even after ReLU's rise?
ASigmoids are required by every loss function used
BSigmoids guarantee convergence of recurrent training
CSigmoids serve as soft gates controlling information flow
DSigmoids are faster than ReLU on GPU hardware now
Answer & Solution
Correct answer: C. Sigmoids serve as soft gates controlling information flow
In LSTMs and GRUs, sigmoid outputs in (0, 1) work as gates that pass or block fractions of a signal. ReLU has displaced sigmoid in hidden layers but not in this gating role.
Related questions
A 'thresholding' activation taking 0 or 1 cleanly is impractical for gradient-based learniDepth often beats width for the same expressive power becauseWhy is the sigmoid prone to the vanishing-gradient problem?For an MLP layer with input X ∈ ℝⁿˣᵈ, hidden width h and output q, the weight shapes areParametrised ReLU (pReLU) adds which capability over plain ReLU?Why has ReLU largely replaced sigmoid in hidden layers?The universal approximation theorem (Cybenko 1989) for MLPs states thatThe derivative of sigmoid(x) equals