Home › Claude › aifoundations › mlp_and_activations › A 'thresholding' activation taking 0 or 1 cleanl…
A 'thresholding' activation taking 0 or 1 cleanly is impractical for gradient-based learning because
AThresholding scales output range beyond unit interval
BThe optimiser needs continuous gradients to update
CThresholding requires a custom CUDA kernel always
DThresholding violates the universal-approximation claim
Answer & Solution
Correct answer: B. The optimiser needs continuous gradients to update
A hard threshold has zero gradient almost everywhere — there is no signal for gradient descent to follow. Sigmoid was originally a smooth differentiable approximation to a thresholding unit.
Related questions
Why are sigmoids still used in recurrent networks (e.g., LSTMs) even after ReLU's rise?Depth often beats width for the same expressive power becauseWhy is the sigmoid prone to the vanishing-gradient problem?For an MLP layer with input X ∈ ℝⁿˣᵈ, hidden width h and output q, the weight shapes areParametrised ReLU (pReLU) adds which capability over plain ReLU?Why has ReLU largely replaced sigmoid in hidden layers?The universal approximation theorem (Cybenko 1989) for MLPs states thatThe derivative of sigmoid(x) equals