Home › Claude › aifoundations › linear_regression_and_supervised_learning › Why is STOCHASTIC gradient descent often preferr…
Why is STOCHASTIC gradient descent often preferred over BATCH gradient descent on very large datasets?
ASGD has no oscillation around the minimum
BSGD always finds a lower loss than batch
CSGD can start making progress immediately without scanning the whole dataset; each step is cheap
DSGD requires no learning rate
Answer & Solution
Correct answer: C. SGD can start making progress immediately without scanning the whole dataset; each step is cheap
On large n, batch's per-step cost (full pass over data) becomes prohibitive. SGD makes progress with each example — often reaching a good neighbourhood of the optimum much faster, at the cost of small oscillations.
Related questions
The resulting per-example log-likelihood is, up to constantsClosed-form normal equations require XᵀX to beLogistic regression uses which hypothesis function?The PROBABILISTIC justification for the least-squares cost assumes the noise ε⁽ⁱ⁾ in y⁽ⁱ⁾ Why is the least-squares optimisation for LINEAR regression guaranteed to converge to the The design matrix X for n training examples with d features (plus intercept) has shapeThe closed-form 'normal equations' solution for least-squares linear regression isThe LMS (Widrow–Hoff) update rule on a single training example is