Home › Claude › aifoundations › alexnet_cnn_imagenet › Why are ReLU networks faster to train than tanh …
Why are ReLU networks faster to train than tanh networks?
AReLU is supported only on modern GPU hardware
BReLU does not saturate for positive inputs
CReLU has a cheaper memory access pattern
DReLU avoids the need for any learning rate
Answer & Solution
Correct answer: B. ReLU does not saturate for positive inputs
Saturating nonlinearities (tanh, sigmoid) produce near-zero gradients for large |x|, slowing learning. ReLU's gradient is 1 for x > 0, so signal passes through cleanly.
Related questions
Image similarity in AlexNet's last hidden layer is measured byThe SGD update rule for AlexNet usesRemoving any single middle convolutional layer of AlexNet costs roughlyAlexNet was trained for roughly how many epochs on ImageNet?At test time dropout is approximated byAlexNet uses overlapping pooling withAlexNet's second form of data augmentation isAlexNet's first form of data augmentation is