Activation functions
Nonlinearities make deep stacks expressive. Two classics: tanh and ReLU.
- tanh: smooth, outputs in $(-1,1)$.
- ReLU: $\max(0,x)$ — cheap; zeros out negatives (“dead” units possible).
Why nonlinearity is mandatory
Two linear layers: $W_2(W_1 x)= (W_2 W_1)x$ — still linear. Insert ReLU or tanh between them and the overall map can bend into complex shapes.
Gradients
tanh can suffer vanishing gradients in deep stacks; ReLU can explode on the positive side. In practice: try both and validate.
Key takeaways
- Nonlinearity enables deep models.
- Know tanh vs ReLU tradeoffs.
- No universal best activation.