Gradient descent
Repeatedly step a little opposite the gradient to drive a loss downhill.
When you cannot solve $\nabla f(x)=0$ by hand, take small downhill steps instead. Each step uses only local slope information.
Update rule
$$x^{(t+1)}=x^{(t)}-\alpha\,\nabla f(x^{(t)})$$
$\\alpha>0$ is the step size (learning rate). Start from some $x^{(0)}$ and iterate until changes are tiny.
1D walkthrough
Suppose $f(x)=(x-3)^2$, so $\nabla f=2(x-3)$. From $x=0$ with $\alpha=0.1$:
- Step 1: $0 - 0.1\cdot 2(0-3)=0.6$
- Step 2: $0.6 - 0.1\cdot 2(0.6-3)=1.08$
You drift toward the minimum at $x=3$. In many dimensions the same idea applies using the full gradient vector.
Key takeaways
- Minus gradient = descent direction.
- Need an initialization.
- With small enough $\alpha$, mild assumptions ⇒ convergence to a local min.