Derivatives and gradients

A derivative is a local slope. A gradient packs all partial slopes into one vector that points uphill.

For $y=f(x)$, the derivative $f'(x)$ says how $y$ changes when $x$ nudges a tiny bit. Geometrically it is the tangent slope.

$$f'(x)=\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}$$

For several inputs, take a partial derivative w.r.t. each coordinate and stack them:

$$\nabla f(x)=\Big(\frac{\partial f}{\partial x_1},\ldots,\frac{\partial f}{\partial x_M}\Big)$$

Hill climbing intuition

Standing on a foggy hillside, the gradient tells you the steepest uphill direction. To minimize a loss, walk the opposite way (downhill).