Automatic differentiation

Autodiff computes exact gradients of whatever forward computation you wrote — the engine inside modern DL frameworks.

You should not hand-derive derivatives for a 50-layer net. Automatic differentiation records each elementary op in the forward pass, then reverse-accumulates gradients with the chain rule.

Tiny graph

Let $a=w\\cdot x$, $y=\\tanh(a)$, $L=(y-t)^2$. Autodiff knows $dL/dy$, $dy/da=1-\\tanh^2(a)$, $da/dw=x$, and multiplies them for $dL/dw$ — exactly, up to floating-point error.

Not the same as

Numerical gradients ($\approx [f(w+\epsilon)-f(w)]/\epsilon$) are slow and approximate. Autodiff is exact for the computational graph and efficient.

Key takeaways

Autodiff ≠ finite differences.
You define forward; gradients come free.
Enables deep, messy models.