Home / 11. Reinforcement learning / Policy and value

Policy and value

A policy says what to do; a value says how good a state is under good play.

Policy

$\pi(s)\rightarrow a$ (or a distribution over actions). The thing we ultimately want to act with.

Value (deterministic sketch)

$$v(s)=\max_a\big(r(s,a)+\gamma v(s')\big)$$ with $s'=f(s,a)$. Optimal action: $\\arg\\max_a(r+\\gamma v(s'))$.

Ladder of states 0–9

Only the ends pay reward 1 for “staying put”. With $\gamma=0.9$, $v(0)=v(9)=10$ and values decay toward the middle. Good policy: walk toward the nearer end.

Key takeaways

  • Policy = behaviour.
  • Value = expected return from a state.
  • Greedy on $v$ yields a strong policy when $v$ is optimal.