Markov decision processes

The standard formal model: states, actions, transitions, rewards — with the Markov property.

State $s$ — what the agent knows (full or partial observation).
Action $a$ — discrete or continuous; may depend on $s$.
Transition — deterministic $s'=f(s,a)$ or stochastic $p(s'|s,a)$.
Reward $r$ — scalar feedback.

The agent–environment loop: action out, next state and reward back.

Discount $\gamma$

$\gamma \in [0,1)$ down-weights future rewards so the agent prefers sooner gains and infinite-horizon returns stay finite. Typical values like $0.9$.

Maze

State = cell. Actions = N/E/S/W. Reward = −1 per step, +10 at the exit. Markov: the next cell depends on the current cell and move, not the full history of wandering.

Key takeaways

MDP = RL’s math stage.
Markov: future depends on present state (given the model).
Discount stabilizes long horizons.