Bellman target
Q-learning uses immediate reward plus the discounted maximum next-state action value as a one-step target.
y = r + γ maxₐ′Q(s′,a′)
0.0episode return
0.00last TD error
20%exploration
0episodes
TD learning bootstraps from an estimate of the future.
TD error
The difference between target and current estimate is a local surprise signal. Positive error raises value; negative error lowers it.
δ = y − Q(s,a)
Incremental update
Learning rate controls how far the estimate moves toward the target after each transition.
Q ← Q + αδ
Discount controls the horizon.
Small γ prioritizes immediate reward. Values near one propagate distant reward farther, but can increase variance and make continuing tasks numerically difficult.
Exploration collects counterfactual evidence.
An epsilon-greedy policy usually chooses the best known action and sometimes tries another. Without exploration, early noise can lock the agent into a poor route.
Off-policy learning separates behavior from target.
Q-learning may explore with one behavior policy while updating toward the greedy target policy. This flexibility also creates instability with function approximation.
Learning questions
Why does reward propagate gradually?
One-step TD updates only the visited state-action pair from its successor. Repeated experience moves information backward through predecessor states.
Can a high learning rate be faster?
Initially, but noisy or changing targets can cause oscillation. Convergence theory typically requires decreasing step sizes under tabular assumptions.
How is Q-learning different from policy gradients?
Q-learning estimates action values and derives a policy. Policy gradients directly optimize policy parameters using sampled returns or advantages.
Primary sources
Watkins and Dayan (1992) established Q-learning convergence. Sutton and Barto provide the standard reinforcement-learning treatment. Mnih et al. (2015) combines Q-learning with deep neural networks. Double DQN addresses value overestimation.