What is Q-learning and where does it fit in reinforcement learning?
Q-learning is a model-free, off-policy, temporal-difference reinforcement learning algorithm that learns the action-value function Q(s, a), the expected discounted return of taking action a in state s and acting optimally afterwards, directly from experience, without a model of the environment’s transitions or rewards. After each step it nudges Q(s, a) toward the target r + γ·max over a′ of Q(s′, a′), and the greedy policy “pick the action with the highest Q” becomes optimal as the estimates converge. It is the foundation of value-based RL and of deep variants such as DQN.
Read full answerHide full answer
Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x
Where It Fits
| Dimension | Q-learning |
|---|---|
| Model-based vs model-free | Model-free: learns from sampled transitions |
| Value-based vs policy-based | Value-based: learns Q, derives the policy greedily |
| On-policy vs off-policy | Off-policy: learns the greedy policy while behaving ε-greedily |
| Monte Carlo vs TD | TD: bootstraps from its own estimate of the next state |
| Tabular vs approximate | Tabular for small problems; neural networks in DQN |
The Update
Q(s, a) ← Q(s, a) + α·[r + γ·max_a′ Q(s′, a′) − Q(s, a)]
-
α (learning rate): step size toward the target.
-
γ (discount factor): weight of future rewards.
-
r + γ·max Q(s′, ·): the TD target.
-
The bracket is the TD error.
Code Example
import numpy as np
# 1-D corridor: states 0..4, start at 0, reward +1 for reaching state 4 (terminal)
n_states, actions = 5, [-1, +1] # action 0 = left, 1 = right
Q = np.zeros((n_states, 2))
alpha, gamma, epsilon = 0.5, 0.9, 0.2
rng = np.random.default_rng(42)
for episode in range(200):
s = 0
while s != 4:
a = rng.integers(2) if rng.random() < epsilon else int(np.argmax(Q[s]))
s_next = min(max(s + actions[a], 0), 4)
r = 1.0 if s_next == 4 else 0.0
target = r if s_next == 4 else r + gamma * Q[s_next].max()
Q[s, a] += alpha * (target - Q[s, a]) # Q-learning update
s = s_next
print(np.round(Q, 3))
print("greedy policy:", ["L" if q[0] > q[1] else "R" for q in Q[:4]]) # ['R', 'R', 'R', 'R']
# Optimal values for "right": 0.729, 0.81, 0.9, 1.0 (gamma^steps to the goal)
Strengths and Limitations
-
Strengths: simple, no model needed, provably converges in the tabular case, learns the optimal policy off-policy (can reuse old data).
-
Limitations: tables don’t scale to large or continuous state spaces; max operator causes overestimation; needs exploration; discrete actions only.
Common Pitfalls
-
Too little exploration, so the agent never finds better actions.
-
Bootstrapping from a terminal state (its value must be 0).
-
Expecting tabular Q-learning to work on raw images or continuous states.
Interview Tips
-
Say “model-free, off-policy, TD control” and write the update.
-
Contrast it with SARSA (on-policy) and policy gradient methods.