Skip to content
Machine Learning & Data Science

Q-Learning interview questions and answers

44 questions, 38 with code. About 2 h 23 min to read them all.

By TestLab · Updated

Plan for an interview
44 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is Q-learning and where does it fit in reinforcement learning?

Q-learning is a model-free, off-policy, temporal-difference reinforcement learning algorithm that learns the action-value function Q(s, a), the expected discounted return of taking action a in state s and acting optimally afterwards, directly from experience, without a model of the environment’s transitions or rewards. After each step it nudges Q(s, a) toward the target r + γ·max over a′ of Q(s′, a′), and the greedy policy “pick the action with the highest Q” becomes optimal as the estimates converge. It is the foundation of value-based RL and of deep variants such as DQN.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Where It Fits

Dimension Q-learning
Model-based vs model-free Model-free: learns from sampled transitions
Value-based vs policy-based Value-based: learns Q, derives the policy greedily
On-policy vs off-policy Off-policy: learns the greedy policy while behaving ε-greedily
Monte Carlo vs TD TD: bootstraps from its own estimate of the next state
Tabular vs approximate Tabular for small problems; neural networks in DQN

The Update

Q(s, a) ← Q(s, a) + α·[r + γ·max_a′ Q(s′, a′) − Q(s, a)]

  • α (learning rate): step size toward the target.

  • γ (discount factor): weight of future rewards.

  • r + γ·max Q(s′, ·): the TD target.

  • The bracket is the TD error.

Code Example

import numpy as np

# 1-D corridor: states 0..4, start at 0, reward +1 for reaching state 4 (terminal)
n_states, actions = 5, [-1, +1]          # action 0 = left, 1 = right
Q = np.zeros((n_states, 2))
alpha, gamma, epsilon = 0.5, 0.9, 0.2
rng = np.random.default_rng(42)

for episode in range(200):
    s = 0
    while s != 4:
        a = rng.integers(2) if rng.random() < epsilon else int(np.argmax(Q[s]))
        s_next = min(max(s + actions[a], 0), 4)
        r = 1.0 if s_next == 4 else 0.0
        target = r if s_next == 4 else r + gamma * Q[s_next].max()
        Q[s, a] += alpha * (target - Q[s, a])     # Q-learning update
        s = s_next

print(np.round(Q, 3))
print("greedy policy:", ["L" if q[0] > q[1] else "R" for q in Q[:4]])   # ['R', 'R', 'R', 'R']
# Optimal values for "right": 0.729, 0.81, 0.9, 1.0 (gamma^steps to the goal)

Strengths and Limitations

  • Strengths: simple, no model needed, provably converges in the tabular case, learns the optimal policy off-policy (can reuse old data).

  • Limitations: tables don’t scale to large or continuous state spaces; max operator causes overestimation; needs exploration; discrete actions only.

Common Pitfalls

  • Too little exploration, so the agent never finds better actions.

  • Bootstrapping from a terminal state (its value must be 0).

  • Expecting tabular Q-learning to work on raw images or continuous states.

Interview Tips

  • Say “model-free, off-policy, TD control” and write the update.

  • Contrast it with SARSA (on-policy) and policy gradient methods.

2

What is a Q-table in Q-learning?

A Q-table is the lookup table that tabular Q-learning uses to store its estimate of the action-value Q(s, a) for every combination of state and action: rows are states, columns are actions, and each cell holds the expected discounted return of taking that action in that state and acting optimally afterwards. The agent reads the table to choose actions (the greedy action is the column with the largest value in the current row) and writes to it after each step with the Q-learning update; the table’s size, |S| × |A|, is why tabular methods only work for small, discrete problems.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x and Gymnasium 1.x

Structure

Action: up Action: down Action: left Action: right
State 0 0.12 0.35 0.10 0.48
State 1 0.20 0.55 0.18 0.30
… … … … …
  • Read: greedy action = argmax over the row; state value V(s) = max over the row.

  • Write: Q[s, a] += α·(r + γ·max Q[s′] − Q[s, a]).

  • Initialize: usually zeros; optimistic values encourage exploration.

Code Example

import gymnasium as gym
import numpy as np

env = gym.make("FrozenLake-v1", is_slippery=False)   # 4x4 grid: 16 states, 4 actions
Q = np.zeros((env.observation_space.n, env.action_space.n))
print("Q-table shape:", Q.shape)                       # (16, 4)

alpha, gamma, epsilon = 0.8, 0.95, 0.1
rng = np.random.default_rng(42)
obs, info = env.reset(seed=42)
for episode in range(2000):
    s, _ = env.reset()
    done = False
    while not done:
        a = env.action_space.sample() if rng.random() < epsilon else int(np.argmax(Q[s]))
        s_next, r, terminated, truncated, _ = env.step(a)
        target = r + gamma * Q[s_next].max() * (not terminated)   # no bootstrap after terminal
        Q[s, a] += alpha * (target - Q[s, a])
        s, done = s_next, terminated or truncated

arrows = np.array(["<", "v", ">", "^"])            # FrozenLake actions: 0 left, 1 down, 2 right, 3 up
print(arrows[Q.argmax(axis=1)].reshape(4, 4))       # greedy action in each cell
env.close()

When a Table Works and When It Doesn’t

Works Doesn’t work
Small discrete state and action spaces (grid worlds, Taxi, small games) Images, continuous sensors, huge combinatorial states
States revisited often States rarely or never seen twice
Need for exact, interpretable values Need to generalize between similar states

Large problems replace the table with a function approximator (linear features or a neural network, as in DQN).

Common Pitfalls

  • Ties in a zero-initialized row: argmax always picks the first action; break ties randomly or explore.

  • Forgetting to zero the bootstrap at terminal states.

  • Discretizing continuous states so finely that the table becomes huge and sparse.

Interview Tips

  • Describe rows, columns, reading (argmax) and writing (update).

  • Explain the |S| × |A| scaling problem and the move to function approximation.

3

How does Q-learning differ from policy gradient methods?

Q-learning is a value-based method: it learns action values Q(s, a) with temporal-difference updates and acts greedily with respect to them, while policy gradient methods (REINFORCE, actor-critic, PPO) directly parameterize the policy π(a | s; θ) and adjust θ by gradient ascent on expected return. Q-learning is off-policy and sample-efficient with discrete actions, but struggles with continuous actions and can be unstable with function approximation; policy gradients handle continuous and stochastic policies naturally and optimize the objective directly, at the cost of higher-variance gradient estimates and usually on-policy data.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Comparison

Aspect Q-learning Policy gradient
What is learned Action-value function Q(s, a) Policy π(a | s; θ) (often plus a value baseline)
Policy Implicit: argmax Q (ε-greedy for exploration) Explicit, can be stochastic
Data Off-policy (replay buffers, old data) Usually on-policy (fresh rollouts), PPO reuses a little
Action spaces Discrete (max over actions) Discrete and continuous
Update TD target with bootstrapping ∇θ log π(a | s)·return or advantage
Variance / bias Lower variance, bias from bootstrapping Higher variance, unbiased (REINFORCE)
Stochastic optimal policies Can’t represent them Can
Convergence Tabular: guaranteed; deep: can diverge Local optimum of the objective
Examples Q-learning, DQN, Double/Dueling DQN, Rainbow REINFORCE, A2C, PPO, TRPO; DDPG/SAC are actor-critic hybrids

Code Example

import numpy as np

rng = np.random.default_rng(42)
true_means = np.array([0.2, 0.8])            # two-armed bandit (one state)

# Value-based: estimate Q for each action, act epsilon-greedily
Q, counts = np.zeros(2), np.zeros(2)
for t in range(2000):
    a = rng.integers(2) if rng.random() < 0.1 else int(np.argmax(Q))
    r = float(rng.random() < true_means[a])
    counts[a] += 1
    Q[a] += (r - Q[a]) / counts[a]
print("Q estimates:", np.round(Q, 2))        # close to [0.2, 0.8]

# Policy gradient (REINFORCE): softmax policy over preferences theta
theta, lr, baseline = np.zeros(2), 0.1, 0.0
for t in range(2000):
    probs = np.exp(theta) / np.exp(theta).sum()
    a = rng.choice(2, p=probs)
    r = float(rng.random() < true_means[a])
    grad_log = -probs
    grad_log[a] += 1.0                        # gradient of log pi(a) for a softmax
    theta += lr * (r - baseline) * grad_log
    baseline += 0.01 * (r - baseline)         # running average reward baseline
print("policy:", np.round(np.exp(theta) / np.exp(theta).sum(), 2))   # most probability on arm 1

When to Use Which

  • Q-learning / DQN: discrete actions, sample efficiency matters, off-policy data available (games, recommendation slates with few actions).

  • Policy gradient / actor-critic: continuous control (robotics), stochastic policies, very large action spaces, or when stable on-policy training (PPO) is preferred.

Common Pitfalls

  • Using DQN for continuous actions by fine discretization, which explodes the action space.

  • Policy gradients without a baseline, which gives very noisy updates.

Interview Tips

  • Contrast “learn values, act greedily” with “learn the policy directly”.

  • Mention actor-critic methods as the bridge between the two.

4

What is the action-value function in Q-learning?

*The action-value function Q^π(s, a) is the expected discounted return when the agent starts in state s, takes action a, and then follows policy π; the optimal action-value function Q(s, a) is the same quantity under the best possible policy, and it is what Q-learning estimates.** Knowing Q* is enough to act optimally without a model of the environment, because the best action in any state is simply argmax over a of Q*(s, a). It differs from the state-value function V(s), which scores states rather than state-action pairs.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Definitions

  • Return: G_t = r{t+1} + γ·r{t+2} + γ²·r_{t+3} + …

  • Q^π(s, a) = E_π[G_t | s_t = s, a_t = a].

  • V^π(s) = E_π[G_t | s_t = s] = Σ_a π(a | s)·Q^π(s, a).

  • Optimal: Q(s, a) = max over π of Q^π(s, a); V(s) = max_a Q*(s, a).

  • Bellman optimality: Q*(s, a) = E[r + γ·max_a′ Q*(s′, a′) | s, a].

Q vs V

Aspect V(s) Q(s, a)
Scores States State-action pairs
Acting greedily needs A model (to look ahead one step) Nothing else: argmax over actions
Size (tabular) |S| |S| × |A|
Used by TD(0) prediction, actor-critic critics Q-learning, SARSA, DQN
Advantage — A(s, a) = Q(s, a) − V(s)

Code Example

import numpy as np

# Two-state example computed exactly: state 0 -> "stay" (r=1, back to 0) or "go" (r=0, to state 1)
# State 1 is terminal with a one-time reward 10 for entering it (given on the "go" transition)
gamma = 0.9
# Value iteration on Q for state 0
Q = np.zeros(2)                      # Q[0] = stay, Q[1] = go
for _ in range(500):
    Q = np.array([1 + gamma * Q.max(),   # stay: reward 1, still in state 0
                  10.0])                 # go: reward 10, then terminal (value 0)
print(np.round(Q, 2))                # stay = 1 + 0.9 * max(stay, go)
# If go is best, stay = 1 + 0.9 * 10 = 10.0 and go = 10.0 (a tie). Always staying would be worth
# 1 / (1 - 0.9) = 10, so both choices are optimal here; with gamma = 0.95 staying (worth 20) wins.

Why It Matters

  • The greedy policy from Q needs no model: useful when transitions are unknown.

  • Q-values compare actions directly, which is what control requires.

  • In deep RL, a network outputs Q(s, ·) for all actions at once (DQN).

Common Pitfalls

  • Confusing Q^π (for a given policy) with Q* (optimal).

  • Forgetting that Q includes the immediate reward of the chosen action.

  • Treating Q-values from function approximation as exact; they are often overestimated.

Interview Tips

  • Define Q with the expectation and the Bellman optimality equation.

  • Explain why Q makes model-free control possible where V doesn’t.

5

What are the roles of the learning rate and discount factor in Q-learning?

The learning rate α (between 0 and 1) controls how far each Q-learning update moves the current estimate toward the new target: Q ← Q + α·(target − Q), so large α learns fast but noisily, and small α learns slowly but averages out randomness; the discount factor γ (between 0 and 1) controls how much future rewards count in that target, r + γ·max Q(s′, ·), so γ near 0 makes the agent short-sighted and γ near 1 makes it value long-term outcomes. α is a learning hyperparameter, while γ is part of the problem definition: it changes which policy is optimal.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Learning Rate α

Value Effect
α = 0 No learning
Small (0.01-0.1) Stable, slow, good for stochastic environments
Large (0.5-1.0) Fast in deterministic environments; noisy in stochastic ones
α = 1 Replace the estimate with the latest target (fine only if deterministic)
Decaying α_t Required for convergence guarantees: Σ α_t = ∞, Σ α_t² < ∞ (e.g. 1/visits)

Discount Factor γ

Value Effect
γ = 0 Only the immediate reward matters
0.9 Rewards 10 steps ahead weigh 0.9¹⁰ ≈ 0.35
0.99 Long horizon; effective horizon ≈ 1 / (1 − γ) = 100 steps
γ = 1 Undiscounted; only safe for episodic tasks that always end

Code Example

import numpy as np

# Effect of alpha: estimating a noisy reward with mean 1.0
rng = np.random.default_rng(42)
rewards = rng.normal(1.0, 1.0, 500)
for alpha in [0.01, 0.1, 0.9]:
    q, history = 0.0, []
    for r in rewards:
        q += alpha * (r - q)                     # same form as the Q-learning update
        history.append(q)
    print(f"alpha={alpha}: final {q:.2f}, std of last 100 estimates {np.std(history[-100:]):.2f}")
# Small alpha: smooth but slow; large alpha: fast but jumps with every noisy reward

# Effect of gamma: value of a reward of 10 received 5 steps from now
for gamma in [0.5, 0.9, 0.99]:
    print(f"gamma={gamma}: present value {10 * gamma ** 5:.2f}")
# 0.31, 5.90, 9.51

How γ Changes the Optimal Policy

  • Small γ: the agent prefers quick small rewards (grab a coin now).

  • Large γ: it accepts short-term costs for larger later rewards (walk around a pit to reach the goal).

  • With step penalties, γ also affects how strongly the agent prefers shorter paths.

Common Pitfalls

  • α too high in stochastic environments, so Q-values never settle.

  • γ = 1 in continuing tasks, so values grow without bound.

  • Tuning γ like a hyperparameter without realizing it changes the objective.

Interview Tips

  • Explain α as a step size and γ as a horizon with 1 / (1 − γ).

  • Mention the Robbins-Monro conditions for α in convergence proofs.

6

How is the exploration-exploitation trade-off handled in Q-learning?

The exploration-exploitation trade-off is the tension between exploiting the action that currently looks best according to the Q-table and exploring other actions that might be better but whose values are still uncertain; Q-learning needs exploration because it can only learn about actions it actually tries. It is usually handled with ε-greedy action selection, taking a random action with probability ε and the greedy action otherwise, with ε decayed over time, and alternatives include softmax (Boltzmann) exploration, optimistic initial values, upper confidence bounds and count-based bonuses.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Strategies

Strategy How Notes
ε-greedy Random action with probability ε Simple; standard default
Decaying ε ε from 1.0 down to 0.01-0.1 over training Explore early, exploit later
Softmax / Boltzmann P(a) ∝ exp(Q(s, a) / τ) Explores promising actions more; temperature τ
Optimistic initialization Start Q high (e.g. max possible return) Untried actions look attractive
UCB-style bonus Q + c·sqrt(ln t / N(s, a)) Prefers rarely tried actions
Count-based / curiosity Intrinsic reward for novel states Sparse-reward problems (deep RL)
Noisy networks Learned parameter noise (NoisyNet DQN) Replaces ε in Rainbow

Code Example

import numpy as np

rng = np.random.default_rng(42)

def epsilon_greedy(q_row, epsilon):
    """Random action with probability epsilon, otherwise greedy (ties broken randomly)."""
    if rng.random() < epsilon:
        return int(rng.integers(len(q_row)))
    best = np.flatnonzero(q_row == q_row.max())
    return int(rng.choice(best))

def softmax_action(q_row, tau):
    prefs = (q_row - q_row.max()) / tau          # subtract max for numerical stability
    probs = np.exp(prefs) / np.exp(prefs).sum()
    return int(rng.choice(len(q_row), p=probs))

def epsilon_schedule(episode, start=1.0, end=0.05, decay_episodes=500):
    """Linear decay from start to end, then constant."""
    fraction = min(episode / decay_episodes, 1.0)
    return start + fraction * (end - start)

q_row = np.array([0.2, 0.5, 0.1])
print([round(epsilon_schedule(e), 3) for e in (0, 250, 500, 1000)])   # [1.0, 0.525, 0.05, 0.05]
picks = [epsilon_greedy(q_row, 0.1) for _ in range(1000)]
print("share of greedy action 1:", round(np.mean(np.array(picks) == 1), 2))   # about 0.93
picks = [softmax_action(q_row, 0.1) for _ in range(1000)]
print("softmax share of action 1:", round(np.mean(np.array(picks) == 1), 2))

With ε = 0.1 and 3 actions, the greedy action is chosen with probability 0.9 + 0.1/3 ≈ 0.93.

Why It Matters

  • Without exploration, early lucky or unlucky rewards lock the agent into a suboptimal policy.

  • Too much exploration wastes experience and lowers reward during training.

  • Q-learning is off-policy: it learns the greedy policy’s values even while exploring.

Common Pitfalls

  • ε decaying too fast, before rare rewarding states are found.

  • Evaluating with ε > 0 and reporting it as the learned policy’s performance (evaluate greedily).

  • Breaking ties always toward the first action with zero-initialized tables.

Interview Tips

  • Describe ε-greedy with decay and one alternative (softmax or UCB).

  • Explain why off-policy learning lets exploration coexist with learning the greedy policy.

7

What is an episode in Q-learning?

An episode is one complete run of interaction between the agent and the environment, from an initial state until a terminal state is reached (the goal, a failure such as falling in a hole, or the end of a game) or until a time limit cuts it off; Q-learning is usually trained over many episodes, resetting the environment at the start of each. Within an episode the agent repeatedly observes a state, chooses an action, receives a reward and the next state, and updates its Q-values; the episode’s total (discounted) reward is the usual measure of how well the agent is doing.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with Gymnasium 1.x and NumPy 2.x

Episode Structure

  1. reset() → initial state s₀.

  2. Loop: choose aₜ (ε-greedy), step(aₜ) → rₜ₊₁, sₜ₊₁, terminated, truncated.

  3. Update Q(sₜ, aₜ).

  4. Stop when terminated (a terminal state of the task) or truncated (time limit or other external cut-off).

Terminated vs Truncated

Signal Meaning Bootstrap in the Q target?
terminated=True Real end of the task (goal reached, agent died) No: the next state’s value is 0
truncated=True Cut off by a time limit Yes: the state still has future value

Gymnasium returns both flags from env.step(); mixing them up biases the Q-values.

Episodic vs Continuing Tasks

Type Example Notes
Episodic Games, mazes, FrozenLake, Taxi Natural end; γ = 1 is possible
Continuing Server load balancing, trading No end; use γ < 1, or average reward

Code Example

import gymnasium as gym
import numpy as np

env = gym.make("FrozenLake-v1", is_slippery=True)    # time limit of 100 steps per episode
Q = np.zeros((env.observation_space.n, env.action_space.n))
alpha, gamma = 0.1, 0.99
rng = np.random.default_rng(42)
env.reset(seed=42)
returns, lengths = [], []

for episode in range(5000):
    s, info = env.reset()                              # start of an episode
    epsilon = max(0.05, 1.0 - episode / 3000)
    total, steps, done = 0.0, 0, False
    while not done:
        a = env.action_space.sample() if rng.random() < epsilon else int(np.argmax(Q[s]))
        s_next, r, terminated, truncated, info = env.step(a)
        target = r + (0.0 if terminated else gamma * Q[s_next].max())   # bootstrap unless terminal
        Q[s, a] += alpha * (target - Q[s, a])
        s, total, steps = s_next, total + r, steps + 1
        done = terminated or truncated                 # end of the episode
    returns.append(total)
    lengths.append(steps)

print("success rate, last 500 episodes:", round(float(np.mean(returns[-500:])), 2))
print("average episode length:", round(float(np.mean(lengths[-500:])), 1))
env.close()

Common Pitfalls

  • Treating time-limit truncation as a terminal state.

  • Not resetting per-episode variables (total reward, eligibility traces).

  • Judging progress by single episodes; average over many (they are noisy).

Interview Tips

  • Define an episode and the terminal/truncated distinction.

  • Mention episodic vs continuing tasks and what that implies for γ.

8

What are the state space and action space in Q-learning?

The state space is the set of all situations the agent can observe (positions on a grid, game boards, sensor readings) and the action space is the set of all moves it can make in a state (up, down, left, right; buy, sell, hold); in tabular Q-learning both must be discrete and small enough that the Q-table, with one entry per state-action pair, fits in memory and every pair can be visited often. The state should contain the information needed to predict future rewards (the Markov property), and the design of both spaces often decides whether Q-learning succeeds.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with Gymnasium 1.x and NumPy 2.x

Types

Space Discrete Continuous
State Grid cells, board positions (Discrete(n)) Positions, velocities, images (Box)
Action Moves, choices (Discrete(n)) Steering angle, torque (Box)
Tabular Q-learning Works Needs discretization or function approximation
DQN Any state (network input) Discrete actions only

Designing the State

  • Markov property: the state should summarize everything relevant; add velocity or recent history if a single snapshot isn’t enough.

  • Compact: drop irrelevant details; |S| grows exponentially with the number of variables.

  • Discretize continuous variables with bins (tile coding for overlapping bins).

  • Encode combinations as one index: e.g. Taxi-v3 encodes taxi row, column, passenger location and destination into 500 states.

Designing the Action Space

  • Keep actions few and meaningful.

  • Mask invalid actions (set their Q to −∞ when selecting).

  • Continuous actions need discretization, or actor-critic methods (DDPG, SAC).

Code Example

import gymnasium as gym
import numpy as np

taxi = gym.make("Taxi-v3")
print(taxi.observation_space, taxi.action_space)        # Discrete(500) Discrete(6)
print("Q-table entries:", taxi.observation_space.n * taxi.action_space.n)   # 3000

cart = gym.make("CartPole-v1")
print(cart.observation_space.shape, cart.action_space)  # (4,) Discrete(2)

# Discretize CartPole's continuous state into bins for a Q-table
bins = [np.linspace(-2.4, 2.4, 7), np.linspace(-3, 3, 7),
        np.linspace(-0.21, 0.21, 7), np.linspace(-3.5, 3.5, 7)]

def discretize(obs):
    idx = [int(np.digitize(x, b)) for x, b in zip(obs, bins)]   # each index in 0..7
    return int(np.ravel_multi_index(idx, (8, 8, 8, 8)))          # one integer state

obs, info = cart.reset(seed=42)
print("discrete state:", discretize(obs), "of", 8 ** 4)          # 4096 states
taxi.close()
cart.close()

Common Pitfalls

  • States that aren’t Markov (missing velocity), so the same state needs different actions.

  • Too many discretization bins, leaving most table entries unvisited.

  • Huge action spaces for tabular methods (combinatorial choices).

Interview Tips

  • Mention the Markov property and the |S| × |A| table size.

  • Explain discretization for continuous states and why DQN still needs discrete actions.

9

How are Q-values updated in Q-learning?

Q-values are updated after every step with the temporal-difference rule Q(s, a) ← Q(s, a) + α·[r + γ·max over a′ of Q(s′, a′) − Q(s, a)]: the agent takes action a in state s, observes reward r and next state s′, forms a target from the reward plus the discounted value of the best next action, and moves its current estimate a fraction α toward that target. If s′ is terminal, the target is just r. Repeating this over many transitions propagates reward information backward through the states until the Q-values approximate the optimal action values.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

The Steps

  1. Observe state s.

  2. Choose action a (ε-greedy on Q[s]).

  3. Take it; observe r and s′ (and whether the episode terminated).

  4. TD target: y = r if terminal, else r + γ·max_a′ Q(s′, a′).

  5. TD error: δ = y − Q(s, a).

  6. Update: Q(s, a) += α·δ.

  7. s ← s′; repeat.

Worked Example

  • α = 0.5, γ = 0.9, current Q(s, a) = 2.0, reward r = 1, and max Q(s′, ·) = 4.0.

  • Target y = 1 + 0.9 × 4.0 = 4.6.

  • TD error δ = 4.6 − 2.0 = 2.6.

  • New Q(s, a) = 2.0 + 0.5 × 2.6 = 3.3.

Code Example

import numpy as np

def q_update(Q, s, a, r, s_next, terminated, alpha=0.5, gamma=0.9):
    """One Q-learning update; returns the TD error."""
    target = r if terminated else r + gamma * np.max(Q[s_next])
    td_error = target - Q[s, a]
    Q[s, a] += alpha * td_error
    return td_error

Q = np.zeros((3, 2))
Q[0, 1] = 2.0
Q[1] = [4.0, 1.0]
delta = q_update(Q, s=0, a=1, r=1.0, s_next=1, terminated=False)
print(round(delta, 2), round(Q[0, 1], 2))          # 2.6 3.3 (the worked example)

# Reward propagation along a chain 0 -> 1 -> 2 (terminal, reward 1 on entering it)
Q = np.zeros((3, 1))
for sweep in range(3):
    q_update(Q, 1, 0, 1.0, 2, terminated=True, alpha=1.0)
    q_update(Q, 0, 0, 0.0, 1, terminated=False, alpha=1.0)
    print(f"after sweep {sweep + 1}: {Q.ravel().round(2).tolist()}")
# after sweep 1: [0.9, 1.0, 0.0]: with alpha = 1 the value reaches state 0 at once,
# because state 1 is updated before state 0 in each sweep

Variants of the Target

Method Target
Q-learning r + γ·max_a′ Q(s′, a′)
SARSA r + γ·Q(s′, a′) for the action actually taken next
Expected SARSA r + γ·Σ π(a′ | s′) Q(s′, a′)
Double Q-learning r + γ·Q_B(s′, argmax_a′ Q_A(s′, a′))
DQN r + γ·max_a′ Q_target(s′, a′) with a frozen target network

Common Pitfalls

  • Bootstrapping from a terminal next state.

  • Using the action the agent will take next (that’s SARSA, not Q-learning).

  • Updating with α too large in stochastic environments.

Interview Tips

  • Write the rule and compute one update by hand.

  • Name the target, the TD error and the role of α and γ.

10

What is the Bellman equation and how does it relate to Q-learning?

The Bellman equation expresses a value recursively: the value of a state (or state-action pair) equals the expected immediate reward plus the discounted value of the next state; the Bellman optimality equation for action values, Q(s, a) = E[r + γ·max over a′ of Q(s′, a′)], says the optimal Q-values are the fixed point of that recursion. Q-learning is a sample-based way to solve it: each update replaces the expectation with one observed transition and moves Q(s, a) a step toward r + γ·max Q(s′, ·), so repeated updates drive Q toward Q* without knowing the transition probabilities.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

The Equations

Equation Form
Bellman expectation (V) V^π(s) = Σ_a π(a | s) Σ_s′ P(s′ | s, a)[R + γ V^π(s′)]
Bellman expectation (Q) Q^π(s, a) = Σ_s′ P(s′ | s, a)[R + γ Σ_a′ π(a′ | s′) Q^π(s′, a′)]
Bellman optimality (V) V*(s) = max_a Σ_s′ P(s′ | s, a)[R + γ V*(s′)]
Bellman optimality (Q) Q*(s, a) = Σ_s′ P(s′ | s, a)[R + γ max_a′ Q*(s′, a′)]

From Bellman to Q-Learning

Method Uses the model P? How it solves Bellman optimality
Value iteration Yes Full expected backups over all next states
Q-value iteration Yes Q ← R + γ Σ P·max Q, for all (s, a)
Q-learning No Sampled backup: Q ← Q + α(r + γ max Q(s′) − Q)

The Bellman operator is a γ-contraction, which is why value iteration converges, and why tabular Q-learning converges under its step-size and exploration conditions.

Code Example

import numpy as np

# Small deterministic MDP: states 0, 1, 2 (2 is terminal); actions 0 = left, 1 = right
next_state = np.array([[0, 1], [0, 2]])      # for states 0 and 1
reward = np.array([[0.0, 0.0], [0.0, 1.0]])  # +1 for moving from 1 into the terminal state 2
gamma = 0.9

# Q-value iteration with the known model (the Bellman optimality backup)
Q = np.zeros((2, 2))
for _ in range(100):
    V_next = np.where(next_state == 2, 0.0, np.max(Q, axis=1)[np.minimum(next_state, 1)])
    Q = reward + gamma * V_next
print(np.round(Q, 3))        # [[0.81, 0.9], [0.81, 1.0]]

# Q-learning from sampled transitions reaches the same fixed point without using the model
rng = np.random.default_rng(42)
Qs = np.zeros((2, 2))
for _ in range(5000):
    s, a = rng.integers(2), rng.integers(2)
    s2, r = next_state[s, a], reward[s, a]
    target = r if s2 == 2 else r + gamma * Qs[s2].max()
    Qs[s, a] += 0.1 * (target - Qs[s, a])
print(np.round(Qs, 3))       # close to the values above

Hand check: Q(1, right) = 1; Q(0, right) = 0 + 0.9 × 1 = 0.9; Q(1, left) = 0.9 × max Q(0, ·) = 0.81; Q(0, left) = 0.9 × 0.9 = 0.81.

Common Pitfalls

  • Mixing the expectation equation (for a given policy) with the optimality equation (max).

  • Forgetting that the terminal state’s value is 0.

  • Assuming the sampled update is unbiased with function approximation; the max introduces overestimation.

Interview Tips

  • Write the Bellman optimality equation for Q and show how Q-learning samples it.

  • Mention the contraction property as the reason for convergence.

11

Why does convergence matter in Q-learning and how is it achieved?

*Convergence matters because Q-learning’s greedy policy is only guaranteed to be optimal once the Q-values have converged to the optimal action values Q; before that, the policy may still pick poor actions, and with function approximation the values can even diverge.** In the tabular case, Watkins and Dayan (1992) proved convergence to Q* with probability 1 if every state-action pair is visited infinitely often and the learning rates satisfy the Robbins-Monro conditions (Σ α = ∞ and Σ α² < ∞). In practice convergence is helped by sufficient exploration, decaying learning rates, and, in deep Q-learning, stabilizers such as experience replay and target networks.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Conditions (Tabular)

Condition Meaning In practice
Finite MDP Finite states and actions Small discrete problems
Infinite visits Every (s, a) tried infinitely often ε-greedy with ε > 0, or other exploration
Step sizes Σ α_t(s, a) = ∞, Σ α_t(s, a)² < ∞ α = 1 / N(s, a) or 1 / N^ω with 0.5 < ω ≤ 1
Bounded rewards Rewards have finite variance Usually true
γ < 1 (or proper episodic tasks) Values are finite Discounting or guaranteed termination

Why It Can Fail

  • Function approximation + bootstrapping + off-policy (the “deadly triad”): DQN-style learning can diverge.

  • Constant α in stochastic environments: values keep fluctuating around Q* rather than settling (often acceptable).

  • Insufficient exploration: some actions are never evaluated correctly.

Practical Stabilizers

  • Decaying α and ε.

  • Experience replay (decorrelates samples) and target networks (fixed targets) in DQN.

  • Double Q-learning against overestimation; reward clipping or normalization; gradient clipping.

Code Example

import numpy as np

# Stochastic chain: action "right" from state 0 reaches the goal with probability 0.8
rng = np.random.default_rng(42)
gamma = 0.9

def run(alpha_rule, steps=20000):
    Q = np.zeros(2)                       # Q[0, right] estimate only (one state-action pair)
    visits = 0
    history = []
    for _ in range(steps):
        visits += 1
        r = 1.0 if rng.random() < 0.8 else 0.0   # expected reward 0.8, terminal afterwards
        alpha = alpha_rule(visits)
        Q[0] += alpha * (r - Q[0])
        history.append(Q[0])
    return np.array(history)

constant = run(lambda n: 0.1)
decaying = run(lambda n: 1.0 / n)          # satisfies Robbins-Monro: the sample average
print("constant alpha, last 1000 std:", round(float(constant[-1000:].std()), 4))   # keeps jittering
print("1/n alpha, last 1000 std:", round(float(decaying[-1000:].std()), 4))        # settles near 0.8
print("final estimates:", round(float(constant[-1]), 3), round(float(decaying[-1]), 3))

Monitoring Convergence

  • Track the maximum change in Q per episode (‖ΔQ‖∞) and stop when it stays below a tolerance.

  • Track the greedy policy: convergence of the policy often happens before values settle.

  • Evaluate the greedy policy periodically without exploration.

Common Pitfalls

  • Expecting deep Q-networks to inherit tabular guarantees.

  • Decaying ε to 0 so early that some pairs are never revisited.

  • Declaring convergence from training rewards measured with exploration on.

Interview Tips

  • State the two tabular conditions (infinite visits, Robbins-Monro step sizes).

  • Mention the deadly triad and the DQN stabilizers.

12

What conditions does Q-learning need to find the optimal policy?

Tabular Q-learning finds the optimal policy when the problem is a finite Markov decision process (the state captures everything needed to predict rewards and transitions), every state-action pair keeps being visited (unbounded exploration), the learning rates decay appropriately (Σ α = ∞ and Σ α² < ∞ for each pair), rewards are bounded, and future rewards are discounted (γ < 1) or episodes always terminate. Under these conditions the Q-values converge to Q* with probability 1, and the greedy policy with respect to Q* is optimal; in practice, with function approximation or partial observability, these guarantees no longer hold.

Read full answer

Applies togeneral concepts

The Conditions

Condition Why it’s needed What breaks it
Markov states Q(s, a) must depend only on s and a Missing information (no velocity, hidden opponent cards)
Finite, discrete S and A Tabular representation Continuous or huge spaces
Infinite exploration Every (s, a) estimated correctly ε decayed to 0 too early, unreachable states
Step sizes (Robbins-Monro) Average out noise while still moving Constant α (oscillates in stochastic tasks)
Bounded rewards Finite values Unbounded or heavy-tailed rewards
γ < 1 or proper termination Finite returns Continuing tasks with γ = 1
Stationary environment Q* doesn’t change Moving targets, other learning agents

GLIE Exploration

  • “Greedy in the Limit with Infinite Exploration”: explore every action infinitely often, but become greedy over time, e.g. ε_t = 1/t per state.

  • For Q-learning (off-policy), exploration just needs to keep visiting all pairs; the learned values are for the greedy policy regardless.

Beyond the Tabular Case

Setting Guarantee
Tabular, conditions met Converges to Q* with probability 1
Linear function approximation, off-policy Can diverge (Baird’s counterexample)
Deep Q-networks No guarantee; stabilized empirically
Partially observable environments Learns values of observations, may be suboptimal
Multi-agent learning Environment is non-stationary; no general guarantee

Practical Checklist

  • Design a Markov state (add history, velocity or features).

  • Use ε-greedy with slow decay, or optimistic initialization.

  • Use α = 1 / N(s, a) or a slowly decaying schedule.

  • Check that the reward signal actually encodes the goal.

Common Pitfalls

  • Reward functions that make an unintended behaviour optimal; Q-learning will find it.

  • Terminating exploration based on training reward that already includes lucky episodes.

  • Assuming convergence of values when only the policy has stabilized (or vice versa).

Interview Tips

  • List Markov states, infinite visits and Robbins-Monro step sizes.

  • Explain what changes with function approximation.

13

What are common strategies for initializing a Q-table?

The common strategies for initializing a Q-table are zeros (simple and neutral), small random values (to break ties between actions), optimistic values above any achievable return (which make untried actions look attractive and drive systematic exploration), pessimistic values (cautious agents), and informed initial values from domain knowledge, a heuristic, a simpler model or a related task (transfer). Terminal states should always have value 0, and invalid actions can be set to a very negative value so they are never chosen.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Strategies

Strategy How Effect
Zeros np.zeros((S, A)) Neutral; needs ε for exploration; ties at the start
Small random rng.uniform(0, 0.01, (S, A)) Breaks ties; small bias
Optimistic Above the maximum return, e.g. r_max / (1 - gamma) Every untried action looks best, so all get tried
Pessimistic Below the minimum return Cautious; risky in sparse-reward tasks
Heuristic / prior From distance to goal, rules, demonstrations Faster learning if the prior is good
Transfer Q-table from a similar task Warm start
Masks -np.inf for invalid actions Never selected

Code Example

import numpy as np

rng = np.random.default_rng(42)
n_states, n_actions = 6, 2
gamma, r_max = 0.9, 1.0

zeros = np.zeros((n_states, n_actions))
small_random = rng.uniform(0, 0.01, (n_states, n_actions))
optimistic = np.full((n_states, n_actions), r_max / (1 - gamma))     # 10.0 > any real return

# Heuristic: in a corridor with the goal at state 5, moving right (action 1) starts closer to its value
heuristic = np.zeros((n_states, n_actions))
heuristic[:, 1] = gamma ** (n_states - 1 - np.arange(n_states))

# Terminal state and invalid actions
optimistic[5] = 0.0                     # terminal state has value 0
optimistic[0, 0] = -np.inf              # e.g. "left" is not allowed in state 0

def greedy(Q, s):
    row = Q[s]
    best = np.flatnonzero(row == row.max())
    return int(rng.choice(best))        # random tie-breaking avoids always picking action 0

print(np.round(heuristic, 3))
print("greedy in state 0 with optimistic table:", greedy(optimistic, 0))   # 1 (left is masked)

Optimistic Initialization in Practice

  • The agent tries an action, gets a realistic (lower) return, and the value drops below untried actions, so it tries those next.

  • Works well in deterministic or mildly stochastic tasks; exploration fades on its own.

  • Less suited to non-stationary problems, because the optimism is used up early.

Common Pitfalls

  • Forgetting to set terminal states to 0 when using optimistic values (they would keep inflating targets).

  • Very large optimistic values with small α, which take a long time to come down.

  • Zeros plus np.argmax without tie-breaking, which always chooses the first action.

Interview Tips

  • Mention optimistic initialization as a form of exploration.

  • Say how you’d handle terminal states and invalid actions.

14

How do you know when a Q-learning agent has trained enough?

You stop training a Q-learning agent when its greedy policy has stopped improving and meets the goal: in practice you monitor the maximum change in Q-values per episode (stop when it stays below a small tolerance), whether the greedy policy has stayed the same for many episodes, and, most importantly, the average return or success rate of the greedy policy measured in separate evaluation episodes without exploration, stopping when it plateaus or reaches a target, or when the training budget runs out. Because returns are noisy, use moving averages over many episodes and confirm with several random seeds.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x and Gymnasium 1.x

Stopping Criteria

Criterion How Note
Value convergence max |ΔQ| per episode < tolerance for k episodes Strict; slow in stochastic tasks with constant α
Policy stability Greedy action unchanged in every state for k checks Policies settle before values
Evaluation return plateau Moving average of greedy-policy returns stops rising The most meaningful criterion
Target performance Success rate or return above a threshold e.g. “solved” thresholds for benchmark tasks
Budget Max episodes, steps or time Always set one
Early stopping with patience Stop after N evaluations without improvement; keep the best Q Avoids late degradation

Code Example

import gymnasium as gym
import numpy as np

env = gym.make("FrozenLake-v1", is_slippery=False)
eval_env = gym.make("FrozenLake-v1", is_slippery=False)
Q = np.zeros((16, 4))
rng = np.random.default_rng(42)
env.reset(seed=42)
eval_env.reset(seed=42)

def evaluate(Q, episodes=20):
    wins = 0
    for _ in range(episodes):
        s, _ = eval_env.reset()
        done = False
        while not done:
            s, r, term, trunc, _ = eval_env.step(int(np.argmax(Q[s])))   # greedy, no exploration
            done = term or trunc
        wins += r
    return wins / episodes

best_score, patience, stale, prev_policy, stable = -1.0, 10, 0, None, 0
for episode in range(1, 5001):
    s, _ = env.reset()
    epsilon = max(0.05, 1 - episode / 1000)
    done = False
    while not done:
        a = int(rng.integers(4)) if rng.random() < epsilon else int(np.argmax(Q[s]))
        s2, r, term, trunc, _ = env.step(a)
        Q[s, a] += 0.5 * (r + (0 if term else 0.95 * Q[s2].max()) - Q[s, a])
        s, done = s2, term or trunc
    if episode % 100 == 0:
        score = evaluate(Q)
        policy = Q.argmax(axis=1)
        stable = stable + 1 if prev_policy is not None and np.array_equal(policy, prev_policy) else 0
        prev_policy = policy
        stale = 0 if score > best_score else stale + 1
        best_score = max(best_score, score)
        if score == 1.0 and stable >= 3 or stale >= patience:
            print(f"stopping at episode {episode}: eval success {score:.2f}, policy stable {stable} checks")
            break
env.close()
eval_env.close()

Common Pitfalls

  • Judging by training returns that include ε-exploration noise.

  • Stopping on one lucky evaluation; average many episodes.

  • Not keeping the best Q-table when performance later degrades (common in deep Q-learning).

Interview Tips

  • Separate training and evaluation episodes.

  • Combine value change, policy stability and evaluation return.

15

How can Q-learning be applied to continuous action spaces?

Standard Q-learning needs a maximization over actions, max over a′ of Q(s′, a′), which is easy for a few discrete actions but not for continuous ones, so it is adapted by discretizing the actions into bins, by choosing a Q-function form whose maximum is analytic (Normalized Advantage Functions, a quadratic in the action), by approximating the max with sampling or optimization (QT-Opt’s cross-entropy method), or, most commonly, by learning a separate actor network that outputs the maximizing action, which gives actor-critic methods such as DDPG, TD3 and SAC. For low-dimensional actions discretization can work; for robotics and control, TD3 and SAC are the usual choices.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x; Stable-Baselines3 2.x for TD3/SAC

Approaches

Approach Idea Trade-off
Discretization Split each action dimension into bins Simple; bins^dims explodes with many dimensions
Branching / factorized discretization One output head per action dimension Scales better than full grids
NAF Q(s, a) = V(s) + quadratic advantage, max at μ(s) Analytic max; restrictive form
Sampling-based max (QT-Opt) Cross-entropy method over sampled actions Flexible; more compute per step
Deterministic actor (DDPG, TD3) Actor μ(s) ≈ argmax Q; critic learns Q Standard for continuous control
Stochastic actor (SAC) Max entropy actor-critic Robust, sample-efficient

Code Example

import numpy as np

rng = np.random.default_rng(42)

# 1) Discretization: a continuous steering angle in [-1, 1] mapped to 7 discrete actions
action_values = np.linspace(-1.0, 1.0, 7)
Q = np.zeros((10, len(action_values)))            # 10 discrete states x 7 actions

def choose(state, epsilon=0.1):
    idx = int(rng.integers(len(action_values))) if rng.random() < epsilon else int(np.argmax(Q[state]))
    return idx, action_values[idx]                # index for the table, real value for the environment

idx, steering = choose(3)
print("chosen steering:", round(float(steering), 3))

# 2) Approximate max over a continuous action with random sampling (QT-Opt idea, simplified)
def q_function(state, action):
    return -(action - 0.3 * state) ** 2           # toy critic: best action is 0.3 * state

def approx_argmax(state, n_samples=64, iters=3):
    mean, std = 0.0, 1.0
    for _ in range(iters):                        # cross-entropy method
        samples = np.clip(rng.normal(mean, std, n_samples), -1, 1)
        elite = samples[np.argsort(q_function(state, samples))[-8:]]
        mean, std = elite.mean(), elite.std() + 1e-3
    return mean

print("approximate best action for state 2:", round(float(approx_argmax(2)), 2))   # near 0.6

With Stable-Baselines3, an actor-critic for continuous actions is one line:

import gymnasium as gym
from stable_baselines3 import TD3

env = gym.make("Pendulum-v1")                     # continuous torque in [-2, 2]
model = TD3("MlpPolicy", env, seed=42, verbose=0)
model.learn(total_timesteps=20_000)

Common Pitfalls

  • Fine discretization of many action dimensions (e.g. 10 bins × 7 joints = 10⁷ actions).

  • Forgetting to scale actions to the environment’s bounds.

  • Using DQN on continuous control benchmarks where TD3/SAC are standard.

Interview Tips

  • Explain why the max over actions is the obstacle.

  • Name discretization, NAF, CEM-based max and actor-critic (DDPG/TD3/SAC).

16

What is experience replay in Q-learning and why is it useful?

Experience replay stores the agent’s past transitions (state, action, reward, next state, done) in a replay buffer and trains the Q-function on random mini-batches sampled from that buffer instead of only on the latest transition. Introduced for deep Q-learning in DQN, it breaks the strong correlation between consecutive samples (which destabilizes neural network training), reuses each experience many times (better sample efficiency), and smooths the training distribution over many past behaviours; it works because Q-learning is off-policy, so data from older policies is still valid for learning the greedy policy.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x and PyTorch 2.x

How It Works

  1. Act in the environment with ε-greedy; store each transition in a fixed-size buffer (oldest dropped first).

  2. Once the buffer has enough samples (warm-up), sample a random mini-batch (e.g. 32-256).

  3. Compute TD targets for the batch, usually with a target network.

  4. Take a gradient step on the loss between Q(s, a) and the targets.

Benefits and Costs

Benefit Why
Decorrelated samples Mini-batches mix many episodes and time steps
Sample efficiency Each transition is reused many times
Stability Avoids feedback loops from training only on recent behaviour
Off-policy learning Old data is still usable
Cost / caveat Note
Memory Large buffers (10⁵-10⁶ transitions)
Stale data Very old transitions may no longer be relevant
Not for on-policy methods SARSA, PPO need fresh on-policy data
Uniform sampling ignores importance Prioritized replay samples useful transitions more

Code Example

import random
from collections import deque

import numpy as np
import torch

class ReplayBuffer:
    def __init__(self, capacity=100_000, seed=42):
        self.buffer = deque(maxlen=capacity)       # oldest transitions drop out automatically
        self.rng = random.Random(seed)

    def push(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))

    def sample(self, batch_size):
        batch = self.rng.sample(self.buffer, batch_size)
        states, actions, rewards, next_states, dones = map(np.array, zip(*batch))
        return (torch.as_tensor(states, dtype=torch.float32),
                torch.as_tensor(actions, dtype=torch.int64),
                torch.as_tensor(rewards, dtype=torch.float32),
                torch.as_tensor(next_states, dtype=torch.float32),
                torch.as_tensor(dones, dtype=torch.float32))

    def __len__(self):
        return len(self.buffer)

buffer = ReplayBuffer(capacity=1000)
rng = np.random.default_rng(42)
for t in range(200):                               # fake transitions with 4-dimensional states
    s, s2 = rng.normal(size=4), rng.normal(size=4)
    buffer.push(s, int(rng.integers(2)), float(rng.random()), s2, bool(rng.random() < 0.05))

states, actions, rewards, next_states, dones = buffer.sample(32)
print(states.shape, actions.shape, len(buffer))   # torch.Size([32, 4]) torch.Size([32]) 200

Common Pitfalls

  • Starting updates before the buffer has enough diverse data.

  • Storing truncated as done, which wrongly zeroes bootstrap targets at time limits.

  • Buffers too small (overfitting to recent data) or too large for memory.

Interview Tips

  • Name decorrelation, reuse and off-policy validity as the three reasons.

  • Mention prioritized experience replay as the main extension.

17

What role do target networks play in deep Q-learning?

A target network is a separate, periodically updated copy of the Q-network used only to compute the TD targets r + γ·max over a′ of Q_target(s′, a′), so the values the online network is trained toward don’t shift with every gradient step. Without it, each update changes both the prediction and the target at once, which can cause oscillation or divergence; with it, the regression target stays fixed for a while (hard updates every C steps, as in the original DQN) or changes slowly (soft Polyak updates θ_target ← τθ + (1 − τ)θ_target, as in DDPG and TD3).

Read full answer

Applies togeneral concepts; examples in Python 3.12 with PyTorch 2.x

Why It Helps

  • Moving target problem: in plain deep Q-learning, the target depends on the same weights being updated, so errors chase themselves.

  • Stability: a frozen target turns each phase into an ordinary supervised regression.

  • Reduced feedback loops: overestimations don’t immediately feed back into targets.

Update Schemes

Scheme Rule Typical setting
Hard update Copy weights every C steps C = 1,000-10,000 environment steps (DQN)
Soft (Polyak) update θ′ ← τθ + (1 − τ)θ′ every step τ = 0.005 (DDPG, TD3, SAC)

Code Example

import copy

import torch
import torch.nn as nn

torch.manual_seed(42)
q_net = nn.Sequential(nn.Linear(4, 64), nn.ReLU(), nn.Linear(64, 2))
target_net = copy.deepcopy(q_net)                # starts as an exact copy
target_net.requires_grad_(False)
optimizer = torch.optim.Adam(q_net.parameters(), lr=1e-3)
gamma = 0.99

def train_step(states, actions, rewards, next_states, dones):
    q_values = q_net(states).gather(1, actions.unsqueeze(1)).squeeze(1)
    with torch.no_grad():                         # targets come from the frozen network
        next_q = target_net(next_states).max(dim=1).values
        targets = rewards + gamma * (1 - dones) * next_q
    loss = nn.functional.smooth_l1_loss(q_values, targets)   # Huber loss, as in DQN
    optimizer.zero_grad()
    loss.backward()
    nn.utils.clip_grad_norm_(q_net.parameters(), 10.0)
    optimizer.step()
    return loss.item()

def hard_update():
    target_net.load_state_dict(q_net.state_dict())

def soft_update(tau=0.005):
    with torch.no_grad():
        for p_target, p in zip(target_net.parameters(), q_net.parameters()):
            p_target.mul_(1 - tau).add_(tau * p)

batch = (torch.randn(32, 4), torch.randint(0, 2, (32,)), torch.randn(32),
         torch.randn(32, 4), torch.zeros(32))
for step in range(1, 1001):
    loss = train_step(*batch)
    if step % 500 == 0:
        hard_update()                             # every C = 500 steps here
print("last loss:", round(loss, 4))

Common Pitfalls

  • Forgetting torch.no_grad() (or .detach()) for the target, so gradients flow into the target.

  • Updating the target too often (no stability gain) or too rarely (slow learning).

  • Confusing the target network with Double DQN; Double DQN uses the online network to choose the next action and the target network to evaluate it.

Interview Tips

  • Explain the moving-target problem in one sentence.

  • Contrast hard and soft updates and give typical values.

18

How do you handle large state spaces in Q-learning?

Large state spaces make a Q-table infeasible, because it needs one entry per state-action pair and most states are visited rarely or never, so they are handled by generalizing across states: function approximation that maps state features to Q-values (linear models with tile coding or other features, or neural networks as in DQN), state abstraction and aggregation (grouping similar states, coarse discretization), and better state design that removes irrelevant detail. Deep Q-networks with experience replay and target networks are the standard approach for very large spaces such as images.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x and PyTorch 2.x

Approaches

Approach How When
State design Keep only relevant variables, use relative features Always first
Discretization / aggregation Bins or clusters of similar states Low-dimensional continuous states
Tile coding Several overlapping offset grids, binary features Classic linear RL on continuous states
Linear function approximation Q(s, a) = wₐ · φ(s) Good features, stable learning
Neural networks (DQN) Network outputs Q(s, ·) Images, high-dimensional inputs
Hierarchical RL Options / sub-goals at a higher level Long-horizon tasks
Model-based planning Learned model + search Sample efficiency

Code Example

import numpy as np

# Linear Q-learning with simple polynomial features on a continuous 1-D state in [0, 1]
rng = np.random.default_rng(42)
n_actions, gamma, alpha = 2, 0.95, 0.05

def features(s):
    return np.array([1.0, s, s ** 2])               # phi(s); one weight vector per action

W = np.zeros((n_actions, 3))

def q(s):
    return W @ features(s)                          # Q(s, .) for all actions

def step(s, a):
    s_next = np.clip(s + (0.1 if a == 1 else -0.1) + rng.normal(0, 0.01), 0.0, 1.0)
    done = s_next >= 1.0
    return s_next, (1.0 if done else -0.01), done   # small step cost, reward at the right end

for episode in range(500):
    s, done, t = rng.random() * 0.5, False, 0
    while not done and t < 200:
        a = int(rng.integers(n_actions)) if rng.random() < 0.1 else int(np.argmax(q(s)))
        s_next, r, done = step(s, a)
        target = r if done else r + gamma * np.max(q(s_next))
        td_error = target - q(s)[a]
        W[a] += alpha * td_error * features(s)      # semi-gradient update
        s, t = s_next, t + 1

print("greedy actions at s = 0.1, 0.5, 0.9:", [int(np.argmax(q(x))) for x in (0.1, 0.5, 0.9)])
# Expected after training: 1 (move right) everywhere

For image-like states, replace features with a convolutional network and train it with a replay buffer and target network (DQN).

Trade-offs

  • Generalization lets the agent act sensibly in states it has never seen.

  • Approximation loses tabular convergence guarantees; off-policy + bootstrapping + approximation can diverge.

  • Feature quality determines how well linear methods work.

Common Pitfalls

  • Discretizing too finely (back to a huge table) or too coarsely (non-Markov states).

  • Unscaled input features for neural networks.

  • No stabilizers (replay, target network) with deep approximation.

Interview Tips

  • Move from “the table doesn’t scale” to function approximation with an example.

  • Mention tile coding for linear methods and DQN for images.

19

What is a Deep Q-Network and how does it relate to Q-learning?

A Deep Q-Network (DQN) is Q-learning with a deep neural network as the Q-function: the network takes the state (for example the last four game frames) and outputs one Q-value per discrete action, and it is trained to minimize the difference between Q(s, a) and the Q-learning target r + γ·max over a′ of Q_target(s′, a′). DeepMind’s 2015 DQN learned to play many Atari games from pixels by adding two stabilizers to Q-learning, experience replay and a separate target network, plus reward clipping and a Huber loss; later variants such as Double DQN, Dueling DQN, prioritized replay and Rainbow improved it further.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with PyTorch 2.x, Gymnasium 1.x, Stable-Baselines3 2.x

From Q-Learning to DQN

Aspect Tabular Q-learning DQN
Q representation Table Q[s, a] Network Q(s, ·; θ)
States Small, discrete High-dimensional (images, vectors)
Update One cell per step Gradient descent on mini-batches
Data Latest transition Replay buffer
Target r + γ max Q(s′) r + γ max Q_target(s′) (frozen network)
Guarantees Converges (tabular conditions) None; empirically stabilized

Loss

L(θ) = E over (s, a, r, s′, done) from the buffer of Huber(Q(s, a; θ) − [r + γ(1 − done) max_a′ Q(s′, a′; θ⁻)]), where θ⁻ are the target network’s weights.

Code Example

import gymnasium as gym
import torch
from stable_baselines3 import DQN

torch.manual_seed(42)
env = gym.make("CartPole-v1")
model = DQN(
    "MlpPolicy", env,
    learning_rate=1e-3,
    buffer_size=50_000,          # experience replay
    learning_starts=1_000,
    batch_size=64,
    gamma=0.99,
    target_update_interval=500,  # hard target network updates
    exploration_fraction=0.2,    # epsilon decays over the first 20% of training
    exploration_final_eps=0.05,
    seed=42, verbose=0,
)
model.learn(total_timesteps=50_000)

obs, info = env.reset(seed=42)
total, done = 0.0, False
while not done:
    action, _ = model.predict(obs, deterministic=True)     # greedy policy
    obs, reward, terminated, truncated, info = env.step(action)
    total += reward
    done = terminated or truncated
print("evaluation return:", total)                         # varies with training; 500 is the maximum
env.close()

Variants

Variant Fix
Double DQN Overestimation: choose with the online network, evaluate with the target
Dueling DQN Separate V(s) and advantage streams
Prioritized replay Sample high-TD-error transitions more often
Noisy nets Learned exploration
Distributional (C51, QR-DQN) Learn the return distribution
Rainbow Combines the above with n-step returns

Common Pitfalls

  • Using DQN for continuous actions (needs actor-critic methods).

  • Training without enough exploration or with too small a replay buffer.

  • Reporting results from a single seed; deep RL results vary a lot.

Interview Tips

  • Say DQN = Q-learning + neural network + replay + target network.

  • Name Double DQN as the first fix for overestimation.

20

How does Double Q-learning reduce overestimation of Q-values?

Double Q-learning reduces overestimation by separating the two jobs that the max in the Q-learning target does at once: selecting the best next action and evaluating it. Standard Q-learning uses max over a′ of Q(s′, a′), and because the max of noisy estimates is biased upward (it picks whichever action happens to be overestimated), values grow too optimistic. Double Q-learning keeps two estimators, Q_A and Q_B, and updates Q_A toward r + γ·Q_B(s′, argmax_a′ Q_A(s′, a′)) (and vice versa), so the noise in selection and evaluation is independent; Double DQN applies the same idea using the online network to select and the target network to evaluate.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x and PyTorch 2.x

Why the Max Overestimates

  • If every action has true value 0 and estimates are noisy with mean 0, E[max of the estimates] > 0.

  • The error compounds through bootstrapping, so whole regions of the state space look better than they are.

  • This can push the policy toward actions whose values are merely uncertain.

The Targets

Method Target
Q-learning r + γ·max_a′ Q(s′, a′)
Double Q-learning (tabular) r + γ·Q_B(s′, argmax_a′ Q_A(s′, a′)); randomly swap roles of A and B
DQN r + γ·max_a′ Q_target(s′, a′)
Double DQN r + γ·Q_target(s′, argmax_a′ Q_online(s′, a′))

Code Example

import numpy as np

rng = np.random.default_rng(42)

# Ten actions whose true values are all 0; each estimate averages 5 noisy samples
n_actions, n_samples, trials = 10, 5, 10_000
single_max, double_est = [], []
for _ in range(trials):
    samples_a = rng.normal(0, 1, (n_actions, n_samples)).mean(axis=1)   # estimator A
    samples_b = rng.normal(0, 1, (n_actions, n_samples)).mean(axis=1)   # independent estimator B
    single_max.append(samples_a.max())                                  # Q-learning style
    double_est.append(samples_b[np.argmax(samples_a)])                  # select with A, evaluate with B

print("true best value: 0.0")
print("single max estimate:", round(float(np.mean(single_max)), 3))     # clearly positive (biased up)
print("double estimate:", round(float(np.mean(double_est)), 3))         # close to 0

Double DQN target in PyTorch:

import torch

def double_dqn_targets(online_net, target_net, rewards, next_states, dones, gamma=0.99):
    with torch.no_grad():
        best_actions = online_net(next_states).argmax(dim=1, keepdim=True)    # select
        next_q = target_net(next_states).gather(1, best_actions).squeeze(1)  # evaluate
        return rewards + gamma * (1 - dones) * next_q

Effect

  • Less optimistic values and often better policies, especially in stochastic environments.

  • Double Q-learning can slightly underestimate; TD3’s clipped double Q (min of two critics) is a related, deliberately conservative idea for continuous control.

Common Pitfalls

  • Using the same network to select and evaluate, which is just DQN again.

  • Forgetting to randomly alternate which table is updated in tabular Double Q-learning.

Interview Tips

  • Explain that the max of noisy estimates is biased upward.

  • Write the Double DQN target with online selection and target evaluation.

21

How does Prioritized Experience Replay improve Q-learning training?

Prioritized Experience Replay (PER, Schaul et al., 2016) samples transitions from the replay buffer in proportion to how much the agent can learn from them, measured by the size of their TD error, instead of uniformly; surprising transitions (large errors), such as rare rewards, are replayed more often, so value information spreads faster and learning needs fewer environment steps. Because non-uniform sampling biases the gradient, PER corrects it with importance-sampling weights that are annealed toward full correction during training, and it stores priorities in a sum tree so sampling stays O(log N).

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

How It Works

  • Priority: p_i = |δ_i| + ε (proportional variant), or 1 / rank(i) (rank-based variant).

  • Sampling probability: P(i) = p_i^α / Σ_k p_k^α, where α ∈ [0, 1] controls how much prioritization is used (0 = uniform; 0.6 is common).

  • Importance-sampling weight: w_i = (N · P(i))^(−β) / max_j w_j, with β annealed from about 0.4 to 1.

  • Update: multiply each sample’s loss by w_i; after the update, set its priority to the new |δ|.

  • New transitions get the current maximum priority so they are replayed at least once.

Code Example

import numpy as np

class SimplePER:
    """Proportional prioritized replay with NumPy arrays (a sum tree is used at scale)."""

    def __init__(self, capacity, alpha=0.6, eps=1e-5, seed=42):
        self.capacity, self.alpha, self.eps = capacity, alpha, eps
        self.data, self.priorities = [], np.zeros(capacity)
        self.pos = 0
        self.rng = np.random.default_rng(seed)

    def push(self, transition):
        max_p = self.priorities.max() if self.data else 1.0
        if len(self.data) < self.capacity:
            self.data.append(transition)
        else:
            self.data[self.pos] = transition
        self.priorities[self.pos] = max_p               # new samples get the highest priority
        self.pos = (self.pos + 1) % self.capacity

    def sample(self, batch_size, beta=0.4):
        p = self.priorities[:len(self.data)] ** self.alpha
        probs = p / p.sum()
        idx = self.rng.choice(len(self.data), batch_size, p=probs)
        weights = (len(self.data) * probs[idx]) ** (-beta)
        weights /= weights.max()                        # normalize for stability
        return idx, [self.data[i] for i in idx], weights

    def update_priorities(self, idx, td_errors):
        self.priorities[idx] = np.abs(td_errors) + self.eps

buffer = SimplePER(capacity=1000)
for t in range(100):
    buffer.push((t, 0, 0.0, t + 1, False))
buffer.update_priorities(np.arange(100), np.r_[np.full(99, 0.01), 5.0])   # one surprising transition
idx, batch, w = buffer.sample(32)
print("times the surprising transition was drawn:", int((idx == 99).sum()), "of 32")
print("its weight vs others:", round(float(w[idx == 99][0]), 3) if (idx == 99).any() else "not drawn")

Benefits and Costs

Benefit Cost
Faster learning, especially with sparse rewards Extra hyperparameters (α, β, ε)
Better final scores in many Atari games (part of Rainbow) Sum-tree bookkeeping, more compute per sample
Focus on informative transitions Can overfit to noisy, high-error transitions

Common Pitfalls

  • Forgetting importance-sampling weights, which biases the learned values.

  • Never updating priorities after training on a sample.

  • Priorities of exactly 0 (add ε) so transitions can still be sampled.

Interview Tips

  • Explain priority by TD error, α for strength, β for bias correction.

  • Mention the sum tree for efficient sampling.

22

How does the Dueling network architecture differ from a standard DQN?

A Dueling DQN (Wang et al., 2016) keeps the same Q-learning training but changes the network’s head: after shared layers, it splits into two streams, one estimating the state value V(s) and one estimating the advantage A(s, a) of each action, and combines them as Q(s, a) = V(s) + A(s, a) − mean over a′ of A(s, a′). Because V(s) is learned from every update regardless of which action was taken, the network learns how good a state is faster, which helps most when many actions have similar values; a standard DQN instead outputs Q(s, a) for each action directly from one stream.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with PyTorch 2.x

Comparison

Aspect Standard DQN Dueling DQN
Output head One stream → Q(s, a) for each action V(s) stream + A(s, a) stream, combined
What’s learned explicitly Action values State value and relative action advantages
Benefit Simple Better value estimates when actions matter little in many states
Training Q-learning loss Same loss, same replay and target network
Combines with Double DQN, PER Double DQN, PER (all in Rainbow)

Why Subtract the Mean Advantage

  • Q = V + A is not identifiable: adding a constant to V and subtracting it from A gives the same Q.

  • Forcing the advantages to have zero mean (or max-advantage zero) fixes V as the state value; the mean version is more stable in practice.

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)

class DuelingQNet(nn.Module):
    def __init__(self, state_dim, n_actions, hidden=128):
        super().__init__()
        self.shared = nn.Sequential(nn.Linear(state_dim, hidden), nn.ReLU())
        self.value = nn.Sequential(nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, 1))
        self.advantage = nn.Sequential(nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, n_actions))

    def forward(self, x):
        h = self.shared(x)
        v = self.value(h)                                  # shape (batch, 1)
        a = self.advantage(h)                              # shape (batch, n_actions)
        return v + a - a.mean(dim=1, keepdim=True)         # Q(s, a)

class StandardQNet(nn.Module):
    def __init__(self, state_dim, n_actions, hidden=128):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(state_dim, hidden), nn.ReLU(),
                                 nn.Linear(hidden, hidden), nn.ReLU(), nn.Linear(hidden, n_actions))

    def forward(self, x):
        return self.net(x)

states = torch.randn(5, 8)
dueling = DuelingQNet(8, 4)
q = dueling(states)
print(q.shape)                                             # torch.Size([5, 4])
h = dueling.shared(states)
print(torch.allclose(q.mean(dim=1, keepdim=True), dueling.value(h), atol=1e-6))   # True: mean Q = V

The second print holds because the advantages are centred: the mean of Q over actions equals V.

Common Pitfalls

  • Combining V and A without the mean (or max) correction, which makes V meaningless.

  • Expecting large gains when actions always matter a lot; benefits vary by task.

Interview Tips

  • Write Q = V + A − mean(A) and explain identifiability.

  • Say it is an architecture change, independent of Double DQN’s target change.

23

How do you implement a basic Q-learning agent for a gridworld?

A basic Q-learning gridworld agent needs three pieces: an environment with discrete states (grid cells), actions (up, down, left, right), rewards and terminal states; a Q-table of shape (states, actions); and a training loop that picks actions ε-greedily, applies the update Q(s, a) += α·(r + γ·max Q(s′) − Q(s, a)), and repeats over many episodes. After training, the greedy action in each cell gives the learned policy, which you can print as arrows on the grid.

Read full answer

Applies toPython 3.12, NumPy 2.x

Code Example

import numpy as np

class GridWorld:
    """4x4 grid. Start top-left, goal bottom-right (+1), a pit (-1), each step costs 0.04."""
    ACTIONS = [(-1, 0), (1, 0), (0, -1), (0, 1)]      # up, down, left, right
    ARROWS = "^v<>"

    def __init__(self, size=4, goal=(3, 3), pit=(1, 1), walls=((2, 1),)):
        self.size, self.goal, self.pit, self.walls = size, goal, pit, set(walls)

    def reset(self):
        self.pos = (0, 0)
        return self.state(self.pos)

    def state(self, pos):
        return pos[0] * self.size + pos[1]

    def step(self, action):
        dr, dc = self.ACTIONS[action]
        r, c = self.pos[0] + dr, self.pos[1] + dc
        if 0 <= r < self.size and 0 <= c < self.size and (r, c) not in self.walls:
            self.pos = (r, c)                          # otherwise bump into the edge or wall and stay
        if self.pos == self.goal:
            return self.state(self.pos), 1.0, True
        if self.pos == self.pit:
            return self.state(self.pos), -1.0, True
        return self.state(self.pos), -0.04, False

env = GridWorld()
n_states, n_actions = env.size ** 2, 4
Q = np.zeros((n_states, n_actions))
alpha, gamma = 0.1, 0.95
rng = np.random.default_rng(42)

for episode in range(3000):
    s, done, steps = env.reset(), False, 0
    epsilon = max(0.05, 1.0 - episode / 1500)          # decaying exploration
    while not done and steps < 100:
        if rng.random() < epsilon:
            a = int(rng.integers(n_actions))
        else:
            a = int(rng.choice(np.flatnonzero(Q[s] == Q[s].max())))   # random tie-breaking
        s_next, r, done = env.step(a)
        target = r if done else r + gamma * Q[s_next].max()
        Q[s, a] += alpha * (target - Q[s, a])
        s, steps = s_next, steps + 1

# Show the greedy policy
grid = []
for row in range(env.size):
    line = []
    for col in range(env.size):
        pos = (row, col)
        if pos == env.goal:
            line.append("G")
        elif pos == env.pit:
            line.append("P")
        elif pos in env.walls:
            line.append("#")
        else:
            line.append(env.ARROWS[int(np.argmax(Q[env.state(pos)]))])
    grid.append(" ".join(line))
print("\n".join(grid))
# A shortest safe route from the top-left to G that avoids P (the exact arrows depend on training)

Design Notes

Choice Reason
Small step cost Encourages shorter paths
Terminal states return done=True No bootstrapping beyond them
Decaying ε Explore first, then exploit
Step limit per episode Prevents endless wandering early on
Random tie-breaking Zero-initialized rows don’t always pick “up”

Common Pitfalls

  • Bootstrapping from terminal states.

  • No step limit, so early episodes can loop for a long time.

  • Reading the policy from cells that were never visited (their Q-values are still 0).

Interview Tips

  • Keep the environment and the agent separate.

  • Explain how the step cost and γ shape the path.

24

How do you write a function that updates a Q-table?

A Q-table update function takes the table, the current state, the action taken, the reward received and the next state (plus whether the episode ended), computes the TD target r + γ·max over a′ of Q[s′, a′] (just r if the next state is terminal), and moves Q[s, a] toward it by the learning rate α: Q[s, a] += α·(target − Q[s, a]). Returning the TD error is useful for monitoring convergence and for prioritized replay.

Read full answer

Applies toPython 3.12, NumPy 2.x

Code Example

import numpy as np

def update_q_table(Q, state, action, reward, next_state, terminated,
                   alpha=0.1, gamma=0.99, valid_next_actions=None):
    """Apply one Q-learning update in place and return the TD error.

    Q: array of shape (n_states, n_actions)
    terminated: True if next_state is terminal (then there is no future value)
    valid_next_actions: optional list of allowed actions in next_state
    """
    if not (0.0 < alpha <= 1.0):
        raise ValueError("alpha must be in (0, 1]")
    if terminated:
        target = reward
    else:
        next_values = Q[next_state] if valid_next_actions is None else Q[next_state, valid_next_actions]
        target = reward + gamma * np.max(next_values)
    td_error = target - Q[state, action]
    Q[state, action] += alpha * td_error
    return float(td_error)

Q = np.zeros((4, 2))
Q[2] = [0.5, 2.0]

err = update_q_table(Q, state=1, action=0, reward=1.0, next_state=2, terminated=False,
                     alpha=0.5, gamma=0.9)
print(round(err, 2), round(float(Q[1, 0]), 2))    # target 1 + 0.9 * 2.0 = 2.8 -> error 2.8, Q = 1.4

err = update_q_table(Q, state=2, action=1, reward=10.0, next_state=3, terminated=True, alpha=0.5)
print(round(err, 2), round(float(Q[2, 1]), 2))    # target 10 -> error 8.0, Q = 2.0 + 4.0 = 6.0

err = update_q_table(Q, state=0, action=1, reward=0.0, next_state=2, terminated=False,
                     alpha=0.5, gamma=0.9, valid_next_actions=[0])
print(round(err, 2), round(float(Q[0, 1]), 2))    # only action 0 allowed: target 0.9 * 0.5 = 0.45

Variants of the Same Function

Algorithm Change to the target
SARSA reward + gamma * Q[next_state, next_action]
Expected SARSA reward + gamma * (pi[next_state] @ Q[next_state])
Double Q-learning Select with Q_a, evaluate with Q_b
Visit-count learning rate alpha = 1 / N[state, action]

Common Pitfalls

  • Forgetting the terminal case, so the agent bootstraps from a meaningless state value.

  • Treating a time-limit truncation as terminal (it should still bootstrap).

  • Accidentally copying the table instead of updating it in place (NumPy arrays update in place here).

Interview Tips

  • Handle terminal states and return the TD error.

  • Mention how the target changes for SARSA and Double Q-learning.

25

How do you simulate a Q-learning agent improving in a stochastic environment?

To show a Q-learning agent improving in a stochastic environment, train it on an environment whose transitions or rewards are random, for example a gridworld where each move slips sideways with some probability or FrozenLake with is_slippery=True, and record a learning curve: the moving average of episode returns or the success rate of the greedy policy, evaluated periodically without exploration. Because outcomes are random, a smaller learning rate, many episodes and averaging over several seeds are needed to see the trend clearly.

Read full answer

Applies toPython 3.12, NumPy 2.x, Gymnasium 1.x, Matplotlib 3.9+

Code Example

import gymnasium as gym
import matplotlib.pyplot as plt
import numpy as np

def evaluate(env, Q, episodes=200):
    wins = 0.0
    for _ in range(episodes):
        s, _ = env.reset()
        done = False
        while not done:
            s, r, term, trunc, _ = env.step(int(np.argmax(Q[s])))
            done = term or trunc
        wins += r
    return wins / episodes

def train(seed, episodes=20000, eval_every=1000):
    env = gym.make("FrozenLake-v1", is_slippery=True)    # intended move succeeds only 1/3 of the time
    eval_env = gym.make("FrozenLake-v1", is_slippery=True)
    env.reset(seed=seed)
    eval_env.reset(seed=seed + 1000)
    rng = np.random.default_rng(seed)
    Q = np.zeros((16, 4))
    curve = []
    for ep in range(1, episodes + 1):
        s, _ = env.reset()
        epsilon = max(0.05, 1.0 - ep / (0.6 * episodes))
        done = False
        while not done:
            a = int(rng.integers(4)) if rng.random() < epsilon else int(np.argmax(Q[s]))
            s2, r, term, trunc, _ = env.step(a)
            target = r + (0.0 if term else 0.99 * Q[s2].max())
            Q[s, a] += 0.05 * (target - Q[s, a])          # small alpha for noisy outcomes
            s, done = s2, term or trunc
        if ep % eval_every == 0:
            curve.append(evaluate(eval_env, Q))
    return np.array(curve)

curves = np.array([train(seed) for seed in (42, 43, 44)])
x = np.arange(1, curves.shape[1] + 1) * 1000
plt.plot(x, curves.mean(axis=0), label="mean of 3 seeds")
plt.fill_between(x, curves.min(axis=0), curves.max(axis=0), alpha=0.2, label="range")
plt.xlabel("training episodes")
plt.ylabel("greedy success rate")
plt.title("Q-learning on slippery FrozenLake")
plt.legend()
plt.savefig("learning_curve.png", dpi=150)
print("final success rates:", curves[:, -1].round(2).tolist())   # typically well above random play

What to Look For

Signal Meaning
Rising success rate The agent is learning
Plateau Values have converged or exploration is too low
Big seed-to-seed spread Stochasticity; more episodes or smaller α
Drop late in training Instability (more common with function approximation)

Why Stochasticity Matters

  • The same action can lead to different outcomes, so Q-values must average many samples (small α).

  • The optimal policy accounts for risk: on slippery FrozenLake it often moves away from holes rather than straight to the goal.

  • No policy reaches 100% success; the optimum is below 1.

Common Pitfalls

  • Judging progress from single episodes or one seed.

  • Large α, which makes values jump with every lucky or unlucky outcome.

  • Evaluating with exploration on.

Interview Tips

  • Show a learning curve with evaluation episodes and several seeds.

  • Explain why α must be smaller in stochastic environments.

26

How do you implement epsilon-greedy action selection in Q-learning?

ε-greedy action selection chooses a random action with probability ε (exploration) and the action with the highest Q-value with probability 1 − ε (exploitation), usually with random tie-breaking among equally valued actions; ε typically starts high (1.0) and decays to a small floor (0.01-0.1) as the agent learns. It is the simplest way to make sure Q-learning keeps visiting all actions while mostly following its current best policy.

Read full answer

Applies toPython 3.12, NumPy 2.x

Code Example

import numpy as np

rng = np.random.default_rng(42)

def epsilon_greedy(q_values, epsilon, rng=rng):
    """Return an action index: random with probability epsilon, else greedy with random tie-breaking."""
    if rng.random() < epsilon:
        return int(rng.integers(len(q_values)))
    best = np.flatnonzero(q_values == np.max(q_values))
    return int(rng.choice(best))

def linear_decay(step, start=1.0, end=0.05, decay_steps=10_000):
    return end + (start - end) * max(0.0, 1.0 - step / decay_steps)

def exponential_decay(episode, start=1.0, end=0.05, rate=0.995):
    return max(end, start * rate ** episode)

q = np.array([1.0, 3.0, 3.0, 0.5])
picks = np.array([epsilon_greedy(q, epsilon=0.2) for _ in range(10_000)])
print("action shares:", np.round(np.bincount(picks, minlength=4) / len(picks), 3).tolist())
# Expected about [0.05, 0.45, 0.45, 0.05]: 0.2/4 random each, plus 0.8 split between the tied actions 1 and 2

print([round(linear_decay(s), 3) for s in (0, 5_000, 10_000, 20_000)])   # [1.0, 0.525, 0.05, 0.05]
print([round(exponential_decay(e), 3) for e in (0, 100, 500, 1000)])     # [1.0, 0.606, 0.082, 0.05]

Hand Check of the Shares

  • Random part: each of 4 actions gets 0.2 / 4 = 0.05.

  • Greedy part: actions 1 and 2 tie, each gets 0.8 / 2 = 0.4.

  • Totals: actions 1 and 2 ≈ 0.45, actions 0 and 3 ≈ 0.05.

  • Decay values: 0.995¹⁰⁰ ≈ 0.606 and 0.995⁵⁰⁰ ≈ 0.082.

Variants

Variant Note
Fixed ε Simple; keeps exploring forever
Linear / exponential decay Standard in DQN (e.g. 1.0 → 0.05)
Per-state ε (e.g. 1 / visits) GLIE: more exploration in rarely seen states
ε-greedy over valid actions only Mask invalid moves
Softmax / UCB Explore promising actions more than useless ones

Common Pitfalls

  • Using np.argmax without tie-breaking, which always picks the first of equal actions.

  • Decaying ε too fast, before rewards are discovered.

  • Leaving ε on during evaluation.

Interview Tips

  • Write the function with tie-breaking and a decay schedule.

  • Compute the expected action probabilities to show understanding.

27

How do you visualize Q-value convergence over episodes in Python?

To visualize Q-value convergence, record during training the quantities that should settle: the maximum absolute change of the Q-table per episode (‖ΔQ‖∞), the values of a few chosen state-action pairs, and the error against known optimal values when they can be computed (for example by value iteration on a small environment); then plot them against the episode number, using a log scale for the changes. Curves that flatten toward zero change, and toward the optimal values, show convergence; a moving average of episode returns adds the performance view.

Read full answer

Applies toPython 3.12, NumPy 2.x, Matplotlib 3.9+

Code Example

import matplotlib.pyplot as plt
import numpy as np

# Deterministic corridor: states 0..5, goal at 5 (reward 1), actions 0 = left, 1 = right
n, gamma = 6, 0.9
def step(s, a):
    s2 = min(max(s + (1 if a == 1 else -1), 0), n - 1)
    return s2, (1.0 if s2 == n - 1 else 0.0), s2 == n - 1

# Optimal Q by value iteration (the model is known here, so we can measure the error exactly)
Q_star = np.zeros((n, 2))
for _ in range(200):
    for s in range(n - 1):
        for a in range(2):
            s2, r, done = step(s, a)
            Q_star[s, a] = r + (0.0 if done else gamma * Q_star[s2].max())

rng = np.random.default_rng(42)
Q = np.zeros((n, 2))
max_change, error, tracked = [], [], []
for episode in range(400):
    before = Q.copy()
    s, done = 0, False
    while not done:
        a = int(rng.integers(2)) if rng.random() < 0.3 else int(np.argmax(Q[s]))
        s2, r, done = step(s, a)
        Q[s, a] += 0.2 * (r + (0.0 if done else gamma * Q[s2].max()) - Q[s, a])
        s = s2
    max_change.append(np.abs(Q - before).max())
    error.append(np.abs(Q - Q_star)[:n - 1].max())
    tracked.append(Q[0, 1])

fig, axes = plt.subplots(1, 3, figsize=(14, 4))
axes[0].semilogy(np.maximum(max_change, 1e-12))
axes[0].set(title="Max |ΔQ| per episode", xlabel="episode")
axes[1].semilogy(np.maximum(error, 1e-12))
axes[1].set(title="Max |Q − Q*|", xlabel="episode")
axes[2].plot(tracked, label="Q(0, right)")
axes[2].axhline(Q_star[0, 1], linestyle="--", color="grey", label="optimal")
axes[2].set(title="One Q-value", xlabel="episode")
axes[2].legend()
fig.tight_layout()
fig.savefig("q_convergence.png", dpi=150)
print("optimal Q(0, right):", round(float(Q_star[0, 1]), 4))   # 0.9^4 = 0.6561

Useful Plots

Plot Shows
Max |ΔQ| per episode (log scale) Whether updates are dying out
Error vs Q* (when known) True convergence
Selected Q-values over time How values propagate from the goal backward
Heatmap of max Q per state Value landscape (gridworlds)
Moving-average return Policy quality

Common Pitfalls

  • With constant α in stochastic environments, ΔQ never reaches zero; look for a stable noise floor.

  • Plotting only returns, which can plateau before values settle.

  • Linear axes that hide small late changes; use a log scale.

Interview Tips

  • Name ‖ΔQ‖∞ and error against value iteration as convergence measures.

  • Explain how values propagate backward from rewarding states.

28

How does function approximation overcome the limits of tabular Q-learning?

Function approximation replaces the Q-table with a parameterized function Q(s, a; w), such as a linear model over features or a neural network, so the agent stores a fixed number of parameters instead of one value per state-action pair and generalizes what it learns to similar states it has never visited. This makes Q-learning usable for huge or continuous state spaces (robot sensors, images), and it is trained with semi-gradient updates w ← w + α·δ·∇_w Q(s, a; w). The cost is the loss of tabular convergence guarantees: combining approximation, bootstrapping and off-policy learning can diverge, which DQN counters with experience replay and target networks.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Tabular vs Approximate

Aspect Tabular Function approximation
Memory |S| × |A| values Number of parameters
Generalization None: each cell learned separately Similar states share values
Continuous states Must discretize Handled directly
Convergence Guaranteed (tabular conditions) Not guaranteed off-policy (“deadly triad”)
Feature design Not needed Needed for linear; learned by deep networks

Approximators

Type Example Note
Linear Q(s, a) = wₐ · φ(s), with tile coding or RBF features Stable-ish, interpretable
Neural network DQN: CNN or MLP → Q(s, ·) Learns features; needs stabilizers
Tree ensembles Fitted Q-iteration with extra-trees Batch (offline) RL
Kernel methods Kernel-based RL Small data

Semi-Gradient Update

  • TD error: δ = r + γ·max_a′ Q(s′, a′; w) − Q(s, a; w).

  • Update: w ← w + α·δ·∇_w Q(s, a; w) (the target is treated as a constant).

  • For linear Q: ∇_w Q = φ(s) for the chosen action’s weights.

Code Example

import numpy as np

# Radial basis features over a continuous state in [0, 1]
centers = np.linspace(0, 1, 10)

def phi(s, width=0.1):
    return np.exp(-((s - centers) ** 2) / (2 * width ** 2))   # 10 features instead of a big table

rng = np.random.default_rng(42)
W = np.zeros((2, len(centers)))                                # weights for actions left/right
alpha, gamma = 0.1, 0.95

for episode in range(300):
    s, done, t = rng.uniform(0, 0.3), False, 0
    while not done and t < 100:
        q = W @ phi(s)
        a = int(rng.integers(2)) if rng.random() < 0.1 else int(np.argmax(q))
        s2 = float(np.clip(s + (0.05 if a == 1 else -0.05), 0, 1))
        done = s2 >= 1.0
        r = 1.0 if done else 0.0
        target = r if done else r + gamma * np.max(W @ phi(s2))
        W[a] += alpha * (target - q[a]) * phi(s)               # semi-gradient step
        s, t = s2, t + 1

for x in (0.123, 0.5, 0.877):                                  # states never seen exactly
    print(x, "->", "right" if np.argmax(W @ phi(x)) == 1 else "left")

Common Pitfalls

  • Expecting tabular guarantees to hold.

  • Poor features (too narrow RBFs act like a table; too wide blur everything).

  • Deep approximation without replay and target networks.

Interview Tips

  • Lead with generalization and memory, then the deadly triad.

  • Write the semi-gradient update.

29

What are eligibility traces in TD learning and how do they relate to Q-learning?

Eligibility traces are short-term memory variables, one per state-action pair (or per weight), that record how recently and how often each pair was visited; when a TD error occurs, every pair is updated in proportion to its trace, so credit for a reward flows back to many earlier steps at once instead of one step at a time. Controlled by λ ∈ [0, 1], they interpolate between one-step TD (λ = 0) and Monte Carlo returns (λ = 1). For Q-learning, Watkins’s Q(λ) uses traces but must cut them to zero whenever an exploratory (non-greedy) action is taken, because Q-learning learns the greedy policy off-policy; SARSA(λ) uses them without cutting.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

How Traces Work (Accumulating Traces)

  1. At each step: δ = r + γ·max Q(s′, ·) − Q(s, a).

  2. Increase the trace of the current pair: e(s, a) += 1 (or set to 1 for replacing traces).

  3. Update all pairs: Q += α·δ·e.

  4. Decay all traces: e *= γλ.

  5. In Watkins’s Q(λ): if the next action is not greedy, set e = 0.

Comparison

Method Credit assignment Off-policy handling
One-step Q-learning Only the last pair Naturally off-policy
SARSA(λ) Recent pairs, decaying by γλ On-policy, no cutting
Watkins’s Q(λ) Recent pairs until an exploratory action Cuts traces on non-greedy actions
Peng’s Q(λ) Mix of both No cutting, slightly biased
Modern deep RL n-step returns, Retrace(λ) Retrace is safe off-policy

Code Example

import numpy as np

def watkins_q_lambda(n_states=8, episodes=50, alpha=0.5, gamma=0.95, lam=0.9, epsilon=0.1, seed=42):
    """Corridor: start at 0, reward 1 for reaching the last state. Actions 0 = left, 1 = right."""
    rng = np.random.default_rng(seed)
    Q = np.zeros((n_states, 2))

    def policy(s):
        if rng.random() < epsilon:
            return int(rng.integers(2))
        return int(rng.choice(np.flatnonzero(Q[s] == Q[s].max())))

    for _ in range(episodes):
        E = np.zeros_like(Q)                         # traces reset every episode
        s, a, done = 0, policy(0), False
        while not done:
            s2 = min(max(s + (1 if a == 1 else -1), 0), n_states - 1)
            done = s2 == n_states - 1
            r = 1.0 if done else 0.0
            a2 = policy(s2)
            greedy_next = int(np.argmax(Q[s2]))
            delta = r + (0.0 if done else gamma * Q[s2, greedy_next]) - Q[s, a]
            E[s, a] += 1.0                           # accumulating trace
            Q += alpha * delta * E                   # update every eligible pair
            if Q[s2, a2] == Q[s2, greedy_next]:      # next action is greedy: keep decaying traces
                E *= gamma * lam
            else:
                E[:] = 0.0                           # exploratory action: cut traces (Watkins)
            s, a = s2, a2
    return Q

Q = watkins_q_lambda()
print(np.round(Q[:, 1], 3))   # values of "right" grow toward the goal; one rewarding episode updates many states

Why They Help

  • Faster propagation of rare or delayed rewards.

  • A tunable bias-variance trade-off through λ.

  • In deep RL, similar benefits come from n-step returns (as in Rainbow) and Retrace.

Common Pitfalls

  • Not cutting traces in Watkins’s Q(λ), which makes it learn a mix of policies.

  • Forgetting to reset traces at the start of each episode.

  • λ close to 1 with long episodes, which raises variance.

Interview Tips

  • Explain traces as a decaying memory of visited pairs and λ as the TD-to-Monte-Carlo dial.

  • Mention why Q(λ) must cut traces after exploratory actions.

30

What is Rainbow DQN and which DQN problems does it address?

Rainbow DQN (Hessel et al., 2018) combines six extensions to DQN in one agent: Double Q-learning (overestimation), prioritized experience replay (inefficient uniform sampling), the dueling architecture (poor state-value learning), multi-step returns (slow reward propagation), distributional RL with C51 (learning only the mean return), and noisy networks (crude ε-greedy exploration). Together they performed far better than any single extension on the Atari 57 benchmark in both learning speed and final score, and ablations showed prioritized replay and multi-step returns contributed the most.

Read full answer

Applies togeneral concepts (deep reinforcement learning)

The Components

Component DQN problem Fix
Double Q-learning Max operator overestimates values Online network selects, target evaluates
Prioritized replay Uniform replay wastes updates on easy transitions Sample by TD error, with importance weights
Dueling network Value of states learned slowly Separate V(s) and A(s, a) streams
Multi-step (n-step) returns One-step targets propagate rewards slowly Target uses n rewards then bootstraps
Distributional RL (C51) Only the expected return is learned Predict a categorical distribution over returns
Noisy networks ε-greedy explores state-independently Learned parametric noise in linear layers

How They Fit Together

  • The network is dueling and noisy, and outputs a return distribution (51 atoms) per action.

  • The loss is the KL divergence between the predicted distribution and the projected n-step distributional target.

  • The target uses Double action selection.

  • Transitions are sampled with prioritized replay, using the KL loss as the priority.

n-Step Return

  • G_t^(n) = r{t+1} + γ r{t+2} + … + γ^(n−1) r{t+n} + γ^n max_a Q(s{t+n}, a); Rainbow used n = 3.

Practical Notes

Aspect Note
Implementation More complex than DQN; use libraries (e.g. Dopamine, CleanRL implementations)
Hyperparameters Many (atoms, v_min/v_max, n, α/β for PER, noise σ)
Compute Heavier per step than DQN, but far more sample-efficient
Later work IQN, R2D2, Agent57, and model-based methods such as MuZero pushed Atari further

Common Pitfalls

  • Mixing n-step returns with off-policy data without care (Rainbow accepts the small bias).

  • Using ε-greedy on top of noisy networks (Rainbow sets ε to 0 or very low).

  • Expecting Rainbow to fix poor reward design or state representation.

Interview Tips

  • List the six components with the problem each solves.

  • Mention the ablation result: prioritized replay and multi-step returns mattered most.

31

How does Q-learning adapt to non-stationary environments?

Q-learning adapts to a non-stationary environment, where rewards or transition probabilities change over time, only if it keeps learning: a constant learning rate α (rather than a decaying 1/n rate) makes Q-values an exponentially weighted average that forgets old experience, and continued exploration (an ε floor or periodic exploration boosts) lets the agent discover that a previously poor action has become good. Stronger adaptations include detecting changes (a spike in TD errors or a drop in returns) and resetting exploration, limiting replay buffer age in deep Q-learning, adding time or context features to the state, or keeping separate models per detected regime.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Why Standard Q-Learning Struggles

Assumption What breaks when the world changes
Stationary MDP The optimal Q-values move, so “convergence” is to a moving target
Decaying α (1/n) Late updates barely change values; the agent stops tracking
Decaying ε to 0 The agent never retries actions that improved
Large replay buffer Old, outdated transitions dominate training

Adaptation Techniques

Technique How it helps
Constant α (e.g. 0.1) Effective memory of roughly 1/α recent updates
ε floor or periodic re-exploration Keeps sampling all actions
Change detection (TD-error or return monitoring) Raise ε or α when a shift is detected
Recency-limited / sliding replay buffer Train on current dynamics
Context in the state (time of day, regime id) Turns a non-stationary problem into a stationary one with a larger state
Optimistic bonuses (UCB with discounting) Encourages revisiting stale estimates
Meta-RL / model-based RL Learn to adapt quickly or re-plan with an updated model

Code Example

import numpy as np

def run(alpha_rule, steps=4000, switch_at=2000, epsilon=0.1, seed=42):
    """Two-action bandit-like task: the better action switches halfway through."""
    rng = np.random.default_rng(seed)
    Q, N = np.zeros(2), np.zeros(2)
    correct = []
    for t in range(steps):
        best = 0 if t < switch_at else 1
        a = int(rng.integers(2)) if rng.random() < epsilon else int(np.argmax(Q))
        r = rng.normal(1.0 if a == best else 0.0, 0.5)
        N[a] += 1
        alpha = 1.0 / N[a] if alpha_rule == "sample-average" else 0.1
        Q[a] += alpha * (r - Q[a])
        correct.append(a == best)
    after = np.array(correct[switch_at:])
    return round(float(after[-500:].mean()), 2)

print("sample-average alpha, best action rate after switch:", run("sample-average"))
print("constant alpha 0.1, best action rate after switch:   ", run("constant"))
# Constant alpha recovers to choosing the new best action most of the time;
# the 1/n rate adapts far more slowly because action 0's long history keeps its estimate high.

Common Pitfalls

  • Decaying both α and ε to near zero in a world that changes.

  • Huge replay buffers full of stale transitions.

  • Treating a change in reward scale as noise instead of a regime change.

Interview Tips

  • Lead with constant α (recency weighting) and persistent exploration.

  • Mention change detection and putting context into the state.

32

How would you model an autonomous vehicle at an intersection for Q-learning?

To model an autonomous vehicle at an intersection for Q-learning, define a compact state from what matters for the decision (the ego vehicle’s distance to the stop line and speed, the traffic signal phase, and the nearest conflicting vehicles’ positions, speeds and time-to-collision), a small discrete action set (accelerate, keep speed, brake, creep, go), and a reward that heavily penalizes collisions and rule violations, rewards safely clearing the intersection, and lightly penalizes waiting time and harsh acceleration. Because real sensor states are continuous and high-dimensional, you either discretize a few features for tabular Q-learning or use DQN, and you train in simulation (SUMO, CARLA, highway-env) with safety constraints layered on top before any real deployment.

Read full answer

Applies togeneral concepts; examples in Python 3.12

State Design

Feature Encoding (tabular example)
Distance to stop line Bins: far, approaching, at line, inside intersection, cleared
Ego speed Bins: stopped, slow, medium, fast
Signal phase Green, yellow, red (or unsignalized)
Nearest conflicting vehicle Time-to-arrival bins: none, > 4 s, 2–4 s, < 2 s
Pedestrian in crosswalk Yes / no
Ego intention Straight, left turn, right turn

That gives 5 × 4 × 3 × 4 × 2 × 3 = 1,440 states, which is small enough for a Q-table.

Actions

  • Discrete: hard brake, brake, maintain, accelerate, creep forward (for occluded views).

  • High-level actions keep the problem tractable; a lower-level controller converts them into throttle and steering.

Reward

Event Reward
Collision −100 (episode ends)
Running a red light or failing to yield −50
Clearing the intersection safely +10
Each time step −0.1 (discourages endless waiting)
Hard braking or jerk −1 (comfort)

Code Example

import numpy as np

DIST = ["far", "approaching", "at_line", "inside", "cleared"]
SPEED = ["stopped", "slow", "medium", "fast"]
SIGNAL = ["green", "yellow", "red"]
GAP = ["none", "gt4s", "2to4s", "lt2s"]
PED = [False, True]
TURN = ["straight", "left", "right"]
ACTIONS = ["hard_brake", "brake", "maintain", "accelerate", "creep"]

def encode(dist, speed, signal, gap, ped, turn):
    idx = (DIST.index(dist), SPEED.index(speed), SIGNAL.index(signal),
           GAP.index(gap), PED.index(ped), TURN.index(turn))
    dims = (len(DIST), len(SPEED), len(SIGNAL), len(GAP), len(PED), len(TURN))
    return int(np.ravel_multi_index(idx, dims))

n_states = len(DIST) * len(SPEED) * len(SIGNAL) * len(GAP) * len(PED) * len(TURN)
Q = np.zeros((n_states, len(ACTIONS)))

def safe_actions(signal, gap, ped, dist):
    """Rule-based shield: forbid entering on red, with a close conflicting car, or with a pedestrian."""
    if dist in ("approaching", "at_line") and (signal == "red" or gap == "lt2s" or ped):
        return [0, 1]                       # only braking actions allowed
    return list(range(len(ACTIONS)))

s = encode("at_line", "slow", "green", "lt2s", False, "left")
allowed = safe_actions("green", "lt2s", False, "at_line")
print(n_states, s, [ACTIONS[a] for a in allowed])   # 1440, a state index, ['hard_brake', 'brake']

Practical Concerns

  • Safety: use a shield or constraints so learning never selects clearly dangerous actions.

  • Partial observability: occlusions mean the true state is hidden; add history or a “creep” action.

  • Sim-to-real gap: randomize traffic, driver behaviour and sensor noise in simulation.

  • Other agents: human drivers react to the ego car, so the environment is non-stationary.

Common Pitfalls

  • Raw continuous states in a tabular setup (state explosion).

  • Rewards that only penalize collisions, producing a car that never moves.

  • Deploying a learned policy without verification and fallback controllers.

Interview Tips

  • Walk through state, actions, reward and terminal conditions in order.

  • Bring up safety shields, simulation and partial observability.

33

How would you teach a Q-learning agent to play a simple video game?

To teach a Q-learning agent a simple video game, wrap the game as an environment with reset and step, choose a state representation (a few game variables for tabular Q-learning, or stacked downscaled frames for a DQN), define actions as the game’s controls, use the score change as the reward (often clipped), and train for many episodes with ε-greedy exploration, experience replay and a target network if you use a neural network. The distinctive challenges are high-dimensional pixel input, sparse or delayed rewards, partial observability from single frames, long training times, and agents exploiting bugs or unintended reward loopholes.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with Gymnasium 1.x and Stable-Baselines3 2.x

Steps

  1. Environment: Gymnasium-style API (reset, step, render); Atari games via ale-py.

  2. State: game variables (paddle x, ball x/y/velocity) or 84×84 grayscale frames, 4 stacked to capture motion.

  3. Actions: the controller inputs (left, right, fire, no-op).

  4. Reward: score changes, clipped to [−1, 1] for Atari; optional shaping.

  5. Algorithm: tabular Q-learning for tiny games; DQN (with Double/Dueling/PER) for pixels.

  6. Training tricks: frame skipping (repeat an action for 4 frames), replay buffer, target network, ε decay.

  7. Evaluation: average score over many episodes with ε near 0, against a random baseline.

Unique Challenges

Challenge Mitigation
Huge pixel state space CNN function approximation (DQN)
One frame hides velocity Stack the last 4 frames
Sparse rewards (e.g. Montezuma’s Revenge) Reward shaping, curiosity / exploration bonuses
Different score scales across games Reward clipping
Slow training Frame skip, vectorized environments, GPU
Reward hacking (exploiting glitches) Inspect replays, adjust reward and termination
Non-deterministic starts No-op starts or sticky actions for robust evaluation

Code Example

import gymnasium as gym
from stable_baselines3 import DQN
from stable_baselines3.common.evaluation import evaluate_policy

# CartPole stands in for a simple game; for Atari use "ALE/Pong-v5" with CnnPolicy and Atari wrappers
env = gym.make("CartPole-v1")
model = DQN(
    "MlpPolicy", env,
    learning_rate=1e-3, buffer_size=50_000, learning_starts=1_000,
    batch_size=64, gamma=0.99, target_update_interval=500,
    exploration_fraction=0.2, exploration_final_eps=0.05,
    train_freq=4, seed=42, verbose=0,
)
model.learn(total_timesteps=60_000)

mean_reward, std_reward = evaluate_policy(model, gym.make("CartPole-v1"), n_eval_episodes=20)
print(f"mean reward {mean_reward:.1f} +/- {std_reward:.1f}")   # random play scores about 20

Common Pitfalls

  • Using raw full-resolution colour frames (slow, no benefit for simple games).

  • Single frames as the state, hiding motion.

  • Judging progress on one episode or with exploration on.

Interview Tips

  • Cover state, actions, reward, algorithm and evaluation.

  • Name frame stacking, reward clipping and reward hacking as video-game specifics.

34

How would you use Q-learning in a multi-agent setting such as doubles tennis?

For doubles tennis, treat each player as an agent in a multi-agent environment where two teammates cooperate against two opponents, and choose a training scheme: independent Q-learning (each agent learns its own Q-function and treats others as part of the environment), centralized training with decentralized execution (agents act on local observations but learn from a shared team value, as in VDN or QMIX), and self-play against past versions of the team to get ever-stronger opponents. The main challenges are non-stationarity (teammates and opponents keep changing their policies), credit assignment (which partner earned the point), coordination (covering the court without both chasing the same ball) and a large continuous state, which calls for deep Q-networks over abstracted actions.

Read full answer

Applies togeneral concepts (multi-agent reinforcement learning)

Environment Design

Element Design
Agents 4 players: 2 teammates per side
Observation (per agent) Own position, partner position, opponents’ positions, ball position and velocity, score state
Actions High-level: move to zone, return cross-court / down the line / lob, approach net, hold position
Reward Team reward: +1 point won, −1 point lost; small shaping for good positioning or rallies
Episode A point, game or set

Training Strategies

Approach Idea Trade-off
Independent Q-learning (IQL) Each agent runs DQN alone Simple; unstable because others change
Parameter sharing Teammates share one network with an agent-id input Faster learning, symmetric roles
VDN Q_team = Q₁ + Q₂ Credit assignment, decentralized acting
QMIX Monotonic mixing network of Q₁, Q₂ conditioned on global state More expressive coordination
Self-play / league Train against snapshots of past teams Avoids overfitting to one opponent

Coordination Ideas

  • Give each agent a role feature (net player vs baseline player) and let the policy learn when to switch.

  • Allow lightweight communication (an intended target zone) if the rules permit it.

  • Reward the team, not the individual, so agents don’t fight over the ball.

Training Plan

  1. Start in a simplified 2D simulator with discrete court zones.

  2. Train teammates against scripted opponents to learn basics.

  3. Switch to self-play with a league of past opponents.

  4. Evaluate with Elo-style ratings across opponent pools.

Common Pitfalls

  • Independent learners with large replay buffers of outdated partner behaviour.

  • Individual rewards that cause selfish play.

  • Training against one fixed opponent and overfitting to it.

Interview Tips

  • Name non-stationarity and credit assignment as the core multi-agent problems.

  • Propose CTDE (VDN/QMIX) plus self-play.

35

What are the current limitations of Q-learning and how does research address them?

Q-learning’s main limitations are that it does not scale to large or continuous state and action spaces without function approximation, becomes unstable when it does use approximation (the “deadly triad”), overestimates values through the max operator, explores inefficiently with ε-greedy, needs many samples, and handles only discrete actions directly. Research responses include deep Q-networks with replay and target networks, Double and distributional Q-learning, better exploration (noisy nets, count-based and curiosity bonuses), model-based methods for sample efficiency, actor-critic variants (DDPG, TD3, SAC) for continuous actions, and conservative offline methods (CQL, IQL) for learning from fixed datasets.

Read full answer

Applies togeneral concepts (reinforcement learning research)

Limitations and Responses

Limitation Why it happens Research directions
State-space explosion One table entry per state-action pair Function approximation, DQN
Instability / divergence Approximation + bootstrapping + off-policy Target networks, replay, regularized and gradient TD methods
Overestimation bias max over noisy estimates Double DQN, clipped double Q (TD3), ensembles
Poor exploration ε-greedy is undirected Noisy networks, RND, count-based bonuses, Go-Explore
Sample inefficiency Model-free, one-step updates Model-based RL (MuZero, Dreamer), n-step returns, data augmentation
Discrete actions only max over actions is hard in continuous spaces DDPG, TD3, SAC, NAF
Learning from fixed data Extrapolation error on unseen actions CQL, IQL, BCQ (offline RL)
Partial observability Assumes Markov states Recurrent DQN (DRQN, R2D2), frame stacking
Reward design Sparse or mis-specified rewards Reward shaping, inverse RL, learning from human feedback
Generalization Overfits training environments Procedural generation, domain randomization

Where Q-Learning Still Shines

  • Small discrete problems where convergence is guaranteed.

  • As the value-learning core of many modern algorithms (critics in SAC and TD3 are Q-functions).

  • Offline RL and recommendation settings, using conservative variants.

Common Pitfalls in Answers

  • Saying Q-learning is “obsolete”: its ideas sit inside most modern value-based and actor-critic methods.

  • Listing limitations without the matching fixes.

Interview Tips

  • Pair each limitation with one concrete technique.

  • Mention the deadly triad and overestimation by name.

36

How has deep learning changed Q-learning methods?

Deep learning turned Q-learning from a tabular method for small problems into one that learns directly from raw, high-dimensional inputs: DQN (Mnih et al., 2015) used a convolutional network to map Atari pixels to Q-values and reached human-level play on many games, made stable by experience replay and a target network. That success started a wave of improvements (Double, Dueling, prioritized replay, distributional and Rainbow DQN, recurrent R2D2, Agent57) and brought Q-functions into actor-critic methods for continuous control (DDPG, TD3, SAC), while introducing new problems: instability, heavy compute, hyperparameter sensitivity and poor interpretability.

Read full answer

Applies togeneral concepts (deep reinforcement learning)

What Changed

Before (tabular / linear) After (deep Q-learning)
Hand-crafted features or small discrete states Learned features from pixels, sensors, text
Exact convergence guarantees (tabular) Empirical stability via engineering tricks
Few thousand states Millions of effective states
Cheap CPU training GPU training, millions of frames

Key Milestones

Method Contribution
DQN (2013/2015) CNN + replay + target network on Atari
Double DQN (2016) Reduced overestimation
Dueling DQN (2016) Separate value and advantage streams
Prioritized replay (2016) Sample informative transitions
C51 / QR-DQN (2017) Learn the return distribution
Rainbow (2018) Combined six improvements
R2D2 (2019) Recurrent networks + distributed replay
Agent57 (2020) Beat the human baseline on all 57 Atari games
DDPG / TD3 / SAC Deep Q-function critics for continuous actions
CQL (2020) Conservative deep Q-learning from offline data

New Challenges Deep Learning Introduced

  • Instability: the deadly triad; needs replay, target networks, gradient clipping and Huber loss.

  • Compute and sample cost: many millions of environment steps.

  • Reproducibility: high variance across seeds; report several seeds.

  • Interpretability: hard to explain why a network values a state.

  • Generalization: agents can overfit to their training environment.

Common Pitfalls in Answers

  • Crediting DQN’s success only to the network, without replay and target networks.

  • Ignoring the continuous-control branch, where Q-functions became critics.

Interview Tips

  • Explain DQN’s two stabilizers and why they were needed.

  • Name a few follow-ups and the problem each solved.

37

How can transfer learning speed up Q-learning across similar tasks?

Transfer learning speeds up Q-learning by reusing what was learned on a source task to start a related target task closer to a good solution: initializing the target Q-table or network with the source values, fine-tuning a pretrained DQN’s feature layers while retraining its head, reusing successful policies or options as exploration guides, or mapping states and actions between tasks when they differ. It works when the tasks share structure, such as the same dynamics with a different goal or a similar visual domain, and can hurt (“negative transfer”) when they don’t, so you compare against training from scratch.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Transfer Techniques

Technique What is reused Best when
Q-value initialization Source Q-table / network weights Same states and actions, different rewards
Feature transfer Early layers of a DQN Similar observations (e.g. same game engine)
Policy reuse / guidance Source policy used during exploration Similar goals
Successor features Dynamics representation, combined with new reward weights Same dynamics, new reward functions
Inter-task mapping Hand or learned mapping of states/actions Different but analogous spaces
Curriculum learning Train on easy tasks first Hard target tasks with sparse rewards
Domain randomization Train on many variations Sim-to-real

Code Example

import numpy as np

def make_step(goal, n=10):
    def step(s, a):
        s2 = min(max(s + (1 if a == 1 else -1), 0), n - 1)
        return s2, (1.0 if s2 == goal else -0.01), s2 == goal
    return step

def train(step, Q, episodes, start=0, seed=42):
    rng = np.random.default_rng(seed)
    lengths = []
    for _ in range(episodes):
        s, done, t = start, False, 0
        while not done and t < 200:
            a = int(rng.integers(2)) if rng.random() < 0.1 else int(rng.choice(np.flatnonzero(Q[s] == Q[s].max())))
            s2, r, done = step(s, a)
            Q[s, a] += 0.2 * (r + (0.0 if done else 0.95 * Q[s2].max()) - Q[s, a])
            s, t = s2, t + 1
        lengths.append(t)
    return Q, lengths

# Source: goal at state 9. Target: goal at state 8 (similar task).
Q_source, _ = train(make_step(goal=9), np.zeros((10, 2)), episodes=300)
_, scratch = train(make_step(goal=8), np.zeros((10, 2)), episodes=20)
_, transfer = train(make_step(goal=8), Q_source.copy(), episodes=20)
print("steps in the first 20 episodes, scratch:", sum(scratch))
print("steps in the first 20 episodes, transfer:", sum(transfer))
# The transferred agent already heads right, so it typically needs far fewer steps early on

Measuring Transfer

  • Jumpstart: better initial performance.

  • Time to threshold: fewer episodes to reach a target score.

  • Asymptotic performance: final score equal or better than scratch.

  • Negative transfer check: always include a from-scratch baseline.

Common Pitfalls

  • Transferring between tasks with different dynamics without adaptation.

  • Overconfident transferred values that suppress exploration (reset ε or scale values down).

  • No scratch baseline, so gains can’t be shown.

Interview Tips

  • Name initialization, feature transfer and curriculum learning.

  • Mention jumpstart and negative transfer.

38

What is Meta Reinforcement Learning and where does Q-learning fit in it?

Meta reinforcement learning (“learning to learn”) trains an agent on a distribution of related tasks so that it can adapt to a new task from that distribution with very little experience, either by learning a good initialization that fine-tunes quickly (MAML), by using a recurrent network whose hidden state performs fast in-context adaptation (RL², the “learning to reinforcement learn” approach), or by inferring a latent task variable from recent experience and conditioning the policy and value function on it (PEARL). Q-learning fits as the off-policy inner learner or the critic: PEARL is built on soft actor-critic, whose Q-function is conditioned on the inferred task, and meta-learned Q-initializations or exploration strategies can speed up ordinary Q-learning on new tasks.

Read full answer

Applies togeneral concepts (meta reinforcement learning)

Main Families

Family Example How adaptation happens
Gradient-based MAML-RL A few gradient steps from a meta-learned initialization
Recurrence / memory-based RL², SNAIL The network’s hidden state adapts within an episode
Context / task inference PEARL, VariBAD Infer a latent task embedding z; condition Q(s, a, z) or π(a | s, z)
Learned learning rules Meta-gradients, learned optimizers Meta-learn hyperparameters or the update itself

Where Q-Learning Fits

  • Task-conditioned Q-functions: Q(s, a, z) where z encodes the current task (PEARL’s critic).

  • Off-policy efficiency: replay makes meta-training far cheaper than on-policy methods.

  • Meta-learned initialization: start Q-learning near good values for the task family.

  • Meta-learned exploration: learn how to explore a new task, then let Q-learning exploit.

  • Meta-gradient tuning: adapt γ, λ or α online for value-based agents.

Example Setting

  • Task family: navigation to a goal whose location varies per task.

  • Meta-training: thousands of goals; the agent learns to infer the goal from the first few rewards.

  • Meta-testing: on a new goal, a few episodes of experience update z, and the Q-function already knows how to act given z.

Challenges

Challenge Note
Task distribution design Test tasks must resemble training tasks
Meta-overfitting Few training tasks → poor generalization
Compute Nested loops over tasks and episodes
Exploration for task identification The agent must gather informative experience fast

Common Pitfalls in Answers

  • Confusing meta-RL with multi-task RL (multi-task solves known tasks; meta-RL adapts to new ones).

  • Claiming tabular Q-learning is meta-learning by itself.

Interview Tips

  • Name the three families with one example each.

  • Place Q-learning as the off-policy critic or inner learner (PEARL on SAC).

39

How do you write a Python function that evaluates a trained Q-learning policy?

A policy evaluation function runs the trained agent greedily (ε = 0 or a very small ε) for many episodes on a separately seeded copy of the environment, without updating the Q-table, and reports summary statistics: mean and standard deviation of the return, success rate, and average episode length, ideally against a random-policy baseline. Separating evaluation from training and using fixed seeds makes the numbers comparable across runs and hyperparameter settings.

Read full answer

Applies toPython 3.12, NumPy 2.x, Gymnasium 1.x

Code Example

import gymnasium as gym
import numpy as np

def evaluate_policy(env, Q, n_episodes=100, epsilon=0.0, seed=42, max_steps=None):
    """Run a Q-table policy without learning and return summary statistics."""
    rng = np.random.default_rng(seed)
    returns, lengths, successes = [], [], 0
    for ep in range(n_episodes):
        state, _ = env.reset(seed=seed + ep)            # reproducible, different start each episode
        total, steps, terminated, truncated = 0.0, 0, False, False
        while not (terminated or truncated):
            if rng.random() < epsilon:
                action = int(env.action_space.sample())
            else:
                action = int(np.argmax(Q[state]))
            state, reward, terminated, truncated, _ = env.step(action)
            total += float(reward)
            steps += 1
            if max_steps is not None and steps >= max_steps:
                break
        returns.append(total)
        lengths.append(steps)
        successes += int(terminated and total > 0)       # task-specific success definition
    returns = np.array(returns)
    return {
        "mean_return": round(float(returns.mean()), 3),
        "std_return": round(float(returns.std()), 3),
        "success_rate": round(successes / n_episodes, 3),
        "mean_length": round(float(np.mean(lengths)), 1),
    }

def random_baseline(env, n_episodes=100, seed=42):
    zero_Q = np.zeros((env.observation_space.n, env.action_space.n))
    return evaluate_policy(env, zero_Q, n_episodes, epsilon=1.0, seed=seed)

env = gym.make("FrozenLake-v1", is_slippery=False)
Q = np.zeros((16, 4))
path = {0: 1, 4: 1, 8: 2, 9: 1, 13: 2, 14: 2}       # a hand-made safe route: down, down, right, down, right, right
for s, a in path.items():
    Q[s, a] = 1.0

print("trained:", evaluate_policy(env, Q))           # success_rate 1.0, mean_length 6.0 on the deterministic map
print("random: ", random_baseline(env))              # success_rate close to 0

What to Report

Metric Why
Mean ± std return Overall performance and variability
Success rate Clearer for goal-reaching tasks
Episode length Efficiency of the policy
Random / heuristic baseline Shows the agent actually learned
Several training seeds Robustness of the training method

Common Pitfalls

  • Evaluating with the training ε, which understates the policy.

  • Updating Q during evaluation.

  • Reusing the training environment’s random state, making results hard to reproduce.

Interview Tips

  • Separate training and evaluation environments, and fix seeds.

  • Report a baseline and variability, not one number.

40

How do you build a Q-learning agent that solves Gymnasium's Taxi-v3?

Taxi-v3 is a 5×5 grid where a taxi must pick up a passenger at one of four locations and drop them at a destination; it has 500 discrete states and 6 actions (south, north, east, west, pickup, dropoff), a reward of −1 per step, +20 for a successful drop-off and −10 for an illegal pickup or drop-off, so tabular Q-learning solves it well. A Q-table of shape (500, 6), α around 0.1–0.2, γ = 0.99 and ε decaying over a few thousand episodes typically gives a greedy policy that completes every episode in roughly 13 steps on average, for an average return of about 8.

Read full answer

Applies toPython 3.12, NumPy 2.x, Gymnasium 1.x (the old OpenAI Gym `Taxi-v3` is the same task)

Code Example

import gymnasium as gym
import numpy as np

env = gym.make("Taxi-v3")
n_states, n_actions = env.observation_space.n, env.action_space.n   # 500, 6
Q = np.zeros((n_states, n_actions))
rng = np.random.default_rng(42)
alpha, gamma = 0.2, 0.99
episodes = 5000

env.reset(seed=42)
for ep in range(episodes):
    state, _ = env.reset()
    epsilon = max(0.01, 1.0 - ep / (0.5 * episodes))
    terminated = truncated = False
    while not (terminated or truncated):
        if rng.random() < epsilon:
            action = int(rng.integers(n_actions))
        else:
            action = int(np.argmax(Q[state]))
        next_state, reward, terminated, truncated, _ = env.step(action)
        target = reward + (0.0 if terminated else gamma * Q[next_state].max())   # bootstrap on truncation
        Q[state, action] += alpha * (target - Q[state, action])
        state = next_state

def evaluate(Q, n=200, seed=1000):
    eval_env = gym.make("Taxi-v3")
    returns, steps, solved = [], [], 0
    for i in range(n):
        s, _ = eval_env.reset(seed=seed + i)
        total, t, term, trunc = 0, 0, False, False
        while not (term or trunc):
            s, r, term, trunc, _ = eval_env.step(int(np.argmax(Q[s])))
            total, t = total + r, t + 1
        returns.append(total)
        steps.append(t)
        solved += int(term)
    return round(float(np.mean(returns)), 2), round(float(np.mean(steps)), 1), solved / n

mean_return, mean_steps, solved = evaluate(Q)
print(f"mean return {mean_return}, mean steps {mean_steps}, solved {solved:.0%}")
# Typically a mean return near 8 and about 13 steps, with every episode solved

Understanding the State

  • 25 taxi positions × 5 passenger locations (four depots or in the taxi) × 4 destinations = 500.

  • env.unwrapped.decode(state) returns (taxi_row, taxi_col, passenger_location, destination) for debugging.

Tuning Notes

Hyperparameter Effect
α too high Values oscillate; 0.1–0.3 works well
ε decay too fast Agent never finds the +20 reward reliably
Episodes A few thousand is usually enough
Truncation at 200 steps Don’t treat it as terminal in the update

Common Pitfalls

  • Using the old Gym four-value step return with Gymnasium.

  • Evaluating with exploration on.

  • Forgetting that illegal pickup/dropoff gives −10, which can make early agents avoid those actions entirely if ε decays too fast.

Interview Tips

  • Explain the 500-state encoding and the reward structure.

  • Show training plus a separate greedy evaluation.

41

How do you implement Q-learning for traffic signal control with variable traffic flow?

For traffic signal control with variable vehicle flow, the agent must learn context-specific rules (switch more often when cross traffic is heavy, hold green on the busy approach at rush hour), so the state has to include the context: queue lengths per approach (discretized), the current phase and how long it has been green, and a flow-regime or time-of-day feature. Actions are “keep phase” or “switch phase” (subject to a minimum green time), and the reward is the negative total queue length or waiting time per step; training over episodes with randomly varying arrival rates teaches one Q-table to act differently in each context.

Read full answer

Applies toPython 3.12, NumPy 2.x (real studies use SUMO via TraCI or sumo-rl)

Model

Element Choice
State (NS queue bin, EW queue bin, current phase, green-time bin, flow regime)
Actions 0 = keep phase, 1 = switch (only allowed after minimum green)
Reward −(NS queue + EW queue) each step
Context Regime: balanced, NS-heavy, EW-heavy; changes between episodes
Episode 300 simulated seconds-steps

Code Example

import numpy as np

REGIMES = {0: (0.3, 0.3), 1: (0.6, 0.15), 2: (0.15, 0.6)}   # arrival probabilities (NS, EW) per step
MIN_GREEN, DEPART = 5, 2                                     # cars leaving per step on green

def qbin(q):
    return 0 if q < 3 else 1 if q < 8 else 2 if q < 15 else 3

class Intersection:
    def __init__(self, rng):
        self.rng = rng

    def reset(self, regime):
        self.regime, self.queues, self.phase, self.green = regime, [0, 0], 0, 0
        return self.state()

    def state(self):
        return (qbin(self.queues[0]), qbin(self.queues[1]), self.phase, min(self.green // MIN_GREEN, 2), self.regime)

    def step(self, action):
        if action == 1 and self.green >= MIN_GREEN:
            self.phase, self.green = 1 - self.phase, 0       # switch (a real model adds an all-red interval)
        else:
            self.green += 1
        p_ns, p_ew = REGIMES[self.regime]
        self.queues[0] += int(self.rng.random() < p_ns) + int(self.rng.random() < p_ns)
        self.queues[1] += int(self.rng.random() < p_ew) + int(self.rng.random() < p_ew)
        self.queues[self.phase] = max(0, self.queues[self.phase] - DEPART)
        return self.state(), -float(sum(self.queues))

rng = np.random.default_rng(42)
env = Intersection(rng)
Q = np.zeros((4, 4, 2, 3, 3, 2))
alpha, gamma = 0.1, 0.95

def run_episode(policy, regime, learn=False, epsilon=0.0, steps=300):
    s, total = env.reset(regime), 0.0
    for _ in range(steps):
        if policy == "fixed":
            a = 1 if env.green >= 15 else 0                  # fixed-time controller: 15 s per phase
        elif rng.random() < epsilon:
            a = int(rng.integers(2))
        else:
            a = int(np.argmax(Q[s]))
        s2, r = env.step(a)
        if learn:
            Q[s + (a,)] += alpha * (r + gamma * Q[s2].max() - Q[s + (a,)])
        s, total = s2, total + r
    return total

for ep in range(3000):
    run_episode("q", regime=int(rng.integers(3)), learn=True, epsilon=max(0.05, 1 - ep / 2000))

for regime in REGIMES:
    q_score = np.mean([run_episode("q", regime) for _ in range(20)])
    fixed_score = np.mean([run_episode("fixed", regime) for _ in range(20)])
    print(f"regime {regime}: Q-learning {q_score:9.0f}   fixed-time {fixed_score:9.0f}")
# Less negative is better; the learned controller usually beats fixed timing, most clearly in the unbalanced regimes

What the Agent Learns

  • In NS-heavy flow, long greens for north-south and short greens for east-west.

  • In balanced flow, switching roughly when queues even out.

  • Because the regime is in the state, one table holds all three rule sets.

Extensions

Extension Benefit
DQN over raw queue counts and speeds No binning; scales to more lanes
Multiple intersections (MARL) Green waves through coordination
Pressure-based reward (incoming − outgoing) Better network-level throughput
Infer regime from recent arrivals No need for a known context label

Common Pitfalls

  • Leaving out the context, so the agent averages incompatible rules.

  • No minimum green or safety constraints, causing rapid flickering.

  • Evaluating only on the regime seen most in training.

Interview Tips

  • Spell out state, action and reward, and why context belongs in the state.

  • Compare against a fixed-time baseline.

42

How do you code a Q-learning agent for a maze with dynamic obstacles?

To handle a maze whose obstacles move, include local obstacle information in the state (for example which of the four neighbouring cells are blocked right now) so the agent learns reactive rules instead of memorizing one layout, keep the learning rate constant and exploration above zero so it keeps adapting, and penalize collisions and each step so it prefers short, safe routes. Training across many episodes with randomly placed or moving obstacles produces a policy that heads toward the goal and steps around whatever blocks its way.

Read full answer

Applies toPython 3.12, NumPy 2.x

Design Choices

Element Choice Why
State (row, col, blocked-up, blocked-down, blocked-left, blocked-right) Agent sees what is around it now
Actions Up, down, left, right, wait “Wait” helps when an obstacle is passing
Reward +10 goal, −1 collision attempt, −0.1 per step Fast, collision-free paths
Obstacles Static walls plus movers that shift each step Changing environment
Learning Constant α = 0.1, ε floor 0.05 Keeps tracking changes

Code Example

import numpy as np

class DynamicMaze:
    MOVES = [(-1, 0), (1, 0), (0, -1), (0, 1), (0, 0)]   # up, down, left, right, wait

    def __init__(self, size=7, n_movers=3, seed=42):
        self.size, self.n_movers = size, n_movers
        self.rng = np.random.default_rng(seed)
        self.walls = {(1, 1), (1, 2), (3, 3), (3, 4), (5, 1), (5, 5)}
        self.goal = (size - 1, size - 1)

    def free(self, cell):
        r, c = cell
        return 0 <= r < self.size and 0 <= c < self.size and cell not in self.walls and cell not in self.movers

    def reset(self):
        self.pos = (0, 0)
        cells = [(r, c) for r in range(self.size) for c in range(self.size)
                 if (r, c) not in self.walls and (r, c) not in (self.pos, self.goal)]
        picks = self.rng.choice(len(cells), self.n_movers, replace=False)
        self.movers = {cells[i] for i in picks}
        return self.state()

    def state(self):
        r, c = self.pos
        blocked = tuple(int(not self.free((r + dr, c + dc))) for dr, dc in self.MOVES[:4])
        return (r, c) + blocked

    def move_obstacles(self):
        new = set()
        for (r, c) in self.movers:
            dr, dc = self.MOVES[int(self.rng.integers(5))]
            cand = (r + dr, c + dc)
            ok = (0 <= cand[0] < self.size and 0 <= cand[1] < self.size and cand not in self.walls
                  and cand not in (self.pos, self.goal) and cand not in new)
            new.add(cand if ok else (r, c))
        self.movers = new

    def step(self, action):
        dr, dc = self.MOVES[action]
        target = (self.pos[0] + dr, self.pos[1] + dc)
        reward = -0.1
        if action != 4:
            if self.free(target):
                self.pos = target
            else:
                reward -= 1.0                                # bumped into a wall or obstacle
        if self.pos == self.goal:
            return self.state(), reward + 10.0, True
        self.move_obstacles()
        return self.state(), reward, False

env = DynamicMaze()
Q = np.zeros((env.size, env.size, 2, 2, 2, 2, 5))
rng = np.random.default_rng(42)
lengths = []
for ep in range(4000):
    s, done, t = env.reset(), False, 0
    epsilon = max(0.05, 1 - ep / 2500)
    while not done and t < 200:
        a = int(rng.integers(5)) if rng.random() < epsilon else int(np.argmax(Q[s]))
        s2, r, done = env.step(a)
        target = r if done else r + 0.95 * Q[s2].max()
        Q[s + (a,)] += 0.1 * (target - Q[s + (a,)])
        s, t = s2, t + 1
    lengths.append(t)

print("mean steps, first 200 episodes:", round(float(np.mean(lengths[:200])), 1))
print("mean steps, last 200 episodes: ", round(float(np.mean(lengths[-200:])), 1))
# Episode length drops toward the shortest path (12 moves on the 7x7 grid) plus occasional detours

How It Manages Change

  • Local sensing in the state makes the same rule (“blocked right, go down”) apply to any layout.

  • Constant α and ε floor let values follow obstacle behaviour if it changes.

  • Randomized obstacle placement per episode prevents memorizing one maze.

  • For larger mazes, replace the table with a DQN over a local view window, or combine learning with re-planning (Dyna-Q).

Common Pitfalls

  • A state with only the position, which cannot represent changing obstacles.

  • Training on one fixed obstacle layout, then testing on new ones.

  • Decaying ε to zero in a world that keeps changing.

Interview Tips

  • Explain why the state must expose the dynamic parts of the environment.

  • Mention Dyna-Q or DQN for scaling up.

43

How can you make Q-learning more computationally efficient?

Q-learning’s computational efficiency improves on three fronts: making each update cheaper (vectorized NumPy or batched GPU updates, compact state encodings, sparse or hashed tables, smaller networks), making each sample count for more (experience replay, prioritized replay, n-step returns, eligibility traces, Dyna-style planning with a learned model), and collecting experience faster (vectorized or parallel environments, frame skipping, distributed actors as in Ape-X or R2D2). Profiling first matters: in deep Q-learning the environment step, not the network, is often the bottleneck.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Techniques

Area Technique Effect
State representation Fewer, well-chosen features; tile coding; abstraction Smaller table or network
Data structures NumPy arrays instead of dicts; integer state indices Fast lookup and max
Updates Batched replay updates on GPU; mixed precision Higher throughput
Sample efficiency PER, n-step returns, eligibility traces Fewer environment steps
Planning Dyna-Q (simulated updates from a model) More learning per real step
Environment speed Vectorized envs (gymnasium.vector), frame skip, JAX-based envs More steps per second
Distribution Ape-X / R2D2: many actors, one learner Scales across machines
Network Smaller architectures, target-network updates less often Less compute per step
Early stopping Stop when evaluation plateaus Saves wasted training

Code Example: Vectorized Batch Updates

import time

import numpy as np

rng = np.random.default_rng(42)
n_states, n_actions, batch = 10_000, 4, 100_000
Q = np.zeros((n_states, n_actions))
s = rng.integers(n_states, size=batch)
a = rng.integers(n_actions, size=batch)
r = rng.normal(size=batch)
s2 = rng.integers(n_states, size=batch)
done = rng.random(batch) < 0.05
alpha, gamma = 0.1, 0.99

Q_loop = Q.copy()
start = time.perf_counter()
for i in range(batch):
    target = r[i] + (0.0 if done[i] else gamma * Q_loop[s2[i]].max())
    Q_loop[s[i], a[i]] += alpha * (target - Q_loop[s[i], a[i]])
loop_time = time.perf_counter() - start

Q_vec = Q.copy()
start = time.perf_counter()
targets = r + gamma * Q_vec[s2].max(axis=1) * (~done)
td = targets - Q_vec[s, a]
np.add.at(Q_vec, (s, a), alpha * td)          # handles repeated (s, a) pairs correctly
vec_time = time.perf_counter() - start

print(f"loop {loop_time:.3f}s, vectorized {vec_time:.3f}s")
# The vectorized batch is typically tens of times faster. It uses the Q-values from before the
# batch (like a replay minibatch with a frozen target), so the two results differ slightly.

Dyna-Q in One Paragraph

  • After each real step, store (s, a) → (r, s′) in a model.

  • Then perform k extra Q-updates on randomly chosen remembered pairs.

  • With k = 5–50, the agent often needs many times fewer real steps in deterministic environments.

Common Pitfalls

  • Optimizing the network when the simulator is the bottleneck.

  • Python dictionaries keyed by tuples for large tables.

  • Tiny batch sizes on a GPU, which waste parallelism.

Interview Tips

  • Separate per-update cost, sample efficiency and data collection speed.

  • Mention vectorization, replay/PER, Dyna-Q and distributed actors.

44

What common issues should you check when debugging a Q-learning agent?

When a Q-learning agent fails to learn, the usual causes are bugs in the update (bootstrapping from terminal states, wrong indexing, the target computed with the wrong network), exploration problems (ε decaying too fast, argmax without tie-breaking), reward problems (wrong sign, too sparse, wrong scale), state problems (missing information, non-Markov observations, unnormalized inputs), and hyperparameters (α too high, γ wrong for the horizon). For deep Q-learning, add divergence from missing target networks or replay, exploding Q-values and gradients, and evaluation mistakes such as testing with exploration on. A methodical approach is to test on a tiny environment with known optimal values first, then log Q-values, TD errors and returns.

Read full answer

Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x

Checklist

Symptom Likely cause Check / fix
Returns never improve Update bug, reward never reached Unit-test the update; inspect reward frequency
Agent always takes the same action argmax on zeros without tie-breaking; ε decayed too fast Random tie-breaking; slower decay
Q-values grow without bound No terminal handling; γ = 1 with endless episodes; divergence Mask terminal targets; γ < 1; target network
Learns, then collapses (DQN) Instability, stale replay, learning rate too high Target network, Huber loss, gradient clipping, smaller LR
Good training score, bad evaluation Evaluation differs (seed, ε, wrappers) Same wrappers, ε ≈ 0, several seeds
Values never settle Stochastic rewards with high α Lower α or decay it
Agent exploits a loophole Reward hacking Watch episodes; fix the reward or termination
Works on one seed only High variance Run 3–5 seeds

Sanity Tests

  1. Known answer: a 3-state chain where Q* can be computed by hand; check the learned values.

  2. Overfit test (DQN): can the network fit a fixed batch of targets?

  3. Random baseline: is the agent better than random?

  4. Reward check: print the rewards of a few random episodes.

  5. Truncation check: make sure time-limit endings still bootstrap.

Code Example

import numpy as np

def q_update(Q, s, a, r, s2, terminated, alpha=0.5, gamma=0.9):
    target = r if terminated else r + gamma * Q[s2].max()
    Q[s, a] += alpha * (target - Q[s, a])

# Chain 0 -> 1 -> 2 (terminal), one action, reward 1 on reaching state 2.
# Known answer: Q*(1) = 1, Q*(0) = gamma * 1 = 0.9.
Q = np.zeros((3, 1))
for _ in range(50):
    q_update(Q, 0, 0, 0.0, 1, False)
    q_update(Q, 1, 0, 1.0, 2, True)
assert abs(Q[1, 0] - 1.0) < 1e-6 and abs(Q[0, 0] - 0.9) < 1e-6, Q.ravel()

# Same chain but the bug of bootstrapping from the terminal state, whose value is non-zero here
Q_bug = np.zeros((3, 1))
Q_bug[2, 0] = 5.0
for _ in range(50):
    target = 1.0 + 0.9 * Q_bug[2].max()                    # forgot the terminal mask
    Q_bug[1, 0] += 0.5 * (target - Q_bug[1, 0])
print("correct Q(1):", round(float(Q[1, 0]), 3), "  buggy Q(1):", round(float(Q_bug[1, 0]), 3))   # 1.0 vs 5.5

What to Log

  • Mean and max Q-values, TD error, episode return and length, ε, and (for DQN) loss and gradient norm.

  • Visit counts per state in tabular problems: never-visited states explain odd policies.

Common Pitfalls

  • Changing many things at once while debugging.

  • Trusting a single training curve.

  • Skipping the small known-answer test.

Interview Tips

  • Give a structured checklist: update, exploration, reward, state, hyperparameters, evaluation.

  • Mention testing on a tiny environment with known Q-values.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 44 Q-Learning answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects