Skip to content
Machine Learning & Data Science

Reinforcement Learning interview questions and answers

70 questions, 69 with code. About 4 h 58 min to read them all.

By TestLab · Updated

Plan for an interview
70 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is reinforcement learning, and how does it differ from supervised and unsupervised learning?

Reinforcement learning (RL) is learning to make sequences of decisions by trial and error: an agent acts in an environment, receives rewards, and learns a policy that maximizes the expected cumulative (discounted) reward. It differs from supervised learning, where a model learns from labelled examples that state the correct output, and from unsupervised learning, where a model finds structure (clusters, components, densities) in unlabelled data. In RL nobody tells the agent the correct action: feedback is a scalar reward that may be delayed (a chess win arrives after many moves), the agent’s own actions change the data it sees, and it must balance exploring new actions with exploiting what already works. Formally, RL problems are Markov decision processes.

Read full answer

Applies toPython 3.10+, NumPy, Gymnasium 1.x

Comparison

Aspect Supervised Unsupervised Reinforcement
Data Labelled (x, y) pairs Unlabelled x Interaction: (s, a, r, s′) generated by the agent
Feedback Correct answer per example None Scalar reward, often delayed
Goal Predict y from x Find structure Maximize expected return
Data distribution Fixed (i.i.d. assumed) Fixed Depends on the agent’s policy
Decisions One-shot predictions — Sequential; actions affect the future
Exploration Not needed Not needed Required
Examples Spam detection, price regression Customer segments, PCA Game playing, robot control, ad bidding, RLHF

Main RL Method Families

Family Idea Examples
Dynamic programming Plan with a known model Value iteration, policy iteration
Monte Carlo Learn from complete episode returns First-visit MC control
Temporal difference Bootstrap from the next state’s estimate SARSA, Q-learning
Deep value-based Neural Q-function DQN, Double DQN
Policy gradient Optimize the policy directly REINFORCE
Actor-critic Policy + learned value baseline A2C, PPO, SAC
Model-based Learn a model, plan with it Dyna, MuZero, Dreamer

Code Example: Learning from Reward Alone (Multi-Armed Bandit)

import numpy as np

rng = np.random.default_rng(42)
true_means = np.array([0.2, 0.5, 0.8])     # unknown to the agent
estimates = np.zeros(3)
counts = np.zeros(3)
epsilon = 0.1

for t in range(2000):
    if rng.random() < epsilon:
        arm = int(rng.integers(3))            # explore
    else:
        arm = int(np.argmax(estimates))       # exploit
    reward = rng.normal(true_means[arm], 1.0) # only a noisy reward, never "the right arm"
    counts[arm] += 1
    estimates[arm] += (reward - estimates[arm]) / counts[arm]   # incremental average

print(np.round(estimates, 2), counts.astype(int))

The agent is never told which arm is best; it learns from noisy rewards that it collects through its own choices. After enough steps the estimates approach the true means and most pulls go to the best arm; the exact numbers depend on the random draws. A full RL problem adds states, so actions also change what the agent will see next.

Applications and Caveats

RL is used for games (AlphaGo), robotics, recommendation and ad bidding, resource and energy control, and RLHF for language models. Caveats: it needs many interactions (usually in simulation), reward design is hard and can be gamed, training is unstable, and simulation-to-real gaps and safety limit deployment.

Common Pitfalls

  • Calling any problem with feedback “RL”; if you have labelled correct actions, supervised learning (imitation) is simpler.

  • Ignoring that the data distribution shifts as the policy changes.

  • Expecting RL to be sample-efficient without a simulator.

Interview Tips

  • Define RL in one sentence with agent, reward and return.

  • Contrast the feedback type and data generation with supervised and unsupervised learning.

  • Mention exploration, delayed rewards and one real application.

2

Define agent, environment, state, action and reward in reinforcement learning.

In reinforcement learning, the agent is the learner and decision-maker; the environment is everything outside the agent that it interacts with; the state is the information describing the current situation; an action is a choice the agent makes; and the reward is a scalar signal the environment returns after each action, telling the agent how good the immediate outcome was. They form a loop: at step t the agent observes state s_t, picks action a_t using its policy π(a|s), and the environment responds with reward r(t+1) and next state s(t+1). The agent’s goal is not to maximize single rewards but the return, the (discounted) sum of future rewards G_t = r(t+1) + γ r(t+2) + γ² r_(t+3) + …, with discount factor 0 ≤ γ ≤ 1. Formally, this loop is a Markov decision process.

Read full answer

Applies toPython 3.10+, Gymnasium 1.x (`import gymnasium as gym`)

The Terms

Term Definition CartPole example Thermostat example
Agent Learner that chooses actions Controller pushing the cart Heating controller
Environment System the agent acts on Cart, pole and physics Building and weather
State s_t Description of the situation Cart position/velocity, pole angle/angular velocity Room temperature, time, outside temperature
Observation What the agent actually sees (may be partial) Same 4 numbers Sensor readings
Action a_t Choice made by the agent Push left or right Heater power level
Reward r_(t+1) Scalar feedback after the action +1 for every step the pole stays up −(energy cost) − comfort penalty
Episode Run from a start state to termination or truncation Until the pole falls or 500 steps One day
Policy π Mapping from states to actions Learned network Learned schedule

Code Example: The Agent-Environment Loop (Gymnasium)

import gymnasium as gym

env = gym.make("CartPole-v1")
print(env.observation_space.shape)        # (4,)  state: position, velocity, angle, angular velocity
print(env.action_space)                   # Discrete(2)  actions: 0 = push left, 1 = push right

observation, info = env.reset(seed=42)
env.action_space.seed(42)
episode_return, steps = 0.0, 0
terminated = truncated = False

while not (terminated or truncated):
    action = env.action_space.sample()                                  # a random "agent"
    observation, reward, terminated, truncated, info = env.step(action)
    episode_return += reward                                            # CartPole gives +1 per step
    steps += 1

print(steps == episode_return)            # True: reward is 1.0 for every step survived
env.close()

terminated means the episode ended inside the task (the pole fell); truncated means it was cut off by a time limit (500 steps). A learning agent would replace env.action_space.sample() with its policy and update that policy from the rewards.

State vs Observation

The state is the full information the environment needs to decide what happens next. The agent often sees only an observation (a camera image, noisy sensors). When the observation leaves out important information, the problem is partially observable (a POMDP), and the agent needs memory such as frame stacking or a recurrent policy.

Common Pitfalls

  • Treating reward as the goal of a single step instead of the return over time.

  • Confusing state with observation.

  • Designing rewards that can be gamed (the agent maximizes exactly what you reward).

  • Using the old Gym API (obs, reward, done, info); Gymnasium splits done into terminated and truncated.

Interview Tips

  • Draw the loop: state → action → reward + next state.

  • Give one concrete example with every term mapped.

  • Mention policy, return and discount factor to show the full picture.

3

What is a Markov Decision Process (MDP) in reinforcement learning?

A Markov Decision Process is the mathematical model of a reinforcement learning problem, defined by the tuple (S, A, P, R, γ): a set of states S, a set of actions A, transition probabilities P(s′ | s, a) of reaching state s′ after taking action a in state s, a reward function R(s, a) (or R(s, a, s′)), and a discount factor γ in [0, 1] that weights future rewards. Its defining assumption is the Markov property: the next state and reward depend only on the current state and action, not on the earlier history, so the state contains everything needed to predict the future. Solving an MDP means finding a policy π that maximizes the expected discounted return E[Σ γᵗ r_(t+1)]. When P and R are known, dynamic programming (value or policy iteration) solves it; when they are not, RL methods learn from sampled transitions.

Read full answer

Applies toPython 3.10+, NumPy

The Components

Symbol Meaning Example (robot vacuum)
S States Location and battery level
A Actions Move, clean, return to dock
P(s′ | s, a) Transition probabilities 0.9 move succeeds, 0.1 wheels slip
R(s, a) Expected immediate reward +1 per dirty cell cleaned, −10 battery empty
γ Discount factor 0.99 (cares about the long run)
π(a | s) Policy (the solution) Cleaning strategy

Why the Discount Factor

γ < 1 keeps the infinite sum finite, expresses preference for sooner rewards, and models uncertainty about the future. With γ = 0.9, a reward 10 steps away is worth 0.9¹⁰ ≈ 0.349 of an immediate one; the “effective horizon” is about 1/(1 − γ) = 10 steps.

Code Example: Evaluating a Policy on a Two-State MDP (NumPy)

import numpy as np

# Fixed policy on two states A=0, B=1:
# in A: reward 1, move to B;  in B: reward 2, move back to A
P_pi = np.array([[0.0, 1.0],
                 [1.0, 0.0]])          # P_pi[s, s'] under the policy
R_pi = np.array([1.0, 2.0])            # expected reward in each state
gamma = 0.9

# Bellman expectation equation in matrix form: V = R + gamma * P V  ->  (I - gamma P) V = R
V = np.linalg.solve(np.eye(2) - gamma * P_pi, R_pi)
print(np.round(V, 2))                  # [14.74 15.26]

By hand: V(A) = 1 + 0.9 V(B) and V(B) = 2 + 0.9 V(A), so V(A) = 2.8 / 0.19 ≈ 14.74 and V(B) = 2 + 0.9 × 14.74 ≈ 15.26.

Code Example: Sampling Trajectories from an MDP with Actions

import numpy as np

rng = np.random.default_rng(42)
# P[s, a, s']: 3 states, 2 actions; rows over s' sum to 1
P = np.array([
    [[0.9, 0.1, 0.0], [0.1, 0.8, 0.1]],
    [[0.5, 0.5, 0.0], [0.0, 0.2, 0.8]],
    [[0.0, 0.0, 1.0], [0.0, 0.0, 1.0]],     # state 2 is absorbing (goal)
])
R = np.array([[0.0, -0.1], [0.0, -0.1], [0.0, 0.0]])
print(np.allclose(P.sum(axis=2), 1.0))      # True

state, ret = 0, 0.0
for t in range(20):
    action = 1                               # policy: always "advance"
    ret += (0.9 ** t) * R[state, action]
    state = int(rng.choice(3, p=P[state, action]))
    if state == 2:
        break
print(state)                                 # 2 (reached the goal), unless 20 steps run out first

Extensions

Variant Change
POMDP Agent sees observations, not states; needs belief or memory
Semi-MDP Actions take variable time (options in hierarchical RL)
Constrained MDP Maximize return subject to cost limits (safe RL)
Multi-agent (Markov game) Several agents act; transitions depend on all

Common Pitfalls

  • Choosing a state that breaks the Markov property (e.g. one camera frame without velocity).

  • Rows of P that do not sum to 1.

  • Setting γ = 1 for continuing tasks (returns can be infinite).

  • Forgetting terminal states should have zero future value.

Interview Tips

  • List the five components and the Markov property.

  • Explain the role of γ with a number.

  • Say that DP solves known MDPs and RL handles unknown ones.

4

What is the role of a policy in reinforcement learning?

The policy is the agent’s behaviour: a rule π that maps each state to an action (deterministic, a = π(s)) or to a probability distribution over actions (stochastic, π(a|s)), and finding a good policy is the goal of reinforcement learning. Every RL method ultimately produces one: value-based methods derive it from a learned Q-function (act greedily, π(s) = argmax_a Q(s, a), with some exploration), policy-gradient and actor-critic methods learn it directly as a parameterized function π_θ (a neural network that outputs action probabilities or the mean and variance of continuous actions), and model-based methods produce it by planning. The policy also decides which data the agent collects, so it controls exploration, and the distinction between the policy being learned (target) and the policy generating data (behaviour) defines on-policy vs off-policy learning.

Read full answer

Applies toPython 3.10+, NumPy, PyTorch 2.x

Types of Policies

Type Form Example Used by
Deterministic tabular Table s → a Grid world: “up” in cell (2, 3) Policy iteration
Greedy from Q argmax_a Q(s, a) Q-learning’s final policy DQN, Q-learning
ε-greedy Greedy with probability 1 − ε, random otherwise Exploration during training Q-learning, SARSA
Softmax / categorical π(a s) = softmax(logits) Discrete action network REINFORCE, PPO
Gaussian a ~ N(μ_θ(s), σ_θ(s)) Robot joint torques PPO, SAC
Deterministic continuous a = μ_θ(s) Continuous control DDPG, TD3

Code Example: Policies Side by Side

import numpy as np
import torch
from torch import nn

rng = np.random.default_rng(42)

# 1. Deterministic tabular policy for 4 states
policy_table = {0: "right", 1: "right", 2: "down", 3: "stay"}
print(policy_table[2])                                   # down

# 2. Greedy and epsilon-greedy policies derived from Q-values
Q = np.array([1.0, 3.0, 2.0])                            # Q(s, a) for three actions in one state
greedy = int(np.argmax(Q))
print(greedy)                                            # 1


def epsilon_greedy(q, eps):
    return int(rng.integers(len(q))) if rng.random() < eps else int(np.argmax(q))


# 3. Stochastic softmax policy
logits = np.array([2.0, 1.0, 0.0])
probs = np.exp(logits) / np.exp(logits).sum()
print(np.round(probs, 3))                                # [0.665 0.245 0.09 ]

# 4. Parameterized neural policy pi_theta(a|s) for a 4-dim state and 2 actions
torch.manual_seed(42)
policy_net = nn.Sequential(nn.Linear(4, 32), nn.Tanh(), nn.Linear(32, 2))
state = torch.randn(1, 4)
dist = torch.distributions.Categorical(logits=policy_net(state))
action = dist.sample()
print(action.shape, dist.log_prob(action).shape)         # torch.Size([1]) torch.Size([1])

The softmax probabilities are e²/(e² + e + 1) ≈ 0.665, e/(e² + e + 1) ≈ 0.245 and 1/(e² + e + 1) ≈ 0.090. The log_prob of the sampled action is what policy-gradient methods use to update θ.

Why Stochastic Policies Matter

Reason Example
Built-in exploration Sampling tries non-greedy actions
Optimal under partial observability Two states that look identical may need different actions
Games against opponents Rock-paper-scissors: the optimal policy is uniformly random
Smooth optimization Policy gradients need differentiable probabilities

Common Pitfalls

  • Confusing the policy (what to do) with the value function (how good a state or action is).

  • Evaluating a stochastic policy with exploration left on, or deploying with exploration on when it should be greedy.

  • Forgetting that changing the policy changes the data distribution.

Interview Tips

  • Define deterministic and stochastic policies with notation.

  • Explain how value-based methods derive a policy and policy-gradient methods learn one directly.

  • Connect the policy to exploration and on-/off-policy learning.

5

What are value functions, and how do they relate to policies?

A value function estimates how good it is to be in a situation under a given policy, measured as expected return: the state-value function V^π(s) = E_π[G_t | s_t = s] is the expected discounted return starting from state s and following π; the action-value function Q^π(s, a) = E_π[G_t | s_t = s, a_t = a] is the expected return after taking action a in s and following π afterwards. They link to policies in both directions. Given a policy, its values are defined by it: V^π(s) = Σ_a π(a|s) Q^π(s, a) (policy evaluation). Given values, you can improve the policy by acting greedily, π′(s) = argmax_a Q^π(s, a) (policy improvement); repeating both steps converges to the optimal values V* and Q* and an optimal policy. The advantage A^π(s, a) = Q^π(s, a) − V^π(s) says how much better an action is than the policy’s average and is the signal used by actor-critic methods.

Read full answer

Applies toPython 3.10+, NumPy

Definitions

Function Formula Answers
V^π(s) E_π[Σ_k γᵏ r_(t+k+1) | s_t = s] How good is this state under π?
Q^π(s, a) E_π[Σ_k γᵏ r_(t+k+1) | s_t = s, a_t = a] How good is this action here, then following π?
A^π(s, a) Q^π(s, a) − V^π(s) Better or worse than average?
V(s), Q(s, a) max over all policies Best achievable

Relationships

Relation Formula
V from Q V^π(s) = Σ_a π(a|s) Q^π(s, a)
Q from V (needs model) Q^π(s, a) = R(s, a) + γ Σ_s′ P(s′|s, a) V^π(s′)
Greedy improvement π′(s) = argmax_a Q^π(s, a)
Optimal policy π(s) = argmax_a Q(s, a)

Code Example: Values, Policy and Advantage (NumPy)

import numpy as np

Q = np.array([[1.0, 3.0],          # Q(s0, a0), Q(s0, a1)
              [2.0, 0.0]])         # Q(s1, a0), Q(s1, a1)
pi = np.array([[0.5, 0.5],         # current stochastic policy pi(a|s)
               [0.25, 0.75]])

V = (pi * Q).sum(axis=1)           # V(s) = sum_a pi(a|s) Q(s, a)
print(V)                           # [2.  0.5]

advantage = Q - V[:, None]
print(advantage)                   # [[-1.   1. ]
                                   #  [ 1.5 -0.5]]

greedy_policy = Q.argmax(axis=1)   # policy improvement step
print(greedy_policy)               # [1 0]

By hand: V(s0) = 0.5·1 + 0.5·3 = 2 and V(s1) = 0.25·2 + 0.75·0 = 0.5. The positive advantages (a1 in s0, a0 in s1) point to the actions the improved policy picks.

Why Q Is Often Preferred

Acting greedily with V needs the model (to look one step ahead through P), while acting greedily with Q needs only an argmax over actions, which is why model-free control methods (Q-learning, SARSA, DQN) learn Q.

How Methods Use Value Functions

Method Value function role
Value/policy iteration Compute V or Q with the model
Q-learning, DQN Learn Q*, act greedily
Actor-critic (A2C, PPO) Critic learns V; advantage weights policy updates
REINFORCE with baseline V(s) baseline reduces variance

Common Pitfalls

  • Mixing up V and Q, or forgetting a value function is always for a particular policy (or optimal).

  • Using V to choose actions in a model-free setting.

  • Ignoring terminal states (their future value is 0).

Interview Tips

  • Write V^π and Q^π as expectations of the return.

  • Show V = Σ π Q and greedy improvement from Q.

  • Mention the advantage and where actor-critic methods use it.

6

What is the difference between on-policy and off-policy learning?

On-policy methods learn the value or improve the policy that is generating the data (the behaviour policy and target policy are the same); off-policy methods learn about a target policy — usually the greedy, optimal one — from data generated by a different behaviour policy, such as an exploratory ε-greedy policy, an older version of the agent, or human logs. The textbook pair is SARSA (on-policy) and Q-learning (off-policy): SARSA’s target uses the action the agent will actually take next, r + γ Q(s′, a′), while Q-learning’s uses the best next action, r + γ max_a Q(s′, a). Off-policy methods can reuse old experience (replay buffers, logged data), so they are more sample-efficient, but they can be less stable and may need corrections such as importance sampling. On-policy methods (SARSA, REINFORCE, A2C, PPO) are usually more stable but must keep collecting fresh data.

Read full answer

Applies toPython 3.10+, NumPy

Comparison

Aspect On-policy Off-policy
Data source Current policy only Any behaviour policy
Experience reuse Little (PPO reuses a batch for a few epochs) Replay buffers, logged data
Sample efficiency Lower Higher
Stability Usually higher Can diverge (with function approximation + bootstrapping)
Corrections needed None Importance sampling or clipped variants for some methods
Learned policy accounts for exploration Yes (SARSA learns a “safer” ε-greedy policy) No (learns the greedy policy)
Examples SARSA, REINFORCE, A2C/A3C, PPO, TRPO Q-learning, DQN, DDPG, TD3, SAC

Code Example: SARSA vs Q-Learning Targets (NumPy)

import numpy as np

gamma, alpha = 0.9, 0.5
Q = {"s": np.array([1.0, 1.0]), "s_next": np.array([2.0, 5.0])}
reward = 1.0
a_taken = 0
a_next_from_behaviour = 0            # epsilon-greedy happened to explore action 0 next

sarsa_target = reward + gamma * Q["s_next"][a_next_from_behaviour]   # on-policy
q_learning_target = reward + gamma * Q["s_next"].max()               # off-policy
print(sarsa_target, q_learning_target)                               # 2.8 5.5

sarsa_new = Q["s"][a_taken] + alpha * (sarsa_target - Q["s"][a_taken])
q_new = Q["s"][a_taken] + alpha * (q_learning_target - Q["s"][a_taken])
print(sarsa_new, q_new)                                              # 1.9 3.25

SARSA evaluates what the exploring agent actually does next (Q = 2 for action 0), while Q-learning assumes the best action from then on (Q = 5). On Sutton and Barto’s “cliff walking” example this makes SARSA learn a safer path away from the cliff edge, since it accounts for its own random steps, while Q-learning learns the shorter, riskier optimal path.

Importance Sampling

To evaluate target policy π with data from behaviour policy b, returns are reweighted by ρ = Π_t π(a_t|s_t) / b(a_t|s_t). The product can have very high variance over long episodes, so practical methods use per-step or clipped ratios (Retrace, V-trace, PPO’s clipped ratio).

Offline RL

Offline (batch) RL is the extreme off-policy case: learn only from a fixed logged dataset with no new interaction. It needs conservative methods (CQL, IQL) because Q-values for actions absent from the data can be wildly overestimated.

Common Pitfalls

  • Calling PPO off-policy because it does several epochs per batch; it is on-policy with a clipped ratio.

  • Using a replay buffer with an on-policy algorithm without corrections.

  • Expecting off-policy learning from logs to work when the logged policy never tried the actions you want to evaluate.

Interview Tips

  • Define behaviour and target policies.

  • Contrast SARSA and Q-learning targets in one line each.

  • Discuss the sample-efficiency vs stability trade-off and importance sampling.

7

What is the exploration vs exploitation trade-off in reinforcement learning?

The exploration-exploitation trade-off is the agent’s dilemma between exploiting — choosing the action it currently believes is best to collect reward now — and exploring — trying other actions to gather information that might reveal something better. Pure exploitation can lock the agent into a suboptimal action because its early estimates were wrong; pure exploration wastes reward on actions known to be bad. Good strategies explore a lot early, when estimates are uncertain, and less later. Common methods are ε-greedy (random action with probability ε, often decayed), softmax/Boltzmann sampling, optimistic initial values, Upper Confidence Bound (UCB) bonuses for rarely tried actions, Thompson sampling from a posterior, entropy bonuses in policy-gradient methods, and curiosity / count-based intrinsic rewards for sparse-reward environments.

Read full answer

Applies toPython 3.10+, NumPy

Strategies

Strategy Rule Pros Cons
Greedy argmax Q Simple Can get stuck
ε-greedy Random with prob. ε Simple, works everywhere Explores blindly, even bad actions
Decaying ε ε from 1.0 to 0.05 over time More exploration early Schedule to tune
Softmax (temperature τ) P(a) ∝ exp(Q(a)/τ) Prefers promising actions τ to tune
Optimistic initial values Start Q high Encourages trying everything Only early exploration
UCB Q(a) + c √(ln t / N(a)) Targets uncertain actions; regret bounds Mostly bandits/tabular
Thompson sampling Sample from posterior, act greedily Strong in practice Needs a posterior
Entropy bonus Add β·H(π) to the objective Keeps policies stochastic β to tune
Intrinsic motivation Bonus for novel states (RND, ICM, counts) Sparse rewards Can chase noise

Code Example: Greedy vs ε-Greedy vs UCB on a 10-Armed Bandit

import numpy as np


def run(strategy, steps=1000, runs=200, eps=0.1, c=2.0, seed=42):
    rng = np.random.default_rng(seed)
    total = 0.0
    for _ in range(runs):
        means = rng.normal(0, 1, 10)                 # new random bandit each run
        Q, N = np.zeros(10), np.zeros(10)
        for t in range(1, steps + 1):
            if strategy == "greedy":
                a = int(np.argmax(Q))
            elif strategy == "eps":
                a = int(rng.integers(10)) if rng.random() < eps else int(np.argmax(Q))
            else:                                    # UCB: try every arm once, then add a bonus
                untried = np.flatnonzero(N == 0)
                a = int(untried[0]) if untried.size else int(np.argmax(Q + c * np.sqrt(np.log(t) / N)))
            r = rng.normal(means[a], 1)
            N[a] += 1
            Q[a] += (r - Q[a]) / N[a]
            total += r
    return total / (runs * steps)


for s in ("greedy", "eps", "ucb"):
    print(s, round(run(s), 3))

On this classic testbed the greedy agent usually earns the lowest average reward (it commits to whichever arm looked good first), while ε-greedy and UCB find better arms; UCB is often best. The exact averages depend on the seed, so run it rather than relying on quoted numbers.

Exploration in Deep RL

Method family Typical exploration
DQN ε-greedy decayed from 1.0 to ~0.05; noisy networks
PPO / A2C Stochastic policy + entropy bonus
SAC Maximum-entropy objective with tuned temperature
Sparse-reward tasks RND, ICM, count-based bonuses, demonstrations

Common Pitfalls

  • Never decaying ε, so the trained agent keeps acting randomly.

  • Evaluating with exploration on (or deploying with it off when online learning is expected).

  • Decaying exploration too fast before the agent has seen rewarding states.

  • In production, exploring in ways that are unsafe or hurt users; use offline evaluation and constrained exploration.

Interview Tips

  • Define both sides and why either extreme fails.

  • Describe ε-greedy with decay, UCB and Thompson sampling.

  • Mention exploration in deep RL (entropy, curiosity) for sparse rewards.

8

What are the Bellman equations, and how are they used in reinforcement learning?

The Bellman equations express the value of a state (or state-action pair) recursively, as the immediate reward plus the discounted value of where you go next. The expectation equation for a policy π is V^π(s) = Σ_a π(a|s) [R(s, a) + γ Σ_s′ P(s′|s, a) V^π(s′)], and the optimality equation is V(s) = max_a [R(s, a) + γ Σ_s′ P(s′|s, a) V(s′)] (with matching versions for Q, such as Q(s, a) = R(s, a) + γ Σ_s′ P(s′|s, a) max_a′ Q(s′, a′)). They turn a sum over an infinite future into a one-step relationship, which almost every RL algorithm uses: policy evaluation and value iteration apply them as update rules with a known model; temporal-difference methods (TD(0), SARSA, Q-learning, DQN) apply them to sampled transitions, moving estimates toward the “Bellman target” r + γ V(s′) or r + γ max Q(s′, a′).

Read full answer

Applies toPython 3.10+, NumPy

The Four Equations

Equation Formula (in words)
Expectation, V Value under π = expected reward + γ × expected next value under π
Expectation, Q Q^π(s, a) = R(s, a) + γ Σ_s′ P(s′|s, a) Σ_a′ π(a′|s′) Q^π(s′, a′)
Optimality, V V*(s) = max over actions of (reward + γ × expected next optimal value)
Optimality, Q Q(s, a) = R(s, a) + γ Σ_s′ P(s′|s, a) max_a′ Q(s′, a′)

How Algorithms Use Them

Algorithm Equation How
Policy evaluation Expectation Iterate or solve linear system
Policy iteration Expectation + greedy step Evaluate, improve, repeat
Value iteration Optimality Apply max-backup until convergence
TD(0) Expectation (sampled) V(s) += α[r + γV(s′) − V(s)]
SARSA Q expectation (sampled) Target r + γQ(s′, a′)
Q-learning, DQN Q optimality (sampled) Target r + γ max Q(s′, ·)

Code Example: Value Iteration with the Optimality Equation (NumPy)

import numpy as np

# Two states, two actions (0 = stay, 1 = switch). Deterministic transitions.
# State 0: stay -> reward 1, stay in 0;  switch -> reward 0, go to 1
# State 1: stay -> reward 2, stay in 1;  switch -> reward 0, go to 0
R = np.array([[1.0, 0.0],
              [2.0, 0.0]])
next_state = np.array([[0, 1],
                       [1, 0]])
gamma = 0.9

V = np.zeros(2)
while True:
    Q = R + gamma * V[next_state]          # Bellman backup for every (s, a)
    V_new = Q.max(axis=1)                  # optimality: take the best action
    if np.max(np.abs(V_new - V)) < 1e-8:
        break
    V = V_new

print(np.round(V_new, 2).tolist())         # [18.0, 20.0]
print(Q.argmax(axis=1).tolist())           # [1, 0]  switch from 0, then stay in 1

By hand: staying in state 1 forever earns 2/(1 − 0.9) = 20. From state 0, staying earns 1/(1 − 0.9) = 10, while switching earns 0 + 0.9 × 20 = 18, so V*(0) = 18 and the optimal policy switches once and then stays.

Why They Work

The Bellman backup is a contraction with factor γ: each application shrinks the maximum error by at least γ, so repeated backups converge to a unique fixed point (V^π or V*). With function approximation and off-policy data that guarantee is lost, which is why DQN needs target networks and replay.

Common Pitfalls

  • Bootstrapping from terminal states (their next value must be 0).

  • Confusing the expectation equation (follows π) with the optimality equation (max).

  • Stopping value iteration on a tiny change in V without considering γ close to 1 (convergence is slow).

Interview Tips

  • Write V* and Q* optimality equations and explain each term.

  • Map algorithms to equations: DP uses them exactly, TD methods use sampled versions.

  • Mention the contraction property as the reason iteration converges.

9

What is the difference between model-based and model-free reinforcement learning?

Model-based reinforcement learning uses a model of the environment — the transition dynamics P(s′|s, a) and reward function R(s, a), either known (chess rules, a physics simulator) or learned from data — and plans with it to choose actions or to generate extra training experience; model-free reinforcement learning skips the model and learns a value function or policy directly from real experience by trial and error. Model-based methods (value iteration with a known model, Dyna, MCTS in AlphaZero, MuZero, PETS, Dreamer, MBPO) are usually far more sample-efficient and can plan for new goals, but they suffer from model errors that the planner can exploit and cost more computation. Model-free methods (Q-learning, SARSA, DQN, REINFORCE, PPO, SAC) are simpler and reach strong final performance, but need many more interactions.

Read full answer

Applies toPython 3.10+, NumPy

Comparison

Aspect Model-based Model-free
Learns Dynamics/reward model (+ often value/policy) Value function and/or policy only
Decision making Planning, lookahead, simulated rollouts Direct policy or argmax Q
Sample efficiency High Low
Computation per decision Higher (planning) Low
Main risk Model bias/compounding errors Many interactions, slow learning
Adapting to new goals Replan with the same model Usually retrain
Examples Value iteration, Dyna-Q, AlphaZero, MuZero, Dreamer, MBPO Q-learning, DQN, PPO, SAC, REINFORCE
Good fit Expensive real data, known rules, robotics with simulators Cheap simulators, hard-to-model dynamics

Code Example: The Same Experience Used Both Ways (NumPy)

import numpy as np
from collections import Counter

# Logged transitions (state, action, reward, next_state); 2 states, 1 action for simplicity
data = [(0, 0, 0.0, 1), (0, 0, 0.0, 1), (0, 0, 0.0, 1), (0, 0, 1.0, 0),
        (1, 0, 2.0, 1), (1, 0, 2.0, 1)]
gamma = 0.9

# Model-based: estimate P and R by counting, then plan (solve the Bellman equation)
counts = Counter((s, s2) for s, _, _, s2 in data)
visits = Counter(s for s, _, _, _ in data)
P_hat = np.array([[counts[(s, s2)] / visits[s] for s2 in range(2)] for s in range(2)])
R_hat = np.array([np.mean([r for s, _, r, _ in data if s == st]) for st in range(2)])
print(P_hat.tolist(), R_hat.tolist())    # [[0.25, 0.75], [0.0, 1.0]] [0.25, 2.0]
V_model = np.linalg.solve(np.eye(2) - gamma * P_hat, R_hat)
print(np.round(V_model, 2))              # [17.74 20.  ]

# Model-free: TD(0) updates straight from the same transitions, no P or R estimated
V_td = np.zeros(2)
alpha = 0.1
for _ in range(500):                     # replay the log many times
    for s, _, r, s2 in data:
        V_td[s] += alpha * (r + gamma * V_td[s2] - V_td[s])
print(np.round(V_td, 1))                 # close to the model-based values; exact digits depend on alpha

By hand for the model-based part: state 1 always returns to itself with reward 2, so V(1) = 2 / 0.1 = 20. State 0 has R̂(0) = 1/4 = 0.25 and V(0) = 0.25 + 0.9(0.25 V(0) + 0.75 × 20), so 0.775 V(0) = 13.75 and V(0) ≈ 17.74. The model-based agent gets these values in one solve; the model-free agent needs many passes over the data to approach them.

Hybrids

Approach Idea
Dyna-Q Learn a model; do extra Q-updates on simulated transitions
MBPO Short model rollouts feed an SAC learner
AlphaZero / MuZero Tree search with a known or learned model, guided by learned policy/value
Dreamer Learn a world model, train an actor-critic in imagination

Common Pitfalls

  • Assuming a learned model is accurate over long rollouts (errors compound; keep rollouts short).

  • Calling DQN model-based because of its replay buffer (replay is stored data, not a model).

  • Ignoring planning cost at decision time.

Interview Tips

  • Define the model as P and R and say what each family learns.

  • Contrast sample efficiency vs model bias.

  • Name examples of each and one hybrid (Dyna, MuZero).

10

What are the advantages and disadvantages of model-based reinforcement learning?

Model-based reinforcement learning learns or is given a model of the environment’s dynamics and rewards and uses it to plan or to generate simulated experience; its main advantage is sample efficiency — it can learn from far fewer real interactions — and its main disadvantage is model error, because a planner will exploit inaccuracies in the model, and errors compound over long imagined rollouts. Other advantages are the ability to plan ahead and look before acting, to transfer the same model to new goals or reward functions, to check safety in simulation, and to explain decisions through predicted outcomes. Other disadvantages are higher computation and implementation complexity, difficulty modelling high-dimensional or stochastic dynamics (images, contact physics), and often lower final performance than well-tuned model-free methods when real data is cheap.

Read full answer

Applies toPython 3.10+, NumPy, Gymnasium 1.x

Advantages and Disadvantages

Advantages Disadvantages
Sample efficiency: learns from few real interactions Model bias: planner exploits model errors
Planning and lookahead (MCTS, MPC) Compounding errors over long rollouts
Reuse the model for new goals/rewards Hard to model complex, stochastic, high-dimensional dynamics
Safer: test actions in the model first More computation at training and decision time
Better use of offline data More components to tune (model, planner, policy)
Interpretability through predicted outcomes Asymptotic performance can lag model-free methods
Uncertainty-aware exploration (ensembles) Uncertainty estimation itself is hard

Code Example: Dyna-Q on a Deterministic Grid (Gymnasium)

import gymnasium as gym
import numpy as np

env = gym.make("FrozenLake-v1", is_slippery=False)      # 4x4 grid, reward 1 at the goal
n_s, n_a = env.observation_space.n, env.action_space.n
rng = np.random.default_rng(42)


def dyna_q(planning_steps, episodes=60, alpha=0.5, gamma=0.95, eps=0.1):
    Q = np.zeros((n_s, n_a))
    model = {}                                         # (s, a) -> (r, s', terminated)
    successes = 0
    for ep in range(episodes):
        s, _ = env.reset(seed=42 + ep)
        done = False
        while not done:
            if rng.random() < eps or not Q[s].any():
                a = int(rng.integers(n_a))
            else:
                a = int(np.argmax(Q[s]))
            s2, r, terminated, truncated, _ = env.step(a)
            Q[s, a] += alpha * (r + gamma * Q[s2].max() * (not terminated) - Q[s, a])   # real update
            model[(s, a)] = (r, s2, terminated)                                         # learn the model
            keys = list(model)
            for _ in range(planning_steps):                                             # simulated updates
                ps, pa = keys[rng.integers(len(keys))]
                pr, ps2, pterm = model[(ps, pa)]
                Q[ps, pa] += alpha * (pr + gamma * Q[ps2].max() * (not pterm) - Q[ps, pa])
            s, done = s2, terminated or truncated
            successes += r
    return successes


print("model-free (0 planning steps):", dyna_q(0))
print("model-based (20 planning steps):", dyna_q(20))

Each real step trains Q once; with planning, the stored model replays 20 extra simulated transitions per real step, so the goal reward spreads back through the grid much faster and more of the 60 episodes typically reach the goal. Counts vary with the random draws. The environment is deterministic, so the table model is exact here; in stochastic or continuous environments the model is only an approximation.

Mitigating the Disadvantages

Problem Mitigation
Model exploitation Ensembles and uncertainty penalties (PETS, MOPO)
Compounding errors Short rollouts branched from real states (MBPO)
High-dimensional observations Latent world models (Dreamer, MuZero)
Computation Amortize planning into a policy network
Wrong model at decision time Model-predictive control: replan every step

Common Pitfalls

  • Trusting long imagined rollouts.

  • Measuring only sample efficiency and ignoring wall-clock compute.

  • Training the model on data from one policy and planning far outside that data.

Interview Tips

  • Lead with sample efficiency vs model bias.

  • Give hybrids that address the weaknesses (Dyna, MBPO, MuZero).

  • Mention when model-free is the better choice (cheap simulators, hard dynamics).

11

How does Q-learning work, and why is it called model-free?

Q-learning learns the optimal action-value function Q(s, a) directly from experience: after each transition (s, a, r, s′) it moves its estimate toward the Bellman optimality target, Q(s, a) ← Q(s, a) + α [r + γ max_a′ Q(s′, a′) − Q(s, a)], and the policy is to act greedily on Q (with ε-greedy exploration while learning). It is model-free* because it never learns or uses the transition probabilities P(s′|s, a) or the reward function: the sampled next state and reward stand in for the expectation in the Bellman equation, and the max over Q-values replaces planning. It is also off-policy*: the target uses the greedy action even when the agent explored, so it learns the optimal policy while behaving exploratorily. With a table, every state-action pair visited infinitely often and suitable step sizes, it converges to Q; DQN extends it to large state spaces with a neural network.

Read full answer

Applies toPython 3.10+, NumPy, Gymnasium 1.x

The Update, Term by Term

Term Meaning
Q(s, a) Current estimate
r + γ max_a′ Q(s′, a′) TD target: reward plus best estimated future value
target − Q(s, a) TD error δ
α Learning rate (step size)
(1 − terminated) Future value is 0 after a terminal state

One update by hand: Q(s, a) = 0, α = 0.5, r = 1, γ = 0.9, max Q(s′, ·) = 2 → target 2.8, new Q = 0 + 0.5 × 2.8 = 1.4.

Code Example: Tabular Q-Learning on FrozenLake (Gymnasium)

import gymnasium as gym
import numpy as np

env = gym.make("FrozenLake-v1", is_slippery=False)
Q = np.zeros((env.observation_space.n, env.action_space.n))
rng = np.random.default_rng(42)
alpha, gamma = 0.5, 0.95
eps, eps_min, eps_decay = 1.0, 0.05, 0.995

for episode in range(2000):
    s, _ = env.reset(seed=42 + episode)
    done = False
    while not done:
        a = int(rng.integers(4)) if rng.random() < eps else int(np.argmax(Q[s]))
        s2, r, terminated, truncated, _ = env.step(a)
        target = r + gamma * np.max(Q[s2]) * (not terminated)   # no future value after terminal
        Q[s, a] += alpha * (target - Q[s, a])
        s, done = s2, terminated or truncated
    eps = max(eps_min, eps * eps_decay)

# Evaluate the greedy policy (no exploration)
s, _ = env.reset(seed=0)
done, total = False, 0.0
while not done:
    s, r, terminated, truncated, _ = env.step(int(np.argmax(Q[s])))
    total += r
    done = terminated or truncated
print("greedy policy reaches the goal:", total == 1.0)
print(np.argmax(Q, axis=1).reshape(4, 4))     # learned action per cell (0=left, 1=down, 2=right, 3=up)

Nothing in the loop uses env.unwrapped.P (FrozenLake’s transition table); the agent learns only from the (s, a, r, s2) samples, which is exactly what “model-free” means. On the deterministic 4×4 map the greedy policy normally reaches the goal after training.

Model-Free vs Model-Based

Q-learning (model-free) Value iteration (model-based)
Needs P and R No Yes
Update uses One sampled transition Expectation over all next states
Data needed Many interactions None (just the model)

Limitations

Limitation Remedy
Tables do not scale Function approximation (DQN)
Maximization bias (overestimation) Double Q-learning
Slow credit assignment n-step returns, eligibility traces
Continuous actions Actor-critic (DDPG, SAC)

Common Pitfalls

  • Bootstrapping from terminal states.

  • Never decaying ε, or evaluating with exploration on.

  • Treating time-limit truncation as termination (it should still bootstrap).

Interview Tips

  • Write the update and name the TD target and TD error.

  • Explain model-free (samples, no P/R) and off-policy (max in the target).

  • Mention convergence conditions and the move to DQN.

12

Describe the Monte Carlo method in reinforcement learning.

Monte Carlo (MC) methods in reinforcement learning estimate value functions by averaging the actual returns observed after visiting states (or state-action pairs) over complete episodes: run an episode with the policy, compute the return G_t = r(t+1) + γ r(t+2) + … from each step to the end, and set V(s) (or Q(s, a)) to the average of the returns that followed visits to s. They are model-free (only sampled episodes are needed), they do not bootstrap from other estimates, so they are unbiased, but their estimates have high variance because a return sums many random rewards, and they can only update after an episode ends, so they need episodic tasks. First-visit MC averages the return after the first visit to s in each episode; every-visit MC uses all visits. MC control alternates evaluation of Q with ε-greedy policy improvement.

Read full answer

Applies toPython 3.10+, NumPy, Gymnasium 1.x

Worked Example (γ = 1)

Episode: s₀ = A (reward 1) → s₁ = B (reward 0) → s₂ = A (reward 2) → end.

Step State Reward after Return G_t
0 A 1 1 + 0 + 2 = 3
1 B 0 0 + 2 = 2
2 A 2 2

First-visit: V(A) = 3, V(B) = 2. Every-visit: V(A) = (3 + 2)/2 = 2.5, V(B) = 2.

Code Example: Returns and First-Visit MC

import numpy as np
from collections import defaultdict


def returns_from(rewards, gamma):
    G, out = 0.0, []
    for r in reversed(rewards):
        G = r + gamma * G
        out.append(G)
    return out[::-1]


states, rewards = ["A", "B", "A"], [1.0, 0.0, 2.0]
G = returns_from(rewards, gamma=1.0)
print(G)                                           # [3.0, 2.0, 2.0]

first_visit, every_visit = {}, defaultdict(list)
for t, s in enumerate(states):
    every_visit[s].append(G[t])
    first_visit.setdefault(s, G[t])
print(first_visit)                                 # {'A': 3.0, 'B': 2.0}
print({s: float(np.mean(v)) for s, v in every_visit.items()})   # {'A': 2.5, 'B': 2.0}

Code Example: First-Visit MC Control on Blackjack (Gymnasium)

import gymnasium as gym
import numpy as np
from collections import defaultdict

env = gym.make("Blackjack-v1")
rng = np.random.default_rng(42)
Q = defaultdict(lambda: np.zeros(2))               # actions: 0 = stick, 1 = hit
N = defaultdict(lambda: np.zeros(2))
eps = 0.1

for episode in range(50_000):
    s, _ = env.reset(seed=episode)
    trajectory, done = [], False
    while not done:
        a = int(rng.integers(2)) if rng.random() < eps else int(np.argmax(Q[s]))
        s2, r, terminated, truncated, _ = env.step(a)
        trajectory.append((s, a, r))
        s, done = s2, terminated or truncated

    G = 0.0
    for t in reversed(range(len(trajectory))):     # gamma = 1 for blackjack
        s_t, a_t, r_t = trajectory[t]
        G = r_t + G
        if (s_t, a_t) not in [(x[0], x[1]) for x in trajectory[:t]]:   # first visit only
            N[s_t][a_t] += 1
            Q[s_t][a_t] += (G - Q[s_t][a_t]) / N[s_t][a_t]             # running average

print(int(np.argmax(Q[(20, 10, False)])))          # with a 20 against a dealer 10, the usual result is 0 (stick)

Blackjack suits MC well: episodes are short, the reward comes only at the end, and the dynamics (card probabilities) are awkward to write down, so learning from sampled games is easier than building a model.

MC vs TD vs DP

Property Monte Carlo TD (SARSA, Q-learning) Dynamic programming
Needs model No No Yes
Bootstraps No Yes Yes
Bias Unbiased Biased (by estimates) —
Variance High Lower —
Update timing End of episode Every step Sweep over states
Continuing tasks No Yes Yes

Common Pitfalls

  • Using MC for very long or continuing tasks (no episode end, huge variance).

  • Forgetting exploration in MC control (exploring starts or ε-soft policies).

  • Using a fixed step size and calling it a sample average (fine for non-stationary problems, but different).

Interview Tips

  • Define the return and first-visit vs every-visit averaging.

  • State unbiased/high variance and episode-end updates.

  • Contrast with TD methods and give a good use case (blackjack, short episodic games).

13

How do temporal-difference methods like SARSA differ from Monte Carlo methods?

Temporal-difference (TD) methods update value estimates after every step using a bootstrapped target — the observed reward plus the discounted current estimate of the next state, r + γ V(s′) (or r + γ Q(s′, a′) in SARSA) — while Monte Carlo methods wait until the episode ends and update toward the full observed return G_t. That difference drives the trade-offs: TD can learn online and in continuing tasks, its targets have lower variance (one random reward plus an estimate), and it usually learns faster in practice, but its targets are biased because they depend on current, possibly wrong, estimates; MC is unbiased but high-variance and needs complete episodes. SARSA is the on-policy TD control method whose update uses the quintuple (s, a, r, s′, a′): Q(s, a) ← Q(s, a) + α [r + γ Q(s′, a′) − Q(s, a)]. n-step TD and TD(λ) sit between the two extremes.

Read full answer

Applies toPython 3.10+, NumPy, Gymnasium 1.x

Comparison

Aspect Monte Carlo TD (TD(0), SARSA, Q-learning)
Target G_t (full return) r + γ V(s′) (bootstrapped)
Update time End of episode Every step
Bias None Yes (depends on estimates)
Variance High Low
Continuing tasks No Yes
Markov property Not exploited Exploited (efficient when it holds)
Sensitivity to initial values Low Higher
Function approximation Stable Can be unstable (deadly triad)

One Update by Hand

V(s) = 0.5, α = 0.1, γ = 1, observed r = 0 and V(s′) = 0.6, and the episode’s eventual return from s is G = 1.

  • TD(0): V(s) ← 0.5 + 0.1 (0 + 0.6 − 0.5) = 0.51

  • MC: V(s) ← 0.5 + 0.1 (1 − 0.5) = 0.55

Code Example: SARSA on CliffWalking (Gymnasium)

import gymnasium as gym
import numpy as np

try:
    env = gym.make("CliffWalking-v1")                 # 4x12 grid, -1 per step, -100 for the cliff
except gym.error.Error:                               # older Gymnasium releases name it v0
    env = gym.make("CliffWalking-v0")
n_s, n_a = env.observation_space.n, env.action_space.n
rng = np.random.default_rng(42)
alpha, gamma, eps = 0.5, 1.0, 0.1


def eps_greedy(Q, s):
    return int(rng.integers(n_a)) if rng.random() < eps else int(np.argmax(Q[s]))


Q = np.zeros((n_s, n_a))
for episode in range(500):
    s, _ = env.reset(seed=42 + episode)
    a = eps_greedy(Q, s)
    done = False
    while not done:
        s2, r, terminated, truncated, _ = env.step(a)
        a2 = eps_greedy(Q, s2)                         # the action SARSA will actually take
        target = r + gamma * Q[s2, a2] * (not terminated)
        Q[s, a] += alpha * (target - Q[s, a])          # updated every step, not at episode end
        s, a, done = s2, a2, terminated or truncated

print(np.argmax(Q, axis=1).reshape(4, 12))             # 0=up, 1=right, 2=down, 3=left

SARSA learns from every step, and because its target includes its own ε-greedy actions, it tends to learn a path that keeps a row away from the cliff (a safer route), as in Sutton and Barto’s classic example.

Between the Extremes

Method Target
TD(0) r(t+1) + γ V(s(t+1))
n-step TD r(t+1) + … + γⁿ⁻¹ r(t+n) + γⁿ V(s_(t+n))
TD(λ) λ-weighted average of all n-step returns (eligibility traces)
Monte Carlo Full return G_t

Common Pitfalls

  • Saying TD is always better; with poor features or non-Markov states, MC can be more reliable.

  • Bootstrapping across a terminal state.

  • Confusing SARSA (on-policy, uses a′) with Q-learning (off-policy, uses max).

Interview Tips

  • Contrast the two targets and the bias-variance trade-off.

  • Write the SARSA update and explain the (s, a, r, s′, a′) name.

  • Mention n-step and TD(λ) as a continuum.

14

What is a Deep Q-Network (DQN), and how does it use deep learning?

A Deep Q-Network is Q-learning with a neural network as the Q-function: the network takes a state (in the original 2015 DeepMind work, a stack of four 84×84 Atari frames) and outputs one Q-value per action, it is trained to minimize the squared or Huber error between Q_θ(s, a) and the target r + γ max_a′ Q_θ⁻(s′, a′), and the agent acts ε-greedily on its outputs. Plain Q-learning with a neural network is unstable, because consecutive samples are correlated and the target moves with every update. DQN made it work with two ideas: experience replay (store transitions in a buffer and train on random mini-batches) and a target network θ⁻ (a periodically copied version of the network used to compute targets). With a convolutional network learning features from raw pixels, one architecture and hyperparameter set reached human-level play on many Atari games. Later improvements include Double DQN, dueling networks, prioritized replay, n-step returns and distributional heads (Rainbow).

Read full answer

Applies toPyTorch 2.x, Gymnasium 1.x; Stable-Baselines3 `DQN`

Components

Component Role
Q-network Q_θ(s, ·) Maps state → Q-value per action
ε-greedy policy Exploration, ε decayed from 1.0 to ~0.05
Replay buffer Breaks correlation, reuses data
Target network θ⁻ Stable targets; copied every C steps (or Polyak-averaged)
Loss Huber (smooth L1) on the TD error
Optimizer Adam / RMSProp, gradient clipping

Code Example: Compact DQN for CartPole (PyTorch)

import random
from collections import deque

import gymnasium as gym
import numpy as np
import torch
from torch import nn

env = gym.make("CartPole-v1")
torch.manual_seed(42)
random.seed(42)


def make_net():
    return nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU(), nn.Linear(128, 2))


q_net, target_net = make_net(), make_net()
target_net.load_state_dict(q_net.state_dict())
optimizer = torch.optim.Adam(q_net.parameters(), lr=1e-3)
buffer = deque(maxlen=50_000)
gamma, batch_size, target_every = 0.99, 64, 500
eps, step = 1.0, 0

s, _ = env.reset(seed=42)
for step in range(1, 30_001):
    if random.random() < eps:
        a = env.action_space.sample()
    else:
        with torch.no_grad():
            a = int(q_net(torch.as_tensor(s, dtype=torch.float32)).argmax())
    s2, r, terminated, truncated, _ = env.step(a)
    buffer.append((s, a, r, s2, terminated))
    s = s2
    if terminated or truncated:
        s, _ = env.reset()
    eps = max(0.05, 1.0 - step / 10_000)                    # linear decay

    if len(buffer) >= 1_000:
        batch = random.sample(buffer, batch_size)
        states, actions, rewards, next_states, dones = map(np.array, zip(*batch))
        states = torch.as_tensor(states, dtype=torch.float32)
        next_states = torch.as_tensor(next_states, dtype=torch.float32)
        actions = torch.as_tensor(actions).unsqueeze(1)
        rewards = torch.as_tensor(rewards, dtype=torch.float32)
        dones = torch.as_tensor(dones, dtype=torch.float32)

        q_sa = q_net(states).gather(1, actions).squeeze(1)
        with torch.no_grad():
            target = rewards + gamma * target_net(next_states).max(1).values * (1 - dones)
        loss = nn.functional.smooth_l1_loss(q_sa, target)
        optimizer.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(q_net.parameters(), 10.0)
        optimizer.step()

    if step % target_every == 0:
        target_net.load_state_dict(q_net.state_dict())      # hard target update

Only terminated zeroes the future value; a time-limit truncated episode still bootstraps. For real projects, stable_baselines3.DQN("MlpPolicy", env).learn(100_000) gives a tested implementation.

Why It Works and Where It Struggles

Strength Weakness
Learns from raw, high-dimensional inputs Discrete actions only
Off-policy, reuses data Overestimates Q (fixed by Double DQN)
One recipe across many games Sample-hungry (millions of frames)
Sensitive to hyperparameters

Common Pitfalls

  • No target network or replay buffer (training diverges or oscillates).

  • Starting training before the buffer has diverse data.

  • Treating truncation as termination.

  • Using DQN for continuous actions (use DDPG, TD3 or SAC).

Interview Tips

  • Describe the network’s input/output and the TD loss.

  • Explain why replay and target networks are needed.

  • Name later improvements (Double, Dueling, PER, Rainbow).

15

What is experience replay in DQN, and why is it important?

Experience replay stores the agent’s transitions (s, a, r, s′, done) in a large buffer and trains the Q-network on mini-batches sampled at random from that buffer instead of on the most recent transition. It matters for three reasons: it breaks temporal correlation (consecutive frames are nearly identical, and training on them in order violates the i.i.d. assumption of gradient descent, causing unstable updates and forgetting); it reuses data, so each costly interaction contributes to many updates (better sample efficiency); and it smooths the training distribution over many past policies instead of only the current one, reducing feedback loops where the policy’s own data reinforces its mistakes. It is possible because Q-learning is off-policy: old transitions from earlier policies are still valid training data. Prioritized experience replay samples transitions with large TD errors more often, with importance-sampling weights to correct the bias.

Read full answer

Applies toPython 3.10+, NumPy, PyTorch 2.x

Design Choices

Choice Typical value Effect
Capacity 10⁵-10⁶ transitions Larger = more diverse, but older (more off-policy) data
Warm-up before learning 1k-50k transitions Avoid overfitting to the first few episodes
Batch size 32-256 Gradient noise vs compute
Replay ratio Updates per environment step Higher = more reuse, risk of overfitting
Sampling Uniform, prioritized PER speeds learning on informative transitions
n-step returns Store n-step transitions Faster credit assignment

Code Example: Uniform Replay Buffer (NumPy)

import numpy as np


class ReplayBuffer:
    def __init__(self, capacity, obs_dim, seed=42):
        self.capacity, self.idx, self.size = capacity, 0, 0
        self.obs = np.zeros((capacity, obs_dim), dtype=np.float32)
        self.next_obs = np.zeros((capacity, obs_dim), dtype=np.float32)
        self.actions = np.zeros(capacity, dtype=np.int64)
        self.rewards = np.zeros(capacity, dtype=np.float32)
        self.dones = np.zeros(capacity, dtype=np.float32)
        self.rng = np.random.default_rng(seed)

    def add(self, s, a, r, s2, done):
        i = self.idx
        self.obs[i], self.actions[i], self.rewards[i], self.next_obs[i], self.dones[i] = s, a, r, s2, done
        self.idx = (self.idx + 1) % self.capacity          # circular: overwrite the oldest
        self.size = min(self.size + 1, self.capacity)

    def sample(self, batch_size):
        ids = self.rng.integers(0, self.size, size=batch_size)
        return self.obs[ids], self.actions[ids], self.rewards[ids], self.next_obs[ids], self.dones[ids]


buf = ReplayBuffer(capacity=5, obs_dim=4)
for t in range(7):                                         # add 7 transitions to a buffer of 5
    buf.add(np.full(4, t), t % 2, 1.0, np.full(4, t + 1), False)
print(buf.size, buf.idx)                                   # 5 2
print(sorted(buf.obs[:, 0].astype(int).tolist()))          # [2, 3, 4, 5, 6]  oldest two were overwritten
s, a, r, s2, d = buf.sample(3)
print(s.shape, a.shape)                                    # (3, 4) (3,)

Prioritized Experience Replay (Schaul et al., 2016)

Formula Meaning
p_i = |δ_i| + ε Priority from the TD error
P(i) = p_i^α / Σ_k p_k^α Sampling probability (α ≈ 0.6)
w_i = (N · P(i))^(−β) / max_j w_j Importance weight, β annealed to 1

High-error transitions are replayed more often, and the weights stop that preference from biasing the learned Q-values. A sum-tree keeps sampling at O(log N).

Common Pitfalls

  • Using replay with on-policy methods (PPO, A2C) without corrections.

  • A buffer so small that it only holds recent, correlated data.

  • Storing terminated and truncated together as one “done” flag.

  • Storing large image observations naively (store frames once and index stacks).

Interview Tips

  • Give the three reasons: decorrelation, data reuse, smoother distribution.

  • Tie it to off-policy learning.

  • Mention PER and its importance-sampling correction.

16

What are the main elements of the Proximal Policy Optimization (PPO) algorithm?

PPO is an on-policy actor-critic algorithm whose core idea is to improve the policy as much as possible from a batch of collected data without moving it too far from the policy that collected it, using a clipped surrogate objective: L^CLIP = E[min(r_t(θ) Â_t, clip(r_t(θ), 1 − ε, 1 + ε) Â_t)], where r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t) is the probability ratio and Â_t the advantage estimate. Its main elements are: rollouts collected with the current policy (often from parallel environments); Generalized Advantage Estimation (GAE) to compute advantages from a learned value function; the clipped policy loss; a value-function loss (often clipped too); an entropy bonus for exploration; and several epochs of mini-batch SGD over each batch. It gets much of TRPO’s stability with only first-order optimization, which is why it is a default for continuous control, games and RLHF fine-tuning of language models.

Read full answer

Applies toPyTorch 2.x, Gymnasium 1.x; Stable-Baselines3 `PPO`

Elements

Element Detail Typical value
Rollout n_steps × n_envs transitions with π_old 2048 × 1-8
GAE Â_t = Σ (γλ)ᵏ δ_(t+k), δ = r + γV(s′) − V(s) γ = 0.99, λ = 0.95
Ratio r_t = exp(log π_θ − log π_old) —
Clipped objective min(r Â, clip(r, 1 ± ε) Â) ε = 0.2
Value loss (V_θ(s) − R_t)², coefficient c₁ 0.5
Entropy bonus + c₂ H(π_θ) 0-0.01
Optimization K epochs, mini-batches, Adam, grad clipping 10 epochs, 64 batch, 3e-4
Advantage normalization Per mini-batch —
Optional KL early stopping, LR annealing target KL ≈ 0.01-0.02

How the Clip Works (by Hand, ε = 0.2)

Case r  r·Â clip®·Â min (objective)
Good action, already more likely 1.5 2 3.0 2.4 2.4 (no further push)
Bad action, already less likely 0.5 −1 −0.5 −0.8 −0.8 (no further push)
Good action, less likely 0.5 2 1.0 1.6 1.0 (still pushed up)

Code Example: The PPO Loss (PyTorch)

import torch


def ppo_loss(new_logp, old_logp, advantages, values, returns, entropy, clip_eps=0.2, vf_coef=0.5, ent_coef=0.01):
    ratio = torch.exp(new_logp - old_logp)
    surr1 = ratio * advantages
    surr2 = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * advantages
    policy_loss = -torch.min(surr1, surr2).mean()          # maximize the objective -> minimize its negative
    value_loss = (values - returns).pow(2).mean()
    return policy_loss + vf_coef * value_loss - ent_coef * entropy.mean()


ratio = torch.tensor([1.5, 0.5, 0.5])
adv = torch.tensor([2.0, -1.0, 2.0])
objective = torch.min(ratio * adv, torch.clamp(ratio, 0.8, 1.2) * adv)
print(objective.tolist())                                  # [2.4000000953674316, -0.800000011920929, 1.0]

loss = ppo_loss(torch.log(ratio), torch.zeros(3), adv,
                values=torch.zeros(3), returns=torch.zeros(3), entropy=torch.zeros(3))
print(round(loss.item(), 4))                               # -0.8667  = -(2.4 - 0.8 + 1.0) / 3

The printed float32 values are 2.4, −0.8 and 1.0 up to rounding.

Code Example: PPO with Stable-Baselines3

import gymnasium as gym
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy

env = gym.make("CartPole-v1")
model = PPO("MlpPolicy", env, n_steps=2048, batch_size=64, n_epochs=10,
            gamma=0.99, gae_lambda=0.95, clip_range=0.2, ent_coef=0.0, seed=42, verbose=0)
model.learn(total_timesteps=100_000)
mean_reward, std_reward = evaluate_policy(model, env, n_eval_episodes=10)
print(f"{mean_reward:.1f} +/- {std_reward:.1f}")

Common Pitfalls

  • Too many epochs or too large a learning rate: the policy drifts despite clipping (monitor approximate KL and clip fraction).

  • Not normalizing advantages or observations.

  • Mixing up the sign of the loss.

  • Treating truncated episodes as terminal in GAE.

Interview Tips

  • Write the clipped objective and explain the ratio.

  • List GAE, value loss, entropy bonus and multiple epochs.

  • Compare with TRPO (trust region via KL constraint) and mention PPO in RLHF.

17

How do actor-critic methods work in reinforcement learning?

Actor-critic methods combine a policy-gradient learner and a value-based learner: the actor is a policy π_θ(a|s) that chooses actions, and the critic is a value function (V_w(s) or Q_w(s, a)) that evaluates them; the critic is trained by temporal-difference learning, and the actor is updated in the direction suggested by the critic, typically ∇θ log π_θ(a|s) · Â, where the advantage  = r + γ V_w(s′) − V_w(s) (the TD error) says whether the action turned out better or worse than expected. Using the critic’s estimate instead of the full Monte Carlo return (as REINFORCE does) greatly reduces variance and allows online, per-step or per-rollout updates, at the cost of some bias. The family includes A2C/A3C, PPO and TRPO (on-policy, with V critics and GAE), and DDPG, TD3 and SAC (off-policy, with Q critics for continuous actions).

Read full answer

Applies toPyTorch 2.x, Gymnasium 1.x; Stable-Baselines3 (`A2C`, `PPO`, `SAC`)

Roles

Part Learns Update
Actor π_θ What to do θ ← θ + α_θ ∇θ log π_θ(a|s) · Â
Critic V_w How good states are w ← w + α_w δ ∇w V_w(s), δ = r + γV_w(s′) − V_w(s)
Advantage  Was the action better than average? δ (one-step), n-step, or GAE

Variants

Method Critic On/off-policy Actions
One-step actor-critic V On Discrete/continuous
A2C / A3C V, n-step On Both
PPO / TRPO V, GAE On Both
DDPG / TD3 Q (TD3: two) Off Continuous
SAC Two Q + entropy Off Continuous (and discrete variants)

Code Example: One-Step Actor-Critic on CartPole (PyTorch)

import gymnasium as gym
import torch
from torch import nn

env = gym.make("CartPole-v1")
torch.manual_seed(42)

actor = nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 2))
critic = nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 1))
actor_opt = torch.optim.Adam(actor.parameters(), lr=1e-3)
critic_opt = torch.optim.Adam(critic.parameters(), lr=5e-3)
gamma = 0.99
episode_lengths = []

for episode in range(500):
    s, _ = env.reset(seed=42 + episode)
    done, steps = False, 0
    while not done:
        s_t = torch.as_tensor(s, dtype=torch.float32)
        dist = torch.distributions.Categorical(logits=actor(s_t))
        a = dist.sample()
        s2, r, terminated, truncated, _ = env.step(int(a))

        v = critic(s_t).squeeze()
        with torch.no_grad():
            v_next = critic(torch.as_tensor(s2, dtype=torch.float32)).squeeze() * (not terminated)
            td_target = r + gamma * v_next
        delta = td_target - v                               # TD error = advantage estimate

        critic_loss = delta.pow(2)
        critic_opt.zero_grad()
        critic_loss.backward()
        critic_opt.step()

        actor_loss = -dist.log_prob(a) * delta.detach()     # push up actions that beat expectations
        actor_opt.zero_grad()
        actor_loss.backward()
        actor_opt.step()

        s, done, steps = s2, terminated or truncated, steps + 1
    episode_lengths.append(steps)

print(sum(episode_lengths[-20:]) / 20)                      # average length of the last 20 episodes

Episode lengths usually rise well above the random-policy level (about 20 steps) as training goes on, though one-step actor-critic can be noisy; the exact average depends on the run. For robust training use PPO("MlpPolicy", env) or A2C("MlpPolicy", env) from Stable-Baselines3.

Actor-Critic vs Pure Methods

Method Strength Weakness
Value-based (DQN) Sample-efficient, off-policy Discrete actions; argmax over actions
Policy gradient (REINFORCE) Any action space, stochastic policies High variance
Actor-critic Lower variance, continuous actions, online Two networks to tune; critic bias

Common Pitfalls

  • Backpropagating the actor loss into the critic (detach the advantage).

  • Bootstrapping from terminal states.

  • Critic learning rate too low, so the actor follows a poor critic.

  • No entropy regularization, leading to premature convergence.

Interview Tips

  • Name the actor and critic roles and write both updates.

  • Explain variance reduction via the advantage/TD error.

  • Place A2C, PPO, DDPG, TD3 and SAC in the family.

18

What improvements does Double DQN make over standard DQN?

Double DQN (van Hasselt, Guez and Silver, 2016) fixes DQN’s overestimation of Q-values by separating action selection from action evaluation in the target: standard DQN uses y = r + γ max_a′ Q_θ⁻(s′, a′), where the same target network both picks the best next action and scores it; Double DQN uses y = r + γ Q_θ⁻(s′, argmax_a′ Q_θ(s′, a′)), where the online network picks the action and the target network evaluates it. The max of noisy estimates is biased upward (it tends to pick actions whose value happens to be overestimated), and with bootstrapping that bias compounds through the value function, leading to over-optimistic values and worse policies. Decoupling the two roles largely removes the bias, at almost no cost — one extra forward pass, and the target network DQN already has — and it improved scores on many Atari games. It is part of Rainbow DQN, and the same idea appears as clipped double Q-learning in TD3 and SAC.

Read full answer

Applies toPyTorch 2.x, NumPy

Why max Overestimates

If all actions have true value 0 and the estimates have independent noise, E[max_a Q̂(a)] > 0 even though max_a Q(a) = 0. For 10 actions with N(0, 1) noise, the expected maximum is about 1.54.

import numpy as np

rng = np.random.default_rng(42)
noisy_q = rng.normal(0.0, 1.0, size=(100_000, 10))   # true value of every action is 0
print(round(noisy_q.max(axis=1).mean(), 2))         # about 1.54, an upward bias from noise alone

Target Computation by Hand

Next-state Q-values: online network Q_θ(s′) = [3, 5, 4], target network Q_θ⁻(s′) = [6, 2, 4]; r = 1, γ = 0.9.

Method Selects Evaluates Target
DQN argmax Q_θ⁻ = action 0 Q_θ⁻ = 6 1 + 0.9 × 6 = 6.4
Double DQN argmax Q_θ = action 1 Q_θ⁻ = 2 1 + 0.9 × 2 = 2.8

Code Example: DQN vs Double DQN Targets (PyTorch)

import torch

online_next = torch.tensor([[3.0, 5.0, 4.0]])
target_next = torch.tensor([[6.0, 2.0, 4.0]])
rewards, dones, gamma = torch.tensor([1.0]), torch.tensor([0.0]), 0.9

dqn_target = rewards + gamma * target_next.max(dim=1).values * (1 - dones)

best_actions = online_next.argmax(dim=1, keepdim=True)              # select with the online network
double_target = rewards + gamma * target_next.gather(1, best_actions).squeeze(1) * (1 - dones)

print([round(v, 4) for v in dqn_target.tolist()], [round(v, 4) for v in double_target.tolist()])   # [6.4] [2.8]

In a training loop the only change from DQN is these two lines; compute both inside torch.no_grad().

Related Improvements

Method Fixes
Double DQN Overestimation bias
Dueling DQN Separate V(s) and advantage streams: better value estimates
Prioritized replay Replays informative transitions more
n-step returns Faster reward propagation
Distributional (C51, QR-DQN) Learns the return distribution
Noisy nets Learned exploration
Rainbow All of the above combined
Clipped double Q (TD3, SAC) min of two critics for continuous control

Common Pitfalls

  • Using the target network for both selection and evaluation and calling it Double DQN.

  • Forgetting torch.no_grad() for the online network’s argmax pass (harmless for gradients through argmax, but wasteful).

  • Expecting Double DQN to fix every instability; it targets overestimation only.

Interview Tips

  • Explain maximization bias with the noisy-max example.

  • Write both targets and show they differ only in who selects the action.

  • Mention TD3’s clipped double Q as the continuous-control version.

19

What role do target networks play in stabilizing deep RL training?

A target network is a separate, slowly changing copy of the value network used only to compute the bootstrapped targets (for example y = r + γ max_a′ Q_θ⁻(s′, a′) in DQN), while the main network Q_θ is trained toward those targets. Without it, the same parameters produce both the prediction and the target, so every gradient step also moves the target — like chasing a moving goalpost — which leads to oscillation, feedback loops and divergence, especially when combined with function approximation and off-policy bootstrapping (the “deadly triad”). The target network holds the targets fixed for a while, turning learning into a sequence of more stable regression problems. It is updated either by a hard copy every C steps (DQN: every 10,000 steps on Atari) or by soft Polyak averaging every step, θ⁻ ← τ θ + (1 − τ) θ⁻ with small τ (DDPG, TD3 and SAC use τ = 0.005).

Read full answer

Applies toPyTorch 2.x

Hard vs Soft Updates

Update Rule Used in Trade-off
Hard θ⁻ ← θ every C steps DQN, Double DQN Simple; targets jump every C steps
Soft (Polyak) θ⁻ ← τθ + (1 − τ)θ⁻ every step DDPG, TD3, SAC Smooth; τ controls lag

A soft update with τ has an effective lag of about 1/τ steps (200 for τ = 0.005).

Code Example: Hard and Soft Updates (PyTorch)

import copy

import torch
from torch import nn

torch.manual_seed(42)
q_net = nn.Linear(4, 2)
target_net = copy.deepcopy(q_net)
for p in target_net.parameters():
    p.requires_grad_(False)                       # the target network is never trained directly


def hard_update(target, online):
    target.load_state_dict(online.state_dict())


@torch.no_grad()
def soft_update(target, online, tau):
    for t_param, o_param in zip(target.parameters(), online.parameters()):
        t_param.mul_(1 - tau).add_(tau * o_param)


# Soft update by hand: target 0, online 1, tau 0.005
t, o = nn.Linear(1, 1, bias=False), nn.Linear(1, 1, bias=False)
with torch.no_grad():
    t.weight.fill_(0.0)
    o.weight.fill_(1.0)
soft_update(t, o, 0.005)
print(round(t.weight.item(), 6))                  # 0.005
soft_update(t, o, 0.005)
print(round(t.weight.item(), 6))                  # 0.009975  = 0.005 + 0.005 * (1 - 0.005)

# Inside a DQN step, targets come from target_net with no gradient
s2, r, done = torch.randn(8, 4), torch.ones(8), torch.zeros(8)
with torch.no_grad():
    y = r + 0.99 * target_net(s2).max(dim=1).values * (1 - done)
print(y.requires_grad)                            # False

Why It Helps

Without target network With target network
Update raises Q(s, a) and similar Q(s′, ·), so the target rises too Target fixed while Q(s, a) moves toward it
Overestimation feeds on itself Feedback loop slowed
Oscillating or diverging losses Closer to supervised regression

Common Pitfalls

  • Computing targets with gradients enabled (gradients flow into the target).

  • Updating too often (no stabilization) or too rarely (slow learning on stale targets).

  • Forgetting to copy the weights at the start, so the target network is random.

  • Using a target network for the actor and forgetting the critic’s (DDPG/TD3 use both).

Interview Tips

  • Explain the moving-target problem and the deadly triad.

  • Give hard and Polyak updates with typical C and τ.

  • Mention that Double DQN reuses the target network to decouple selection from evaluation.

20

How does the Asynchronous Advantage Actor-Critic (A3C) algorithm work?

A3C (Mnih et al., 2016) trains an actor-critic network with many parallel workers: each worker has its own copy of the environment and of the network, collects a short rollout (for example 5-20 steps), computes n-step returns and advantages, calculates gradients of the actor loss (−log π(a|s) · advantage), the critic loss (squared error of V(s)) and an entropy bonus, and applies them asynchronously to a shared global network, then copies the global weights back and continues. Running different workers in different parts of the environment at the same time decorrelates the data, which takes the place of a replay buffer, and it ran efficiently on multi-core CPUs. Its synchronous version, A2C, waits for all workers and makes one batched update; it performs as well or better and uses GPUs more efficiently, so A2C (and later PPO) mostly replaced A3C in practice.

Read full answer

Applies toPyTorch 2.x (`torch.multiprocessing`), Gymnasium 1.x; Stable-Baselines3 `A2C`

Algorithm (One Worker)

Step Action
1 Copy global parameters θ, θ_v into the local network
2 Run the policy for t_max steps (or until episode end)
3 Bootstrap R = V(s_last) (0 if terminal)
4 Backward over steps: R ← r_i + γR; advantage A_i = R − V(s_i)
5 Loss = −Σ log π(a_i|s_i) A_i + c_v Σ (R_i − V(s_i))² − β Σ H(π(·|s_i))
6 Compute gradients locally; apply them to the global network (no lock, Hogwild style)
7 Repeat

n-Step Returns by Hand

Rewards [1, 1, 1], bootstrap V(s₃) = 5, γ = 0.9, critic values V(s₀..s₂) = [6, 5, 5]:

R₂ = 1 + 0.9 × 5 = 5.5; R₁ = 1 + 0.9 × 5.5 = 5.95; R₀ = 1 + 0.9 × 5.95 = 6.355; advantages = [0.355, 0.95, 0.5].

Code Example: The Worker’s Loss Computation (PyTorch)

import numpy as np
import torch
from torch import nn


def n_step_returns(rewards, bootstrap, gamma):
    R, out = bootstrap, []
    for r in reversed(rewards):
        R = r + gamma * R
        out.append(R)
    return out[::-1]


returns = n_step_returns([1.0, 1.0, 1.0], bootstrap=5.0, gamma=0.9)
values = np.array([6.0, 5.0, 5.0])
print(np.round(returns, 3).tolist())                       # [6.355, 5.95, 5.5]
print(np.round(np.array(returns) - values, 3).tolist())    # [0.355, 0.95, 0.5]


class ActorCritic(nn.Module):
    def __init__(self, obs_dim, n_actions):
        super().__init__()
        self.body = nn.Sequential(nn.Linear(obs_dim, 128), nn.ReLU())
        self.pi = nn.Linear(128, n_actions)
        self.v = nn.Linear(128, 1)

    def forward(self, x):
        h = self.body(x)
        return self.pi(h), self.v(h).squeeze(-1)


def worker_loss(model, states, actions, returns, value_coef=0.5, entropy_coef=0.01):
    logits, values = model(states)
    dist = torch.distributions.Categorical(logits=logits)
    advantages = returns - values
    actor_loss = -(dist.log_prob(actions) * advantages.detach()).mean()   # no gradient into the critic here
    critic_loss = advantages.pow(2).mean()
    return actor_loss + value_coef * critic_loss - entropy_coef * dist.entropy().mean()


torch.manual_seed(42)
global_model = ActorCritic(4, 2)
global_model.share_memory()                                # A3C: workers update shared parameters
local_model = ActorCritic(4, 2)
local_model.load_state_dict(global_model.state_dict())

loss = worker_loss(local_model, torch.randn(3, 4), torch.tensor([0, 1, 0]), torch.tensor(returns, dtype=torch.float32))
loss.backward()
for g_param, l_param in zip(global_model.parameters(), local_model.parameters()):
    g_param.grad = l_param.grad                            # hand local gradients to the global model
print(all(p.grad is not None for p in global_model.parameters()))   # True

In full A3C, several processes started with torch.multiprocessing each run this loop and call a shared optimizer’s step() on the global model. For practical use, stable_baselines3.A2C("MlpPolicy", env) with vectorized environments gives the synchronous version.

A3C vs A2C vs PPO

Aspect A3C A2C PPO
Updates Asynchronous, per worker Synchronous, batched Synchronous, several epochs per batch
Hardware Many CPU cores GPU-friendly GPU-friendly
Stale gradients Yes No No
Data reuse One update per rollout One update per rollout Multiple epochs (clipped)
Today Mostly historical Simple baseline Default choice

Common Pitfalls

  • Letting actor-loss gradients flow into the critic through the advantage (detach it).

  • Forgetting the entropy bonus, so the policy collapses early.

  • Bootstrapping from V at terminal states.

  • Assuming asynchrony is what made A3C work; parallel data collection was the main benefit.

Interview Tips

  • Describe workers, n-step advantages, the three loss terms and asynchronous updates.

  • Explain why parallel workers replace experience replay.

  • Mention A2C and PPO as the successors.

21

What is reward shaping, and how does it affect an RL agent?

Reward shaping adds extra reward signals to the environment’s original reward to guide the agent toward useful behaviour sooner, typically when the true reward is sparse (for example, +1 only on reaching a goal) and random exploration would almost never find it. Done well, it speeds up learning dramatically by giving intermediate feedback (“getting closer is good”). Done badly, it changes the optimal policy: the agent learns to collect the shaping bonus instead of solving the task (reward hacking), such as circling a checkpoint forever to farm points. The safe form is potential-based reward shaping (Ng, Harada and Russell, 1999): add F(s, s′) = γΦ(s′) − Φ(s) for any potential function Φ over states; this provably leaves the optimal policy unchanged, because the shaping terms telescope along any trajectory and only shift values by Φ(s₀).

Read full answer

Applies toPython 3.10+, NumPy, Gymnasium 1.x (`gym.Wrapper`)

Kinds of Shaping

Type Example Risk
Potential-based Φ(s) = −distance to goal; F = γΦ(s′) − Φ(s) Safe: optimal policy preserved
Ad-hoc bonus +0.1 for moving right Can create loops and exploits
Penalties −0.01 per step, −1 for collisions Too large: agent ends episodes early on purpose
Intrinsic motivation Curiosity/novelty bonus Agent chases noise (“noisy TV”)
Curriculum of rewards Dense early, sparse later Needs a schedule
Learned rewards Inverse RL, RLHF reward models Reward model can be exploited

Why Potential-Based Shaping Is Safe

Over an episode, Σ_t γᵗ [γΦ(s_(t+1)) − Φ(s_t)] = γᵀΦ(s_T) − Φ(s₀), which telescopes. With Φ = 0 at terminal states, the shaping changes every return from s₀ by the same constant −Φ(s₀), so the ranking of policies, and therefore the optimal policy, does not change.

Code Example: Potential-Based Shaping Wrapper (Gymnasium)

import gymnasium as gym


class DistanceShaping(gym.Wrapper):
    """Adds gamma * phi(s') - phi(s), with phi = -Manhattan distance to the goal on FrozenLake."""

    def __init__(self, env, gamma=0.99, size=4):
        super().__init__(env)
        self.gamma, self.size = gamma, size
        self.goal = (size - 1, size - 1)

    def phi(self, s, terminal=False):
        if terminal:
            return 0.0                                     # potential is 0 at terminal states
        row, col = divmod(int(s), self.size)
        return -float(abs(row - self.goal[0]) + abs(col - self.goal[1]))

    def reset(self, **kwargs):
        obs, info = self.env.reset(**kwargs)
        self.last_obs = obs
        return obs, info

    def step(self, action):
        obs, reward, terminated, truncated, info = self.env.step(action)
        shaping = self.gamma * self.phi(obs, terminated) - self.phi(self.last_obs)
        self.last_obs = obs
        info["shaping"] = shaping
        return obs, reward + shaping, terminated, truncated, info


env = DistanceShaping(gym.make("FrozenLake-v1", is_slippery=False))
obs, _ = env.reset(seed=42)
obs, r, terminated, truncated, info = env.step(2)          # move right: one step closer to the goal
print(obs, round(info["shaping"], 2))                       # 1 1.05

Moving from cell 0 (distance 6) to cell 1 (distance 5) gives γΦ(s′) − Φ(s) = 0.99 × (−5) + 6 = 1.05. Moving away from the goal gives a negative bonus (0.99 × (−6) + 5 = −0.94). Over any loop the discounted bonuses telescope to a small constant, so walking back and forth never beats reaching the goal: the optimal policy is unchanged.

Common Pitfalls

  • Ad-hoc bonuses that can be farmed (agent loops near a checkpoint).

  • Step penalties so large that ending the episode early (falling in a hole) beats trying.

  • Forgetting Φ = 0 at terminal states, which breaks the policy-invariance guarantee.

  • Evaluating the agent on the shaped reward instead of the true task reward.

Interview Tips

  • Define shaping and why sparse rewards need it.

  • State the potential-based formula and the policy-invariance result.

  • Give a concrete reward-hacking example.

22

What are policy gradients, and how are they used to learn policies?

Policy gradient methods learn a parameterized policy π_θ(a|s) directly by gradient ascent on the expected return J(θ) = E_(τ∼π_θ)[G(τ)], using the policy gradient theorem: ∇θ J(θ) = E[Σ_t ∇θ log π_θ(a_t|s_t) · Ψ_t], where Ψ_t is a measure of how good the action was — the return G_t (REINFORCE), the return minus a baseline G_t − b(s_t), or an advantage estimate Â_t (actor-critic). Intuitively, it increases the log-probability of actions that led to better-than-expected outcomes and decreases it for worse ones. No model of the environment is needed, and the “log-derivative trick” means the gradient only needs samples and the policy’s own log-probabilities. Policy gradients handle continuous and high-dimensional actions and stochastic policies naturally, and they are the basis of REINFORCE, A2C/A3C, TRPO, PPO and the RL stage of RLHF. Their main weakness is high variance, reduced with baselines, advantages, GAE and large batches.

Read full answer

Applies toPyTorch 2.x, Gymnasium 1.x

The Log-Derivative Trick

∇θ E_(a∼π_θ)[f(a)] = E[f(a) ∇θ log π_θ(a)], because ∇θ π_θ = π_θ ∇θ log π_θ. For trajectories, the environment’s transition probabilities do not depend on θ, so they drop out of ∇θ log p(τ), leaving Σ_t ∇θ log π_θ(a_t|s_t).

Choices for Ψ_t

Ψ_t Method Variance
G(τ), total return Basic REINFORCE Highest
G_t, reward-to-go REINFORCE (causality) High
G_t − V(s_t) REINFORCE with baseline Lower
Q(s_t, a_t) Actor-critic with Q critic Lower, biased
Â_t (TD error, n-step, GAE) A2C, PPO Low, tunable bias

Code Example: Policy Gradient Step on a Bandit (PyTorch)

import torch

torch.manual_seed(42)
logits = torch.zeros(3, requires_grad=True)          # policy over 3 actions, starts uniform
true_rewards = torch.tensor([1.0, 0.0, 2.0])
optimizer = torch.optim.SGD([logits], lr=0.1)

for step in range(300):
    dist = torch.distributions.Categorical(logits=logits)
    actions = dist.sample((64,))
    rewards = true_rewards[actions] + 0.1 * torch.randn(64)
    baseline = rewards.mean()                         # reduces variance, keeps the gradient unbiased in expectation
    loss = -(dist.log_prob(actions) * (rewards - baseline)).mean()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print(torch.softmax(logits, 0).argmax().item())       # 2: the policy concentrates on the best action

The loss is the negative of the policy-gradient objective, so minimizing it with SGD performs gradient ascent on expected reward. The same loss, with states as network inputs and returns or advantages as weights, gives REINFORCE and actor-critic methods.

Advantages and Disadvantages

Advantages Disadvantages
Continuous and large action spaces High-variance gradients
Stochastic policies (exploration, POMDPs, games) Sample-inefficient (on-policy)
Smooth policy changes, convergence to local optimum Local optima, sensitive step size
Works with any differentiable policy Needs many episodes

Common Pitfalls

  • Wrong sign (descending on reward instead of ascending).

  • No baseline or advantage normalization, causing very noisy training.

  • Steps too large, collapsing the policy (motivates TRPO/PPO).

  • Reusing old trajectories without importance correction.

Interview Tips

  • Write the policy gradient theorem and explain the log-derivative trick.

  • Explain variance reduction with baselines and advantages.

  • Connect to REINFORCE, actor-critic, PPO and RLHF.

23

What are common challenges in designing reward functions for RL?

Designing reward functions is hard because the agent optimizes exactly what you reward, not what you meant: the main challenges are sparse rewards (the signal is too rare to learn from), reward hacking or specification gaming (the agent finds loopholes that score well without doing the task), delayed rewards that make credit assignment difficult, balancing several objectives with weights, unintended side effects that the reward does not penalize, reward scale that destabilizes learning, and rewards that are hard to measure at all (helpfulness, comfort, safety). Classic examples: a boat-racing agent that circled to collect respawning targets instead of finishing the race (OpenAI’s CoastRunners example), and a simulated robot exploiting physics bugs. Mitigations include potential-based shaping, careful normalization, constraints instead of penalties (constrained RL), learning rewards from demonstrations or human preferences (inverse RL, RLHF), and thorough evaluation with metrics separate from the training reward.

Read full answer

Applies toRL problem design in any framework

Challenges and Mitigations

Challenge Example Mitigation
Sparse reward +1 only when a robot assembles a part Potential-based shaping, curriculum, demonstrations, HER, intrinsic motivation
Reward hacking Agent loops for points instead of finishing Reward the outcome, red-team the reward, separate evaluation metrics
Delayed reward Chess win after 60 moves Value functions, n-step/TD(λ), discount choice
Multiple objectives Speed vs energy vs comfort Explicit weights tuned on held-out evaluation; constrained RL; Pareto analysis
Side effects Robot knocks over a vase en route Impact penalties, constraints, human oversight
Reward scale Rewards of ±1000 destabilize value learning Normalize/clip rewards, PopArt
Hard-to-measure goals “Helpful” chatbot answers Learned reward models from human preferences (RLHF), with KL penalty
Proxy metrics Clicks instead of satisfaction Combine proxies; long-term metrics; guardrails
Non-stationarity Reward based on users who change Monitoring and periodic retraining

Code Example: A Multi-Objective Reward with Logged Components

from dataclasses import dataclass


@dataclass
class RewardWeights:
    progress: float = 1.0
    energy: float = 0.05
    collision: float = 10.0


def reward_fn(progress_m, energy_j, collided, w=RewardWeights()):
    components = {
        "progress": w.progress * progress_m,
        "energy": -w.energy * energy_j,
        "collision": -w.collision * float(collided),
    }
    return sum(components.values()), components         # log components, not just the total


total, parts = reward_fn(progress_m=0.5, energy_j=2.0, collided=False)
print(round(total, 3), parts)        # 0.4 {'progress': 0.5, 'energy': -0.1, 'collision': -0.0}

total, parts = reward_fn(progress_m=0.5, energy_j=2.0, collided=True)
print(round(total, 3))               # -9.6

Logging each component makes it possible to spot hacking (progress rising while collisions also rise) and to see which term dominates learning.

A Checklist for Reward Design

Question Why
Would a policy that maximizes this reward do what I want? Catch loopholes early
Can the agent get reward without the outcome? Specification gaming
Are penalties so large that quitting is optimal? Early-termination exploits
Is the scale reasonable (≈ −10 to 10 per step)? Stable value learning
What metric will I use to evaluate, separate from the reward? Detect hacking

Common Pitfalls

  • Rewarding actions (“press the button”) instead of outcomes.

  • Tuning reward weights on the same runs used to report results.

  • Assuming a learned reward model is correct far from its training data (RLHF needs a KL penalty to stay close).

Interview Tips

  • Name sparse rewards, hacking, delayed credit and multi-objective trade-offs.

  • Give a concrete specification-gaming example.

  • Offer mitigations, including learning rewards from humans.

24

Describe TRPO and how it differs from other policy gradient methods.

Trust Region Policy Optimization (Schulman et al., 2015) is a policy gradient method that limits how much the policy can change in each update: it maximizes the surrogate objective E[(π_θ(a|s) / π_θ_old(a|s)) Â(s, a)] subject to a constraint on the average KL divergence between the old and new policies, E_s[KL(π_θ_old(·|s) ‖ π_θ(·|s))] ≤ δ (for example δ = 0.01). Vanilla policy gradient methods take steps of a fixed size in parameter space, but a small parameter change can cause a huge change in behaviour, collapsing performance and ruining the data future updates depend on. TRPO measures step size in policy space instead, which gives approximate monotonic improvement. It solves the constrained problem with a second-order method: approximate the KL with the Fisher information matrix, compute the natural gradient direction with conjugate gradient (using Fisher-vector products, never forming the matrix), then do a backtracking line search to make sure the constraint holds and the surrogate improves. PPO later replaced the hard constraint with a clipped objective that is simpler and usually as good.

Read full answer

Applies toPyTorch 2.x; `sb3_contrib.TRPO` (sb3-contrib package)

TRPO vs Other Policy Gradient Methods

Aspect Vanilla PG / REINFORCE TRPO PPO
Step control Learning rate only Hard KL constraint (trust region) Clipped ratio (soft)
Optimizer First-order (SGD/Adam) Natural gradient + conjugate gradient + line search First-order, several epochs
Data reuse One gradient step per batch One update per batch Several epochs per batch
Guarantees None on improvement Approximate monotonic improvement Empirical
Complexity Low High Low
Works with shared actor-critic nets, dropout Yes Awkward Yes

The Update

Step What happens
1 Collect trajectories with π_old; estimate  with GAE
2 Compute the policy gradient g of the surrogate
3 Solve F x = g with conjugate gradient (F = Fisher matrix, via Fisher-vector products)
4 Step size β = √(2δ / (xᵀ F x)); proposed θ_new = θ_old + β x
5 Backtracking line search: shrink the step until KL ≤ δ and the surrogate improves
6 Fit the value function separately (regression on returns)

Code Example: Conjugate Gradient and the Step Size (NumPy)

import numpy as np


def conjugate_gradient(Avp, b, iters=10, tol=1e-10):
    x = np.zeros_like(b)
    r, p = b.copy(), b.copy()
    rr = r @ r
    for _ in range(iters):
        Ap = Avp(p)
        alpha = rr / (p @ Ap)
        x += alpha * p
        r -= alpha * Ap
        rr_new = r @ r
        if rr_new < tol:
            break
        p = r + (rr_new / rr) * p
        rr = rr_new
    return x


F = np.array([[4.0, 1.0],
              [1.0, 3.0]])                     # stand-in Fisher matrix (symmetric positive definite)
g = np.array([1.0, 2.0])                       # policy gradient
x = conjugate_gradient(lambda v: F @ v, g)     # natural gradient direction F^-1 g
print(np.round(x, 4))                          # [0.0909 0.6364]

delta = 0.01
beta = np.sqrt(2 * delta / (x @ F @ x))        # largest step with 0.5 * step^T F step = delta
step = beta * x
print(round(0.5 * step @ F @ step, 6))         # 0.01  (quadratic KL estimate equals delta)

By hand, F⁻¹g solves 4x₁ + x₂ = 1 and x₁ + 3x₂ = 2: x₁ = 1/11 ≈ 0.0909, x₂ = 7/11 ≈ 0.6364. In real TRPO, F @ v is replaced by a Fisher-vector product computed with two automatic-differentiation passes through the KL.

Common Pitfalls

  • Forming the Fisher matrix explicitly (impossible for large networks).

  • Skipping the line search; the quadratic KL approximation can be wrong.

  • Choosing δ too large (unstable) or too small (slow learning).

  • Expecting TRPO to beat PPO by default; in practice PPO is usually preferred for its simplicity.

Interview Tips

  • State the constrained objective with the KL trust region.

  • Explain natural gradient, conjugate gradient and line search.

  • Contrast with vanilla PG (step size in parameter space) and PPO (clipping).

25

How do you scale reinforcement learning to high-dimensional state spaces?

Tabular methods break down when states are high-dimensional (images, many sensors, text) because almost every state is new; RL scales by replacing tables with function approximation — neural networks that generalize across similar states — and by learning compact representations of the observations. In practice that means: convolutional encoders for images (DQN on Atari pixels), MLPs for vector states, transformers or recurrent networks for sequences and partial observability; preprocessing such as frame stacking, downsampling, grayscale conversion and observation normalization; representation learning with auxiliary or self-supervised losses (CURL, SPR, contrastive or reconstruction objectives) and data augmentation (DrQ, RAD); world models that learn a low-dimensional latent state and plan or train in it (Dreamer, MuZero); pretrained encoders from vision or language models; and engineering at scale — vectorized environments, distributed actors and learners (IMPALA, Ape-X), and large replay buffers.

Read full answer

Applies toPyTorch 2.x, Gymnasium 1.x (wrappers), Stable-Baselines3 (`CnnPolicy`)

Techniques

Technique What it does Examples
Deep function approximation Generalizes values/policies across states DQN, PPO, SAC
CNN encoders Spatial features from pixels Nature DQN, IMPALA ResNet
Preprocessing Fewer input dimensions, Markov state Grayscale, 84×84 resize, 4-frame stack
Normalization Stable input scales Running mean/std of observations
Data augmentation Better sample efficiency from pixels DrQ, RAD (random shifts)
Self-supervised auxiliary losses Richer representations CURL, SPR, UNREAL
World models / latent planning Learn and plan in compact latent space Dreamer, MuZero, PlaNet
Pretrained encoders Reuse vision/language features R3M, VC-1, CLIP features
Memory Handle partial observability LSTM/GRU policies, transformers
Distributed training More data per hour IMPALA, Ape-X, SEED RL

Code Example: Pixel Encoder and Preprocessing (PyTorch + Gymnasium)

import torch
from torch import nn


class NatureCNN(nn.Module):
    """Encoder from the DQN paper: 4 stacked 84x84 grayscale frames -> features."""

    def __init__(self, in_frames=4, n_actions=6):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(in_frames, 32, kernel_size=8, stride=4), nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2), nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, stride=1), nn.ReLU(),
            nn.Flatten(),
        )
        self.head = nn.Sequential(nn.Linear(64 * 7 * 7, 512), nn.ReLU(), nn.Linear(512, n_actions))

    def forward(self, frames_uint8):
        x = frames_uint8.float() / 255.0                  # scale pixels to [0, 1]
        return self.head(self.features(x))


torch.manual_seed(42)
net = NatureCNN()
batch = torch.randint(0, 256, (2, 4, 84, 84), dtype=torch.uint8)
print(net(batch).shape)                                   # torch.Size([2, 6])
print(84 * 84 * 4, 64 * 7 * 7)                            # 28224 3136  raw inputs vs conv features

# Typical Atari preprocessing with Gymnasium wrappers (needs ale-py installed):
# import ale_py, gymnasium as gym
# gym.register_envs(ale_py)
# env = gym.make("ALE/Breakout-v5", frameskip=1)
# env = gym.wrappers.AtariPreprocessing(env, frame_skip=4, screen_size=84, grayscale_obs=True)
# env = gym.wrappers.FrameStackObservation(env, stack_size=4)

Feature sizes by hand: 84 → (84 − 8)/4 + 1 = 20 → (20 − 4)/2 + 1 = 9 → 9 − 3 + 1 = 7, so the encoder outputs 64 × 7 × 7 = 3,136 features. With Stable-Baselines3, PPO("CnnPolicy", env) uses this encoder by default.

Common Pitfalls

  • Feeding single frames when velocity matters (stack frames or add memory).

  • Unnormalized vector observations of very different scales.

  • Expecting tabular-style convergence guarantees; deep RL needs stabilizers (target networks, clipping, replay).

  • Overfitting to visual details of training levels; use augmentation and many training levels.

Interview Tips

  • Lead with function approximation and representation learning.

  • Mention preprocessing, augmentation, world models and pretrained encoders.

  • Add the systems angle: vectorized and distributed training.

26

What strategies transfer knowledge across tasks in reinforcement learning?

Transfer in reinforcement learning reuses something learned on source tasks — representations, policies, value functions, models, or data — to learn a target task faster or better. Common strategies are: fine-tuning a pretrained policy or encoder on the new task; feature/representation transfer (freeze or partially freeze a pretrained visual encoder); policy distillation and behaviour cloning from source-task policies or demonstrations as initialization; reward and value transfer such as successor features, which separate environment dynamics from the reward so a new reward can reuse the learned dynamics; multi-task and goal-conditioned learning (one policy conditioned on a task or goal, as in universal value functions and hindsight experience replay); meta-RL (learn to adapt quickly); sim-to-real transfer with domain randomization; and model transfer, reusing a learned world model with a new reward. Watch for negative transfer, where source knowledge hurts, and measure jump-start, asymptotic performance and time-to-threshold against training from scratch.

Read full answer

Applies toPyTorch 2.x, Stable-Baselines3

Strategies

Strategy Transfers Good when
Fine-tuning Whole policy/value net Similar tasks, same action space
Frozen/partially frozen encoder Representation Same observation type, different goals
Policy distillation Teacher policies → student Combining several experts
Behaviour cloning + RL Demonstrations Expert data available
Successor features Dynamics-related features Same dynamics, new rewards
Goal-conditioned / multi-task Shared policy across goals Family of related tasks
Meta-RL Learning-to-adapt Many tasks from a distribution
Domain randomization Robust policy Sim-to-real
World-model reuse Dynamics model New reward, same environment

Code Example: Fine-Tuning with a Frozen Encoder (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)


class Policy(nn.Module):
    def __init__(self, obs_dim, n_actions):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(obs_dim, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU())
        self.head = nn.Linear(128, n_actions)

    def forward(self, x):
        return self.head(self.encoder(x))


source = Policy(obs_dim=8, n_actions=4)            # imagine this was trained on the source task
target = Policy(obs_dim=8, n_actions=6)            # new task: same observations, different actions

target.encoder.load_state_dict(source.encoder.state_dict())   # transfer the representation
for p in target.encoder.parameters():
    p.requires_grad_(False)                        # stage 1: train only the new head

trainable = sum(p.numel() for p in target.parameters() if p.requires_grad)
print(trainable)                                   # 774 = 128 * 6 + 6
optimizer = torch.optim.Adam([p for p in target.parameters() if p.requires_grad], lr=3e-4)

# Stage 2 (later): unfreeze the encoder with a smaller learning rate
for p in target.encoder.parameters():
    p.requires_grad_(True)
optimizer = torch.optim.Adam([
    {"params": target.encoder.parameters(), "lr": 3e-5},
    {"params": target.head.parameters(), "lr": 3e-4},
])

With Stable-Baselines3, the simplest transfer is model = PPO.load("source.zip", env=new_env) followed by model.learn(...), which fine-tunes all weights when observation and action spaces match.

Measuring Transfer

Metric Question
Jump-start Is initial performance better than from scratch?
Time to threshold How much sooner is a target score reached?
Asymptotic performance Is the final score better (or worse)?
Total reward / area under curve Overall benefit over training

Common Pitfalls

  • Negative transfer from dissimilar tasks; always compare with training from scratch.

  • Catastrophic forgetting of the source task when that matters (use distillation or replay of source data).

  • Value functions transferred with a different reward scale.

  • Mismatched observation normalization statistics between source and target.

Interview Tips

  • List several strategies with what each transfers.

  • Mention successor features or goal-conditioned policies for reward changes, and domain randomization for sim-to-real.

  • Explain how to measure transfer and detect negative transfer.

27

How do you make RL agents generalize to unseen environments?

RL agents often overfit to their training environments — memorizing specific levels, visual details or simulator quirks — so generalization has to be designed in: train on a broad distribution of environments (procedurally generated levels, many seeds, randomized physics and visuals), regularize and augment the observations, use architectures and representations that capture task-relevant structure, and always evaluate on held-out environments that the agent never trained on. Key techniques are domain randomization (vary friction, masses, lighting, textures, sensor noise), procedural content generation (Procgen, NetHack, MiniGrid), data augmentation of observations (random crops/shifts, colour jitter), regularization (L2, dropout, batch norm, entropy bonus, information bottlenecks), representation learning that ignores distractors, domain adaptation and meta-RL for quick adaptation, and robust/adversarial training against worst-case perturbations. Research on Procgen (Cobbe et al.) showed that agents need thousands of training levels before test performance approaches training performance.

Read full answer

Applies toGymnasium 1.x, PyTorch 2.x, Stable-Baselines3

Techniques

Technique Addresses Example
Many training levels/seeds Memorizing specific layouts Procgen with 10k+ levels
Domain randomization Sim-to-real, physics/visual shifts Randomized friction, mass, textures
Observation augmentation Visual overfitting DrQ / RAD random shifts
Regularization Overfitting in networks L2, dropout, batch norm, entropy
Invariant representations Distractors (backgrounds) Bisimulation metrics, contrastive learning
Memory and system identification Unknown dynamics Recurrent policies that infer parameters
Meta-RL Fast adaptation to new tasks RL², PEARL
Robust/adversarial training Worst-case perturbations Adversarial disturbances, RARL
Held-out evaluation Measuring generalization Disjoint train/test level seeds

Code Example: Domain Randomization Wrapper (Gymnasium)

import gymnasium as gym
import numpy as np


class RandomizedCartPole(gym.Wrapper):
    """Samples new pole length and cart mass at every reset."""

    def __init__(self, env, length_range=(0.3, 0.8), masscart_range=(0.5, 2.0), seed=42):
        super().__init__(env)
        self.length_range, self.masscart_range = length_range, masscart_range
        self.rng = np.random.default_rng(seed)

    def reset(self, **kwargs):
        u = self.env.unwrapped
        u.length = float(self.rng.uniform(*self.length_range))      # half pole length
        u.masscart = float(self.rng.uniform(*self.masscart_range))
        u.total_mass = u.masspole + u.masscart                      # keep derived values consistent
        u.polemass_length = u.masspole * u.length
        return self.env.reset(**kwargs)


train_env = RandomizedCartPole(gym.make("CartPole-v1"))
obs, _ = train_env.reset(seed=42)
u = train_env.unwrapped
print(0.3 <= u.length <= 0.8, 0.5 <= u.masscart <= 2.0)            # True True

# Evaluate on settings outside the training range (held out)
test_env = RandomizedCartPole(gym.make("CartPole-v1"), length_range=(0.9, 1.0), seed=7)

The training wrapper changes physics at every episode, so a policy has to work across a family of carts rather than one; the test wrapper uses lengths it never saw. CartPole’s internal attribute names (length, masscart, total_mass, polemass_length) are those of the current Gymnasium implementation; check them for your version.

Evaluation Protocol

Split Contents
Train Levels/seeds/parameters used for learning
Validation Held-out levels for model selection
Test Untouched levels, plus shifted conditions (out-of-distribution)
Report Train vs test gap, mean ± std over several training seeds

Common Pitfalls

  • Evaluating on the training levels and calling it generalization.

  • Randomizing so widely that the task becomes unlearnable.

  • Reporting one training seed.

  • Ignoring the gap between simulator and real sensors (latency, noise).

Interview Tips

  • Say clearly that RL agents overfit to training environments.

  • Name domain randomization, procedural generation, augmentation and regularization.

  • Describe a held-out evaluation protocol.

28

How can RL agents overfit, and how do you prevent it?

Overfitting in reinforcement learning means the agent performs well in the exact conditions it trained on but poorly on slightly different ones: it memorizes specific levels, start states, random seeds, simulator artefacts or visual details instead of learning the underlying skill. It shows up in several forms: environment overfitting (a policy that only works on its training levels or physics settings), overfitting to early data (the value network fits the first experiences in the replay buffer so strongly that it loses plasticity), critic overfitting in offline RL (overestimating values for actions absent from the dataset), and exploiting simulator bugs that do not exist in the real world. Prevention: train on diverse environments (many seeds and procedurally generated levels, domain randomization), augment observations, regularize networks (L2, dropout, layer norm, entropy bonus), limit updates per sample or periodically reset parts of the network, use conservative methods for offline RL, and evaluate on held-out environments with several training seeds.

Read full answer

Applies toPyTorch 2.x, Gymnasium 1.x, Stable-Baselines3

Forms and Remedies

Form Symptom Remedy
Level/seed memorization Train return high, held-out return low More levels, procedural generation, held-out evaluation
Visual overfitting Fails when background/colour changes Augmentation (random shift, colour jitter)
Physics overfitting Fails in the real robot Domain randomization, system identification
Primacy bias / plasticity loss Learning stalls after early data Lower replay ratio, periodic network resets, layer norm
Offline critic overestimation Policy picks unseen actions with fake high Q CQL, IQL, behaviour regularization
Simulator exploits Uses glitches for reward Fix the sim, add penalties, validate in a different simulator
Hyperparameter overfitting Tuned settings fail on new seeds Tune on separate seeds, report variance

Code Example: Measuring the Generalization Gap

import gymnasium as gym
import numpy as np


def evaluate(policy, env_id, seeds, episodes_per_seed=1, **env_kwargs):
    env = gym.make(env_id, **env_kwargs)
    returns = []
    for seed in seeds:
        for ep in range(episodes_per_seed):
            obs, _ = env.reset(seed=seed * 1000 + ep)
            done, total = False, 0.0
            while not done:
                obs, r, terminated, truncated, _ = env.step(policy(obs))
                total += r
                done = terminated or truncated
            returns.append(total)
    env.close()
    return float(np.mean(returns))


def random_policy(obs, rng=np.random.default_rng(42)):
    return int(rng.integers(4))


train_seeds, test_seeds = range(0, 20), range(1000, 1020)          # disjoint
map_kwargs = {"map_name": "8x8", "is_slippery": True}
train_score = evaluate(random_policy, "FrozenLake-v1", train_seeds, **map_kwargs)
test_score = evaluate(random_policy, "FrozenLake-v1", test_seeds, **map_kwargs)
print(f"train {train_score:.2f}  test {test_score:.2f}  gap {train_score - test_score:.2f}")

The function is the important part: replace random_policy with your trained agent and keep the train and test seeds (or levels) disjoint. FrozenLake’s map is fixed, so here seeds only change the random slips; with procedurally generated environments (Procgen, MiniGrid) seeds change the level layout, and the train-test gap measures overfitting directly.

Common Pitfalls

  • Reporting training-environment returns as the result.

  • Using one training seed and one evaluation seed.

  • Updating too many times per sample (high replay ratio) without resets or regularization.

  • Leaving exploration noise on during evaluation (or off during training).

Interview Tips

  • Describe several kinds of overfitting, not only memorizing levels.

  • Give remedies matched to each kind.

  • Stress disjoint train/test environments and multiple seeds.

29

How does REINFORCE update policies, and how does it handle variance?

REINFORCE (Williams, 1992) is the basic Monte Carlo policy-gradient algorithm: run a full episode with the current stochastic policy π_θ, compute the return-to-go G_t from every step, and update the parameters by gradient ascent with θ ← θ + α Σ_t γᵗ G_t ∇θ log π_θ(a_t|s_t) (the γᵗ factor is often dropped in practice). Actions followed by high returns become more likely, actions followed by low returns less likely. The estimate is unbiased but has very high variance: returns sum many random rewards, and every action in an episode gets credit for rewards it may not have caused. Variance is reduced by using reward-to-go instead of the total episode return (actions cannot affect past rewards), subtracting a baseline b(s_t) — typically a learned value function V(s_t), which leaves the gradient unbiased — normalizing returns or advantages in each batch, averaging over many episodes per update, a smaller discount factor, and, moving beyond pure REINFORCE, actor-critic methods that replace G_t with bootstrapped advantage estimates.

Read full answer

Applies toPyTorch 2.x, Gymnasium 1.x

Variance-Reduction Techniques

Technique Change Bias?
Reward-to-go Use G_t from step t, not total return No
Baseline b(s) Weight by G_t − b(s_t) No (b must not depend on the action)
Learned value baseline b(s) = V_w(s) No (actor)
Return/advantage normalization Zero mean, unit std per batch Slight, helps in practice
More episodes per update Average over a batch No
Lower γ Shorter effective horizon Yes (changes objective)
Actor-critic / GAE Bootstrapped advantages Yes, but much lower variance

Why a Baseline Is Unbiased

E_(a∼π)[b(s) ∇θ log π_θ(a|s)] = b(s) ∇θ Σ_a π_θ(a|s) = b(s) ∇θ 1 = 0. Subtracting b(s) therefore removes no signal on average but cancels much of the noise when b(s) is close to the expected return.

Code Example: REINFORCE with a Value Baseline on CartPole (PyTorch)

import gymnasium as gym
import torch
from torch import nn

env = gym.make("CartPole-v1")
torch.manual_seed(42)

policy = nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 2))
value = nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 1))
pi_opt = torch.optim.Adam(policy.parameters(), lr=1e-3)
v_opt = torch.optim.Adam(value.parameters(), lr=1e-3)
gamma = 0.99


def returns_to_go(rewards, gamma):
    G, out = 0.0, []
    for r in reversed(rewards):
        G = r + gamma * G
        out.append(G)
    return torch.tensor(out[::-1], dtype=torch.float32)


for episode in range(600):
    obs, _ = env.reset(seed=42 + episode)
    states, log_probs, rewards = [], [], []
    done = False
    while not done:
        s = torch.as_tensor(obs, dtype=torch.float32)
        dist = torch.distributions.Categorical(logits=policy(s))
        a = dist.sample()
        obs, r, terminated, truncated, _ = env.step(int(a))
        states.append(s)
        log_probs.append(dist.log_prob(a))
        rewards.append(r)
        done = terminated or truncated

    G = returns_to_go(rewards, gamma)                       # reward-to-go
    S = torch.stack(states)
    baseline = value(S).squeeze(-1)
    advantages = G - baseline.detach()                      # baseline subtraction
    advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)   # normalization

    policy_loss = -(torch.stack(log_probs) * advantages).sum()
    pi_opt.zero_grad()
    policy_loss.backward()
    pi_opt.step()

    value_loss = (baseline - G).pow(2).mean()               # fit the baseline to the returns
    v_opt.zero_grad()
    value_loss.backward()
    v_opt.step()

print(len(rewards))                                         # length of the last episode

Episodes typically get much longer as training goes on (CartPole’s maximum is 500), but single-run REINFORCE is noisy, so the final length varies from run to run. Removing the baseline and normalization lines usually makes learning visibly slower and less stable.

Common Pitfalls

  • Using the total episode return for every step instead of reward-to-go.

  • A baseline that depends on the action (that introduces bias).

  • Not detaching the baseline in the policy loss.

  • Single-episode updates on long tasks (very noisy); batch several episodes.

Interview Tips

  • Write the REINFORCE update and explain it in words.

  • Explain why variance is high and why a baseline keeps the estimate unbiased.

  • Mention the step to actor-critic and GAE.

30

How are eligibility traces used in reinforcement learning?

Eligibility traces are a short-term memory vector that records which states (or state-action pairs, or weights) were visited recently, so that when a TD error occurs, credit or blame is passed back to all of them at once in proportion to how recently and how often they were visited. Each step, every trace decays by γλ, and the trace of the current state is increased; then every value is updated by α · δ · e(s), where δ is the one-step TD error. This implements TD(λ), which interpolates between one-step TD (λ = 0, only the current state is updated) and Monte Carlo (λ = 1, credit reaches all earlier states as if the full return were used), with an efficient online, incremental update instead of waiting for the episode to end. Variants are accumulating traces (e += 1), replacing traces (e = 1), Dutch traces (true online TD(λ)), and control versions SARSA(λ) and Watkins’s Q(λ). In deep RL the same idea appears in the λ-return used by GAE and in Retrace/TD(λ) targets.

Read full answer

Applies toPython 3.10+, NumPy

The Update (Tabular TD(λ), Accumulating Traces)

Step Formula
TD error δ_t = r(t+1) + γ V(s(t+1)) − V(s_t)
Decay all traces e(s) ← γλ e(s)
Mark current state e(s_t) ← e(s_t) + 1
Update all states V(s) ← V(s) + α δ_t e(s)
Episode start e ← 0

Worked Example

Chain s0 → s1 → s2 → terminal, rewards 0, 0, 1; γ = 1, λ = 0.5, α = 0.5, V = 0 everywhere. The first two TD errors are 0. At the last step δ = 1 and the traces are e(s2) = 1, e(s1) = 0.5, e(s0) = 0.25, so V becomes [0.125, 0.25, 0.5]. One-step TD (λ = 0) would only update V(s2) = 0.5 in that episode.

Code Example: TD(λ) on the Chain (NumPy)

import numpy as np


def td_lambda_episode(V, states, rewards, alpha, gamma, lam):
    e = np.zeros_like(V)
    for t, s in enumerate(states):
        next_value = V[states[t + 1]] if t + 1 < len(states) else 0.0   # terminal after the last state
        delta = rewards[t] + gamma * next_value - V[s]
        e *= gamma * lam
        e[s] += 1.0                                                      # accumulating trace
        V += alpha * delta * e                                           # every traced state is updated
    return V


states, rewards = [0, 1, 2], [0.0, 0.0, 1.0]
print(td_lambda_episode(np.zeros(3), states, rewards, alpha=0.5, gamma=1.0, lam=0.5))   # [0.125 0.25  0.5  ]
print(td_lambda_episode(np.zeros(3), states, rewards, alpha=0.5, gamma=1.0, lam=0.0))   # [0.  0.  0.5]
print(td_lambda_episode(np.zeros(3), states, rewards, alpha=0.5, gamma=1.0, lam=1.0))   # [0.5 0.5 0.5]

With λ = 1 every visited state moves halfway toward the episode’s return (1), which matches the Monte Carlo update; with λ = 0 only the last state learns.

Forward vs Backward View

View Description
Forward Update each state toward the λ-return G^λ_t = (1 − λ) Σ_n λⁿ⁻¹ G_t^(n) (needs future rewards)
Backward Traces + TD errors, computed online (equivalent for offline updates; true online TD(λ) makes it exact online)

Choosing λ

λ Behaviour
0 One-step TD: low variance, slow credit propagation
0.8-0.95 Common sweet spot (GAE uses λ = 0.95)
1 Monte Carlo: unbiased, high variance

Common Pitfalls

  • Not resetting traces at episode start.

  • In Watkins’s Q(λ), forgetting to cut traces after an exploratory (non-greedy) action.

  • Using traces with experience replay naively (traces assume sequential on-policy data).

  • Accumulating traces with large α on states revisited many times (replacing traces are steadier).

Interview Tips

  • Explain traces as decaying records of recent visits.

  • Write δ, the trace decay and the update.

  • Connect λ to the TD-MC spectrum and to GAE.

31

How is hierarchical reinforcement learning used for complex tasks?

Hierarchical reinforcement learning (HRL) splits a long, complex task into levels of decision-making: a high-level policy chooses subgoals or skills (“go to the kitchen”, “pick up the cup”) at a coarse time scale, and low-level policies carry them out with primitive actions over many steps. This attacks the main difficulties of long-horizon, sparse-reward problems: temporal abstraction shortens the effective horizon the high level must reason over, exploration improves because the agent explores in the space of meaningful skills rather than random joint torques, and reuse comes for free because a learned skill (open a door) can serve many tasks. The main frameworks are the options framework (Sutton, Precup and Singh: an option has an initiation set, an internal policy and a termination condition; learned end-to-end in Option-Critic), feudal / goal-conditioned hierarchies (FeUdal Networks, HIRO, HAC: the manager outputs goals and the worker is rewarded for reaching them), and skill discovery (DIAYN learns diverse skills without rewards). Challenges include non-stationarity between levels, designing or discovering good subgoals, and skills collapsing into trivial ones.

Read full answer

Applies toPyTorch 2.x, Gymnasium 1.x (conceptual code)

Frameworks

Framework High level outputs Low level Notes
Options (semi-MDP) Which option to run Option’s internal policy until termination Option-Critic learns options end-to-end
Feudal / goal-conditioned (HIRO, HAC) A goal state every k steps Goal-conditioned policy, intrinsic reward −‖s − g‖ Off-policy corrections for changing workers
MAXQ Task decomposition graph Subtask policies Value function decomposition
Skill discovery (DIAYN, DADS) Skill ID Skill-conditioned policy Skills learned without task reward
LLM planners + skills Plan in language Learned or scripted controllers SayCan-style robotics

Code Example: A Two-Level Agent Skeleton (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)
OBS, GOAL, ACT, K = 6, 2, 3, 10                 # manager acts every K steps


class Manager(nn.Module):                        # pi_hi(goal | state)
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(OBS, 64), nn.ReLU(), nn.Linear(64, GOAL))

    def forward(self, s):
        return torch.tanh(self.net(s))           # goal = desired (x, y) offset in [-1, 1]


class Worker(nn.Module):                         # pi_lo(action | state, goal)
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(OBS + GOAL, 64), nn.ReLU(), nn.Linear(64, ACT))

    def forward(self, s, g):
        return torch.distributions.Categorical(logits=self.net(torch.cat([s, g], -1)))


def intrinsic_reward(state_xy, start_xy, goal_offset):
    return -torch.norm(state_xy - (start_xy + goal_offset)).item()   # worker: reach the subgoal


manager, worker = Manager(), Worker()
s = torch.randn(OBS)
goal, start = manager(s), s[:2].clone()
for t in range(25):
    if t % K == 0 and t > 0:
        goal, start = manager(s), s[:2].clone()  # new subgoal every K steps
    action = worker(s, goal).sample()
    s = s + 0.1 * torch.randn(OBS)               # stand-in for env.step(action)
    r_lo = intrinsic_reward(s[:2], start, goal)
print(goal.shape, action.shape, isinstance(r_lo, float))    # torch.Size([2]) torch.Size([]) True

The manager is trained on the environment’s reward summed over its K-step windows; the worker is trained on the intrinsic reward for reaching the manager’s goal. Both can use standard algorithms (PPO, SAC, TD3).

Benefits and Challenges

Benefits Challenges
Shorter effective horizon Non-stationarity: the worker changes under the manager
Structured exploration Choosing goal spaces and time scales
Reusable skills Options collapsing (always one option, or very short ones)
Interpretability of subgoals More components and hyperparameters

Common Pitfalls

  • Hand-picking a goal space that does not cover what the task needs.

  • Training both levels simultaneously without corrections (HIRO relabels old goals).

  • Using HRL when a flat agent with good reward shaping would do.

Interview Tips

  • Explain temporal abstraction and why it helps with long horizons and sparse rewards.

  • Describe options and goal-conditioned manager/worker designs.

  • Give a robotics or navigation example and mention the non-stationarity challenge.

32

What is inverse reinforcement learning?

Inverse reinforcement learning (IRL) works backwards from standard RL: instead of being given a reward function and learning a policy, it observes an expert’s behaviour (demonstrations) and infers a reward function that explains it — the reward under which the expert’s behaviour would be (near-)optimal. The learned reward can then train a policy with ordinary RL, and because rewards capture why the expert acts rather than what it did, they often transfer better to new situations than simply copying actions (behaviour cloning). The core difficulty is that the problem is ill-posed: many rewards (including the trivial all-zero reward) explain the same behaviour. Methods resolve this differently: max-margin / apprenticeship learning (Abbeel and Ng: match expert feature expectations), Maximum Entropy IRL (Ziebart: prefer the reward under which the expert’s trajectories are most likely, with a max-entropy trajectory distribution), and adversarial methods (GAIL, AIRL) that train a discriminator between expert and agent behaviour. Learning reward models from human preference comparisons, as in RLHF, is a closely related idea.

Read full answer

Applies toPython 3.10+, NumPy

Approaches

Method Idea Output
Behaviour cloning (baseline) Supervised learning of actions Policy only
Apprenticeship / max-margin Reward w·φ(s) making expert better than others by a margin Linear reward + policy
MaxEnt IRL Maximize likelihood of demos under p(τ) ∝ exp(R(τ)) Reward
Deep MaxEnt IRL Neural reward network Nonlinear reward
GAIL Discriminator as implicit reward; policy fools it Policy (reward not reusable)
AIRL Discriminator structured to recover a transferable reward Reward + policy
Preference learning (RLHF) Fit reward from human pairwise comparisons Reward model

Code Example: MaxEnt IRL Gradient as Feature Matching (NumPy)

import numpy as np

# Linear reward R(s) = w . phi(s); MaxEnt IRL gradient = expert features - expected features under the model
phi = np.array([[1.0, 0.0],          # state 0 features
                [0.0, 1.0],          # state 1
                [1.0, 1.0]])         # state 2
expert_visits = np.array([0.1, 0.2, 0.7])         # expert spends most time in state 2


def model_visits(w):
    # Toy stand-in for the soft-optimal policy's state visitation: softmax over state rewards
    r = phi @ w
    p = np.exp(r - r.max())
    return p / p.sum()


w = np.zeros(2)
for _ in range(500):
    grad = expert_visits @ phi - model_visits(w) @ phi     # match feature expectations
    w += 0.5 * grad

print(np.round(expert_visits @ phi, 3), np.round(model_visits(w) @ phi, 3))   # features now match closely
print(bool(phi[2] @ w > phi[0] @ w and phi[2] @ w > phi[1] @ w))              # True: state 2 gets the highest reward

In real MaxEnt IRL, model_visits comes from solving a soft-RL problem (soft value iteration in small MDPs, or a policy optimization loop in large ones) with the current reward, which is why IRL is expensive: every reward update needs an RL solve.

IRL vs Behaviour Cloning

Aspect Behaviour cloning IRL
Learns Actions Reward (then policy)
Needs environment interaction No Yes (usually)
Compounding errors off the expert’s path Yes Less, the policy learns to recover
Transfer to new dynamics Poor Better (reward is reused)
Cost Low High

Applications and Caveats

Applications: autonomous driving (learning human driving preferences), robotics from demonstrations, modelling pedestrian or customer behaviour, and preference-based reward models for language models. Caveats: reward ambiguity, assumed expert optimality, needing many demonstrations, and learned rewards that can be exploited by the policy.

Common Pitfalls

  • Ignoring reward ambiguity (the recovered reward is not “the” true reward).

  • Assuming demonstrations are optimal when humans are noisy or inconsistent.

  • Using GAIL when a reusable reward is needed.

Interview Tips

  • Define IRL as inferring a reward from demonstrations, and why that is ill-posed.

  • Name MaxEnt IRL, apprenticeship learning and GAIL/AIRL.

  • Contrast with behaviour cloning and connect to RLHF reward models.

33

What is partial observability in RL, and how can it be addressed?

Partial observability means the agent’s observation does not contain the full state of the environment, so the Markov property fails for observations: two situations that look identical can require different actions, and the best action may depend on what happened before. Examples: a single video frame without velocity, a robot whose camera cannot see behind it, a card game with hidden hands, a recommendation system that cannot observe a user’s mood. The formal model is a POMDP: an MDP plus an observation function O(o | s′, a); the optimal agent acts on a belief state — a probability distribution over hidden states updated with Bayes’ rule. Practical remedies: stack recent observations (Atari frame stacking), give the policy memory (LSTM/GRU or transformer policies such as DRQN or recurrent PPO), maintain explicit beliefs with Bayes filters (Kalman or particle filters) when a model exists, add information-gathering actions, use privileged information during training only (asymmetric actor-critic), and engineer observations that include missing quantities when possible.

Read full answer

Applies toNumPy, PyTorch 2.x, Gymnasium 1.x; `sb3_contrib.RecurrentPPO`

Remedies

Remedy How Good for
Frame/observation stacking Concatenate last k observations Velocities, short memory
Recurrent policy LSTM/GRU hidden state as memory Longer dependencies
Transformer policy Attention over observation history Long contexts, more compute
Belief state (Bayes filter) Track P(s | history) Known models (navigation, tracking)
Information-gathering actions Agent learns to look/probe Active sensing
Asymmetric actor-critic Critic sees the true state in simulation Sim-to-real robotics
Better sensors / features Add missing variables When feasible

Code Example: Bayes Belief Update (Tiger Problem, NumPy)

import numpy as np

# Hidden state: tiger behind left (0) or right (1) door. "Listen" is correct 85% of the time.
belief = np.array([0.5, 0.5])
p_hear_left = np.array([0.85, 0.15])          # P(hear "left" | tiger left), P(hear "left" | tiger right)


def update(belief, heard_left):
    likelihood = p_hear_left if heard_left else 1 - p_hear_left
    posterior = likelihood * belief
    return posterior / posterior.sum()


belief = update(belief, heard_left=True)
print(np.round(belief, 3))                    # [0.85 0.15]
belief = update(belief, heard_left=True)
print(np.round(belief, 3))                    # [0.97 0.03]

After two “left” observations: 0.85² / (0.85² + 0.15²) = 0.7225 / 0.745 ≈ 0.970. A POMDP agent decides on this belief, for example opening a door only once the belief is confident enough.

Code Example: Frame Stacking and a Recurrent Policy

import gymnasium as gym
import torch
from torch import nn

env = gym.wrappers.FrameStackObservation(gym.make("CartPole-v1"), stack_size=4)
obs, _ = env.reset(seed=42)
print(obs.shape)                               # (4, 4): last 4 observations


class RecurrentPolicy(nn.Module):
    def __init__(self, obs_dim, n_actions, hidden=64):
        super().__init__()
        self.gru = nn.GRU(obs_dim, hidden, batch_first=True)
        self.head = nn.Linear(hidden, n_actions)

    def forward(self, obs_seq, h=None):
        out, h = self.gru(obs_seq, h)          # h carries memory between calls during an episode
        return self.head(out[:, -1]), h


torch.manual_seed(42)
policy = RecurrentPolicy(obs_dim=2, n_actions=2)   # e.g. observations with velocities removed
logits, h = policy(torch.randn(1, 1, 2))
logits, h = policy(torch.randn(1, 1, 2), h)        # next step reuses the hidden state
print(logits.shape, h.shape)                        # torch.Size([1, 2]) torch.Size([1, 1, 64])

For a ready-made option, sb3_contrib.RecurrentPPO("MlpLstmPolicy", env) trains an LSTM policy; reset the hidden state at episode boundaries.

Common Pitfalls

  • Treating observations as states and blaming the algorithm when it fails.

  • Not resetting recurrent state between episodes, or training on shuffled single transitions with an RNN.

  • Stacking too few frames for the memory the task needs.

  • Letting the actor use privileged simulator state that will not exist at deployment.

Interview Tips

  • Define POMDPs and belief states, with an example.

  • List practical remedies: stacking, recurrent policies, belief tracking.

  • Mention asymmetric actor-critic for sim-to-real.

34

Implement the epsilon-greedy action selection strategy in Python.

Epsilon-greedy action selection picks a uniformly random action with probability ε (explore) and the action with the highest estimated value with probability 1 − ε (exploit). A good implementation also handles a few details: ties among best actions should be broken randomly (otherwise an all-zero Q-table always picks action 0), it should take a seeded random generator for reproducibility, it may support action masks for invalid actions, and it often comes with a decay schedule for ε (linear or exponential, from about 1.0 down to 0.01-0.05). Note that with probability ε the random action can still be the greedy one, so the greedy action is chosen with total probability 1 − ε + ε/|A|.

Read full answer

Applies toPython 3.10+, NumPy, PyTorch 2.x

Code Example: Tabular ε-Greedy with Random Tie-Breaking and Masks

import numpy as np


def epsilon_greedy(q_values, epsilon, rng, mask=None):
    """Return an action index. mask: boolean array, True = action allowed."""
    q = np.asarray(q_values, dtype=float)
    allowed = np.ones_like(q, dtype=bool) if mask is None else np.asarray(mask, dtype=bool)
    valid = np.flatnonzero(allowed)
    if rng.random() < epsilon:
        return int(rng.choice(valid))                        # explore among valid actions
    best_value = q[valid].max()
    best = valid[q[valid] == best_value]                     # all actions tied for best
    return int(rng.choice(best))                             # random tie-break


rng = np.random.default_rng(42)
q = [1.0, 3.0, 3.0, 0.5]

greedy_picks = [epsilon_greedy(q, 0.0, rng) for _ in range(1000)]
print(sorted(set(greedy_picks)))                             # [1, 2]  ties broken randomly

print(epsilon_greedy(q, 0.0, rng, mask=[True, False, False, True]))   # 0  best among allowed actions

picks = np.array([epsilon_greedy([0.0, 1.0, 0.0, 0.0], 0.2, rng) for _ in range(100_000)])
print(round((picks == 1).mean(), 2))                         # about 0.85 = 1 - 0.2 + 0.2 / 4

Decay Schedules

import math


def linear_epsilon(step, start=1.0, end=0.05, decay_steps=10_000):
    fraction = min(step / decay_steps, 1.0)
    return start + fraction * (end - start)


def exponential_epsilon(step, start=1.0, end=0.05, rate=1e-3):
    return end + (start - end) * math.exp(-rate * step)


print([round(linear_epsilon(n), 3) for n in (0, 5_000, 20_000)])             # [1.0, 0.525, 0.05]
print(round(exponential_epsilon(0), 3), round(exponential_epsilon(1_000), 3))  # 1.0 0.399

By hand: halfway through the linear schedule, 1.0 + 0.5 × (0.05 − 1.0) = 0.525; the exponential schedule at step 1,000 gives 0.05 + 0.95 × e⁻¹ ≈ 0.399.

Code Example: Batched ε-Greedy for a DQN (PyTorch)

import torch


def epsilon_greedy_batch(q_values, epsilon, generator=None):
    """q_values: (batch, n_actions) tensor -> (batch,) actions."""
    greedy = q_values.argmax(dim=1)
    random_actions = torch.randint(q_values.shape[1], greedy.shape, generator=generator)
    explore = torch.rand(greedy.shape, generator=generator) < epsilon
    return torch.where(explore, random_actions, greedy)


g = torch.Generator().manual_seed(42)
q = torch.tensor([[0.1, 0.9], [0.8, 0.2]])
print(epsilon_greedy_batch(q, 0.0, g).tolist())              # [1, 0]

Common Pitfalls

  • np.argmax alone: always picks the first of tied actions (biased early exploration).

  • Forgetting to decay ε, or decaying it before the agent has seen any reward.

  • Leaving ε > 0 during evaluation (or using ε = 0 in environments that need exploration to avoid getting stuck).

  • Exploring invalid actions in environments with action masks.

Interview Tips

  • Write the function with random tie-breaking.

  • State the true probability of the greedy action, 1 − ε + ε/|A|.

  • Add a decay schedule and mention alternatives (softmax, UCB).

35

Write a Python script that simulates a simple MDP using a transition matrix.

To simulate an MDP from a transition matrix, store the dynamics as an array P[s, a, s′] whose entries are the probabilities of moving from state s to s′ when taking action a (each P[s, a, :] must sum to 1), store rewards as R[s, a] (or R[s, a, s′]), and then repeatedly sample the next state with rng.choice(n_states, p=P[s, a]), add the discounted reward, and stop at a terminal state or a step limit. Wrapping this in a small class with reset and step methods that follow the Gymnasium convention makes it usable by RL code; the same arrays can be used to compute exact values with dynamic programming, which is a good way to check the simulation (sampled returns should average to the computed values).

Read full answer

Applies toPython 3.10+, NumPy

Code Example: MDP Simulator with Exact Value Check

import numpy as np


class TabularMDP:
    def __init__(self, P, R, terminal, gamma=0.9, max_steps=100, seed=42):
        self.P, self.R = np.asarray(P, dtype=float), np.asarray(R, dtype=float)
        assert np.allclose(self.P.sum(axis=2), 1.0), "each P[s, a, :] must sum to 1"
        self.n_states, self.n_actions = self.R.shape
        self.terminal, self.gamma, self.max_steps = set(terminal), gamma, max_steps
        self.rng = np.random.default_rng(seed)

    def reset(self, start=0):
        self.state, self.t = start, 0
        return self.state, {}

    def step(self, action):
        reward = self.R[self.state, action]
        self.state = int(self.rng.choice(self.n_states, p=self.P[self.state, action]))
        self.t += 1
        terminated = self.state in self.terminal
        truncated = self.t >= self.max_steps
        return self.state, reward, terminated, truncated, {}

    def policy_value(self, policy):
        """Exact V for a deterministic policy: solve (I - gamma P_pi) V = R_pi."""
        idx = np.arange(self.n_states)
        P_pi = self.P[idx, policy].copy()
        R_pi = self.R[idx, policy].copy()
        for s in self.terminal:                     # terminal: no reward, no future
            P_pi[s], R_pi[s] = 0.0, 0.0
        return np.linalg.solve(np.eye(self.n_states) - self.gamma * P_pi, R_pi)


# States: 0 = start, 1 = middle, 2 = goal (terminal). Actions: 0 = safe, 1 = risky
P = [
    [[0.0, 1.0, 0.0], [0.5, 0.0, 0.5]],    # from 0: safe -> 1; risky -> 0 or goal (50/50)
    [[0.0, 0.0, 1.0], [0.0, 0.0, 1.0]],    # from 1: both reach the goal
    [[0.0, 0.0, 1.0], [0.0, 0.0, 1.0]],    # goal is absorbing
]
R = [[-1.0, -1.0],
     [5.0, 5.0],
     [0.0, 0.0]]
mdp = TabularMDP(P, R, terminal=[2], gamma=0.9)

policy = np.array([0, 0, 0])                  # always "safe"
print(np.round(mdp.policy_value(policy), 2))  # [3.5 5.  0. ]

returns = []
for episode in range(5_000):
    s, _ = mdp.reset()
    G, discount, done = 0.0, 1.0, False
    while not done:
        s, r, terminated, truncated, _ = mdp.step(policy[s])
        G += discount * r
        discount *= mdp.gamma
        done = terminated or truncated
    returns.append(G)
print(round(float(np.mean(returns)), 2))      # 3.5 (the safe path is deterministic)

By hand for the “safe” policy: from state 1 the agent earns 5 and reaches the goal, so V(1) = 5; from state 0 it earns −1 then V(1): V(0) = −1 + 0.9 × 5 = 3.5. Every simulated episode follows the same path, so the sample mean equals 3.5. Switching to the risky action at state 0 (policy = [1, 0, 0]) gives V(0) = −1 + 0.9 × 0.5 × V(0), so V(0) = −1 / 0.55 ≈ −1.82, and the simulated average then varies around that value.

Extensions

Extension Change
R[s, a, s′] Reward = R[s, a, next_state] after sampling
Stochastic policy Sample a ~ π[s] before each step
Gymnasium compatibility Subclass gym.Env with spaces.Discrete
Speed Pre-compute cumulative probabilities and use searchsorted

Common Pitfalls

  • Rows of P that do not sum to 1 (rng.choice raises an error).

  • Counting rewards after reaching a terminal state.

  • Using np.random.seed globally instead of a Generator.

  • No step limit for policies that never reach a terminal state.

Interview Tips

  • Show the P[s, a, s′] layout and the validation check.

  • Sample next states with rng.choice(..., p=...).

  • Verify the simulation against the exact Bellman solution.

36

Code a Q-learning algorithm in Python to solve a grid-world problem.

A grid-world Q-learning solution needs three parts: a small environment (states are grid cells, actions are up/down/left/right, rewards for reaching the goal and penalties for each step or for traps), a Q-table of shape (number of states × number of actions), and the training loop that picks ε-greedy actions and applies Q(s, a) ← Q(s, a) + α [r + γ max_a′ Q(s′, a′) − Q(s, a)], with no future term at terminal states. After training, the greedy policy argmax_a Q(s, a) gives an arrow per cell, which should trace a shortest safe path to the goal. Because the grid is tiny and deterministic, the result can be checked against value iteration or by counting steps.

Read full answer

Applies toPython 3.10+, NumPy

Code Example: Grid World + Q-Learning

import numpy as np

GRID = [
    "S..#",
    ".#..",
    "...X",
    "#..G",
]                                       # S start, G goal (+10), X trap (-10), # wall, -1 per step
ROWS, COLS = len(GRID), len(GRID[0])
ACTIONS = [(-1, 0), (1, 0), (0, -1), (0, 1)]          # up, down, left, right
ARROWS = "^v<>"


def find(ch):
    for r, row in enumerate(GRID):
        if ch in row:
            return r, row.index(ch)


START = find("S")


def step(pos, a):
    r, c = pos[0] + ACTIONS[a][0], pos[1] + ACTIONS[a][1]
    if not (0 <= r < ROWS and 0 <= c < COLS) or GRID[r][c] == "#":
        r, c = pos                                    # bump into wall: stay
    cell = GRID[r][c]
    if cell == "G":
        return (r, c), 10.0, True
    if cell == "X":
        return (r, c), -10.0, True
    return (r, c), -1.0, False


def idx(pos):
    return pos[0] * COLS + pos[1]


rng = np.random.default_rng(42)
Q = np.zeros((ROWS * COLS, len(ACTIONS)))
alpha, gamma = 0.5, 0.95
eps, eps_min, eps_decay = 1.0, 0.05, 0.99

for episode in range(1_000):
    pos, done, steps = START, False, 0
    while not done and steps < 100:
        s = idx(pos)
        if rng.random() < eps:
            a = int(rng.integers(4))
        else:
            best = np.flatnonzero(Q[s] == Q[s].max())
            a = int(rng.choice(best))                 # random tie-break
        pos, r, done = step(pos, a)
        target = r + (0.0 if done else gamma * Q[idx(pos)].max())
        Q[s, a] += alpha * (target - Q[s, a])
        steps += 1
    eps = max(eps_min, eps * eps_decay)

# Print the greedy policy
for r in range(ROWS):
    print(" ".join(GRID[r][c] if GRID[r][c] in "#GX" else ARROWS[int(np.argmax(Q[idx((r, c))]))]
                   for c in range(COLS)))

# Follow the greedy policy from the start
pos, path, done = START, [START], False
while not done and len(path) < 20:
    pos, _, done = step(pos, int(np.argmax(Q[idx(pos)])))
    path.append(pos)
print(path[-1] == find("G"), len(path) - 1)    # True 6  (a shortest path has 6 moves)

The shortest route from S at (0, 0) to G at (3, 3) needs 6 moves (3 down and 3 right, avoiding the walls and the trap at (2, 3)), for example down, down, right, right, down, right. With 1,000 episodes on this 16-cell grid, the greedy policy reliably finds such a path; several 6-move paths exist, so the printed arrows can differ from run to run with other seeds while the path length stays 6.

Design Notes

Choice Reason
−1 per step Encourages shortest paths
Terminal: no future term Correct Bellman target
Random tie-breaking Untrained Q-rows are all 0
ε decay 0.99 per episode Explores early, exploits later
Step cap 100 Prevents endless wandering early on

Common Pitfalls

  • Bootstrapping from terminal states (values leak past the goal or trap).

  • Always choosing action 0 on ties.

  • Letting the agent walk through walls or off the grid.

  • Evaluating with ε > 0.

Interview Tips

  • Build the environment, Q-table and update clearly separated.

  • Explain the reward design (step cost, goal, trap) and terminal handling.

  • Verify the result (path length, compare with value iteration).

37

Implement value iteration for a given MDP in Python.

Value iteration finds the optimal value function and policy of a known MDP by repeatedly applying the Bellman optimality backup to every state, V(s) ← max_a Σ_s′ P(s′|s, a)[R(s, a, s′) + γ V(s′)], until the largest change in a sweep falls below a threshold θ; the optimal policy is then greedy with respect to the final values, π(s) = argmax_a Q(s, a). With NumPy the backup for all states and actions is one tensor expression: Q = R + γ P·V. The algorithm converges because the backup is a γ-contraction; a stopping threshold of θ(1 − γ)/(2γ) on the change guarantees the greedy policy’s values are within θ of optimal. Each sweep costs O(|S|²|A|).

Read full answer

Applies toPython 3.10+, NumPy; Gymnasium 1.x (FrozenLake's `env.unwrapped.P` model)

Code Example: Vectorized Value Iteration

import numpy as np


def value_iteration(P, R, gamma=0.9, theta=1e-8, max_iters=10_000):
    """P: (S, A, S') transition probs. R: (S, A) expected rewards. Returns V, policy, iterations."""
    V = np.zeros(P.shape[0])
    for i in range(1, max_iters + 1):
        Q = R + gamma * P @ V                  # (S, A): Bellman backup for every state-action
        V_new = Q.max(axis=1)
        if np.max(np.abs(V_new - V)) < theta:
            return V_new, Q.argmax(axis=1), i
        V = V_new
    return V, Q.argmax(axis=1), max_iters


# Example: states 0, 1, 2 (2 = terminal goal). Actions: 0 = safe, 1 = risky
P = np.array([
    [[0.0, 1.0, 0.0], [0.5, 0.0, 0.5]],
    [[0.0, 0.0, 1.0], [0.0, 0.0, 1.0]],
    [[0.0, 0.0, 1.0], [0.0, 0.0, 1.0]],
])
R = np.array([[-1.0, -1.0],
              [5.0, 5.0],
              [0.0, 0.0]])
V, policy, iters = value_iteration(P, R, gamma=0.9)
print(np.round(V, 2), policy)                  # [3.5 5.  0. ] [0 0 0]

By hand: V(2) = 0 (absorbing, no reward), V(1) = 5, and in state 0 the safe action gives −1 + 0.9 × 5 = 3.5 while the risky action gives −1 + 0.9 × (0.5 × 3.5 + 0.5 × 0) = 0.575, so the optimal value is 3.5 with the safe action. (For state 1 both actions tie, and argmax returns the first.)

Code Example: Value Iteration on FrozenLake’s Model (Gymnasium)

import gymnasium as gym
import numpy as np

env = gym.make("FrozenLake-v1", is_slippery=True)
model = env.unwrapped.P                         # model[s][a] = [(prob, next_s, reward, terminated), ...]
S, A = env.observation_space.n, env.action_space.n

P = np.zeros((S, A, S))
R = np.zeros((S, A))
for s in range(S):
    for a in range(A):
        for prob, s2, r, terminated in model[s][a]:
            if not terminated:
                P[s, a, s2] += prob             # terminal transitions have no future value
            R[s, a] += prob * r

gamma, theta = 0.99, 1e-10
V = np.zeros(S)
while True:
    Q = R + gamma * P @ V
    V_new = Q.max(axis=1)
    if np.max(np.abs(V_new - V)) < theta:
        break
    V = V_new
policy = Q.argmax(axis=1)
print(np.round(V_new.reshape(4, 4), 3))
print(policy.reshape(4, 4))                     # 0=left, 1=down, 2=right, 3=up

Leaving terminal transitions out of P makes their future value zero without special-casing terminal states. On the slippery map the optimal policy often points away from holes rather than straight at the goal, because each move only goes in the intended direction one third of the time.

Value Iteration vs Policy Iteration

Aspect Value iteration Policy iteration
Per iteration One max-backup sweep Full policy evaluation + improvement
Iterations Many Few
Cost per iteration Low High (solve a linear system)
Typical use Simple, large-ish tabular MDPs Small MDPs, exact solutions

Common Pitfalls

  • Shapes: P @ V must contract over the last axis (s′).

  • Counting future value after terminal transitions.

  • θ too large when γ is close to 1 (the policy may still be suboptimal).

  • Using in-place updates and thinking the result differs (Gauss-Seidel variants converge to the same values, often faster).

Interview Tips

  • Write the Bellman optimality backup and the stopping rule.

  • Vectorize with NumPy and extract the greedy policy.

  • Compare with policy iteration and mention the contraction argument.

38

Write a function to compute discounted rewards for a reward sequence.

The discounted return from each time step t is G_t = r_t + γ r(t+1) + γ² r(t+2) + … to the end of the episode, and it is computed efficiently in one backward pass using the recursion G_t = r_t + γ G_(t+1) with G after the last step equal to 0 (or to a bootstrap value V(s_T) if the episode was cut off rather than finished). That gives O(T) time instead of the O(T²) of summing each tail separately. For rollouts that contain several episodes, the recursion must reset at episode boundaries using the done flags. The same backward pattern computes GAE advantages in PPO. In policy-gradient code, the returns are often normalized (zero mean, unit variance) afterwards to reduce variance.

Read full answer

Applies toPython 3.10+, NumPy, SciPy (optional `lfilter` trick), PyTorch 2.x

Code Example: Discounted Returns

import numpy as np
from scipy.signal import lfilter


def discounted_returns(rewards, gamma, bootstrap=0.0):
    """G_t = r_t + gamma * G_{t+1}; bootstrap = V(s_T) if the episode was truncated."""
    returns = np.zeros(len(rewards), dtype=float)
    running = bootstrap
    for t in reversed(range(len(rewards))):
        running = rewards[t] + gamma * running
        returns[t] = running
    return returns


rewards = [1.0, 0.0, 0.0, 10.0]
print(discounted_returns(rewards, 0.9))                 # [ 8.29  8.1   9.   10.  ]
print(discounted_returns(rewards, 0.0))                 # [ 1.  0.  0. 10.]
print(discounted_returns(rewards, 1.0))                 # [11. 10. 10. 10.]
print(discounted_returns([1.0, 1.0], 0.9, bootstrap=5.0))   # [5.95 5.5 ]


def discounted_returns_lfilter(rewards, gamma):
    # Same recursion as an IIR filter run on the reversed sequence
    return lfilter([1.0], [1.0, -gamma], np.asarray(rewards, dtype=float)[::-1])[::-1]


print(np.allclose(discounted_returns_lfilter(rewards, 0.9), discounted_returns(rewards, 0.9)))   # True

By hand with γ = 0.9: G₃ = 10, G₂ = 0 + 0.9 × 10 = 9, G₁ = 0 + 0.9 × 9 = 8.1, G₀ = 1 + 0.9 × 8.1 = 8.29.

Code Example: Several Episodes in One Rollout (Done Flags)

import numpy as np


def discounted_returns_with_dones(rewards, dones, gamma, last_value=0.0):
    """dones[t] = True if the episode ended at step t (terminal); resets the running return."""
    returns = np.zeros(len(rewards))
    running = last_value
    for t in reversed(range(len(rewards))):
        if dones[t]:
            running = 0.0
        running = rewards[t] + gamma * running
        returns[t] = running
    return returns


rewards = [1.0, 1.0, 1.0, 2.0, 2.0]
dones = [False, False, True, False, False]      # episode 1 ends at t=2; episode 2 continues
print(discounted_returns_with_dones(rewards, dones, 0.5, last_value=4.0))   # [1.75 1.5  1.   4.   4.  ]

By hand: episode 1 gives G₂ = 1, G₁ = 1 + 0.5 = 1.5, G₀ = 1 + 0.75 = 1.75; episode 2 is unfinished, so it bootstraps from 4: G₄ = 2 + 0.5 × 4 = 4, G₃ = 2 + 0.5 × 4 = 4.

Normalizing for Policy Gradients

import numpy as np

G = np.array([8.29, 8.1, 9.0, 10.0])
G_norm = (G - G.mean()) / (G.std() + 1e-8)
print(abs(round(G_norm.mean(), 6)), round(G_norm.std(), 4))     # 0.0 1.0

Common Pitfalls

  • Looping forward and summing tails (O(T²), and easy to get wrong).

  • Not resetting at episode boundaries in multi-episode buffers.

  • Treating time-limit truncation as termination (should bootstrap from V).

  • Integer arrays: np.zeros_like(rewards) on an int list silently truncates returns.

Interview Tips

  • Give the recursion and do the backward loop.

  • Handle done flags and bootstrapping.

  • Mention normalization and the link to GAE.

39

Develop a SARSA agent in Python for the Taxi-v3 environment.

SARSA learns Q-values on-policy from the quintuple (s, a, r, s′, a′): the agent picks a′ with the same ε-greedy policy it is following and updates Q(s, a) ← Q(s, a) + α [r + γ Q(s′, a′) − Q(s, a)]. Taxi-v3 (now in Gymnasium; the old OpenAI Gym package is no longer maintained) is a good fit: 500 discrete states (taxi position × passenger location × destination), 6 actions (south, north, east, west, pickup, dropoff), −1 per step, +20 for a correct drop-off, −10 for illegal pickup/drop-off, and a 200-step time limit. A table of 500 × 6 Q-values, ε decayed over a few thousand episodes, α ≈ 0.1-0.5 and γ ≈ 0.99 typically learns a near-optimal policy whose evaluation return is about +8 per episode on average (20 minus the steps taken).

Read full answer

Applies toPython 3.10+, NumPy, Gymnasium 1.x

Code Example: SARSA on Taxi-v3

import gymnasium as gym
import numpy as np

env = gym.make("Taxi-v3")
n_states, n_actions = env.observation_space.n, env.action_space.n
print(n_states, n_actions)                               # 500 6

rng = np.random.default_rng(42)
Q = np.zeros((n_states, n_actions))
alpha, gamma = 0.1, 0.99
eps, eps_min, eps_decay = 1.0, 0.01, 0.999


def choose(state, epsilon):
    if rng.random() < epsilon:
        return int(rng.integers(n_actions))
    best = np.flatnonzero(Q[state] == Q[state].max())
    return int(rng.choice(best))


for episode in range(10_000):
    s, _ = env.reset(seed=int(rng.integers(1_000_000)))
    a = choose(s, eps)
    done = False
    while not done:
        s2, r, terminated, truncated, _ = env.step(a)
        a2 = choose(s2, eps)                             # next action from the same policy
        target = r if terminated else r + gamma * Q[s2, a2]   # truncated still bootstraps
        Q[s, a] += alpha * (target - Q[s, a])
        s, a = s2, a2
        done = terminated or truncated
    eps = max(eps_min, eps * eps_decay)


def evaluate(episodes=100):
    total, successes = 0.0, 0
    for i in range(episodes):
        s, _ = env.reset(seed=100_000 + i)              # held-out seeds
        done, ep_return = False, 0.0
        while not done:
            s, r, terminated, truncated, _ = env.step(int(np.argmax(Q[s])))
            ep_return += r
            done = terminated or truncated
        total += ep_return
        successes += int(terminated)
    return total / episodes, successes / episodes


mean_return, success_rate = evaluate()
print(f"mean return {mean_return:.2f}, success rate {success_rate:.2f}")

After training, the greedy policy usually completes nearly every evaluation episode with a mean return in the high single digits (an optimal taxi averages around +8); the exact figure depends on the run.

Hyperparameters

Parameter Value Effect
α 0.1 Stable updates in a stochastic start-state setting
γ 0.99 Plans across the whole trip
ε 1.0 → 0.01, ×0.999 per episode ε ≈ 0.37 after 1,000 episodes, reaches 0.01 near episode 4,600
Episodes 10,000 Enough for 500 × 6 table

SARSA vs Q-Learning on Taxi

Both converge here; Q-learning’s target uses max Q(s′, ·) and often learns a little faster, while SARSA’s on-policy target makes it account for its own exploration (more cautious in environments with large penalties near the optimal path).

Common Pitfalls

  • Using import gym and the old 4-value step API.

  • Choosing a′ greedily (that is Q-learning, not SARSA).

  • Treating the 200-step truncation as terminal.

  • Evaluating with ε > 0 or on the same seeds as training.

Interview Tips

  • Write the SARSA update and point out where a′ comes from.

  • Describe Taxi’s state, actions and rewards.

  • Report evaluation with a greedy policy on held-out seeds.

40

Build a neural network in PyTorch or TensorFlow that approximates a policy.

A policy network maps a state to a distribution over actions: for discrete actions it outputs one logit per action and a categorical distribution (softmax) is built from them; for continuous actions it outputs the mean of a Gaussian, with a learned log standard deviation, often squashed by tanh to respect action bounds. The network should provide three things to the training algorithm: sampling (act) for exploration, log-probabilities of taken actions for the policy-gradient loss, and the entropy for exploration bonuses; for deterministic evaluation it returns the most likely action (argmax or the mean). Good defaults: two hidden layers of 64-256 units with tanh or ReLU, orthogonal initialization with a small gain (0.01) on the output layer so the initial policy is close to uniform, and observation normalization.

Read full answer

Applies toPyTorch 2.x; Keras 3 version included

Code Example: Discrete and Continuous Policy Networks (PyTorch)

import torch
from torch import nn
from torch.distributions import Categorical, Normal


def layer_init(layer, gain=2 ** 0.5):
    nn.init.orthogonal_(layer.weight, gain)
    nn.init.zeros_(layer.bias)
    return layer


class DiscretePolicy(nn.Module):
    def __init__(self, obs_dim, n_actions, hidden=64):
        super().__init__()
        self.net = nn.Sequential(
            layer_init(nn.Linear(obs_dim, hidden)), nn.Tanh(),
            layer_init(nn.Linear(hidden, hidden)), nn.Tanh(),
            layer_init(nn.Linear(hidden, n_actions), gain=0.01),   # near-uniform initial policy
        )

    def distribution(self, obs):
        return Categorical(logits=self.net(obs))

    def act(self, obs, deterministic=False):
        dist = self.distribution(obs)
        action = dist.probs.argmax(-1) if deterministic else dist.sample()
        return action, dist.log_prob(action), dist.entropy()


class GaussianPolicy(nn.Module):
    def __init__(self, obs_dim, act_dim, hidden=64):
        super().__init__()
        self.mu = nn.Sequential(
            layer_init(nn.Linear(obs_dim, hidden)), nn.Tanh(),
            layer_init(nn.Linear(hidden, hidden)), nn.Tanh(),
            layer_init(nn.Linear(hidden, act_dim), gain=0.01),
        )
        self.log_std = nn.Parameter(torch.zeros(act_dim))          # state-independent std

    def act(self, obs, deterministic=False):
        dist = Normal(self.mu(obs), self.log_std.exp())
        action = dist.mean if deterministic else dist.sample()
        return action, dist.log_prob(action).sum(-1), dist.entropy().sum(-1)


torch.manual_seed(42)
pi = DiscretePolicy(obs_dim=4, n_actions=2)
obs = torch.randn(5, 4)
a, logp, ent = pi.act(obs)
print(a.shape, logp.shape, ent.shape)                              # torch.Size([5]) torch.Size([5]) torch.Size([5])
print(torch.allclose(pi.distribution(obs).probs, torch.full((5, 2), 0.5), atol=0.05))   # True: near uniform

gpi = GaussianPolicy(obs_dim=3, act_dim=2)
a, logp, ent = gpi.act(torch.randn(5, 3))
print(a.shape, logp.shape)                                         # torch.Size([5, 2]) torch.Size([5])

# One policy-gradient step: increase log-prob of actions with positive advantage
advantages = torch.tensor([1.0, -1.0, 0.5, 0.0, 2.0])
a, logp, ent = pi.act(obs)
loss = -(logp * advantages).mean() - 0.01 * ent.mean()
loss.backward()
print(all(p.grad is not None for p in pi.parameters()))            # True

The small output gain keeps the initial logits near zero, so both CartPole actions start with probability close to 0.5; the policy then learns from the loss above (REINFORCE, A2C) or from PPO’s clipped version of it.

Keras 3 Equivalent (Discrete)

import keras
from keras import layers

policy = keras.Sequential([
    keras.Input(shape=(4,)),
    layers.Dense(64, activation="tanh"),
    layers.Dense(64, activation="tanh"),
    layers.Dense(2),                                   # logits; sample with keras.random.categorical
])
print(policy.output_shape)                             # (None, 2)

Design Choices

Choice Discrete Continuous
Output Logits per action Mean (and log std)
Distribution Categorical Gaussian (tanh-squashed for SAC)
Deterministic action argmax Mean
Bounds — tanh squash or clip to action space
Invalid actions Mask logits to −∞ —

Common Pitfalls

  • Applying softmax and then passing the probabilities as logits=.

  • Summing log-probs over action dimensions for Gaussian policies (forgetting it).

  • Clipping sampled actions but computing log-prob of the unclipped action without care.

  • Large output initialization, so the initial policy is nearly deterministic.

Interview Tips

  • Explain logits → categorical and mean/std → Gaussian.

  • Show act returning action, log-prob and entropy.

  • Mention initialization, normalization and action masking.

41

Write a Python implementation of the REINFORCE algorithm.

REINFORCE is implemented in four steps per update: run one or more full episodes with the current stochastic policy while recording the log-probability of each chosen action and each reward; compute the discounted return-to-go G_t for every step with a backward pass; form the loss −Σ_t log π_θ(a_t|s_t) · G_t (usually with returns normalized or a baseline subtracted, to reduce variance); and take a gradient step on that loss. Minimizing the negative of the objective is gradient ascent on expected return. The implementation below uses PyTorch for the policy network and Gymnasium’s CartPole-v1, batches several episodes per update for steadier gradients, and evaluates greedily at the end. Expect noisy learning curves; REINFORCE is the simplest policy-gradient method, not the most efficient.

Read full answer

Applies toPython 3.10+, PyTorch 2.x, Gymnasium 1.x

Algorithm

Step Detail
1 Sample N episodes with π_θ, storing log π_θ(a_t|s_t) and r_t
2 G_t = r_t + γ G_(t+1), backward per episode
3 Normalize G across the batch (or subtract a baseline)
4 loss = −mean over steps of [log π_θ(a_t|s_t) · G_t]
5 Backpropagate, optimizer step, repeat

Code Example (PyTorch + Gymnasium)

import gymnasium as gym
import numpy as np
import torch
from torch import nn

SEED, GAMMA, LR = 42, 0.99, 1e-2
EPISODES_PER_UPDATE, UPDATES = 8, 150

torch.manual_seed(SEED)
env = gym.make("CartPole-v1")
obs_dim, n_actions = env.observation_space.shape[0], env.action_space.n

policy = nn.Sequential(nn.Linear(obs_dim, 64), nn.Tanh(), nn.Linear(64, n_actions))
optimizer = torch.optim.Adam(policy.parameters(), lr=LR)


def returns_to_go(rewards, gamma):
    G, out = 0.0, []
    for r in reversed(rewards):
        G = r + gamma * G
        out.append(G)
    return out[::-1]


def run_episode(seed):
    obs, _ = env.reset(seed=seed)
    log_probs, rewards, done = [], [], False
    while not done:
        logits = policy(torch.as_tensor(obs, dtype=torch.float32))
        dist = torch.distributions.Categorical(logits=logits)
        action = dist.sample()
        obs, reward, terminated, truncated, _ = env.step(action.item())
        log_probs.append(dist.log_prob(action))
        rewards.append(reward)
        done = terminated or truncated
    return log_probs, rewards


history = []
for update in range(UPDATES):
    all_log_probs, all_returns, lengths = [], [], []
    for i in range(EPISODES_PER_UPDATE):
        log_probs, rewards = run_episode(seed=SEED + update * EPISODES_PER_UPDATE + i)
        all_log_probs += log_probs
        all_returns += returns_to_go(rewards, GAMMA)
        lengths.append(len(rewards))

    returns = torch.tensor(all_returns, dtype=torch.float32)
    returns = (returns - returns.mean()) / (returns.std() + 1e-8)   # variance reduction
    loss = -(torch.stack(all_log_probs) * returns).mean()

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    history.append(np.mean(lengths))
    if (update + 1) % 25 == 0:
        print(f"update {update + 1:3d}  mean episode length {history[-1]:.1f}")


def evaluate(episodes=10):
    lengths = []
    for i in range(episodes):
        obs, _ = env.reset(seed=10_000 + i)
        done, steps = False, 0
        while not done:
            with torch.no_grad():
                action = policy(torch.as_tensor(obs, dtype=torch.float32)).argmax().item()
            obs, _, terminated, truncated, _ = env.step(action)
            steps += 1
            done = terminated or truncated
        lengths.append(steps)
    return float(np.mean(lengths))


print("greedy evaluation length:", evaluate())

A random CartPole policy lasts about 20 steps; as REINFORCE trains, the printed mean episode length should rise substantially toward the 500-step cap, but the curve is noisy and depends on the seed, so judge it over several runs.

Checking the Return Calculation by Hand

For rewards [1, 1, 1] and γ = 0.9: G₂ = 1, G₁ = 1 + 0.9 = 1.9, G₀ = 1 + 0.9 × 1.9 = 2.71.

Improvements

Improvement Effect
Learned value baseline Lower variance without bias
Entropy bonus Prevents premature collapse
Larger batches Smoother gradients
Actor-critic / PPO Lower variance and better sample efficiency

Common Pitfalls

  • Detaching log_prob or computing it under torch.no_grad() (no gradient reaches the policy).

  • Using the episode’s total return for every step instead of return-to-go.

  • Forgetting the minus sign (gradient descent on reward).

  • Learning rate too high: the policy collapses to one action.

Interview Tips

  • Walk through collect → returns → loss → step.

  • Explain why the loss is −log π · G.

  • Mention baselines and normalization for variance.

42

Code an epsilon-decreasing exploration strategy for an RL agent.

An epsilon-decreasing strategy starts with a high exploration rate (ε close to 1, nearly random actions) and lowers it over time toward a small floor (0.01-0.05), so the agent explores widely while its value estimates are poor and exploits more as they improve. The common schedules are linear (ε drops by a fixed amount per step until the floor), exponential (ε multiplied by a decay factor each episode, or ε = ε_min + (ε_0 − ε_min)·e^(−kt)), inverse-time (ε = ε_0 / (1 + kt), which satisfies the conditions for convergence of tabular methods), and step-wise schedules. Implement the schedule as a small class with a value(step) method so the agent, logging and tests share one definition, and choose the decay length relative to the total training budget (for DQN, typically the first 10% of steps).

Read full answer

Applies toPython 3.10+, NumPy, Gymnasium 1.x

Schedules

Schedule Formula Notes
Linear ε = max(ε_min, ε_0 − (ε_0 − ε_min) t / T) DQN default; reaches the floor at step T
Exponential (per episode) ε ← max(ε_min, ε × d) Simple; d = 0.995-0.9999
Exponential (continuous) ε = ε_min + (ε_0 − ε_min) e^(−kt) Smooth approach to the floor
Inverse time ε = ε_0 / (1 + kt) GLIE-style: decays to 0 slowly
Step-wise ε halves every N episodes Easy to reason about

Code Example: Schedules and an Agent That Uses Them

import math

import gymnasium as gym
import numpy as np


class LinearDecay:
    def __init__(self, start=1.0, end=0.05, decay_steps=10_000):
        self.start, self.end, self.decay_steps = start, end, decay_steps

    def value(self, step):
        frac = min(step / self.decay_steps, 1.0)
        return self.start + frac * (self.end - self.start)


class ExponentialDecay:
    def __init__(self, start=1.0, end=0.01, rate=1e-3):
        self.start, self.end, self.rate = start, end, rate

    def value(self, step):
        return self.end + (self.start - self.end) * math.exp(-self.rate * step)


class InverseTimeDecay:
    def __init__(self, start=1.0, k=1e-3):
        self.start, self.k = start, k

    def value(self, step):
        return self.start / (1 + self.k * step)


for sched in (LinearDecay(), ExponentialDecay(), InverseTimeDecay()):
    print(type(sched).__name__, [round(sched.value(t), 3) for t in (0, 1_000, 5_000, 20_000)])
# LinearDecay [1.0, 0.905, 0.525, 0.05]
# ExponentialDecay [1.0, 0.374, 0.017, 0.01]
# InverseTimeDecay [1.0, 0.5, 0.167, 0.048]


class EpsilonGreedyQAgent:
    def __init__(self, n_states, n_actions, schedule, alpha=0.5, gamma=0.95, seed=42):
        self.Q = np.zeros((n_states, n_actions))
        self.schedule, self.alpha, self.gamma = schedule, alpha, gamma
        self.rng = np.random.default_rng(seed)
        self.steps = 0

    def act(self, s, greedy=False):
        eps = 0.0 if greedy else self.schedule.value(self.steps)
        if self.rng.random() < eps:
            return int(self.rng.integers(self.Q.shape[1]))
        best = np.flatnonzero(self.Q[s] == self.Q[s].max())
        return int(self.rng.choice(best))

    def learn(self, s, a, r, s2, terminated):
        target = r + (0.0 if terminated else self.gamma * self.Q[s2].max())
        self.Q[s, a] += self.alpha * (target - self.Q[s, a])
        self.steps += 1                                   # schedule advances per environment step


env = gym.make("FrozenLake-v1", is_slippery=False)
agent = EpsilonGreedyQAgent(env.observation_space.n, env.action_space.n, LinearDecay(decay_steps=5_000))
for episode in range(1_000):
    s, _ = env.reset(seed=episode)
    done = False
    while not done:
        a = agent.act(s)
        s2, r, terminated, truncated, _ = env.step(a)
        agent.learn(s, a, r, s2, terminated)
        s, done = s2, terminated or truncated
print(round(agent.schedule.value(agent.steps), 3))    # current epsilon after training

By hand: linear at step 1,000 is 1 − 0.1 × 0.95 = 0.905; exponential at 1,000 is 0.01 + 0.99e⁻¹ ≈ 0.374 and at 5,000 is 0.01 + 0.99e⁻⁵ ≈ 0.017; inverse-time at 5,000 is 1/6 ≈ 0.167 and at 20,000 is 1/21 ≈ 0.048.

Choosing a Schedule

Situation Suggestion
DQN on a fixed step budget Linear over the first ~10% of steps to 0.01-0.05
Small tabular problems Exponential per episode, or inverse time
Sparse rewards Slower decay, or add intrinsic exploration
Non-stationary environment Keep a non-zero floor

Common Pitfalls

  • Decaying per episode with a decay rate tuned for per-step decay (decay far too slow or too fast).

  • Reaching the floor before the agent has ever seen a reward.

  • Forgetting to use ε = 0 for evaluation.

  • Resetting the schedule’s step counter when resuming training from a checkpoint.

Interview Tips

  • Show at least linear and exponential schedules with formulas.

  • Explain why high early exploration and a non-zero floor make sense.

  • Mention GLIE conditions for convergence in tabular methods.

43

Implement a policy gradient method with a neural network in PyTorch.

A practical neural policy-gradient implementation is an advantage actor-critic (A2C-style) agent: a shared network body with a policy head that outputs action logits and a value head that estimates V(s); rollouts of fixed length are collected from several vectorized environments; advantages are computed with Generalized Advantage Estimation (GAE); and one combined loss — policy gradient −log π(a|s)·Â, plus a value regression term, minus an entropy bonus — is minimized with Adam and gradient clipping. Compared with plain REINFORCE, the critic baseline and GAE reduce variance, and vectorized environments provide more, less correlated data per update. The code below follows Gymnasium’s vector API, which automatically resets finished sub-environments.

Read full answer

Applies toPython 3.10+, PyTorch 2.x, Gymnasium 1.x (`gym.make_vec`)

Code Example: A2C with GAE on CartPole

import gymnasium as gym
import numpy as np
import torch
from torch import nn

SEED, N_ENVS, N_STEPS, UPDATES = 42, 8, 32, 400
GAMMA, LAM, LR = 0.99, 0.95, 7e-4
torch.manual_seed(SEED)

envs = gym.make_vec("CartPole-v1", num_envs=N_ENVS, vectorization_mode="sync")
obs, _ = envs.reset(seed=SEED)


class ActorCritic(nn.Module):
    def __init__(self, obs_dim, n_actions):
        super().__init__()
        self.body = nn.Sequential(nn.Linear(obs_dim, 64), nn.Tanh(), nn.Linear(64, 64), nn.Tanh())
        self.pi = nn.Linear(64, n_actions)
        self.v = nn.Linear(64, 1)

    def forward(self, x):
        h = self.body(x)
        return torch.distributions.Categorical(logits=self.pi(h)), self.v(h).squeeze(-1)


model = ActorCritic(envs.single_observation_space.shape[0], envs.single_action_space.n)
optimizer = torch.optim.Adam(model.parameters(), lr=LR)
episode_returns, running = [], np.zeros(N_ENVS)

for update in range(UPDATES):
    log_probs, values, entropies, rewards, terminals = [], [], [], [], []
    for _ in range(N_STEPS):
        dist, value = model(torch.as_tensor(obs, dtype=torch.float32))
        actions = dist.sample()
        obs, r, terminated, truncated, _ = envs.step(actions.numpy())
        log_probs.append(dist.log_prob(actions))
        values.append(value)
        entropies.append(dist.entropy())
        rewards.append(torch.as_tensor(r, dtype=torch.float32))
        terminals.append(torch.as_tensor(terminated | truncated, dtype=torch.float32))
        running += r
        for i in np.flatnonzero(terminated | truncated):
            episode_returns.append(running[i])
            running[i] = 0.0

    with torch.no_grad():
        _, next_value = model(torch.as_tensor(obs, dtype=torch.float32))

    # GAE, backwards over the rollout
    advantages = torch.zeros(N_STEPS, N_ENVS)
    gae = torch.zeros(N_ENVS)
    for t in reversed(range(N_STEPS)):
        v_next = next_value if t == N_STEPS - 1 else values[t + 1].detach()
        nonterminal = 1.0 - terminals[t]
        delta = rewards[t] + GAMMA * v_next * nonterminal - values[t].detach()
        gae = delta + GAMMA * LAM * nonterminal * gae
        advantages[t] = gae
    values_t = torch.stack(values)
    returns = advantages + values_t.detach()

    policy_loss = -(torch.stack(log_probs) * advantages).mean()
    value_loss = (returns - values_t).pow(2).mean()
    entropy = torch.stack(entropies).mean()
    loss = policy_loss + 0.5 * value_loss - 0.01 * entropy

    optimizer.zero_grad()
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 0.5)
    optimizer.step()

    if (update + 1) % 100 == 0 and episode_returns:
        print(f"update {update + 1}: mean return of last 20 episodes {np.mean(episode_returns[-20:]):.1f}")
envs.close()

The mean episode return should climb well above the random-policy level of about 20 as training proceeds; the exact values depend on the seed. For simplicity this treats time-limit truncation like termination (no bootstrap). Also, Gymnasium 1.x vector environments reset a finished sub-environment on the following step call, so each episode boundary adds one dummy transition with reward 0; a production implementation (or Stable-Baselines3’s A2C/PPO) masks those steps out and bootstraps truncated episodes from the final observation’s value.

Key Pieces

Piece Purpose
Vectorized envs More data per update, decorrelated samples
Value head Baseline/critic for advantages
GAE (λ = 0.95) Bias-variance balance in advantages
Entropy bonus Keeps exploring
Gradient clipping Stability

Common Pitfalls

  • Letting the policy loss backpropagate into the value estimates (advantages must be detached).

  • Not masking the bootstrap at episode ends.

  • Too large a learning rate: performance collapses after improving.

  • Expecting identical numbers between runs; report several seeds.

Interview Tips

  • Describe the actor and critic heads and the three-term loss.

  • Explain GAE in one sentence.

  • Mention vectorized environments and PPO as the next step.

44

How would you use RL to optimize traffic signals in a simulated city?

I would frame each intersection’s signal controller as an RL agent in a traffic simulator such as SUMO (via its TraCI API or the sumo-rl package, which provides Gymnasium and PettingZoo environments) or CityFlow: the state is the local traffic picture (queue lengths and vehicle counts per incoming lane, current phase and how long it has been active, optionally neighbours’ states), the action is choosing the next green phase (or keeping/switching the current one) every few seconds subject to minimum-green and yellow constraints, and the reward is negative delay — for example the reduction in cumulative waiting time, or negative queue length or pressure. Train with DQN or PPO first on a single intersection, then scale to a network using independent agents with parameter sharing or cooperative multi-agent methods (neighbour observations, MA2C, CoLight-style graph attention). Train on realistic and randomized demand, compare against fixed-time and actuated baselines and max-pressure control on held-out traffic scenarios, and enforce safety constraints in the action layer before any field pilot.

Read full answer

Applies toSUMO + TraCI / `sumo-rl`, Gymnasium/PettingZoo, Stable-Baselines3

MDP Design

Element Choice Notes
State Queue length, vehicle count, approaching vehicles per lane; current phase (one-hot); time in phase Detector-based features are deployable; avoid perfect information
Action Next green phase every 5 s (or keep/switch) Enforce min green, yellow and all-red in the environment
Reward −Δ(total waiting time), −queue length, or −pressure Pressure = upstream − downstream vehicles
Episode 1 h of simulated traffic Several demand profiles
Agents One per intersection Parameter sharing for scale

Algorithms

Option When
DQN / Double DQN Single intersection, discrete phases
PPO with shared parameters Many similar intersections
MA2C, CoLight (graph attention) Coordinating neighbours, green waves
Max-pressure (non-RL) Strong baseline to beat

Code Example: Single Intersection with sumo-rl and Stable-Baselines3

# Requires SUMO installed (SUMO_HOME set) and: pip install sumo-rl stable-baselines3
from stable_baselines3 import DQN
from sumo_rl import SumoEnvironment

env = SumoEnvironment(
    net_file="single-intersection.net.xml",      # road network
    route_file="single-intersection.rou.xml",    # traffic demand
    single_agent=True,
    use_gui=False,
    num_seconds=3600,                            # one simulated hour per episode
    delta_time=5,                                # agent acts every 5 s
    yellow_time=3,
    min_green=5,
    reward_fn="diff-waiting-time",               # reduction in cumulative waiting time
)

model = DQN("MlpPolicy", env, learning_rate=1e-3, buffer_size=50_000,
            exploration_fraction=0.2, exploration_final_eps=0.02, seed=42, verbose=0)
model.learn(total_timesteps=100_000)
model.save("dqn_single_intersection")

The .net.xml and .rou.xml files come from your SUMO scenario (the sumo-rl repository ships example networks). For many intersections, sumo_rl.parallel_env(...) gives a PettingZoo multi-agent environment.

Evaluation

Metric Why
Average travel time and delay Main objective
Queue length, stops per vehicle Congestion and comfort
Throughput Network capacity
Worst-approach delay Fairness to side streets
Emissions / fuel Secondary objective
Robustness Unseen demand, incidents, sensor failures

Deployment Concerns

Simulation-to-real gaps (driver behaviour, detector noise), safety rules enforced outside the agent, fallback to actuated control if detectors fail, explainability for traffic engineers, and gradual pilots at a few intersections with before/after studies.

Common Pitfalls

  • Rewards that starve low-volume approaches (add fairness or maximum-wait terms).

  • Training on one demand pattern only.

  • Using states that cannot be measured with real detectors.

  • Comparing only against fixed-time plans rather than actuated or max-pressure control.

Interview Tips

  • Define state, action and reward clearly, with safety constraints.

  • Start single-intersection, then scale with parameter sharing or MARL.

  • Describe evaluation against strong non-RL baselines and a careful rollout plan.

45

What should you consider when applying RL to real-world robotics?

Real-world robotics breaks many assumptions of RL benchmarks: real interaction is slow, expensive and can damage hardware or hurt people, so the main considerations are sample efficiency, safety, the gap between simulation and reality, and reliable sensing and resets. In practice teams train mostly in simulation (MuJoCo, Isaac Sim/Isaac Lab, PyBullet) with domain randomization of physics, latency, sensor noise and visuals, then transfer to hardware with fine-tuning; they start from demonstrations (behaviour cloning, offline RL) rather than random policies; they design safe action spaces (joint limits, velocity and torque limits, safety controllers that override the policy); they handle partial observability and delays with observation history or recurrent policies; they use smooth rewards and action penalties to avoid jerky motions; they plan automatic resets; and they monitor wear, temperature and failures. Modern systems often combine RL for low-level skills (locomotion, dexterous manipulation) with classical control and learned or language-based planners.

Read full answer

Applies toMuJoCo / Isaac Lab / PyBullet simulation, Gymnasium 1.x, Stable-Baselines3 (SAC, PPO)

Considerations

Area Issue Practice
Sample efficiency Hours of real data are costly Simulation, off-policy RL (SAC), demonstrations, model-based RL
Sim-to-real Physics, latency, sensor gaps Domain randomization, system identification, real-data fine-tuning
Safety Damage, collisions, people Action limits, safety layers/shields, constrained RL, emergency stop
Observation Noise, occlusion, delays Filtering, history stacking, recurrent policies
Action space Torque vs position/velocity control Higher-level actions (end-effector deltas) are easier and safer
Control frequency Policy at 20-100 Hz, low-level PD at 1 kHz Separate policy and controller loops
Rewards Sparse success signals Shaping, demonstrations, HER for goals
Resets Someone must reset the scene Reset policies, reset-free RL, automation
Smoothness Jerky actions wear motors Action-rate penalties, low-pass filters
Generalization New objects, lighting Diverse training, augmentation
Evaluation Real success rate Many trials, confidence intervals

Code Example: A Safety and Smoothness Action Wrapper (Gymnasium)

import gymnasium as gym
import numpy as np


class SafeSmoothActions(gym.ActionWrapper):
    """Clips actions to safe limits and limits how fast they can change."""

    def __init__(self, env, max_abs=0.5, max_delta=0.1):
        super().__init__(env)
        self.max_abs, self.max_delta = max_abs, max_delta
        self.prev = np.zeros(env.action_space.shape, dtype=np.float32)

    def reset(self, **kwargs):
        self.prev[:] = 0.0
        return self.env.reset(**kwargs)

    def action(self, action):
        a = np.clip(action, -self.max_abs, self.max_abs)                       # safe range
        a = np.clip(a, self.prev - self.max_delta, self.prev + self.max_delta)  # rate limit
        self.prev = a.astype(np.float32)
        return self.prev


env = SafeSmoothActions(gym.make("Pendulum-v1"), max_abs=1.0, max_delta=0.25)
obs, _ = env.reset(seed=42)
print(env.action(np.array([2.0])), env.action(np.array([2.0])))   # [0.25] [0.5]

The first commanded torque of 2.0 is first clipped to the safe limit of 1.0 and then rate-limited to 0.25; the next one can rise only to 0.5. The same wrapper runs in simulation and on the robot, so the policy learns under the limits it will face.

Typical Workflow

Stage Activity
1 Build and calibrate the simulator; identify system parameters
2 Collect demonstrations; pretrain with behaviour cloning
3 Train with RL in simulation with domain randomization
4 Evaluate in simulation on held-out randomizations
5 Deploy on hardware with safety limits and supervision
6 Fine-tune with real data (offline-to-online RL), monitor

Common Pitfalls

  • Policies exploiting simulator bugs (contacts, friction) that fail on hardware.

  • Ignoring actuation latency in simulation.

  • Rewarding success only, producing violent motions.

  • Reporting a handful of real trials as a success rate.

Interview Tips

  • Lead with sample efficiency, safety and sim-to-real.

  • Give concrete practices: domain randomization, demonstrations, safety layers.

  • Mention control architecture (policy frequency vs low-level controllers).

46

How can reinforcement learning be used to build an autonomous trading agent?

An RL trading agent is framed as an MDP in which the state contains market features (recent returns, volatility, order-book or technical indicators) plus the agent’s own position, cash and unrealized P&L; the action sets a target position (for example short, flat or long, or a continuous portfolio weight) or places orders; and the reward is the change in portfolio value net of transaction costs, often risk-adjusted (differential Sharpe ratio, drawdown penalties). Training uses a historical-data simulator (backtesting environment) with realistic costs, slippage and position limits, and algorithms such as PPO, SAC or DQN; tasks with clearer structure, such as optimal execution (splitting a large order to minimize impact) and market making, are where RL has the most credible results. The hard part is not the algorithm but non-stationary, low signal-to-noise data: overfitting, look-ahead bias and unrealistic simulators produce backtests that fail live. Rigorous walk-forward evaluation, conservative assumptions and strict risk controls outside the agent are essential.

Read full answer

Applies toPython 3.10+, NumPy, Gymnasium 1.x, Stable-Baselines3

MDP Design

Element Choice
State Window of log returns, volatility, volume features; current position; time of day
Action Discrete target position {−1, 0, +1} or continuous weight in [−1, 1]
Reward position × next return − cost × |position change| (optionally risk-adjusted)
Episode One trading day or a rolling window
Constraints Position and leverage limits, stop-loss, trading hours

Code Example: A Minimal Trading Environment (Gymnasium)

import gymnasium as gym
import numpy as np
from gymnasium import spaces


class SingleAssetEnv(gym.Env):
    """Target position in {-1, 0, +1}; reward = position * next return - costs."""

    def __init__(self, returns, window=10, cost=0.0005):
        super().__init__()
        self.returns, self.window, self.cost = np.asarray(returns, dtype=np.float32), window, cost
        self.action_space = spaces.Discrete(3)                         # 0 short, 1 flat, 2 long
        self.observation_space = spaces.Box(-np.inf, np.inf, shape=(window + 1,), dtype=np.float32)

    def _obs(self):
        past = self.returns[self.t - self.window:self.t]               # only past data: no look-ahead
        return np.append(past, self.position).astype(np.float32)

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        self.t, self.position = self.window, 0.0
        return self._obs(), {}

    def step(self, action):
        new_position = float(action) - 1.0
        trade_cost = self.cost * abs(new_position - self.position)
        self.position = new_position
        reward = self.position * float(self.returns[self.t]) - trade_cost   # return realized after acting
        self.t += 1
        terminated = self.t >= len(self.returns)
        obs = self._obs() if not terminated else np.zeros(self.window + 1, dtype=np.float32)
        return obs, reward, terminated, False, {}


rng = np.random.default_rng(42)
env = SingleAssetEnv(rng.normal(0.0, 0.01, 500))
obs, _ = env.reset(seed=42)
obs, reward, terminated, truncated, _ = env.step(2)                    # go long
print(obs.shape, obs[-1])                                              # (11,) 1.0

With synthetic random-walk returns there is nothing to learn, which makes this a useful sanity check: a trained agent should not show a profit here, and if it does, the environment leaks information. With real data, train PPO("MlpPolicy", env) on one period and evaluate on later, untouched periods.

Evaluation Protocol

Practice Why
Walk-forward splits (train past, test future) No look-ahead
Realistic costs, slippage, latency Backtests otherwise overstate profit
Benchmarks: buy-and-hold, simple rules Show real added value
Risk metrics: Sharpe, max drawdown, turnover Profit alone is misleading
Many seeds and periods Results are noisy
Paper trading before capital Catch live-data differences

Common Pitfalls

  • Look-ahead bias (features using future prices, normalizing with full-period statistics).

  • Ignoring transaction costs, leading to hyperactive policies.

  • Survivorship bias in the asset universe.

  • Overfitting by tuning on the test period.

  • Assuming the market does not react to the agent’s trades (impact).

Interview Tips

  • Define state, action and reward with costs and risk.

  • Emphasize walk-forward evaluation and realistic simulation.

  • Mention execution and market making as stronger RL use cases, and risk controls outside the agent.

47

How is reinforcement learning applied to personalization and recommendations?

Reinforcement learning treats recommendation as sequential decision-making: the system (agent) chooses what to show a user (action) given what it knows about the user and context (state), observes the response — clicks, watch time, purchases, returns (reward) — and learns a policy that maximizes long-term value such as retention or satisfaction rather than the next click alone. In practice there is a spectrum. Contextual bandits (LinUCB, Thompson sampling) handle one-step decisions with exploration and are the most widely deployed form (news article selection, ranking of promotions, notification timing). Full RL models the effect of recommendations on future user state: slate recommendation (SlateQ at YouTube), session-based policies, notification and email cadence, and long-term engagement optimization (REINFORCE with off-policy correction for YouTube recommendations, Chen et al. 2019). Because live exploration is risky, these systems rely heavily on logged data, off-policy evaluation (inverse propensity scoring, doubly robust estimators) and careful A/B testing.

Read full answer

Applies toPython 3.10+, NumPy

Formulation

Element Example
State User features, recent interactions, session context, time
Action Item, ranked slate, notification yes/no, discount level
Reward Click, watch time, purchase; delayed: return visits, churn
Policy Scoring model + exploration
Horizon One step (bandit) or session/lifetime (RL)

Bandits vs Full RL

Aspect Contextual bandit Full RL
Models future effects No Yes
Data needs Moderate High
Stability Good Harder
Typical use Article/promo selection, ad creatives Session optimization, long-term engagement, notification policy

Code Example: Thompson Sampling for Item Selection (NumPy)

import numpy as np

rng = np.random.default_rng(42)
true_ctr = np.array([0.04, 0.05, 0.07, 0.03])       # unknown click-through rates
alpha = np.ones(4)                                  # Beta(1, 1) prior per item
beta = np.ones(4)

shown = np.zeros(4, dtype=int)
for user in range(20_000):
    sampled = rng.beta(alpha, beta)                 # sample a plausible CTR for each item
    item = int(np.argmax(sampled))                  # show the item that looks best in this sample
    click = rng.random() < true_ctr[item]
    alpha[item] += click
    beta[item] += 1 - click
    shown[item] += 1

print(shown)                                        # most impressions go to item 2 (CTR 0.07)
print(np.round(alpha / (alpha + beta), 3))          # posterior mean CTRs

Thompson sampling explores items it is uncertain about and shifts traffic to the best item as evidence accumulates. A contextual version replaces the per-item Beta posteriors with a Bayesian model of CTR given user features.

Off-Policy Evaluation (Before Going Live)

Estimator Idea
IPS Reweight logged rewards by π_new(a|x) / π_logged(a|x)
Self-normalized IPS Lower variance
Direct method Predict rewards with a model
Doubly robust Combine model and IPS; unbiased if either is right

These need logged propensities (the probability with which the old policy chose each item), so log them from day one.

Risks

Feedback loops and popularity bias, filter bubbles, clickbait from short-term rewards, delayed and noisy rewards, fairness to item providers, and privacy. Use long-term metrics, diversity constraints and guardrail metrics in experiments.

Common Pitfalls

  • Optimizing clicks only and harming long-term satisfaction.

  • Not logging action propensities, which rules out off-policy evaluation.

  • Training on logged data without correcting for the old policy’s bias.

  • Exploring in ways that hurt user experience (cap exploration, explore within safe candidates).

Interview Tips

  • Map the problem to state, action and reward with a long-term objective.

  • Distinguish contextual bandits from full RL and say which is used more.

  • Mention off-policy evaluation and feedback-loop risks.

48

In what ways can reinforcement learning be used in healthcare?

Healthcare has many sequential decisions — choosing and adjusting treatments over time — so RL is studied for dynamic treatment regimes: sepsis fluid and vasopressor dosing in intensive care (the “AI Clinician” work), mechanical ventilation and sedation weaning, insulin dosing and closed-loop glucose control, chemotherapy and radiotherapy scheduling, HIV and chronic disease therapy sequencing, and adaptive clinical trial designs. Other uses include personalized mobile-health interventions (just-in-time reminders, often with contextual bandits), resource allocation such as bed and staff scheduling, and robotic surgery assistance. Because experimenting on patients is unacceptable, almost all work uses offline RL on electronic health records, with conservative algorithms (CQL, BCQ, IQL), off-policy evaluation, and clinician review. The main obstacles are confounding and missing data in records, the difficulty of defining rewards (survival, quality of life, side effects), distribution shift between hospitals, interpretability, fairness across patient groups, and regulatory requirements; deployed systems are decision support, not autonomous agents.

Read full answer

Applies toOffline RL concepts; Python tooling such as d3rlpy for offline RL

Applications

Area Decision Reward signal Typical approach
Sepsis management Fluids, vasopressor doses every 4 h Survival, organ function Offline RL on ICU records (MIMIC)
Diabetes Insulin dose Time in glucose range, no hypoglycaemia Simulator (UVA/Padova) + safe RL
Ventilation Settings and weaning timing Successful extubation Offline RL
Oncology Dose and schedule Tumour control vs toxicity Model-based RL, simulations
Mobile health When and what to prompt Activity, adherence Contextual bandits (micro-randomized trials)
Operations Beds, staff, appointments Wait times, utilization RL with simulators

Why Offline RL and Its Risks

Challenge Mitigation
No online exploration Offline RL with conservative value estimates
Policies choose actions rare in data Behaviour regularization, action support constraints
Confounding (sicker patients get more treatment) Careful state design, causal methods
Evaluating a new policy Off-policy evaluation (WIS, doubly robust), clinician review
Reward specification Clinical outcome definitions with experts; multiple objectives
Distribution shift between sites External validation

Code Example: Off-Policy Evaluation with Weighted Importance Sampling (NumPy)

import numpy as np

# Logged trajectories: for each step, behaviour-policy prob of the logged action,
# target-policy prob of the same action, and the episode's outcome reward.
episodes = [
    {"b": [0.5, 0.5], "pi": [0.9, 0.8], "reward": 1.0},
    {"b": [0.5, 0.5], "pi": [0.1, 0.2], "reward": 0.0},
    {"b": [0.25, 0.5], "pi": [0.5, 0.5], "reward": 1.0},
]

weights = np.array([np.prod(np.array(e["pi"]) / np.array(e["b"])) for e in episodes])
rewards = np.array([e["reward"] for e in episodes])
print(np.round(weights, 2))                              # [2.88 0.08 2.  ]
wis = (weights * rewards).sum() / weights.sum()
print(round(wis, 3))                                     # 0.984
print(round(rewards.mean(), 3))                          # 0.667 (behaviour policy's average)

By hand: weights are (0.9/0.5)(0.8/0.5) = 2.88, (0.1/0.5)(0.2/0.5) = 0.08 and (0.5/0.25)(0.5/0.5) = 2; WIS = (2.88 + 2) / (2.88 + 0.08 + 2) = 4.88 / 4.96 ≈ 0.984. The target policy looks better than the logged one (0.667), but the estimate rests on just two heavily weighted episodes, which is exactly the situation in which off-policy estimates mislead.

Common Pitfalls

  • Trusting a high off-policy estimate based on a few heavily weighted episodes (check effective sample size).

  • Using rewards that ignore side effects or quality of life.

  • Ignoring that records reflect clinicians’ decisions and biases.

  • Presenting a research policy as ready for autonomous use.

Interview Tips

  • Give specific applications (sepsis, insulin, ventilation, mobile health).

  • Stress offline RL, off-policy evaluation and safety.

  • Discuss reward design, confounding, fairness and regulation.

49

How would you tune hyperparameters for a reinforcement learning model?

RL hyperparameter tuning is harder than in supervised learning because results vary a lot between random seeds, each trial is an expensive training run, and the evaluation signal (episode return) is noisy — so tune systematically with a fixed budget, multiple seeds and a separate evaluation protocol. Start from published defaults for the algorithm and environment class (for example the RL Baselines3 Zoo configurations), identify the few parameters that matter most — learning rate, discount factor γ, batch/rollout size, entropy coefficient or exploration schedule, network size, and algorithm-specific ones such as PPO’s clip range and GAE λ or DQN’s replay size and target update interval — and search them with Bayesian optimization (Optuna’s TPE) using early stopping/pruning of bad trials. Score each configuration by the mean evaluation return over several seeds (or a robust statistic such as the interquartile mean), check the best configurations on fresh seeds, and report variability rather than one best run.

Read full answer

Applies toOptuna 3.x/4.x, Stable-Baselines3, Gymnasium 1.x

Important Hyperparameters

Parameter Typical range Effect
Learning rate 1e-5 to 1e-3 (log scale) Most sensitive; too high causes collapse
γ 0.9-0.999 Effective horizon ≈ 1/(1 − γ)
Rollout / batch size 128-4096 steps; 32-512 batch Gradient noise vs speed
PPO clip range 0.1-0.3 Update size
GAE λ 0.9-0.99 Bias vs variance of advantages
Entropy coefficient 0-0.01 Exploration
DQN ε schedule Decay over 5-20% of training Exploration
Replay size, target update 1e4-1e6; 1k-10k steps or τ = 0.005 Stability
Network size 2 × 64 to 2 × 256 Capacity
Observation/reward normalization On/off Often decisive

Code Example: Optuna + Stable-Baselines3 with Pruning

import gymnasium as gym
import numpy as np
import optuna
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy


def objective(trial):
    params = {
        "learning_rate": trial.suggest_float("learning_rate", 1e-5, 1e-3, log=True),
        "gamma": trial.suggest_float("gamma", 0.95, 0.999),
        "n_steps": trial.suggest_categorical("n_steps", [256, 512, 1024, 2048]),
        "ent_coef": trial.suggest_float("ent_coef", 1e-8, 1e-2, log=True),
        "clip_range": trial.suggest_float("clip_range", 0.1, 0.3),
        "gae_lambda": trial.suggest_float("gae_lambda", 0.9, 0.99),
    }
    scores = []
    for seed in (0, 1, 2):                                   # several seeds per configuration
        env = gym.make("CartPole-v1")
        model = PPO("MlpPolicy", env, seed=seed, verbose=0, **params)
        for chunk in range(4):
            model.learn(total_timesteps=10_000, reset_num_timesteps=False)
            mean_reward, _ = evaluate_policy(model, gym.make("CartPole-v1"), n_eval_episodes=5)
            if seed == 0:
                trial.report(mean_reward, step=chunk)
                if trial.should_prune():                     # stop clearly bad trials early
                    raise optuna.TrialPruned()
        mean_reward, _ = evaluate_policy(model, gym.make("CartPole-v1"), n_eval_episodes=10)
        scores.append(mean_reward)
    return float(np.mean(scores))


study = optuna.create_study(direction="maximize",
                            sampler=optuna.samplers.TPESampler(seed=42),
                            pruner=optuna.pruners.MedianPruner(n_warmup_steps=1))
study.optimize(objective, n_trials=30)
print(study.best_params)

After the search, retrain the best configuration (and the runners-up) on new seeds and compare their mean ± standard deviation or confidence intervals; the top trial of a noisy search is often partly lucky.

Practical Workflow

Step Action
1 Reproduce a known result with default hyperparameters
2 Fix evaluation: deterministic policy, held-out seeds, N episodes
3 Search 3-6 key parameters, log scale for rates
4 Prune early, run in parallel
5 Re-evaluate top configurations on fresh seeds
6 Report variability (IQM, confidence intervals across ≥ 5 seeds)

Common Pitfalls

  • Choosing the best of many single-seed runs (selection on noise).

  • Tuning on the evaluation seeds used for the final report.

  • Changing environment wrappers or reward scale between trials without noticing.

  • Ignoring normalization, which often matters more than any single parameter.

Interview Tips

  • Name the most sensitive parameters and their typical ranges.

  • Describe Optuna/TPE with pruning and multi-seed scoring.

  • Stress robust evaluation and reporting across seeds.

50

How would you design an RL agent to learn optimal strategies for a game?

I would design the agent by first formalizing the game as an environment — state (or observation), legal actions, reward, episode end, and whether it is single-agent, two-player zero-sum or multi-player, perfect or imperfect information — then choose a method matched to that structure: DQN or PPO for single-player arcade-style games, self-play with tree search and a policy-value network (AlphaZero-style) for perfect-information board games, and self-play with population-based training or counterfactual-regret methods for imperfect-information games like poker. The environment should be fast and follow a standard interface (Gymnasium for single-agent, PettingZoo for multi-agent), with action masking for illegal moves. Rewards start with the true outcome (win/loss or score) and add careful potential-based shaping only if learning stalls. Training uses vectorized environments, curricula or self-play against past versions to avoid strategy cycling, and evaluation measures strength against fixed baselines (random, scripted, earlier checkpoints) with Elo ratings.

Read full answer

Applies toGymnasium 1.x, PettingZoo, Stable-Baselines3 / sb3-contrib (`MaskablePPO`), PyTorch 2.x

Design Steps (Example: Connect Four)

Step Decision for Connect Four
Game type Two-player, zero-sum, perfect information, deterministic
Observation 2 × 6 × 7 planes: own discs, opponent discs (from the current player’s view)
Actions 7 columns; mask full columns
Reward +1 win, −1 loss, 0 draw, at game end
Algorithm PPO self-play with masking, or AlphaZero (MCTS + policy-value net)
Opponents Pool of past checkpoints + scripted heuristics
Evaluation Win rate vs random, vs one-step lookahead bot, Elo across checkpoints

Method by Game Type

Game type Method
Single-player arcade / puzzle DQN, PPO; curriculum over levels
Perfect-information two-player AlphaZero / MuZero (MCTS + network), self-play
Imperfect information (poker) CFR variants, ReBeL, deep CFR
Real-time multi-agent PPO self-play with league training (AlphaStar, OpenAI Five)
Huge action spaces Action decomposition, masking, autoregressive heads

Code Example: Masked Legal Moves and a Self-Play Opponent Pool

import numpy as np
import torch
from torch import nn

ROWS, COLS = 6, 7


def legal_mask(board):
    """board: (6, 7) with 0 empty, 1/-1 players. A column is legal if its top cell is empty."""
    return board[0] == 0


class PolicyValueNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.body = nn.Sequential(nn.Conv2d(2, 32, 3, padding=1), nn.ReLU(),
                                  nn.Conv2d(32, 32, 3, padding=1), nn.ReLU(), nn.Flatten())
        self.policy = nn.Linear(32 * ROWS * COLS, COLS)
        self.value = nn.Sequential(nn.Linear(32 * ROWS * COLS, 1), nn.Tanh())

    def forward(self, planes, mask):
        h = self.body(planes)
        logits = self.policy(h).masked_fill(~mask, float("-inf"))   # illegal moves get probability 0
        return torch.distributions.Categorical(logits=logits), self.value(h).squeeze(-1)


def to_planes(board, player):
    return torch.tensor(np.stack([board == player, board == -player]), dtype=torch.float32).unsqueeze(0)


torch.manual_seed(42)
net = PolicyValueNet()
board = np.zeros((ROWS, COLS), dtype=int)
board[:, 3] = [1, -1, 1, -1, 1, -1]            # column 3 is full
mask = torch.tensor(legal_mask(board)).unsqueeze(0)
dist, value = net(to_planes(board, player=1), mask)
print(dist.probs[0, 3].item(), mask.sum().item())     # 0.0 6

opponent_pool = []                                    # past checkpoints for self-play
opponent_pool.append({k: v.clone() for k, v in net.state_dict().items()})
rng = np.random.default_rng(42)
opponent_weights = opponent_pool[rng.integers(len(opponent_pool))]   # sample an opponent each game

Masking with −∞ before the softmax gives illegal columns exactly zero probability, so the agent never wastes experience on invalid moves. Sampling opponents from a pool of past checkpoints keeps the agent from overfitting to its latest self and cycling between strategies.

Common Pitfalls

  • Penalizing illegal moves instead of masking them (slow learning).

  • Pure self-play against only the latest version (strategy cycling, forgetting).

  • Observations not from the current player’s perspective in two-player games.

  • Evaluating only against itself; use fixed external baselines and Elo.

Interview Tips

  • Start with the game’s structure (players, information, determinism).

  • Choose the algorithm family to match, and justify it.

  • Cover masking, self-play opponent pools, reward design and evaluation.

51

Propose an RL framework for energy management in a smart grid.

I would build the energy management system as a constrained, possibly multi-agent RL problem trained in a grid simulator: agents control flexible resources — battery storage charge/discharge, EV charging rates, heat pumps and HVAC setpoints, flexible industrial loads, and dispatchable generation — and observe the state of the system (prices or tariffs, demand and renewable forecasts, battery state of charge, temperatures, network voltages); the reward combines negative operating cost, penalties for unmet demand and comfort violations, battery degradation, emissions, and network constraint violations. Training uses historical and synthetic scenarios in simulators such as CityLearn (building demand response), Grid2Op (transmission operation) or pandapower-based distribution models; algorithms are SAC or PPO for continuous control, with safety layers or constrained RL (Lagrangian methods) for hard limits. A centralized-training, decentralized-execution multi-agent design scales to many buildings or feeders. Deployment adds forecasting, a rule-based or optimization fallback (model predictive control), human oversight, and staged rollout, and the RL policy must beat an MPC baseline to be worth it.

Read full answer

Applies toGymnasium 1.x, Stable-Baselines3 (SAC/PPO); CityLearn, Grid2Op, pandapower simulators

MDP Design (Building Cluster with Batteries and EVs)

Element Design
State Hour, day type, price now and forecast, PV forecast, load forecast, battery SoC, EV SoC and departure times, indoor temperature, feeder load
Actions Battery power in [−P_max, P_max], EV charge rates, HVAC setpoint offsets (continuous)
Reward −(energy cost + carbon cost) − λ₁ comfort violation − λ₂ unmet EV energy − λ₃ battery degradation − λ₄ peak demand above limit
Constraints SoC bounds, charger limits, voltage/thermal limits, comfort band
Time step 15 min to 1 h, episodes of one day to one year
Agents One per building, shared critic with grid-level information during training

Architecture

Layer Role
Forecasting Load, PV, price forecasts feed the state
RL policy Proposes setpoints
Safety layer Projects actions into feasible region (SoC, power, voltage)
Fallback MPC or rule-based controller if forecasts or policy fail
Monitoring Cost, peak, comfort, constraint violations vs baseline

Code Example: Battery Arbitrage Environment with a Safety Projection (Gymnasium)

import gymnasium as gym
import numpy as np
from gymnasium import spaces


class BatteryEnv(gym.Env):
    """Hourly battery control against a price signal. Action in [-1, 1] = fraction of max power."""

    def __init__(self, prices, capacity_kwh=10.0, max_kw=5.0, efficiency=0.95):
        super().__init__()
        self.prices = np.asarray(prices, dtype=np.float32)
        self.capacity, self.max_kw, self.eff = capacity_kwh, max_kw, efficiency
        self.action_space = spaces.Box(-1.0, 1.0, shape=(1,), dtype=np.float32)
        self.observation_space = spaces.Box(-np.inf, np.inf, shape=(3,), dtype=np.float32)

    def _obs(self):
        hour = self.t % 24
        return np.array([self.soc / self.capacity, self.prices[self.t], hour / 23], dtype=np.float32)

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        self.t, self.soc = 0, 0.5 * self.capacity
        return self._obs(), {}

    def step(self, action):
        power = float(np.clip(action[0], -1, 1)) * self.max_kw            # + charge, - discharge (kW for 1 h)
        # Safety projection: never exceed the battery's energy limits
        power = min(power, (self.capacity - self.soc) / self.eff)
        power = max(power, -self.soc * self.eff)
        self.soc += power * self.eff if power > 0 else power / self.eff
        grid_energy = power                                                 # kWh bought (+) or sold (-)
        reward = -self.prices[self.t] * grid_energy - 0.01 * abs(power)     # cost + degradation proxy
        self.t += 1
        terminated = self.t >= len(self.prices)
        obs = self._obs() if not terminated else np.zeros(3, dtype=np.float32)
        return obs, float(reward), terminated, False, {"soc": self.soc}


prices = np.array([0.10] * 6 + [0.30] * 6 + [0.15] * 6 + [0.40] * 6)        # cheap night, expensive evening
env = BatteryEnv(prices)
obs, _ = env.reset(seed=42)
_, r, *_ , info = env.step(np.array([1.0]))
print(round(info["soc"], 2), round(r, 3))                                   # 9.75 -0.55

By hand: starting at 5 kWh, a full-power charge is capped at (10 − 5)/0.95 ≈ 5.26 kW, so the requested 5 kW is allowed; the battery stores 5 × 0.95 = 4.75 kWh (SoC 9.75) and the reward is −0.10 × 5 − 0.01 × 5 = −0.55. A trained SAC agent (SAC("MlpPolicy", env)) should learn to charge in cheap hours and discharge in the expensive evening; compare it with the optimal schedule from a linear program on the same prices.

Evaluation

Metric Baseline
Energy cost, peak demand, emissions Rule-based (time-of-use) and MPC with perfect/forecast information
Comfort and EV service violations Must not increase
Constraint violations Zero tolerated in deployment
Robustness Unseen weather, price spikes, forecast errors

Common Pitfalls

  • Ignoring battery degradation, so the agent cycles the battery constantly.

  • Training on perfect forecasts and deploying with noisy ones.

  • Relying on reward penalties alone for hard grid constraints (use a safety layer).

  • Not comparing with MPC, which is often hard to beat.

Interview Tips

  • Define state, action, reward and constraints concretely.

  • Address safety (projection, fallback) and multi-agent scaling.

  • Describe evaluation against rule-based and MPC baselines.

52

How would you set up an RL environment for chess?

A chess RL environment needs a correct game engine for the rules (the python-chess library handles legal moves, check, castling, en passant, promotion and all draw rules), an observation encoding of the board from the perspective of the player to move, an action space with masking of illegal moves, a reward at the end of the game (+1 win, 0 draw, −1 loss), and a multi-agent or self-play interface so the agent can play against itself and against fixed opponents. Following AlphaZero, the observation is a stack of 8×8 planes (one per piece type and colour, plus planes for side to move, castling rights, repetition counts and the 50-move counter, often with a few past positions), and the action space is a fixed set of 4,672 move encodings (73 move types from each of 64 squares: queen-like moves, knight moves and underpromotions), of which only the legal ones are unmasked. Training typically uses self-play with Monte Carlo Tree Search guided by a policy-value network, or PPO self-play for a simpler baseline; evaluation measures Elo against engines such as Stockfish at fixed strengths.

Read full answer

Applies toPython 3.10+, `python-chess` 1.x, Gymnasium 1.x or PettingZoo (`pettingzoo.classic.chess_v6`), NumPy

Environment Components

Component Choice
Rules engine python-chess (legal moves, terminal detection)
Observation 8×8×N planes from the mover’s perspective (board flipped for Black)
Actions AlphaZero 4,672-move encoding, or index into the list of all UCI moves
Action mask Legal moves only
Reward +1 / 0 / −1 at game end, from the mover’s perspective
Termination Checkmate, stalemate, insufficient material, threefold repetition, 50-move rule
Truncation Move cap (e.g. 512 plies) to bound episode length
Opponents Self-play pool, random, Stockfish at low depth

Code Example: Board Encoding and Legal-Move Mask (python-chess + NumPy)

import chess
import numpy as np

PIECES = [chess.PAWN, chess.KNIGHT, chess.BISHOP, chess.ROOK, chess.QUEEN, chess.KING]


def encode(board):
    """12 piece planes from the side-to-move's perspective + 1 plane for castling rights."""
    planes = np.zeros((13, 8, 8), dtype=np.float32)
    me, them = board.turn, not board.turn
    for i, piece in enumerate(PIECES):
        for color, offset in ((me, 0), (them, 6)):
            for sq in board.pieces(piece, color):
                rank, file = divmod(sq, 8)
                if me == chess.BLACK:
                    rank = 7 - rank                       # flip so the mover always plays "up"
                planes[offset + i, rank, file] = 1.0
    planes[12, 0, 0] = float(board.has_kingside_castling_rights(me))
    planes[12, 0, 1] = float(board.has_queenside_castling_rights(me))
    return planes


# Simple action space: index over all from-square x to-square pairs (64 * 64) + promotion handled by queen default
def move_to_index(move):
    return move.from_square * 64 + move.to_square


def legal_mask(board):
    mask = np.zeros(64 * 64, dtype=bool)
    for move in board.legal_moves:
        mask[move_to_index(move)] = True
    return mask


board = chess.Board()
print(encode(board).shape, int(encode(board)[:12].sum()))   # (13, 8, 8) 32
print(legal_mask(board).sum())                               # 20 legal moves at the start


def reward_for(board, mover):
    outcome = board.outcome(claim_draw=True)
    if outcome is None:
        return 0.0, False
    if outcome.winner is None:
        return 0.0, True
    return (1.0 if outcome.winner == mover else -1.0), True


board = chess.Board()
for uci in ["f2f3", "e7e5", "g2g4", "d8h4"]:                 # fool's mate
    board.push_uci(uci)
print(reward_for(board, chess.BLACK))                        # (1.0, True)

This simple 4,096-index space folds all promotions into one index per square pair; a full implementation adds underpromotion indices (or uses AlphaZero’s 4,672 encoding). For a ready-made multi-agent version, PettingZoo’s chess_v6 environment provides observations and action masks in this style.

Training Approach

Approach Notes
AlphaZero (MCTS + policy-value net) Strongest; needs large compute for self-play
Supervised pretraining on master games Fast start; then RL fine-tuning
PPO self-play with masks Simpler baseline; weaker than search-based methods
Evaluation Elo vs Stockfish levels, tournaments between checkpoints

Common Pitfalls

  • Hand-coding chess rules (bugs in castling, en passant, repetition).

  • Encoding the board in absolute colours so the network must learn two mirror-image games.

  • Penalizing illegal moves instead of masking them.

  • No move cap, leading to extremely long early games.

Interview Tips

  • Describe observation planes, the action encoding and masking.

  • Explain self-play with MCTS (AlphaZero) and evaluation by Elo.

  • Mention using an established rules library.

53

What are recent advances in multi-agent reinforcement learning?

Multi-agent reinforcement learning (MARL) studies several agents learning at the same time in a shared environment, cooperatively, competitively or both; major advances of recent years include: centralized training with decentralized execution (CTDE) through value factorization (VDN, QMIX, QPLEX) and centralized critics (MADDPG, COMA), the surprisingly strong performance of simple on-policy methods with parameter sharing (MAPPO, IPPO); large-scale self-play and league training that reached top human level in StarCraft II (AlphaStar), Dota 2 (OpenAI Five) and Stratego (DeepNash); sequence-model approaches that treat agents as tokens (Multi-Agent Transformer); population-based methods for zero-shot coordination with unseen partners (Other-Play, Fictitious Co-Play); and combining language models with RL for negotiation and cooperation (Meta’s CICERO in Diplomacy). Active research areas include communication learning, credit assignment, scalability to hundreds of agents with mean-field and graph-based methods, robustness to adversarial agents, and LLM-based multi-agent systems. Benchmarks such as SMAC/SMACv2, Google Research Football, Melting Pot, Overcooked and PettingZoo environments drive progress.

Read full answer

Applies toPettingZoo, PyTorch 2.x; libraries such as EPyMARL, BenchMARL, RLlib, JaxMARL

Key Advances

Area Methods Idea
Value factorization VDN, QMIX, QPLEX Team Q = monotonic combination of per-agent Qs; decentralized argmax
Centralized critics MADDPG, COMA, MAPPO Critic sees global state during training only
Simple strong baselines IPPO, MAPPO with parameter sharing Careful PPO tuning rivals complex methods
Self-play leagues AlphaStar, OpenAI Five, DeepNash Diverse opponent populations avoid cycling
Zero-shot coordination Other-Play, FCP, Hidden-Utility Self-Play Work with unseen human/AI partners
Transformers Multi-Agent Transformer Sequence modelling over agents’ actions
Language + RL CICERO Planning plus dialogue with humans
Scale Mean-field MARL, graph neural networks Hundreds of agents
Hardware acceleration JAX environments (JaxMARL) Orders of magnitude faster training

Core Challenges

Challenge Why
Non-stationarity Other agents keep changing their policies
Credit assignment Which agent caused the team reward?
Scalability Joint action space grows exponentially
Partial observability Each agent sees only part of the state
Equilibrium selection Many equilibria; coordination failures

Code Example: QMIX-Style Monotonic Mixer (PyTorch)

import torch
from torch import nn


class MonotonicMixer(nn.Module):
    """Combines per-agent Q-values into Q_tot with non-negative weights produced from the global state."""

    def __init__(self, n_agents, state_dim, embed=32):
        super().__init__()
        self.n_agents, self.embed = n_agents, embed
        self.hyper_w1 = nn.Linear(state_dim, n_agents * embed)
        self.hyper_b1 = nn.Linear(state_dim, embed)
        self.hyper_w2 = nn.Linear(state_dim, embed)
        self.hyper_b2 = nn.Sequential(nn.Linear(state_dim, embed), nn.ReLU(), nn.Linear(embed, 1))

    def forward(self, agent_qs, state):
        b = agent_qs.shape[0]
        w1 = torch.abs(self.hyper_w1(state)).view(b, self.n_agents, self.embed)   # abs keeps it monotonic
        hidden = torch.nn.functional.elu(torch.bmm(agent_qs.unsqueeze(1), w1).squeeze(1) + self.hyper_b1(state))
        w2 = torch.abs(self.hyper_w2(state)).unsqueeze(-1)
        return (torch.bmm(hidden.unsqueeze(1), w2).squeeze(-1) + self.hyper_b2(state)).squeeze(-1)


torch.manual_seed(42)
mixer = MonotonicMixer(n_agents=3, state_dim=10)
qs = torch.randn(4, 3, requires_grad=True)
q_tot = mixer(qs, torch.randn(4, 10))
q_tot.sum().backward()
print(q_tot.shape, bool((qs.grad >= 0).all()))     # torch.Size([4]) True

Because ∂Q_tot/∂Q_i ≥ 0, each agent maximizing its own Q_i also maximizes the team value, so agents can act independently at execution time while training uses the global state.

Common Pitfalls

  • Treating other agents as part of a stationary environment.

  • Comparing methods with different tuning budgets (MAPPO results showed tuning matters a lot).

  • Self-play without opponent diversity.

Interview Tips

  • Name CTDE, value factorization and centralized critics.

  • Mention landmark systems (AlphaStar, OpenAI Five, CICERO) and MAPPO’s lesson.

  • Explain the core challenges of non-stationarity and credit assignment.

54

How does curriculum learning work in reinforcement learning?

Curriculum learning in RL trains an agent on a sequence of tasks that starts easy and becomes harder, so the agent gets learning signal early and builds skills that transfer to the final, hard task, instead of facing a sparse-reward problem it almost never solves by chance. Curricula can vary the start state (begin near the goal and move away, “reverse curriculum”), the goal distance, environment parameters (terrain roughness, obstacle density, opponent strength), the reward density, or the task itself (walk, then run, then jump). They can be hand-designed with performance thresholds, or automatic: teacher algorithms pick tasks where the agent’s learning progress is highest (Teacher-Student curriculum learning), Prioritized Level Replay replays levels with high value error, goal generators propose goals of intermediate difficulty (GoalGAN), and unsupervised environment design (PAIRED, ACCEL) generates environments at the edge of the agent’s ability. Self-play is a natural curriculum because the opponent improves with the agent.

Read full answer

Applies toPython 3.10+, NumPy, Gymnasium 1.x

Curriculum Types

Type What changes Example
Start-state curriculum Distance from goal at reset Reverse curriculum for robot assembly
Goal curriculum Goal difficulty GoalGAN: goals the agent solves 10-90% of the time
Environment parameters Physics, terrain, obstacles Legged locomotion on increasingly rough terrain
Opponent strength Opponent skill Self-play, league training
Task sequence Sub-skills Navigate → pick → place
Reward density Shaped → sparse Remove shaping gradually
Automatic (UED/PLR) Generated or replayed levels PAIRED, ACCEL, Prioritized Level Replay

Code Example: Success-Rate-Driven Curriculum Wrapper (Gymnasium)

from collections import deque

import gymnasium as gym
import numpy as np


class DistanceCurriculum(gym.Wrapper):
    """Grid-style task whose start distance from the goal grows when the agent succeeds often enough."""

    def __init__(self, env, levels=(1, 2, 4, 8), promote_at=0.8, window=50):
        super().__init__(env)
        self.levels, self.level = levels, 0
        self.promote_at = promote_at
        self.recent = deque(maxlen=window)

    @property
    def start_distance(self):
        return self.levels[self.level]

    def record(self, success):
        self.recent.append(float(success))
        full = len(self.recent) == self.recent.maxlen
        if full and np.mean(self.recent) >= self.promote_at and self.level < len(self.levels) - 1:
            self.level += 1                    # promote to a harder level
            self.recent.clear()

    def reset(self, **kwargs):
        options = dict(kwargs.pop("options", None) or {})
        options["start_distance"] = self.start_distance   # the wrapped env reads this option
        return self.env.reset(options=options, **kwargs)


curriculum = DistanceCurriculum(gym.make("CartPole-v1"))   # stand-in env; a real one would use the option
for _ in range(50):
    curriculum.record(success=True)
print(curriculum.start_distance)                            # 2  promoted after 50 successes
for _ in range(50):
    curriculum.record(success=False)
print(curriculum.start_distance)                            # 2  stays while performance is low

The wrapper passes the current difficulty to the environment through reset(options=...) and promotes the agent only after a full window of episodes above 80% success. Some curricula also demote when performance drops, and many mix in a share of easier tasks to avoid forgetting.

Design Guidelines

Guideline Reason
Keep tasks at intermediate difficulty Maximum learning signal
Promote on measured success, not on a fixed schedule Adapts to the agent
Replay earlier tasks occasionally Prevents forgetting
Always evaluate on the final target task The curriculum is a means, not the goal

Common Pitfalls

  • Promoting too fast, so the agent never masters the basics.

  • Easy tasks that teach habits that hurt on the hard task.

  • Forgetting easy cases after moving on.

  • Reporting curriculum-level returns instead of target-task performance.

Interview Tips

  • Explain why curricula help with sparse rewards.

  • Give hand-designed and automatic examples (PLR, PAIRED, GoalGAN).

  • Mention self-play as an implicit curriculum.

55

Explain the concept of meta-reinforcement learning.

Meta-reinforcement learning (“learning to learn” for RL) trains an agent on a distribution of related tasks so that it can adapt to a new task from the same distribution with very little experience — a few episodes or even within one episode — rather than learning from scratch. There are two main families. Gradient-based meta-RL (MAML, Finn et al. 2017) learns initial network parameters from which a few policy-gradient steps on a new task give good performance; the outer loop optimizes performance after adaptation. Context- or memory-based meta-RL (RL², Duan et al. 2016; L2RL, Wang et al.) uses a recurrent or transformer policy that receives past observations, actions and rewards, so the learning algorithm is encoded in its activations: it explores, infers the task, and exploits, without weight updates at test time. Inference-based methods such as PEARL learn a latent task variable from context with a probabilistic encoder, which supports efficient off-policy training and posterior sampling for exploration. Applications include robots adapting to new payloads or damage, personalized recommendations and fast adaptation in games; challenges are designing task distributions, meta-overfitting, and exploration for task identification.

Read full answer

Applies toPyTorch 2.x, Gymnasium 1.x (conceptual code)

Families

Family Adapts by Examples Pros Cons
Gradient-based A few gradient steps from a meta-learned initialization MAML, ProMP Generalizes beyond training tasks somewhat Second-order gradients, high variance
Recurrent / context Hidden-state updates from history RL², L2RL, transformer agents (AdA) Fast, no test-time gradients Hard to train; limited to similar tasks
Inference-based Latent task posterior PEARL, VariBAD Sample-efficient, principled exploration Needs good task encoders

Setup

Level Contents
Task distribution p(T) E.g. goal positions, payloads, bandit arm probabilities
Inner loop Adapt to one task (gradient steps or recurrent updates)
Outer loop Update meta-parameters to maximize post-adaptation return
Meta-test New tasks from p(T); measure return after k episodes

Code Example: RL²-Style Policy Input (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)
N_ARMS, HIDDEN = 5, 64


class RL2Policy(nn.Module):
    """Recurrent policy whose input includes the previous action and reward, so it can infer the task."""

    def __init__(self, n_actions, obs_dim=1):
        super().__init__()
        self.gru = nn.GRU(obs_dim + n_actions + 1, HIDDEN, batch_first=True)
        self.pi = nn.Linear(HIDDEN, n_actions)
        self.n_actions = n_actions

    def forward(self, obs, prev_action, prev_reward, h=None):
        a_onehot = nn.functional.one_hot(prev_action, self.n_actions).float()
        x = torch.cat([obs, a_onehot, prev_reward.unsqueeze(-1)], dim=-1).unsqueeze(1)
        out, h = self.gru(x, h)
        return torch.distributions.Categorical(logits=self.pi(out[:, 0])), h


policy = RL2Policy(N_ARMS)
arm_probs = torch.rand(N_ARMS)                    # one sampled task: a new bandit
h, prev_a, prev_r = None, torch.zeros(1, dtype=torch.long), torch.zeros(1)
for t in range(10):                               # hidden state is NOT reset between pulls within a task
    dist, h = policy(torch.zeros(1, 1), prev_a, prev_r, h)
    prev_a = dist.sample()
    prev_r = (torch.rand(1) < arm_probs[prev_a]).float()
print(h.shape)                                    # torch.Size([1, 1, 64])

During meta-training, the policy is trained (for example with PPO) on many sampled bandits, with returns summed across all pulls of a task, so it learns an exploration-then-exploitation strategy in its hidden state. At meta-test time it faces a new bandit and adapts with no gradient updates. The hidden state is reset only when a new task is sampled.

Common Pitfalls

  • Resetting the recurrent state between episodes of the same task (it forgets what it inferred).

  • Narrow task distributions that make the meta-learner memorize instead of adapt.

  • Evaluating on tasks from the training set.

  • Comparing with baselines that were not given the same number of adaptation episodes.

Interview Tips

  • Define meta-RL as fast adaptation across a task distribution.

  • Contrast MAML (gradients) with RL² (memory) and PEARL (latent inference).

  • Give a concrete use case such as robot adaptation.

56

What are the challenges of safe RL in sensitive domains like healthcare or driving?

Safe reinforcement learning aims to learn good policies while avoiding unacceptable outcomes during both training and deployment, and in domains like healthcare and autonomous driving that is hard because standard RL learns by trial and error, maximizes a proxy reward, and offers no built-in guarantees. The main challenges are: safe exploration (you cannot try dangerous actions on patients or roads, so learning must happen offline or in simulation); specifying safety (rewards miss rare but catastrophic events, and penalties can be traded off against reward); distribution shift (situations unseen in training, such as new patient populations or unusual road users, where the policy may behave unpredictably); rare events and long tails that need far more testing than average-case metrics show; offline evaluation of a new policy from logged data; sim-to-real gaps; verification and interpretability for clinicians, engineers and regulators; and accountability and regulation (medical device rules, automotive safety standards). Techniques include constrained MDPs with Lagrangian methods (CPO, PPO-Lagrangian), safety layers and shields that override unsafe actions, conservative offline RL, uncertainty estimation and abstention, risk-sensitive objectives (CVaR), formal verification for limited components, human-in-the-loop operation, and staged deployment.

Read full answer

Applies toSafe RL concepts; constrained RL libraries (e.g. OmniSafe, Safety-Gymnasium)

Challenges and Techniques

Challenge Example Techniques
Safe exploration Cannot test overdoses on patients Offline RL, simulators, shielded exploration
Specifying safety Reward ignores near-misses Constrained MDPs (cost limits), explicit safety costs
Hard constraints Never exceed max dose, never run a red light Safety layers, action projection, rule-based shields
Distribution shift New hospital, snow on the road Uncertainty estimation, OOD detection, fallback policies
Rare catastrophic events Child running into the street Scenario-based testing, adversarial testing, CVaR objectives
Evaluation Policy never tried live Off-policy evaluation, simulation, shadow mode
Interpretability Clinician must understand advice Interpretable policies, explanations, decision support only
Accountability Who is responsible for an error? Human oversight, audit logs, regulation

Constrained RL Formulation

Maximize E[Σ γᵗ r_t] subject to E[Σ γᵗ c_t] ≤ d, where c_t is a safety cost. The Lagrangian method alternates policy updates on r − λc with a dual update λ ← max(0, λ + η(Ĵ_c − d)), so λ rises while the cost exceeds its limit.

Code Example: Lagrange Multiplier Update and a Safety Shield (Python)

def update_lambda(lam, measured_cost, cost_limit, lr=0.05):
    return max(0.0, lam + lr * (measured_cost - cost_limit))


lam = 0.0
for cost in [30.0, 30.0, 20.0, 10.0]:          # episode safety costs, limit 25
    lam = update_lambda(lam, cost, cost_limit=25.0)
    print(round(lam, 2), end=" ")
print()                                         # 0.25 0.5 0.25 0.0


def shield(proposed_dose_mg, max_dose_mg=10.0, last_dose_mg=0.0, max_increase_mg=2.0):
    """Rule-based safety layer: clip to the maximum dose and to a maximum increase per step."""
    safe = min(proposed_dose_mg, max_dose_mg, last_dose_mg + max_increase_mg)
    return max(safe, 0.0)


print(shield(15.0, last_dose_mg=5.0))          # 7.0

By hand: λ = 0 + 0.05 × 5 = 0.25, then 0.5, then 0.5 + 0.05 × (−5) = 0.25, then max(0, 0.25 − 0.75) = 0. The shield clips a proposed 15 mg to the 5 + 2 = 7 mg allowed increase; it is illustrative only, as real dosing limits come from clinical protocols.

Deployment Practices

Practice Purpose
Decision support, not autonomy Human makes the final decision
Shadow mode Compare recommendations with real decisions without acting
Staged rollout with kill switch Limit exposure
Monitoring of safety metrics and drift Detect problems early
External validation Different sites, populations, conditions

Common Pitfalls

  • Encoding safety only as a reward penalty that can be outweighed.

  • Trusting simulation results without real-world validation.

  • Ignoring fairness: safe on average but unsafe for subgroups.

  • Treating rare-event performance as covered by average metrics.

Interview Tips

  • List safe exploration, specification, shift, rare events and evaluation.

  • Describe constrained MDPs and safety layers.

  • Emphasize human oversight and staged deployment.

57

Why does interpretability matter in RL, and how can it be achieved?

Interpretability matters in reinforcement learning because RL agents make sequences of decisions, often with neural policies, in settings where people must trust, debug, audit or override them — and because RL agents are known to find unexpected strategies, including reward hacking, that are invisible in aggregate return curves. Understanding why an agent acts helps catch exploits and spurious correlations before deployment, satisfies regulators and domain experts (clinicians, operators, traders), supports safe human-agent collaboration, and speeds up debugging. It can be achieved at different levels: intrinsically interpretable policies (decision trees distilled from neural policies, as in VIPER; linear or rule-based policies; programmatic policies); post-hoc explanations of neural policies (saliency maps over observations, SHAP for state features, attention weights); value and reward decomposition (separate Q-heads per reward component, so you can see that “avoid collision” dominates a choice); outcome-based explanations (predicted future states, counterfactual “what if” rollouts); summaries of behaviour (representative trajectories, critical states where the action matters most); and mechanistic analysis of internal representations.

Read full answer

Applies toPyTorch 2.x, scikit-learn 1.5+, Gymnasium 1.x

Methods

Method Explains Notes
Policy distillation to a decision tree (VIPER) Whole policy Verifiable, small trees; may lose performance
Reward decomposition Why an action is preferred Needs multi-component rewards
Saliency maps (gradient, perturbation) Which inputs mattered Can be noisy or misleading
SHAP / feature attribution State-feature importance Works on vector observations
Counterfactual rollouts “What if the agent did X?” Needs a simulator
Critical-state identification When decisions matter Large Q-value spread across actions
Behaviour summaries Typical strategy Selected trajectories for review

Code Example: Distilling a Policy into a Decision Tree (scikit-learn)

import gymnasium as gym
import numpy as np
from sklearn.tree import DecisionTreeClassifier, export_text


def teacher_policy(obs):
    """Stand-in for a trained neural policy on CartPole: push toward the falling side."""
    x, x_dot, theta, theta_dot = obs
    return int(theta + 0.5 * theta_dot > 0)


env = gym.make("CartPole-v1")
states, actions = [], []
for ep in range(30):
    obs, _ = env.reset(seed=ep)
    done = False
    while not done:
        a = teacher_policy(obs)
        states.append(obs)
        actions.append(a)
        obs, _, terminated, truncated, _ = env.step(a)
        done = terminated or truncated

X, y = np.array(states), np.array(actions)
tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X, y)
print(f"agreement with teacher: {tree.score(X, y):.3f}")
print(export_text(tree, feature_names=["x", "x_dot", "theta", "theta_dot"]))

The tree’s rules (thresholds on the pole angle and angular velocity) can be read and checked by a person. VIPER improves on this simple behaviour cloning by collecting states visited by the tree itself (DAgger-style) and weighting states by how much the action matters (Q-value gaps). Always check the tree’s return in the environment, not only its agreement with the teacher.

Code Example: Critical States from Q-Values

import numpy as np

q_values = np.array([[1.0, 1.1, 0.9],      # actions barely matter here
                     [5.0, -3.0, 0.0],     # wrong action is very costly here
                     [2.0, 2.0, 1.5]])
importance = q_values.max(axis=1) - q_values.min(axis=1)
print(np.round(importance, 3).tolist(), int(np.argmax(importance)))   # [0.2, 8.0, 0.5] 1

States with a large spread are where explanations and human review add the most value.

Common Pitfalls

  • Treating saliency maps as proof of what the agent “knows”; validate with interventions.

  • Distilled policies that look simple but perform worse; report their performance.

  • Explaining average behaviour while rare critical states go unexamined.

Interview Tips

  • Give reasons: trust, debugging, regulation, reward-hacking detection.

  • Name intrinsic (trees, decomposition) and post-hoc (saliency, counterfactuals) methods.

  • Show a concrete technique such as policy distillation.

58

What ethical concerns arise when deploying reinforcement learning systems?

Reinforcement learning raises ethical concerns beyond those of predictive models because RL systems act in the world, optimize objectives over time and learn from interaction: they can cause harm through their actions, pursue a mis-specified reward in ways people did not intend, change the behaviour of the people and markets they interact with, and experiment on users while exploring. Key concerns include: safety and harm (physical harm from robots or vehicles, clinical harm from treatment policies); reward mis-specification and manipulation (engagement-optimizing recommenders that promote addictive or extreme content, or nudge users against their interests); fairness (policies that serve some groups worse, often inherited from historical data or proxy rewards); consent and experimentation (exploration means some users receive deliberately suboptimal or untested treatments); transparency and accountability (who is responsible when an autonomous policy causes harm, and can decisions be explained and contested?); privacy (logging detailed interaction data); feedback loops and societal effects (algorithmic pricing that can tacitly collude, trading agents that amplify volatility); and dual use (autonomous weapons, surveillance). Responsible practice combines careful objective design, constraints, human oversight, audits, impact assessments, transparency to users, and staged, monitored deployment.

Read full answer

Applies toRL system design and governance

Concerns and Mitigations

Concern Example Mitigation
Physical/clinical harm Robot collision, dosing error Safety constraints, shields, human oversight, testing
Reward mis-specification Recommender maximizes watch time via outrage Long-term well-being metrics, guardrail metrics, value-sensitive design
Manipulation Dark-pattern nudging, persuasion Policies against manipulative actions, user control, review
Fairness Worse service for minority users Subgroup metrics, fairness constraints, audits
Exploration on people Random treatments in a live system Informed consent where applicable, ethics review, bounded exploration
Accountability Unclear responsibility for decisions Documented ownership, audit logs, appeal processes
Transparency Users unaware an adaptive system steers them Disclosure, explanations
Privacy Fine-grained behavioural logs Data minimization, consent, differential privacy
Market/societal effects Pricing agents converge on high prices Monitoring, regulation-aware design, simulation studies
Dual use Military autonomy Use policies, export controls, refusal

A Pre-Deployment Checklist

Question Evidence
What exactly is optimized, and who might it harm? Reward spec review with stakeholders
What can the agent do that it must never do? Hard constraints, tested
How does performance differ across groups? Subgroup evaluation
How is exploration bounded and disclosed? Exploration policy, ethics/IRB review where relevant
Who can override or switch it off? Runbook, kill switch
How will harms be detected after launch? Monitoring, incident reporting

Common Pitfalls

  • Treating ethics as a post-launch review instead of a design input.

  • Measuring success only by the optimized reward.

  • Assuming offline evaluation captures how people will adapt to the system.

Interview Tips

  • Explain why acting and optimizing over time makes RL ethics distinctive.

  • Cover reward mis-specification, fairness, exploration on people and accountability.

  • Offer concrete mitigations and governance practices.

59

How can the alignment problem be tackled in reinforcement learning?

The alignment problem in RL is the gap between what we want an agent to do and what it actually optimizes: we specify a reward that is only a proxy for our goals, and a capable optimizer finds ways to score highly on the proxy that we did not intend (specification gaming, reward hacking), may learn goals that only coincidentally matched the reward during training (goal misgeneralization), and may pursue instrumentally useful behaviour such as avoiding correction. Approaches to narrowing the gap include: learning rewards from people rather than hand-coding them — inverse RL from demonstrations and preference-based RL, which became RLHF for language models (a reward model trained on human comparisons, then policy optimization such as PPO with a KL penalty to a reference model), plus AI-feedback variants such as Constitutional AI / RLAIF and direct methods such as DPO; regularization against over-optimization (KL constraints, early stopping, reward model ensembles, uncertainty penalties); scalable oversight (debate, recursive reward modelling, AI-assisted evaluation) so humans can supervise tasks they cannot judge directly; conservatism and impact limits (avoid side effects, constrained RL, corrigibility research); interpretability to inspect what objectives a model has learned; and red-teaming and evaluation for hacking, deception and misgeneralization under distribution shift.

Read full answer

Applies toRL and RLHF concepts; PyTorch 2.x for the illustration

Failure Modes

Failure Example
Specification gaming Boat circles to collect points instead of finishing a race
Reward model over-optimization Language model output scores higher on the reward model but humans rate it worse
Goal misgeneralization Agent trained with coins at the level’s end learns “go right” rather than “get the coin” (CoinRun)
Side effects Agent breaks things en route because the reward ignores them
Sycophancy Chat model agrees with users because raters liked agreement

Approaches

Approach Idea
Preference learning / RLHF Reward model from pairwise human comparisons
KL-regularized optimization Stay near a trusted reference policy
Reward model ensembles and uncertainty Penalize disagreement to resist over-optimization
RLAIF / Constitutional AI AI feedback guided by written principles
Direct preference optimization (DPO) Optimize the policy on preferences without an explicit RL loop
Scalable oversight (debate, recursive reward modelling) Help humans evaluate hard tasks
Impact regularization, constraints Limit side effects
Interpretability and evaluations Inspect learned goals; red-team for hacking and deception

Code Example: Bradley-Terry Reward Model Loss and KL-Penalized Reward (PyTorch)

import torch
import torch.nn.functional as F

# Reward model scores for (preferred, rejected) responses in a batch of 3 comparisons
r_chosen = torch.tensor([2.0, 0.5, 1.0])
r_rejected = torch.tensor([1.0, 0.5, 2.0])
loss = -F.logsigmoid(r_chosen - r_rejected).mean()      # Bradley-Terry preference loss
print(round(loss.item(), 4))                             # 0.7732

# RLHF-style reward: reward model score minus a KL penalty to the reference policy
beta = 0.1
rm_score = torch.tensor(3.0)
logp_policy, logp_reference = torch.tensor(-10.0), torch.tensor(-14.0)
shaped = rm_score - beta * (logp_policy - logp_reference)   # per-sample KL estimate
print(round(shaped.item(), 2))                           # 2.6

By hand: the three logsigmoid terms are log σ(1) ≈ −0.3133, log σ(0) ≈ −0.6931 and log σ(−1) ≈ −1.3133, so the loss is (0.3133 + 0.6931 + 1.3133) / 3 ≈ 0.7732. The third pair, where the reward model prefers the rejected answer, contributes most of the loss.

Common Pitfalls

  • Treating the reward model as ground truth (it is a learned, exploitable proxy).

  • Optimizing too long against a fixed reward model.

  • Assuming good training performance shows the intended goal was learned (test under distribution shift).

Interview Tips

  • Define the alignment gap with specification gaming and goal misgeneralization examples.

  • Explain RLHF with the reward model and KL penalty, and its limits.

  • Mention scalable oversight, interpretability and evaluation.

60

Why do fairness and bias matter in reinforcement learning?

Fairness matters in reinforcement learning because RL systems allocate opportunities and resources over time — loans, prices, jobs shown, medical treatments, police or inspection resources, content exposure — and an agent that maximizes total reward can systematically serve some groups worse, either by inheriting historical bias from logged data and proxy rewards or by creating new bias through feedback loops. RL adds dynamics that one-shot fairness does not capture: decisions change future states and data (a group that gets fewer loans builds less credit history, so the agent learns even less about it), exploration may be spent unevenly, and short-term fairness constraints can have different long-term effects (Liu et al., 2018, “Delayed Impact of Fair Machine Learning”). Addressing it involves defining fairness for the application (equal exposure, equal outcomes, equal error rates, individual fairness), measuring per-group returns and outcomes over time, adding fairness constraints to the MDP (constrained RL), rewarding long-term group welfare, ensuring adequate exploration for all groups, correcting biased logged data in offline training, and auditing with simulations of long-term effects.

Read full answer

Applies toRL design and evaluation; Python/NumPy for metrics

Sources of Bias in RL

Source Example
Historical logged data Offline policy learns past discriminatory lending decisions
Proxy rewards Clicks reward sensational content from certain creators
Feedback loops Less exposure → fewer interactions → even less exposure
Unequal exploration Agent stops exploring options for a smaller group early
State representation Features act as proxies for protected attributes
Simulator bias Simulated users don’t reflect real populations

Approaches

Approach How
Constrained RL Maximize reward s.t. per-group metric gaps ≤ ε
Reward design Include group welfare or exposure terms
Fair exploration Guarantee minimum exploration per group/item
Data correction Reweighting, causal adjustments for offline data
Long-term simulation Study effects on group states over many steps
Auditing Per-group returns, error rates, exposure over time

Code Example: Exposure Fairness in a Recommender Slate (NumPy)

import numpy as np

# Exposure each item gets from a ranking: position bias 1 / log2(rank + 1)
positions = np.arange(1, 6)
position_weight = 1 / np.log2(positions + 1)
item_group = np.array(["A", "A", "A", "B", "B"])        # ranking order: group A items on top

exposure = {g: position_weight[item_group == g].sum() for g in ("A", "B")}
share = {g: round(float(e / sum(exposure.values())), 3) for g, e in exposure.items()}
print(share)                                           # {'A': 0.723, 'B': 0.277}

# Interleaved ranking: A, B, A, B, A
interleaved = np.array(["A", "B", "A", "B", "A"])
exposure2 = {g: position_weight[interleaved == g].sum() for g in ("A", "B")}
print({g: round(float(e / sum(exposure2.values())), 3) for g, e in exposure2.items()})   # {'A': 0.64, 'B': 0.36}

By hand: position weights are 1, 0.631, 0.5, 0.431 and 0.387 (sum 2.948); group A in the top three gets 2.131 / 2.948 ≈ 0.723; interleaving gives A (positions 1, 3, 5) 1.887 / 2.948 ≈ 0.64. An RL ranker can be constrained or rewarded so that each group’s exposure tracks its relevance over time rather than compounding early advantages.

Common Pitfalls

  • Measuring fairness only at one time step.

  • Removing protected attributes and assuming the policy is fair (proxies remain).

  • Constraints that look fair short term but harm the group long term.

  • Ignoring fairness to suppliers/creators as well as users.

Interview Tips

  • Explain how sequential decisions and feedback loops create bias.

  • Name fairness definitions and constrained RL as a remedy.

  • Cite delayed impact and stress long-term, per-group evaluation.

61

What role does reinforcement learning play in NLP?

Reinforcement learning plays its largest role in NLP today in fine-tuning language models toward goals that are hard to express as supervised labels: Reinforcement Learning from Human Feedback (RLHF) trains a reward model on human preference comparisons and then optimizes the language model against it (InstructGPT, ChatGPT, Claude), and RL with verifiable rewards trains models on maths and coding tasks where answers can be checked automatically, which has driven recent reasoning models (for example DeepSeek-R1 with the GRPO algorithm). In these setups the language model is the policy, the prompt and text generated so far are the state, each token (or the full response) is the action, and the reward comes at the end. RL is also used to optimize non-differentiable sequence metrics such as BLEU or ROUGE (self-critical sequence training for summarization and captioning), for task-oriented dialogue policies, for learning when to call tools or retrieve documents, and in text-based games. Because RL fine-tuning can exploit a learned reward, it is paired with a KL penalty to a reference model, and simpler preference methods without an RL loop, such as DPO, are now widely used alongside it.

Read full answer

Applies toPyTorch 2.x; Hugging Face TRL (PPO, DPO, GRPO trainers)

Uses of RL in NLP

Use Reward Method
RLHF for assistants Reward model from human comparisons PPO with KL penalty; DPO as RL-free alternative
Reasoning (maths, code) Verifiable correctness, unit tests GRPO, PPO
Summarization / translation ROUGE, BLEU, learned quality metrics, human preferences Self-critical sequence training, RLHF
Dialogue systems Task success, user satisfaction Policy learning over dialogue acts
Tool use and retrieval Answer quality, cost RL over tool calls
Text games / agents Game score DQN-style or LLM agents

How the RLHF Loop Works

Step Detail
1. Supervised fine-tuning Train on demonstrations of good responses
2. Reward model Pairwise comparisons → Bradley-Terry loss
3. RL fine-tuning Maximize RM score − β·KL(π ‖ π_ref) with PPO or GRPO
4. Evaluation Human and automatic evaluations, safety tests

Code Example: Group-Relative Advantages (GRPO-Style) and KL-Penalized Reward

import numpy as np

# Four sampled answers to the same maths prompt, scored by a verifier (1 = correct)
rewards = np.array([1.0, 0.0, 1.0, 0.0])
advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-8)   # compare answers within the group
print(np.round(advantages, 3).tolist())                             # [1.0, -1.0, 1.0, -1.0]

# PPO-style RLHF reward for one response: reward-model score minus a KL penalty to the reference model
beta = 0.05
rm_score = 1.2
logp_policy = np.array([-1.0, -0.5, -2.0])      # per-token log-probs of the sampled response
logp_ref = np.array([-1.5, -0.7, -2.1])
kl_estimate = (logp_policy - logp_ref).sum()    # 0.5 + 0.2 + 0.1 = 0.8
print(round(rm_score - beta * kl_estimate, 3))  # 1.16

GRPO removes the critic network by using the group mean as a baseline: correct answers get positive advantages and incorrect ones negative, and the policy gradient raises the probability of the tokens in better answers. The KL term keeps the model close to its starting point so it does not drift into text that fools the reward model.

Challenges

Challenge Mitigation
Reward hacking (verbose, sycophantic outputs) KL penalty, reward model ensembles, length control, better data
Huge action space (vocabulary) Start from a pretrained policy; RL as fine-tuning only
Sparse, sequence-level reward Per-token KL shaping, value models or group baselines
Cost of human feedback AI feedback (RLAIF), verifiable rewards
Instability Small learning rates, clipping, careful monitoring

Common Pitfalls

  • Optimizing against a reward model for too long (over-optimization).

  • Forgetting the KL penalty or reference model.

  • Using automatic metrics like BLEU as the only reward; models learn to game them.

Interview Tips

  • Explain RLHF step by step and the role of the KL penalty.

  • Mention verifiable rewards and GRPO for reasoning, and DPO as an RL-free alternative.

  • Map NLP generation onto states, actions and rewards.

62

How is reinforcement learning used to improve data centre energy efficiency?

Data centres spend a large share of their energy on cooling, and RL is used to control cooling equipment — chillers, cooling towers, pumps, fans, water and air setpoints — in response to IT load, weather and equipment state, minimizing energy use (or PUE, power usage effectiveness) while keeping temperatures and other conditions within safe limits. The best-known case is Google and DeepMind: in 2016 they reported that a machine-learning system recommending cooling settings cut cooling energy by up to 40%, and in 2018 they moved to an autonomous system that directly adjusts the plant under safety constraints, reporting average savings of about 30% over its first months. Their approach used neural networks trained on sensor history to predict future energy use and temperatures, choosing actions that minimize predicted energy subject to constraints, with multiple safety layers: actions verified against constraints, confidence checks, a local control system that can override, and operators able to take over at any time. Other uses include workload scheduling and placement (shifting batch jobs to cooler times or greener power), server power management (DVFS, sleep states) and demand response with the grid.

Read full answer

Applies toRL for building/HVAC control; Gymnasium-style simulators (e.g. Sinergym with EnergyPlus)

Problem Formulation

Element Cooling control
State IT load, outside temperature and humidity, supply/return temperatures, pump and fan speeds, chiller status, time
Actions Setpoints: chilled water temperature, cooling tower fan speed, number of chillers, air handler setpoints
Reward −energy (or −PUE) − penalty for temperature/humidity violations
Constraints Server inlet temperature limits, equipment operating ranges, rate of change limits
Control interval Minutes

Safety Architecture

Layer Purpose
Action constraint checks Reject unsafe setpoints before sending
Uncertainty/confidence estimates Do nothing if the model is unsure
Rate limits Change setpoints gradually
Local control fallback Existing building management system takes over on failure
Human override Operators can switch to manual
Continuous monitoring Compare with baseline performance

Code Example: Reward with Comfort Constraints and a Safe Setpoint Filter

import numpy as np


def cooling_reward(energy_kwh, inlet_temps_c, limit_c=27.0, penalty=10.0):
    violation = np.clip(np.asarray(inlet_temps_c) - limit_c, 0, None).sum()
    return -energy_kwh - penalty * violation


def safe_setpoint(proposed_c, current_c, low=6.0, high=14.0, max_step=0.5):
    """Chilled-water setpoint: keep within equipment range and change at most max_step per interval."""
    bounded = min(max(proposed_c, low), high)
    return float(np.clip(bounded, current_c - max_step, current_c + max_step))


print(round(cooling_reward(120.0, [24.5, 26.0, 27.4]), 6))   # -124.0  (0.4 C over the limit costs 4)
print(safe_setpoint(16.0, current_c=10.0))           # 10.5
print(safe_setpoint(9.8, current_c=10.0))            # 9.8

By hand: one rack inlet is 0.4 °C over the 27 °C limit, so the penalty is 10 × 0.4 = 4 and the reward is −124. The filter turns a large proposed jump to 16 °C into a 0.5 °C step.

Lessons from Practice

Lesson Detail
Start with recommendations Operators apply suggestions before autonomy
Use models trained on historical sensor data Offline/model-based learning before live control
Constrain hard Safety limits enforced outside the learned model
Measure carefully Normalize savings for weather and load

Common Pitfalls

  • Optimizing energy while letting server inlet temperatures drift.

  • Evaluating savings without adjusting for weather and IT load changes.

  • Exploring live with large setpoint changes.

  • Neglecting sensor failures and drift in the inputs.

Interview Tips

  • Describe the state, actions, reward and constraints of cooling control.

  • Cite the DeepMind/Google results as reported figures, with their safety layers.

  • Mention workload scheduling and grid demand response as other uses.

64

What are the challenges of deploying RL models in production?

Deploying reinforcement learning is harder than deploying supervised models because an RL policy acts, changes the data it will later see, and was usually trained in a simulator or on logged data that differs from reality. The main challenges: offline evaluation (you must estimate how a new policy performs before it acts, using simulators or off-policy evaluation, both of which can be wrong); simulation-to-real and distribution shift (dynamics, users or markets differ from training and change over time); safety and constraints (exploration and mistakes have real costs); reward mis-specification (production metrics reveal that the proxy reward rewarded the wrong behaviour); delayed and noisy feedback (long-term outcomes such as retention take weeks to observe); non-reproducibility and instability (results vary by seed, and retraining can produce a very different policy); online learning risks (continual updates can degrade quickly); latency and infrastructure (real-time inference, logging states, actions and propensities); and explainability and governance. Successful deployments use conservative policies, safety layers and fallbacks, shadow mode and gradual rollout with A/B tests, strong monitoring, and logged propensities for future off-policy evaluation.

Read full answer

Applies toRL systems engineering (MLOps for RL)

Challenges and Practices

Challenge Practice
Can’t safely test online Simulators, off-policy evaluation (IPS, doubly robust), shadow mode
Sim-to-real gap Domain randomization, calibrate simulators with real data, fine-tuning
Distribution shift Monitoring of state distributions, retraining triggers, OOD detection
Safety Action constraints, rule-based guards, fallback policy, kill switch
Reward/metric mismatch Guardrail metrics, long-term metrics, stakeholder review
Delayed feedback Surrogate metrics validated against long-term outcomes
Instability across retrains Multiple seeds, policy regression tests, champion-challenger
Exploration in production Bounded ε, exploration budget, randomized but logged
Logging Store state, action, action probability, reward, model version
Latency Small networks, batching, distilled policies

Code Example: Production Action Logging and a Guarded Policy Call

import json
import time
from dataclasses import asdict, dataclass

import numpy as np


@dataclass
class DecisionLog:
    request_id: str
    model_version: str
    state: list
    action: int
    propensity: float           # probability of the chosen action: needed for off-policy evaluation
    fallback_used: bool
    ts: float


def decide(policy_probs_fn, state, rng, fallback_action=0, allowed=None, model_version="v1"):
    try:
        probs = np.asarray(policy_probs_fn(state), dtype=float)
        if allowed is not None:
            probs = probs * allowed
        if not np.isfinite(probs).all() or probs.sum() <= 0:
            raise ValueError("invalid policy output")
        probs = probs / probs.sum()
        action = int(rng.choice(len(probs), p=probs))
        log = DecisionLog("req-1", model_version, list(map(float, state)), action, float(probs[action]), False, time.time())
    except Exception:
        log = DecisionLog("req-1", model_version, list(map(float, state)), fallback_action, 1.0, True, time.time())
    return log.action, json.dumps(asdict(log))


rng = np.random.default_rng(42)
action, record = decide(lambda s: [0.2, 0.8, 0.0], [0.5, 1.0], rng, allowed=np.array([1, 1, 0]))
print(action in (0, 1), json.loads(record)["fallback_used"])      # True False
action, record = decide(lambda s: [np.nan, 1.0], [0.5, 1.0], rng)
print(action, json.loads(record)["fallback_used"])                # 0 True

Every decision logs the action’s probability and the model version, so later policies can be evaluated off-policy from production data, and any invalid model output falls back to a known safe action.

Rollout Stages

Stage Exposure
Offline evaluation None
Shadow mode Policy computes actions, the old system acts
Canary / small A/B 1-5% of traffic with guardrails
Gradual ramp Increase if metrics hold
Full deployment + monitoring Champion-challenger for future versions

Common Pitfalls

  • Not logging propensities, which blocks off-policy evaluation later.

  • Retraining on data generated by the policy itself without correcting for its bias.

  • No fallback when the model service fails.

  • Judging success on short-term metrics only.

Interview Tips

  • Contrast RL deployment with supervised deployment (actions change the data).

  • Cover offline evaluation, safety, shift and monitoring.

  • Describe a staged rollout with shadow mode and A/B tests.

65

What are common pitfalls when scaling reinforcement learning applications?

Scaling RL — to bigger environments, more agents, more compute or production traffic — fails in recurring ways: the simulator becomes the bottleneck, so expensive GPUs sit idle waiting for environment steps; distributed actors generate stale, off-policy data that destabilizes on-policy algorithms; hyperparameters tuned at small scale (learning rate, batch size, entropy) stop working at large batch sizes; reward hacking and simulator exploits appear once the agent is strong enough to find them; results are dominated by seed variance, so small-scale wins do not replicate; and engineering problems (non-determinism, memory blow-ups from replay buffers, bugs in vectorized resets, logging overload) multiply. Other pitfalls are treating truncation as termination at scale, normalization statistics that differ between actors, evaluating only on training environments, and scaling compute instead of fixing a weak reward or representation. Remedies include profiling throughput (steps per second at each stage), GPU-accelerated or batched simulators, architectures designed for asynchrony (IMPALA with V-trace, Ape-X, SEED RL), careful scaling of batch size and learning rate, many seeds with robust statistics, and strong evaluation and monitoring.

Read full answer

Applies toDistributed RL systems (RLlib, Sample Factory, EnvPool, Isaac Lab, JAX environments)

Pitfalls and Remedies

Pitfall Symptom Remedy
Simulator bottleneck Low GPU utilization Vectorized/GPU simulators (EnvPool, Isaac Lab, Brax), more CPU actors
Stale data in distributed actors Instability, worse returns Off-policy corrections (V-trace), sync more often, bound staleness
Hyperparameters don’t transfer Collapse at large batch Retune learning rate/batch; LR warm-up; normalize advantages
Reward hacking emerges Return rises, task fails Behavioural evaluation, reward audits
Seed variance Unreproducible wins ≥ 5 seeds, IQM and confidence intervals (rliable)
Replay buffer memory Out-of-memory crashes Compressed storage, frame dedup, sharded buffers
Vectorized reset bugs Wrong bootstrapping Unit tests for done/truncation handling
Observation normalization drift Policy works in training, fails in eval Share/freeze normalization statistics, save with the model
Logging overload Slow training, huge storage Sample logs, aggregate metrics
Compute instead of insight Expensive, little gain Fix reward/representation first; ablations

Code Example: Measuring Environment Throughput (Gymnasium Vector Environments)

import time

import gymnasium as gym


def steps_per_second(num_envs, mode, steps=2_000):
    envs = gym.make_vec("CartPole-v1", num_envs=num_envs, vectorization_mode=mode)
    envs.reset(seed=42)
    start = time.perf_counter()
    for _ in range(steps):
        envs.step(envs.action_space.sample())
    elapsed = time.perf_counter() - start
    envs.close()
    return steps * num_envs / elapsed


if __name__ == "__main__":                       # needed for the "async" mode's subprocesses
    for n in (1, 8):
        print(n, "envs sync:", int(steps_per_second(n, "sync")))
    print(8, "envs async:", int(steps_per_second(8, "async")))

Measure before scaling: for cheap environments such as CartPole, synchronous vectorization in one process is often faster than subprocesses (inter-process overhead dominates), while expensive simulators benefit from async workers or GPU simulation. Numbers depend on the machine.

Scaling Checklist

Check Why
Steps/s per component (env, inference, learner) Find the bottleneck
Learner throughput vs data generation Avoid stale or wasted data
Hyperparameters retuned at target scale Small-scale settings can fail
Evaluation on held-out environments, many seeds Real gains, not noise
Reproducible configs and versioned environments Debuggable results

Common Pitfalls (Summary)

  • Scaling an algorithm that is still buggy at small scale.

  • Comparing methods at different compute budgets.

  • Ignoring the cost of inference latency in actors.

Interview Tips

  • Name throughput bottlenecks, stale data, hyperparameter transfer and seed variance.

  • Mention IMPALA/V-trace and GPU simulators.

  • Stress profiling and robust evaluation.

66

How do you monitor and manage a deployed reinforcement learning system?

Monitoring a deployed RL system means watching four things continuously: the business or task outcome the policy is meant to improve (and guardrail metrics it must not harm), the policy’s behaviour (action distribution, entropy, how often the fallback or safety layer intervenes), the inputs (state distribution shift relative to training data, missing or broken sensors/features), and the learning signal (reward distribution, delayed outcomes, value estimates versus realized returns). Management means having the controls to act on what monitoring shows: versioned policies with a champion-challenger setup, safe rollback to a previous policy or rule-based fallback, kill switches, retraining pipelines with off-policy evaluation gates, bounded and logged exploration, alerting with clear owners, and periodic audits for reward hacking and fairness. Because RL policies influence the data they receive, also watch for feedback loops and evaluate new versions against logged propensities before rollout.

Read full answer

Applies toRL MLOps; Python/NumPy for drift metrics

What to Monitor

Category Metrics Alert example
Outcomes Task KPI, cumulative reward, guardrails (complaints, safety incidents) KPI below control group
Behaviour Action frequencies, policy entropy, exploration rate, fallback/shield interventions One action > 90% of decisions
Inputs Feature drift (PSI, KS), missing values, sensor health PSI > 0.2 on a key feature
Learning signal Reward distribution, delayed outcome rates Mean reward shifts 3σ
Value calibration Predicted value vs realized return Systematic overestimation
System Latency, errors, throughput p99 latency over budget
Fairness Per-group outcomes and exposure Gap exceeds threshold

Code Example: Action Distribution Drift (Population Stability Index)

import numpy as np


def psi(expected_counts, actual_counts, eps=1e-6):
    e = np.asarray(expected_counts, dtype=float)
    a = np.asarray(actual_counts, dtype=float)
    e, a = e / e.sum() + eps, a / a.sum() + eps
    return float(np.sum((a - e) * np.log(a / e)))


training_actions = [500, 300, 200]       # action counts during validation of the policy
this_week = [480, 310, 210]
after_incident = [900, 50, 50]

print(round(psi(training_actions, this_week), 4))       # 0.0016  stable
print(round(psi(training_actions, after_incident), 3))  # 0.891  large shift: investigate

By hand for the stable week: shares move from (0.5, 0.3, 0.2) to (0.48, 0.31, 0.21), giving (−0.02)·ln(0.96) + 0.01·ln(1.0333) + 0.01·ln(1.05) ≈ 0.00082 + 0.00033 + 0.00049 ≈ 0.0016. A common rule of thumb treats PSI above 0.2 as a significant shift.

Management Practices

Practice Purpose
Model registry and versioning Know which policy acted when
Champion-challenger / A/B Compare new policies safely
Off-policy evaluation gate before rollout Catch regressions early
Rollback and fallback policy Recover quickly
Retraining schedule + triggers Handle drift
Incident runbooks and owners Respond consistently
Periodic behaviour audits Detect reward hacking, unfairness

Common Pitfalls

  • Monitoring only average reward and missing behaviour changes.

  • Retraining automatically on drifted data without evaluation gates.

  • No record of action propensities.

  • Alert thresholds without owners or runbooks.

Interview Tips

  • Organize monitoring into outcomes, behaviour, inputs and learning signal.

  • Give one concrete drift metric and alert.

  • Describe rollback, champion-challenger and retraining gates.

67

Discuss a recent RL research paper and its implications.

A recent paper with broad implications is “Mastering Diverse Domains through World Models” by Danijar Hafner, Jurgis Pasukonis, Jimmy Ba and Timothy Lillicrap, which introduced DreamerV3 (preprint 2023, published in Nature in 2025). DreamerV3 is a model-based agent that learns a world model — a recurrent state-space model that encodes observations into compact latent states and predicts future latents, rewards and episode ends — and then trains an actor and critic entirely on imagined trajectories in that latent space. Its main contribution is robustness: with one fixed set of hyperparameters it performed strongly across more than 150 tasks spanning Atari, continuous control, DMLab, procedurally generated games and Minecraft, and it was reported as the first algorithm to collect diamonds in Minecraft from scratch, without human data or curricula. The techniques that made this possible include symlog transformations of inputs and rewards to handle very different scales, two-hot encoded discrete regression for rewards and values, percentile-based return normalization, KL balancing with free bits in the world model, and an architecture that improves predictably with model size. Implications: RL practitioners have historically retuned hyperparameters for each domain; DreamerV3 suggests general-purpose agents with fixed settings are achievable, that model-based RL can be both sample-efficient and general, and that world models are a promising path to agents that plan and learn from imagination, with open questions about compute cost, long-horizon prediction errors and real-world transfer.

Read full answer

Applies toModel-based RL concepts; NumPy for the illustration

Key Ideas

Component Idea
World model (RSSM) Encoder → stochastic + deterministic latent state; predicts next latent, reward, continuation
Imagination training Actor-critic trained on latent rollouts (horizon ~15)
Symlog symlog(x) = sign(x) ln(1 + |x|) squashes large values while keeping sign
Two-hot regression Predict values as a distribution over fixed bins
Return normalization Scale advantages by a percentile range of returns, robust to reward scale
Fixed hyperparameters One configuration across domains

Code Example: Symlog and Its Inverse (NumPy)

import numpy as np


def symlog(x):
    return np.sign(x) * np.log1p(np.abs(x))


def symexp(x):
    return np.sign(x) * np.expm1(np.abs(x))


rewards = np.array([-1000.0, -1.0, 0.0, 0.5, 1000.0])
squashed = symlog(rewards)
print(np.round(squashed, 3).tolist())                    # [-6.909, -0.693, 0.0, 0.405, 6.909]
print(np.allclose(symexp(squashed), rewards))           # True

Rewards spanning six orders of magnitude become values between about −7 and 7 while remaining exactly invertible, so the same network and loss settings work whether a game gives rewards of 1 or 1,000: one reason a single hyperparameter set could cover so many domains.

Implications and Open Questions

Implication Open question
Less per-task tuning Compute cost of world-model training
Model-based RL as a general method Long-horizon prediction errors
Imagination-based planning Transfer to real robots with noisy sensors
Scaling laws in RL How far scaling goes

Common Pitfalls (When Discussing a Paper)

  • Restating the abstract without the technical idea that made it work.

  • Overstating results (check exactly what was reported and under which conditions).

  • Ignoring limitations and follow-up work.

Interview Tips

  • Pick a paper you can explain in depth: problem, idea, results, limitations.

  • Connect it to practical implications for your work.

  • Be accurate about venues, dates and claims.

69

How is adversarial robustness being addressed in current RL research?

Adversarial robustness in RL addresses agents whose behaviour can be broken by small, deliberate changes — perturbed observations, manipulated rewards or poisoned training data, shifted dynamics, or adversarial opponents in multi-agent settings — and research has shown each of these is a real threat: tiny pixel perturbations can make Atari policies fail (Huang et al., 2017), “adversarial policies” that act strangely can defeat strong agents in multi-agent games without being good players (Gleave et al., 2020), and adversarial strategies have beaten superhuman Go programs such as KataGo (Wang et al., 2023). Defences being studied include: adversarial training against attacked observations; the state-adversarial MDP (SA-MDP) framework with policy-smoothness regularizers (Zhang et al., 2020) and alternating training against a learned adversary (ATLA); robust adversarial RL (RARL), which trains the protagonist against an adversary that applies disturbance forces (Pinto et al., 2017); robust MDPs that optimize for the worst case over an uncertainty set of dynamics; certified robustness methods that give provable bounds on actions or returns under bounded perturbations (randomized smoothing, interval bound propagation); population and league training so agents face diverse opponents; and defences against reward and data poisoning for offline and RLHF settings.

Read full answer

Applies toPyTorch 2.x (attack/defence illustration)

Threats and Defences

Threat Example Defences
Observation perturbations FGSM noise on game frames, sensor spoofing Adversarial training, SA-MDP regularization, certified bounds, input filtering
Adversarial opponents Adversarial policies in competitive games League/population training, diverse opponents, exploitability testing
Dynamics shift / disturbances Wind, payload changes RARL, robust MDPs, domain randomization
Reward/data poisoning Corrupted logged rewards, poisoned preferences Robust statistics, data auditing, anomaly detection
Backdoors (trojans) Trigger pattern causes harmful action Data provenance, trigger detection

Code Example: FGSM Attack on a Policy and a Smoothness Regularizer (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)
policy = nn.Sequential(nn.Linear(4, 64), nn.Tanh(), nn.Linear(64, 2))
obs = torch.randn(16, 4)


def fgsm_attack(policy, obs, eps=0.05):
    """Perturb observations to reduce the probability of the action the policy currently prefers."""
    obs_adv = obs.clone().requires_grad_(True)
    logits = policy(obs_adv)
    preferred = logits.argmax(dim=1)
    loss = nn.functional.cross_entropy(logits, preferred)    # increase this loss = move away from preferred action
    loss.backward()
    return (obs + eps * obs_adv.grad.sign()).detach()


obs_adv = fgsm_attack(policy, obs)
print(bool(((obs_adv - obs).abs() <= 0.05 + 1e-6).all()))    # True: perturbation stays within eps
with torch.no_grad():
    flipped = (policy(obs).argmax(1) != policy(obs_adv).argmax(1)).float().mean()
print(0.0 <= flipped.item() <= 1.0)                           # True (fraction of actions changed)


def smoothness_penalty(policy, obs, obs_adv):
    """SA-MDP-style regularizer: KL between action distributions on clean and perturbed states."""
    p_clean = torch.log_softmax(policy(obs), dim=1)
    p_adv = torch.log_softmax(policy(obs_adv), dim=1)
    return nn.functional.kl_div(p_adv, p_clean, log_target=True, reduction="batchmean")


penalty = smoothness_penalty(policy, obs, obs_adv)
print(penalty.item() >= 0)                                    # True

Adding this penalty (times a coefficient) to the usual RL loss encourages the policy to choose similar actions for nearby observations, which makes small perturbations less effective. The fraction of flipped actions for an untrained policy depends on the random weights; for trained agents, evaluations report return under attack at several budgets ε.

Evaluating Robustness

Evaluation Report
Return vs perturbation budget ε Curve, not one number
Against learned adversaries Stronger than fixed attacks
Exploitability in games Best-response win rate
Clean performance Robustness should not destroy normal returns

Common Pitfalls

  • Testing only against weak, fixed attacks.

  • Reporting robustness while clean performance collapsed.

  • Assuming robustness to one perturbation type transfers to others.

Interview Tips

  • List threat types: observations, opponents, dynamics, rewards/data.

  • Name defences: adversarial training, SA-MDP, RARL, robust MDPs, certification.

  • Cite a concrete attack result, such as adversarial policies against KataGo.

70

Describe an end-to-end pipeline for training, validating and deploying RL commercially.

An end-to-end commercial RL pipeline turns a business decision problem into a monitored, continuously improving policy through these stages: problem framing (decision, objective, constraints, and whether a simpler bandit or optimization method suffices), data and environment (logged interaction data with propensities, and a simulator calibrated to it), offline training and evaluation (algorithm selection, hyperparameter search over seeds, off-policy evaluation and simulator tests against strong baselines), pre-deployment validation (safety constraints, stress tests, fairness and robustness checks, review with stakeholders), staged deployment (shadow mode, canary A/B test, gradual ramp with guardrails and fallbacks), and operations (monitoring, drift detection, logging, retraining with evaluation gates, rollback). Every stage is versioned and reproducible — environment, data, code, configs, seeds and model artefacts — and the decision log (state, action, action probability, reward, model version) is designed in from the start because it powers later evaluation and retraining.

Read full answer

Applies toGymnasium 1.x, Stable-Baselines3, MLflow (or similar) for tracking; any serving stack

Pipeline Stages

Stage Activities Output / gate
1. Framing Define decision, state, action, reward, constraints, KPIs; baseline policies Problem spec signed off
2. Data Collect logs with propensities; feature pipelines shared with serving Versioned datasets
3. Environment Build/calibrate simulator; validate it against held-out logs Simulator fidelity report
4. Training Algorithms (PPO/SAC/DQN, offline RL), Optuna search, multiple seeds Candidate policies
5. Offline evaluation Simulator returns, OPE (IPS/DR), baselines, subgroup metrics Must beat champion with confidence
6. Safety validation Constraint tests, stress scenarios, adversarial tests, review Safety sign-off
7. Packaging Export policy (TorchScript/ONNX), deterministic inference, fallback logic Registered model version
8. Shadow mode Run alongside current system, compare decisions No critical disagreements
9. Online test Canary/A-B with guardrails, kill switch Statistically significant KPI lift
10. Operations Monitoring, drift, periodic retraining, rollback SLOs, runbooks

Code Example: Train, Evaluate Against a Baseline, and Gate Promotion (Stable-Baselines3)

import gymnasium as gym
import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy

ENV_ID, SEEDS, EVAL_EPISODES = "CartPole-v1", [0, 1, 2], 20


def baseline_return(episodes=EVAL_EPISODES):
    env, returns = gym.make(ENV_ID), []
    for i in range(episodes):
        obs, _ = env.reset(seed=10_000 + i)                      # held-out evaluation seeds
        done, total = False, 0.0
        while not done:
            action = int(obs[2] + 0.5 * obs[3] > 0)               # simple rule-based champion
            obs, r, terminated, truncated, _ = env.step(action)
            total += r
            done = terminated or truncated
        returns.append(total)
    return float(np.mean(returns))


candidates = []
for seed in SEEDS:
    model = PPO("MlpPolicy", gym.make(ENV_ID), seed=seed, verbose=0)
    model.learn(total_timesteps=50_000)
    eval_env = gym.make(ENV_ID)
    eval_env.reset(seed=10_000)
    mean_r, std_r = evaluate_policy(model, eval_env, n_eval_episodes=EVAL_EPISODES, deterministic=True)
    candidates.append((mean_r, std_r, seed, model))

champion = baseline_return()
best_mean, best_std, best_seed, best_model = max(candidates, key=lambda c: c[0])
all_seeds_ok = all(c[0] >= champion for c in candidates)        # robust across seeds, not one lucky run
promote = all_seeds_ok and best_mean > champion
print(f"champion {champion:.1f}, candidates {[round(c[0], 1) for c in candidates]}, promote={promote}")
if promote:
    best_model.save(f"ppo_cartpole_seed{best_seed}")            # then register the version with its metrics

The gate requires every training seed to match the current champion and the best to exceed it on held-out seeds; in a real system the same gate also checks off-policy estimates on production logs, guardrail metrics and safety tests before the policy reaches shadow mode.

Engineering Practices

Practice Why
Version everything (env, data, code, config, seeds) Reproducibility and audits
Same feature code for training and serving Avoid training-serving skew
Experiment tracking (MLflow/W&B) Compare runs and seeds
Automated tests for environment logic Bugs in rewards and resets are common
Decision logging with propensities Enables OPE and retraining
Fallback policy and kill switch Safe failure

Common Pitfalls

  • Skipping the simple baseline (rules, bandits, MPC) that the RL policy must beat.

  • Trusting a simulator that was never validated against real logs.

  • Promoting the best single seed.

  • Retraining on policy-generated data without correcting for its bias.

Interview Tips

  • Walk through framing → data/simulator → training → offline evaluation → safety → staged rollout → operations.

  • Emphasize gates between stages and logging of propensities.

  • Mention baselines, multi-seed evaluation and rollback.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 70 Reinforcement Learning answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects