What is reinforcement learning, and how does it differ from supervised and unsupervised learning?
KnownRevise
Reinforcement learning (RL) is learning to make sequences of decisions by trial and error: an agent acts in an environment, receives rewards, and learns a policy that maximizes the expected cumulative (discounted) reward. It differs from supervised learning, where a model learns from labelled examples that state the correct output, and from unsupervised learning, where a model finds structure (clusters, components, densities) in unlabelled data. In RL nobody tells the agent the correct action: feedback is a scalar reward that may be delayed (a chess win arrives after many moves), the agent’s own actions change the data it sees, and it must balance exploring new actions with exploiting what already works. Formally, RL problems are Markov decision processes.
Read full answerHide full answer
Applies toPython 3.10+, NumPy, Gymnasium 1.x
Comparison
Aspect
Supervised
Unsupervised
Reinforcement
Data
Labelled (x, y) pairs
Unlabelled x
Interaction: (s, a, r, s′) generated by the agent
Feedback
Correct answer per example
None
Scalar reward, often delayed
Goal
Predict y from x
Find structure
Maximize expected return
Data distribution
Fixed (i.i.d. assumed)
Fixed
Depends on the agent’s policy
Decisions
One-shot predictions
—
Sequential; actions affect the future
Exploration
Not needed
Not needed
Required
Examples
Spam detection, price regression
Customer segments, PCA
Game playing, robot control, ad bidding, RLHF
Main RL Method Families
Family
Idea
Examples
Dynamic programming
Plan with a known model
Value iteration, policy iteration
Monte Carlo
Learn from complete episode returns
First-visit MC control
Temporal difference
Bootstrap from the next state’s estimate
SARSA, Q-learning
Deep value-based
Neural Q-function
DQN, Double DQN
Policy gradient
Optimize the policy directly
REINFORCE
Actor-critic
Policy + learned value baseline
A2C, PPO, SAC
Model-based
Learn a model, plan with it
Dyna, MuZero, Dreamer
Code Example: Learning from Reward Alone (Multi-Armed Bandit)
import numpy as np
rng = np.random.default_rng(42)
true_means = np.array([0.2, 0.5, 0.8]) # unknown to the agent
estimates = np.zeros(3)
counts = np.zeros(3)
epsilon = 0.1
for t in range(2000):
if rng.random() < epsilon:
arm = int(rng.integers(3)) # explore
else:
arm = int(np.argmax(estimates)) # exploit
reward = rng.normal(true_means[arm], 1.0) # only a noisy reward, never "the right arm"
counts[arm] += 1
estimates[arm] += (reward - estimates[arm]) / counts[arm] # incremental average
print(np.round(estimates, 2), counts.astype(int))
The agent is never told which arm is best; it learns from noisy rewards that it collects through its own choices. After enough steps the estimates approach the true means and most pulls go to the best arm; the exact numbers depend on the random draws. A full RL problem adds states, so actions also change what the agent will see next.
Applications and Caveats
RL is used for games (AlphaGo), robotics, recommendation and ad bidding, resource and energy control, and RLHF for language models. Caveats: it needs many interactions (usually in simulation), reward design is hard and can be gamed, training is unstable, and simulation-to-real gaps and safety limit deployment.
Common Pitfalls
Calling any problem with feedback “RL”; if you have labelled correct actions, supervised learning (imitation) is simpler.
Ignoring that the data distribution shifts as the policy changes.
Expecting RL to be sample-efficient without a simulator.
Interview Tips
Define RL in one sentence with agent, reward and return.
Contrast the feedback type and data generation with supervised and unsupervised learning.
Mention exploration, delayed rewards and one real application.
2
Define agent, environment, state, action and reward in reinforcement learning.
KnownRevise
In reinforcement learning, the agent is the learner and decision-maker; the environment is everything outside the agent that it interacts with; the state is the information describing the current situation; an action is a choice the agent makes; and the reward is a scalar signal the environment returns after each action, telling the agent how good the immediate outcome was. They form a loop: at step t the agent observes state s_t, picks action a_t using its policy π(a|s), and the environment responds with reward r(t+1) and next state s(t+1). The agent’s goal is not to maximize single rewards but the return, the (discounted) sum of future rewards G_t = r(t+1) + γ r(t+2) + γ² r_(t+3) + …, with discount factor 0 ≤ γ ≤ 1. Formally, this loop is a Markov decision process.
Read full answerHide full answer
Applies toPython 3.10+, Gymnasium 1.x (`import gymnasium as gym`)
The Terms
Term
Definition
CartPole example
Thermostat example
Agent
Learner that chooses actions
Controller pushing the cart
Heating controller
Environment
System the agent acts on
Cart, pole and physics
Building and weather
State s_t
Description of the situation
Cart position/velocity, pole angle/angular velocity
Room temperature, time, outside temperature
Observation
What the agent actually sees (may be partial)
Same 4 numbers
Sensor readings
Action a_t
Choice made by the agent
Push left or right
Heater power level
Reward r_(t+1)
Scalar feedback after the action
+1 for every step the pole stays up
−(energy cost) − comfort penalty
Episode
Run from a start state to termination or truncation
Until the pole falls or 500 steps
One day
Policy π
Mapping from states to actions
Learned network
Learned schedule
Code Example: The Agent-Environment Loop (Gymnasium)
import gymnasium as gym
env = gym.make("CartPole-v1")
print(env.observation_space.shape) # (4,) state: position, velocity, angle, angular velocity
print(env.action_space) # Discrete(2) actions: 0 = push left, 1 = push right
observation, info = env.reset(seed=42)
env.action_space.seed(42)
episode_return, steps = 0.0, 0
terminated = truncated = False
while not (terminated or truncated):
action = env.action_space.sample() # a random "agent"
observation, reward, terminated, truncated, info = env.step(action)
episode_return += reward # CartPole gives +1 per step
steps += 1
print(steps == episode_return) # True: reward is 1.0 for every step survived
env.close()
terminated means the episode ended inside the task (the pole fell); truncated means it was cut off by a time limit (500 steps). A learning agent would replace env.action_space.sample() with its policy and update that policy from the rewards.
State vs Observation
The state is the full information the environment needs to decide what happens next. The agent often sees only an observation (a camera image, noisy sensors). When the observation leaves out important information, the problem is partially observable (a POMDP), and the agent needs memory such as frame stacking or a recurrent policy.
Common Pitfalls
Treating reward as the goal of a single step instead of the return over time.
Confusing state with observation.
Designing rewards that can be gamed (the agent maximizes exactly what you reward).
Using the old Gym API (obs, reward, done, info); Gymnasium splits done into terminated and truncated.
Interview Tips
Draw the loop: state → action → reward + next state.
Give one concrete example with every term mapped.
Mention policy, return and discount factor to show the full picture.
3
What is a Markov Decision Process (MDP) in reinforcement learning?
KnownRevise
A Markov Decision Process is the mathematical model of a reinforcement learning problem, defined by the tuple (S, A, P, R, γ): a set of states S, a set of actions A, transition probabilities P(s′ | s, a) of reaching state s′ after taking action a in state s, a reward function R(s, a) (or R(s, a, s′)), and a discount factor γ in [0, 1] that weights future rewards. Its defining assumption is the Markov property: the next state and reward depend only on the current state and action, not on the earlier history, so the state contains everything needed to predict the future. Solving an MDP means finding a policy π that maximizes the expected discounted return E[Σ γᵗ r_(t+1)]. When P and R are known, dynamic programming (value or policy iteration) solves it; when they are not, RL methods learn from sampled transitions.
Read full answerHide full answer
Applies toPython 3.10+, NumPy
The Components
Symbol
Meaning
Example (robot vacuum)
S
States
Location and battery level
A
Actions
Move, clean, return to dock
P(s′ | s, a)
Transition probabilities
0.9 move succeeds, 0.1 wheels slip
R(s, a)
Expected immediate reward
+1 per dirty cell cleaned, −10 battery empty
γ
Discount factor
0.99 (cares about the long run)
π(a | s)
Policy (the solution)
Cleaning strategy
Why the Discount Factor
γ < 1 keeps the infinite sum finite, expresses preference for sooner rewards, and models uncertainty about the future. With γ = 0.9, a reward 10 steps away is worth 0.9¹⁰ ≈ 0.349 of an immediate one; the “effective horizon” is about 1/(1 − γ) = 10 steps.
Code Example: Evaluating a Policy on a Two-State MDP (NumPy)
import numpy as np
# Fixed policy on two states A=0, B=1:
# in A: reward 1, move to B; in B: reward 2, move back to A
P_pi = np.array([[0.0, 1.0],
[1.0, 0.0]]) # P_pi[s, s'] under the policy
R_pi = np.array([1.0, 2.0]) # expected reward in each state
gamma = 0.9
# Bellman expectation equation in matrix form: V = R + gamma * P V -> (I - gamma P) V = R
V = np.linalg.solve(np.eye(2) - gamma * P_pi, R_pi)
print(np.round(V, 2)) # [14.74 15.26]
By hand: V(A) = 1 + 0.9 V(B) and V(B) = 2 + 0.9 V(A), so V(A) = 2.8 / 0.19 ≈ 14.74 and V(B) = 2 + 0.9 × 14.74 ≈ 15.26.
Code Example: Sampling Trajectories from an MDP with Actions
import numpy as np
rng = np.random.default_rng(42)
# P[s, a, s']: 3 states, 2 actions; rows over s' sum to 1
P = np.array([
[[0.9, 0.1, 0.0], [0.1, 0.8, 0.1]],
[[0.5, 0.5, 0.0], [0.0, 0.2, 0.8]],
[[0.0, 0.0, 1.0], [0.0, 0.0, 1.0]], # state 2 is absorbing (goal)
])
R = np.array([[0.0, -0.1], [0.0, -0.1], [0.0, 0.0]])
print(np.allclose(P.sum(axis=2), 1.0)) # True
state, ret = 0, 0.0
for t in range(20):
action = 1 # policy: always "advance"
ret += (0.9 ** t) * R[state, action]
state = int(rng.choice(3, p=P[state, action]))
if state == 2:
break
print(state) # 2 (reached the goal), unless 20 steps run out first
Extensions
Variant
Change
POMDP
Agent sees observations, not states; needs belief or memory
Semi-MDP
Actions take variable time (options in hierarchical RL)
Constrained MDP
Maximize return subject to cost limits (safe RL)
Multi-agent (Markov game)
Several agents act; transitions depend on all
Common Pitfalls
Choosing a state that breaks the Markov property (e.g. one camera frame without velocity).
Rows of P that do not sum to 1.
Setting γ = 1 for continuing tasks (returns can be infinite).
Forgetting terminal states should have zero future value.
Interview Tips
List the five components and the Markov property.
Explain the role of γ with a number.
Say that DP solves known MDPs and RL handles unknown ones.
4
What is the role of a policy in reinforcement learning?
KnownRevise
The policy is the agent’s behaviour: a rule π that maps each state to an action (deterministic, a = π(s)) or to a probability distribution over actions (stochastic, π(a|s)), and finding a good policy is the goal of reinforcement learning. Every RL method ultimately produces one: value-based methods derive it from a learned Q-function (act greedily, π(s) = argmax_a Q(s, a), with some exploration), policy-gradient and actor-critic methods learn it directly as a parameterized function π_θ (a neural network that outputs action probabilities or the mean and variance of continuous actions), and model-based methods produce it by planning. The policy also decides which data the agent collects, so it controls exploration, and the distinction between the policy being learned (target) and the policy generating data (behaviour) defines on-policy vs off-policy learning.
Read full answerHide full answer
Applies toPython 3.10+, NumPy, PyTorch 2.x
Types of Policies
Type
Form
Example
Used by
Deterministic tabular
Table s → a
Grid world: “up” in cell (2, 3)
Policy iteration
Greedy from Q
argmax_a Q(s, a)
Q-learning’s final policy
DQN, Q-learning
ε-greedy
Greedy with probability 1 − ε, random otherwise
Exploration during training
Q-learning, SARSA
Softmax / categorical
π(a
s) = softmax(logits)
Discrete action network
REINFORCE, PPO
Gaussian
a ~ N(μ_θ(s), σ_θ(s))
Robot joint torques
PPO, SAC
Deterministic continuous
a = μ_θ(s)
Continuous control
DDPG, TD3
Code Example: Policies Side by Side
import numpy as np
import torch
from torch import nn
rng = np.random.default_rng(42)
# 1. Deterministic tabular policy for 4 states
policy_table = {0: "right", 1: "right", 2: "down", 3: "stay"}
print(policy_table[2]) # down
# 2. Greedy and epsilon-greedy policies derived from Q-values
Q = np.array([1.0, 3.0, 2.0]) # Q(s, a) for three actions in one state
greedy = int(np.argmax(Q))
print(greedy) # 1
def epsilon_greedy(q, eps):
return int(rng.integers(len(q))) if rng.random() < eps else int(np.argmax(q))
# 3. Stochastic softmax policy
logits = np.array([2.0, 1.0, 0.0])
probs = np.exp(logits) / np.exp(logits).sum()
print(np.round(probs, 3)) # [0.665 0.245 0.09 ]
# 4. Parameterized neural policy pi_theta(a|s) for a 4-dim state and 2 actions
torch.manual_seed(42)
policy_net = nn.Sequential(nn.Linear(4, 32), nn.Tanh(), nn.Linear(32, 2))
state = torch.randn(1, 4)
dist = torch.distributions.Categorical(logits=policy_net(state))
action = dist.sample()
print(action.shape, dist.log_prob(action).shape) # torch.Size([1]) torch.Size([1])
The softmax probabilities are e²/(e² + e + 1) ≈ 0.665, e/(e² + e + 1) ≈ 0.245 and 1/(e² + e + 1) ≈ 0.090. The log_prob of the sampled action is what policy-gradient methods use to update θ.
Why Stochastic Policies Matter
Reason
Example
Built-in exploration
Sampling tries non-greedy actions
Optimal under partial observability
Two states that look identical may need different actions
Games against opponents
Rock-paper-scissors: the optimal policy is uniformly random
Smooth optimization
Policy gradients need differentiable probabilities
Common Pitfalls
Confusing the policy (what to do) with the value function (how good a state or action is).
Evaluating a stochastic policy with exploration left on, or deploying with exploration on when it should be greedy.
Forgetting that changing the policy changes the data distribution.
Interview Tips
Define deterministic and stochastic policies with notation.
Explain how value-based methods derive a policy and policy-gradient methods learn one directly.
Connect the policy to exploration and on-/off-policy learning.
5
What are value functions, and how do they relate to policies?
KnownRevise
A value function estimates how good it is to be in a situation under a given policy, measured as expected return: the state-value function V^π(s) = E_π[G_t | s_t = s] is the expected discounted return starting from state s and following π; the action-value function Q^π(s, a) = E_π[G_t | s_t = s, a_t = a] is the expected return after taking action a in s and following π afterwards. They link to policies in both directions. Given a policy, its values are defined by it: V^π(s) = Σ_a π(a|s) Q^π(s, a) (policy evaluation). Given values, you can improve the policy by acting greedily, π′(s) = argmax_a Q^π(s, a) (policy improvement); repeating both steps converges to the optimal values V* and Q* and an optimal policy. The advantage A^π(s, a) = Q^π(s, a) − V^π(s) says how much better an action is than the policy’s average and is the signal used by actor-critic methods.
Read full answerHide full answer
Applies toPython 3.10+, NumPy
Definitions
Function
Formula
Answers
V^π(s)
E_π[Σ_k γᵏ r_(t+k+1) | s_t = s]
How good is this state under π?
Q^π(s, a)
E_π[Σ_k γᵏ r_(t+k+1) | s_t = s, a_t = a]
How good is this action here, then following π?
A^π(s, a)
Q^π(s, a) − V^π(s)
Better or worse than average?
V(s), Q(s, a)
max over all policies
Best achievable
Relationships
Relation
Formula
V from Q
V^π(s) = Σ_a π(a|s) Q^π(s, a)
Q from V (needs model)
Q^π(s, a) = R(s, a) + γ Σ_s′ P(s′|s, a) V^π(s′)
Greedy improvement
π′(s) = argmax_a Q^π(s, a)
Optimal policy
π(s) = argmax_a Q(s, a)
Code Example: Values, Policy and Advantage (NumPy)
By hand: V(s0) = 0.5·1 + 0.5·3 = 2 and V(s1) = 0.25·2 + 0.75·0 = 0.5. The positive advantages (a1 in s0, a0 in s1) point to the actions the improved policy picks.
Why Q Is Often Preferred
Acting greedily with V needs the model (to look one step ahead through P), while acting greedily with Q needs only an argmax over actions, which is why model-free control methods (Q-learning, SARSA, DQN) learn Q.
How Methods Use Value Functions
Method
Value function role
Value/policy iteration
Compute V or Q with the model
Q-learning, DQN
Learn Q*, act greedily
Actor-critic (A2C, PPO)
Critic learns V; advantage weights policy updates
REINFORCE with baseline
V(s) baseline reduces variance
Common Pitfalls
Mixing up V and Q, or forgetting a value function is always for a particular policy (or optimal).
Using V to choose actions in a model-free setting.
Ignoring terminal states (their future value is 0).
Interview Tips
Write V^π and Q^π as expectations of the return.
Show V = Σ π Q and greedy improvement from Q.
Mention the advantage and where actor-critic methods use it.
6
What is the difference between on-policy and off-policy learning?
KnownRevise
On-policy methods learn the value or improve the policy that is generating the data (the behaviour policy and target policy are the same); off-policy methods learn about a target policy — usually the greedy, optimal one — from data generated by a different behaviour policy, such as an exploratory ε-greedy policy, an older version of the agent, or human logs. The textbook pair is SARSA (on-policy) and Q-learning (off-policy): SARSA’s target uses the action the agent will actually take next, r + γ Q(s′, a′), while Q-learning’s uses the best next action, r + γ max_a Q(s′, a). Off-policy methods can reuse old experience (replay buffers, logged data), so they are more sample-efficient, but they can be less stable and may need corrections such as importance sampling. On-policy methods (SARSA, REINFORCE, A2C, PPO) are usually more stable but must keep collecting fresh data.
Read full answerHide full answer
Applies toPython 3.10+, NumPy
Comparison
Aspect
On-policy
Off-policy
Data source
Current policy only
Any behaviour policy
Experience reuse
Little (PPO reuses a batch for a few epochs)
Replay buffers, logged data
Sample efficiency
Lower
Higher
Stability
Usually higher
Can diverge (with function approximation + bootstrapping)
Corrections needed
None
Importance sampling or clipped variants for some methods
SARSA evaluates what the exploring agent actually does next (Q = 2 for action 0), while Q-learning assumes the best action from then on (Q = 5). On Sutton and Barto’s “cliff walking” example this makes SARSA learn a safer path away from the cliff edge, since it accounts for its own random steps, while Q-learning learns the shorter, riskier optimal path.
Importance Sampling
To evaluate target policy π with data from behaviour policy b, returns are reweighted by ρ = Π_t π(a_t|s_t) / b(a_t|s_t). The product can have very high variance over long episodes, so practical methods use per-step or clipped ratios (Retrace, V-trace, PPO’s clipped ratio).
Offline RL
Offline (batch) RL is the extreme off-policy case: learn only from a fixed logged dataset with no new interaction. It needs conservative methods (CQL, IQL) because Q-values for actions absent from the data can be wildly overestimated.
Common Pitfalls
Calling PPO off-policy because it does several epochs per batch; it is on-policy with a clipped ratio.
Using a replay buffer with an on-policy algorithm without corrections.
Expecting off-policy learning from logs to work when the logged policy never tried the actions you want to evaluate.
Interview Tips
Define behaviour and target policies.
Contrast SARSA and Q-learning targets in one line each.
Discuss the sample-efficiency vs stability trade-off and importance sampling.
7
What is the exploration vs exploitation trade-off in reinforcement learning?
KnownRevise
The exploration-exploitation trade-off is the agent’s dilemma between exploiting — choosing the action it currently believes is best to collect reward now — and exploring — trying other actions to gather information that might reveal something better. Pure exploitation can lock the agent into a suboptimal action because its early estimates were wrong; pure exploration wastes reward on actions known to be bad. Good strategies explore a lot early, when estimates are uncertain, and less later. Common methods are ε-greedy (random action with probability ε, often decayed), softmax/Boltzmann sampling, optimistic initial values, Upper Confidence Bound (UCB) bonuses for rarely tried actions, Thompson sampling from a posterior, entropy bonuses in policy-gradient methods, and curiosity / count-based intrinsic rewards for sparse-reward environments.
Read full answerHide full answer
Applies toPython 3.10+, NumPy
Strategies
Strategy
Rule
Pros
Cons
Greedy
argmax Q
Simple
Can get stuck
ε-greedy
Random with prob. ε
Simple, works everywhere
Explores blindly, even bad actions
Decaying ε
ε from 1.0 to 0.05 over time
More exploration early
Schedule to tune
Softmax (temperature τ)
P(a) ∝ exp(Q(a)/τ)
Prefers promising actions
τ to tune
Optimistic initial values
Start Q high
Encourages trying everything
Only early exploration
UCB
Q(a) + c √(ln t / N(a))
Targets uncertain actions; regret bounds
Mostly bandits/tabular
Thompson sampling
Sample from posterior, act greedily
Strong in practice
Needs a posterior
Entropy bonus
Add β·H(π) to the objective
Keeps policies stochastic
β to tune
Intrinsic motivation
Bonus for novel states (RND, ICM, counts)
Sparse rewards
Can chase noise
Code Example: Greedy vs ε-Greedy vs UCB on a 10-Armed Bandit
import numpy as np
def run(strategy, steps=1000, runs=200, eps=0.1, c=2.0, seed=42):
rng = np.random.default_rng(seed)
total = 0.0
for _ in range(runs):
means = rng.normal(0, 1, 10) # new random bandit each run
Q, N = np.zeros(10), np.zeros(10)
for t in range(1, steps + 1):
if strategy == "greedy":
a = int(np.argmax(Q))
elif strategy == "eps":
a = int(rng.integers(10)) if rng.random() < eps else int(np.argmax(Q))
else: # UCB: try every arm once, then add a bonus
untried = np.flatnonzero(N == 0)
a = int(untried[0]) if untried.size else int(np.argmax(Q + c * np.sqrt(np.log(t) / N)))
r = rng.normal(means[a], 1)
N[a] += 1
Q[a] += (r - Q[a]) / N[a]
total += r
return total / (runs * steps)
for s in ("greedy", "eps", "ucb"):
print(s, round(run(s), 3))
On this classic testbed the greedy agent usually earns the lowest average reward (it commits to whichever arm looked good first), while ε-greedy and UCB find better arms; UCB is often best. The exact averages depend on the seed, so run it rather than relying on quoted numbers.
Exploration in Deep RL
Method family
Typical exploration
DQN
ε-greedy decayed from 1.0 to ~0.05; noisy networks
PPO / A2C
Stochastic policy + entropy bonus
SAC
Maximum-entropy objective with tuned temperature
Sparse-reward tasks
RND, ICM, count-based bonuses, demonstrations
Common Pitfalls
Never decaying ε, so the trained agent keeps acting randomly.
Evaluating with exploration on (or deploying with it off when online learning is expected).
Decaying exploration too fast before the agent has seen rewarding states.
In production, exploring in ways that are unsafe or hurt users; use offline evaluation and constrained exploration.
Interview Tips
Define both sides and why either extreme fails.
Describe ε-greedy with decay, UCB and Thompson sampling.
Mention exploration in deep RL (entropy, curiosity) for sparse rewards.
8
What are the Bellman equations, and how are they used in reinforcement learning?
KnownRevise
The Bellman equations express the value of a state (or state-action pair) recursively, as the immediate reward plus the discounted value of where you go next. The expectation equation for a policy π is V^π(s) = Σ_a π(a|s) [R(s, a) + γ Σ_s′ P(s′|s, a) V^π(s′)], and the optimality equation is V(s) = max_a [R(s, a) + γ Σ_s′ P(s′|s, a) V(s′)] (with matching versions for Q, such as Q(s, a) = R(s, a) + γ Σ_s′ P(s′|s, a) max_a′ Q(s′, a′)). They turn a sum over an infinite future into a one-step relationship, which almost every RL algorithm uses: policy evaluation and value iteration apply them as update rules with a known model; temporal-difference methods (TD(0), SARSA, Q-learning, DQN) apply them to sampled transitions, moving estimates toward the “Bellman target” r + γ V(s′) or r + γ max Q(s′, a′).
Read full answerHide full answer
Applies toPython 3.10+, NumPy
The Four Equations
Equation
Formula (in words)
Expectation, V
Value under π = expected reward + γ × expected next value under π
Expectation, Q
Q^π(s, a) = R(s, a) + γ Σ_s′ P(s′|s, a) Σ_a′ π(a′|s′) Q^π(s′, a′)
Optimality, V
V*(s) = max over actions of (reward + γ × expected next optimal value)
Optimality, Q
Q(s, a) = R(s, a) + γ Σ_s′ P(s′|s, a) max_a′ Q(s′, a′)
How Algorithms Use Them
Algorithm
Equation
How
Policy evaluation
Expectation
Iterate or solve linear system
Policy iteration
Expectation + greedy step
Evaluate, improve, repeat
Value iteration
Optimality
Apply max-backup until convergence
TD(0)
Expectation (sampled)
V(s) += α[r + γV(s′) − V(s)]
SARSA
Q expectation (sampled)
Target r + γQ(s′, a′)
Q-learning, DQN
Q optimality (sampled)
Target r + γ max Q(s′, ·)
Code Example: Value Iteration with the Optimality Equation (NumPy)
import numpy as np
# Two states, two actions (0 = stay, 1 = switch). Deterministic transitions.
# State 0: stay -> reward 1, stay in 0; switch -> reward 0, go to 1
# State 1: stay -> reward 2, stay in 1; switch -> reward 0, go to 0
R = np.array([[1.0, 0.0],
[2.0, 0.0]])
next_state = np.array([[0, 1],
[1, 0]])
gamma = 0.9
V = np.zeros(2)
while True:
Q = R + gamma * V[next_state] # Bellman backup for every (s, a)
V_new = Q.max(axis=1) # optimality: take the best action
if np.max(np.abs(V_new - V)) < 1e-8:
break
V = V_new
print(np.round(V_new, 2).tolist()) # [18.0, 20.0]
print(Q.argmax(axis=1).tolist()) # [1, 0] switch from 0, then stay in 1
By hand: staying in state 1 forever earns 2/(1 − 0.9) = 20. From state 0, staying earns 1/(1 − 0.9) = 10, while switching earns 0 + 0.9 × 20 = 18, so V*(0) = 18 and the optimal policy switches once and then stays.
Why They Work
The Bellman backup is a contraction with factor γ: each application shrinks the maximum error by at least γ, so repeated backups converge to a unique fixed point (V^π or V*). With function approximation and off-policy data that guarantee is lost, which is why DQN needs target networks and replay.
Common Pitfalls
Bootstrapping from terminal states (their next value must be 0).
Confusing the expectation equation (follows π) with the optimality equation (max).
Stopping value iteration on a tiny change in V without considering γ close to 1 (convergence is slow).
Interview Tips
Write V* and Q* optimality equations and explain each term.
Map algorithms to equations: DP uses them exactly, TD methods use sampled versions.
Mention the contraction property as the reason iteration converges.
9
What is the difference between model-based and model-free reinforcement learning?
KnownRevise
Model-based reinforcement learning uses a model of the environment — the transition dynamics P(s′|s, a) and reward function R(s, a), either known (chess rules, a physics simulator) or learned from data — and plans with it to choose actions or to generate extra training experience; model-free reinforcement learning skips the model and learns a value function or policy directly from real experience by trial and error. Model-based methods (value iteration with a known model, Dyna, MCTS in AlphaZero, MuZero, PETS, Dreamer, MBPO) are usually far more sample-efficient and can plan for new goals, but they suffer from model errors that the planner can exploit and cost more computation. Model-free methods (Q-learning, SARSA, DQN, REINFORCE, PPO, SAC) are simpler and reach strong final performance, but need many more interactions.
Read full answerHide full answer
Applies toPython 3.10+, NumPy
Comparison
Aspect
Model-based
Model-free
Learns
Dynamics/reward model (+ often value/policy)
Value function and/or policy only
Decision making
Planning, lookahead, simulated rollouts
Direct policy or argmax Q
Sample efficiency
High
Low
Computation per decision
Higher (planning)
Low
Main risk
Model bias/compounding errors
Many interactions, slow learning
Adapting to new goals
Replan with the same model
Usually retrain
Examples
Value iteration, Dyna-Q, AlphaZero, MuZero, Dreamer, MBPO
Q-learning, DQN, PPO, SAC, REINFORCE
Good fit
Expensive real data, known rules, robotics with simulators
Cheap simulators, hard-to-model dynamics
Code Example: The Same Experience Used Both Ways (NumPy)
import numpy as np
from collections import Counter
# Logged transitions (state, action, reward, next_state); 2 states, 1 action for simplicity
data = [(0, 0, 0.0, 1), (0, 0, 0.0, 1), (0, 0, 0.0, 1), (0, 0, 1.0, 0),
(1, 0, 2.0, 1), (1, 0, 2.0, 1)]
gamma = 0.9
# Model-based: estimate P and R by counting, then plan (solve the Bellman equation)
counts = Counter((s, s2) for s, _, _, s2 in data)
visits = Counter(s for s, _, _, _ in data)
P_hat = np.array([[counts[(s, s2)] / visits[s] for s2 in range(2)] for s in range(2)])
R_hat = np.array([np.mean([r for s, _, r, _ in data if s == st]) for st in range(2)])
print(P_hat.tolist(), R_hat.tolist()) # [[0.25, 0.75], [0.0, 1.0]] [0.25, 2.0]
V_model = np.linalg.solve(np.eye(2) - gamma * P_hat, R_hat)
print(np.round(V_model, 2)) # [17.74 20. ]
# Model-free: TD(0) updates straight from the same transitions, no P or R estimated
V_td = np.zeros(2)
alpha = 0.1
for _ in range(500): # replay the log many times
for s, _, r, s2 in data:
V_td[s] += alpha * (r + gamma * V_td[s2] - V_td[s])
print(np.round(V_td, 1)) # close to the model-based values; exact digits depend on alpha
By hand for the model-based part: state 1 always returns to itself with reward 2, so V(1) = 2 / 0.1 = 20. State 0 has R̂(0) = 1/4 = 0.25 and V(0) = 0.25 + 0.9(0.25 V(0) + 0.75 × 20), so 0.775 V(0) = 13.75 and V(0) ≈ 17.74. The model-based agent gets these values in one solve; the model-free agent needs many passes over the data to approach them.
Hybrids
Approach
Idea
Dyna-Q
Learn a model; do extra Q-updates on simulated transitions
MBPO
Short model rollouts feed an SAC learner
AlphaZero / MuZero
Tree search with a known or learned model, guided by learned policy/value
Dreamer
Learn a world model, train an actor-critic in imagination
Common Pitfalls
Assuming a learned model is accurate over long rollouts (errors compound; keep rollouts short).
Calling DQN model-based because of its replay buffer (replay is stored data, not a model).
Ignoring planning cost at decision time.
Interview Tips
Define the model as P and R and say what each family learns.
Contrast sample efficiency vs model bias.
Name examples of each and one hybrid (Dyna, MuZero).
10
What are the advantages and disadvantages of model-based reinforcement learning?
KnownRevise
Model-based reinforcement learning learns or is given a model of the environment’s dynamics and rewards and uses it to plan or to generate simulated experience; its main advantage is sample efficiency — it can learn from far fewer real interactions — and its main disadvantage is model error, because a planner will exploit inaccuracies in the model, and errors compound over long imagined rollouts. Other advantages are the ability to plan ahead and look before acting, to transfer the same model to new goals or reward functions, to check safety in simulation, and to explain decisions through predicted outcomes. Other disadvantages are higher computation and implementation complexity, difficulty modelling high-dimensional or stochastic dynamics (images, contact physics), and often lower final performance than well-tuned model-free methods when real data is cheap.
Read full answerHide full answer
Applies toPython 3.10+, NumPy, Gymnasium 1.x
Advantages and Disadvantages
Advantages
Disadvantages
Sample efficiency: learns from few real interactions
Model bias: planner exploits model errors
Planning and lookahead (MCTS, MPC)
Compounding errors over long rollouts
Reuse the model for new goals/rewards
Hard to model complex, stochastic, high-dimensional dynamics
Safer: test actions in the model first
More computation at training and decision time
Better use of offline data
More components to tune (model, planner, policy)
Interpretability through predicted outcomes
Asymptotic performance can lag model-free methods
Uncertainty-aware exploration (ensembles)
Uncertainty estimation itself is hard
Code Example: Dyna-Q on a Deterministic Grid (Gymnasium)
import gymnasium as gym
import numpy as np
env = gym.make("FrozenLake-v1", is_slippery=False) # 4x4 grid, reward 1 at the goal
n_s, n_a = env.observation_space.n, env.action_space.n
rng = np.random.default_rng(42)
def dyna_q(planning_steps, episodes=60, alpha=0.5, gamma=0.95, eps=0.1):
Q = np.zeros((n_s, n_a))
model = {} # (s, a) -> (r, s', terminated)
successes = 0
for ep in range(episodes):
s, _ = env.reset(seed=42 + ep)
done = False
while not done:
if rng.random() < eps or not Q[s].any():
a = int(rng.integers(n_a))
else:
a = int(np.argmax(Q[s]))
s2, r, terminated, truncated, _ = env.step(a)
Q[s, a] += alpha * (r + gamma * Q[s2].max() * (not terminated) - Q[s, a]) # real update
model[(s, a)] = (r, s2, terminated) # learn the model
keys = list(model)
for _ in range(planning_steps): # simulated updates
ps, pa = keys[rng.integers(len(keys))]
pr, ps2, pterm = model[(ps, pa)]
Q[ps, pa] += alpha * (pr + gamma * Q[ps2].max() * (not pterm) - Q[ps, pa])
s, done = s2, terminated or truncated
successes += r
return successes
print("model-free (0 planning steps):", dyna_q(0))
print("model-based (20 planning steps):", dyna_q(20))
Each real step trains Q once; with planning, the stored model replays 20 extra simulated transitions per real step, so the goal reward spreads back through the grid much faster and more of the 60 episodes typically reach the goal. Counts vary with the random draws. The environment is deterministic, so the table model is exact here; in stochastic or continuous environments the model is only an approximation.
Mitigating the Disadvantages
Problem
Mitigation
Model exploitation
Ensembles and uncertainty penalties (PETS, MOPO)
Compounding errors
Short rollouts branched from real states (MBPO)
High-dimensional observations
Latent world models (Dreamer, MuZero)
Computation
Amortize planning into a policy network
Wrong model at decision time
Model-predictive control: replan every step
Common Pitfalls
Trusting long imagined rollouts.
Measuring only sample efficiency and ignoring wall-clock compute.
Training the model on data from one policy and planning far outside that data.
Interview Tips
Lead with sample efficiency vs model bias.
Give hybrids that address the weaknesses (Dyna, MBPO, MuZero).
Mention when model-free is the better choice (cheap simulators, hard dynamics).
11
How does Q-learning work, and why is it called model-free?
KnownRevise
Q-learning learns the optimal action-value function Q(s, a) directly from experience: after each transition (s, a, r, s′) it moves its estimate toward the Bellman optimality target, Q(s, a) ← Q(s, a) + α [r + γ max_a′ Q(s′, a′) − Q(s, a)], and the policy is to act greedily on Q (with ε-greedy exploration while learning). It is model-free* because it never learns or uses the transition probabilities P(s′|s, a) or the reward function: the sampled next state and reward stand in for the expectation in the Bellman equation, and the max over Q-values replaces planning. It is also off-policy*: the target uses the greedy action even when the agent explored, so it learns the optimal policy while behaving exploratorily. With a table, every state-action pair visited infinitely often and suitable step sizes, it converges to Q; DQN extends it to large state spaces with a neural network.
Read full answerHide full answer
Applies toPython 3.10+, NumPy, Gymnasium 1.x
The Update, Term by Term
Term
Meaning
Q(s, a)
Current estimate
r + γ max_a′ Q(s′, a′)
TD target: reward plus best estimated future value
target − Q(s, a)
TD error δ
α
Learning rate (step size)
(1 − terminated)
Future value is 0 after a terminal state
One update by hand: Q(s, a) = 0, α = 0.5, r = 1, γ = 0.9, max Q(s′, ·) = 2 → target 2.8, new Q = 0 + 0.5 × 2.8 = 1.4.
Code Example: Tabular Q-Learning on FrozenLake (Gymnasium)
import gymnasium as gym
import numpy as np
env = gym.make("FrozenLake-v1", is_slippery=False)
Q = np.zeros((env.observation_space.n, env.action_space.n))
rng = np.random.default_rng(42)
alpha, gamma = 0.5, 0.95
eps, eps_min, eps_decay = 1.0, 0.05, 0.995
for episode in range(2000):
s, _ = env.reset(seed=42 + episode)
done = False
while not done:
a = int(rng.integers(4)) if rng.random() < eps else int(np.argmax(Q[s]))
s2, r, terminated, truncated, _ = env.step(a)
target = r + gamma * np.max(Q[s2]) * (not terminated) # no future value after terminal
Q[s, a] += alpha * (target - Q[s, a])
s, done = s2, terminated or truncated
eps = max(eps_min, eps * eps_decay)
# Evaluate the greedy policy (no exploration)
s, _ = env.reset(seed=0)
done, total = False, 0.0
while not done:
s, r, terminated, truncated, _ = env.step(int(np.argmax(Q[s])))
total += r
done = terminated or truncated
print("greedy policy reaches the goal:", total == 1.0)
print(np.argmax(Q, axis=1).reshape(4, 4)) # learned action per cell (0=left, 1=down, 2=right, 3=up)
Nothing in the loop uses env.unwrapped.P (FrozenLake’s transition table); the agent learns only from the (s, a, r, s2) samples, which is exactly what “model-free” means. On the deterministic 4×4 map the greedy policy normally reaches the goal after training.
Model-Free vs Model-Based
Q-learning (model-free)
Value iteration (model-based)
Needs P and R
No
Yes
Update uses
One sampled transition
Expectation over all next states
Data needed
Many interactions
None (just the model)
Limitations
Limitation
Remedy
Tables do not scale
Function approximation (DQN)
Maximization bias (overestimation)
Double Q-learning
Slow credit assignment
n-step returns, eligibility traces
Continuous actions
Actor-critic (DDPG, SAC)
Common Pitfalls
Bootstrapping from terminal states.
Never decaying ε, or evaluating with exploration on.
Treating time-limit truncation as termination (it should still bootstrap).
Interview Tips
Write the update and name the TD target and TD error.
Explain model-free (samples, no P/R) and off-policy (max in the target).
Mention convergence conditions and the move to DQN.
12
Describe the Monte Carlo method in reinforcement learning.
KnownRevise
Monte Carlo (MC) methods in reinforcement learning estimate value functions by averaging the actual returns observed after visiting states (or state-action pairs) over complete episodes: run an episode with the policy, compute the return G_t = r(t+1) + γ r(t+2) + … from each step to the end, and set V(s) (or Q(s, a)) to the average of the returns that followed visits to s. They are model-free (only sampled episodes are needed), they do not bootstrap from other estimates, so they are unbiased, but their estimates have high variance because a return sums many random rewards, and they can only update after an episode ends, so they need episodic tasks. First-visit MC averages the return after the first visit to s in each episode; every-visit MC uses all visits. MC control alternates evaluation of Q with ε-greedy policy improvement.
Read full answerHide full answer
Applies toPython 3.10+, NumPy, Gymnasium 1.x
Worked Example (γ = 1)
Episode: s₀ = A (reward 1) → s₁ = B (reward 0) → s₂ = A (reward 2) → end.
import numpy as np
from collections import defaultdict
def returns_from(rewards, gamma):
G, out = 0.0, []
for r in reversed(rewards):
G = r + gamma * G
out.append(G)
return out[::-1]
states, rewards = ["A", "B", "A"], [1.0, 0.0, 2.0]
G = returns_from(rewards, gamma=1.0)
print(G) # [3.0, 2.0, 2.0]
first_visit, every_visit = {}, defaultdict(list)
for t, s in enumerate(states):
every_visit[s].append(G[t])
first_visit.setdefault(s, G[t])
print(first_visit) # {'A': 3.0, 'B': 2.0}
print({s: float(np.mean(v)) for s, v in every_visit.items()}) # {'A': 2.5, 'B': 2.0}
Code Example: First-Visit MC Control on Blackjack (Gymnasium)
import gymnasium as gym
import numpy as np
from collections import defaultdict
env = gym.make("Blackjack-v1")
rng = np.random.default_rng(42)
Q = defaultdict(lambda: np.zeros(2)) # actions: 0 = stick, 1 = hit
N = defaultdict(lambda: np.zeros(2))
eps = 0.1
for episode in range(50_000):
s, _ = env.reset(seed=episode)
trajectory, done = [], False
while not done:
a = int(rng.integers(2)) if rng.random() < eps else int(np.argmax(Q[s]))
s2, r, terminated, truncated, _ = env.step(a)
trajectory.append((s, a, r))
s, done = s2, terminated or truncated
G = 0.0
for t in reversed(range(len(trajectory))): # gamma = 1 for blackjack
s_t, a_t, r_t = trajectory[t]
G = r_t + G
if (s_t, a_t) not in [(x[0], x[1]) for x in trajectory[:t]]: # first visit only
N[s_t][a_t] += 1
Q[s_t][a_t] += (G - Q[s_t][a_t]) / N[s_t][a_t] # running average
print(int(np.argmax(Q[(20, 10, False)]))) # with a 20 against a dealer 10, the usual result is 0 (stick)
Blackjack suits MC well: episodes are short, the reward comes only at the end, and the dynamics (card probabilities) are awkward to write down, so learning from sampled games is easier than building a model.
MC vs TD vs DP
Property
Monte Carlo
TD (SARSA, Q-learning)
Dynamic programming
Needs model
No
No
Yes
Bootstraps
No
Yes
Yes
Bias
Unbiased
Biased (by estimates)
—
Variance
High
Lower
—
Update timing
End of episode
Every step
Sweep over states
Continuing tasks
No
Yes
Yes
Common Pitfalls
Using MC for very long or continuing tasks (no episode end, huge variance).
Forgetting exploration in MC control (exploring starts or ε-soft policies).
Using a fixed step size and calling it a sample average (fine for non-stationary problems, but different).
Interview Tips
Define the return and first-visit vs every-visit averaging.
State unbiased/high variance and episode-end updates.
Contrast with TD methods and give a good use case (blackjack, short episodic games).
13
How do temporal-difference methods like SARSA differ from Monte Carlo methods?
KnownRevise
Temporal-difference (TD) methods update value estimates after every step using a bootstrapped target — the observed reward plus the discounted current estimate of the next state, r + γ V(s′) (or r + γ Q(s′, a′) in SARSA) — while Monte Carlo methods wait until the episode ends and update toward the full observed return G_t. That difference drives the trade-offs: TD can learn online and in continuing tasks, its targets have lower variance (one random reward plus an estimate), and it usually learns faster in practice, but its targets are biased because they depend on current, possibly wrong, estimates; MC is unbiased but high-variance and needs complete episodes. SARSA is the on-policy TD control method whose update uses the quintuple (s, a, r, s′, a′): Q(s, a) ← Q(s, a) + α [r + γ Q(s′, a′) − Q(s, a)]. n-step TD and TD(λ) sit between the two extremes.
Read full answerHide full answer
Applies toPython 3.10+, NumPy, Gymnasium 1.x
Comparison
Aspect
Monte Carlo
TD (TD(0), SARSA, Q-learning)
Target
G_t (full return)
r + γ V(s′) (bootstrapped)
Update time
End of episode
Every step
Bias
None
Yes (depends on estimates)
Variance
High
Low
Continuing tasks
No
Yes
Markov property
Not exploited
Exploited (efficient when it holds)
Sensitivity to initial values
Low
Higher
Function approximation
Stable
Can be unstable (deadly triad)
One Update by Hand
V(s) = 0.5, α = 0.1, γ = 1, observed r = 0 and V(s′) = 0.6, and the episode’s eventual return from s is G = 1.
TD(0): V(s) ← 0.5 + 0.1 (0 + 0.6 − 0.5) = 0.51
MC: V(s) ← 0.5 + 0.1 (1 − 0.5) = 0.55
Code Example: SARSA on CliffWalking (Gymnasium)
import gymnasium as gym
import numpy as np
try:
env = gym.make("CliffWalking-v1") # 4x12 grid, -1 per step, -100 for the cliff
except gym.error.Error: # older Gymnasium releases name it v0
env = gym.make("CliffWalking-v0")
n_s, n_a = env.observation_space.n, env.action_space.n
rng = np.random.default_rng(42)
alpha, gamma, eps = 0.5, 1.0, 0.1
def eps_greedy(Q, s):
return int(rng.integers(n_a)) if rng.random() < eps else int(np.argmax(Q[s]))
Q = np.zeros((n_s, n_a))
for episode in range(500):
s, _ = env.reset(seed=42 + episode)
a = eps_greedy(Q, s)
done = False
while not done:
s2, r, terminated, truncated, _ = env.step(a)
a2 = eps_greedy(Q, s2) # the action SARSA will actually take
target = r + gamma * Q[s2, a2] * (not terminated)
Q[s, a] += alpha * (target - Q[s, a]) # updated every step, not at episode end
s, a, done = s2, a2, terminated or truncated
print(np.argmax(Q, axis=1).reshape(4, 12)) # 0=up, 1=right, 2=down, 3=left
SARSA learns from every step, and because its target includes its own ε-greedy actions, it tends to learn a path that keeps a row away from the cliff (a safer route), as in Sutton and Barto’s classic example.
Between the Extremes
Method
Target
TD(0)
r(t+1) + γ V(s(t+1))
n-step TD
r(t+1) + … + γⁿ⁻¹ r(t+n) + γⁿ V(s_(t+n))
TD(λ)
λ-weighted average of all n-step returns (eligibility traces)
Monte Carlo
Full return G_t
Common Pitfalls
Saying TD is always better; with poor features or non-Markov states, MC can be more reliable.
Bootstrapping across a terminal state.
Confusing SARSA (on-policy, uses a′) with Q-learning (off-policy, uses max).
Interview Tips
Contrast the two targets and the bias-variance trade-off.
Write the SARSA update and explain the (s, a, r, s′, a′) name.
Mention n-step and TD(λ) as a continuum.
14
What is a Deep Q-Network (DQN), and how does it use deep learning?
KnownRevise
A Deep Q-Network is Q-learning with a neural network as the Q-function: the network takes a state (in the original 2015 DeepMind work, a stack of four 84×84 Atari frames) and outputs one Q-value per action, it is trained to minimize the squared or Huber error between Q_θ(s, a) and the target r + γ max_a′ Q_θ⁻(s′, a′), and the agent acts ε-greedily on its outputs. Plain Q-learning with a neural network is unstable, because consecutive samples are correlated and the target moves with every update. DQN made it work with two ideas: experience replay (store transitions in a buffer and train on random mini-batches) and a target network θ⁻ (a periodically copied version of the network used to compute targets). With a convolutional network learning features from raw pixels, one architecture and hyperparameter set reached human-level play on many Atari games. Later improvements include Double DQN, dueling networks, prioritized replay, n-step returns and distributional heads (Rainbow).
Stable targets; copied every C steps (or Polyak-averaged)
Loss
Huber (smooth L1) on the TD error
Optimizer
Adam / RMSProp, gradient clipping
Code Example: Compact DQN for CartPole (PyTorch)
import random
from collections import deque
import gymnasium as gym
import numpy as np
import torch
from torch import nn
env = gym.make("CartPole-v1")
torch.manual_seed(42)
random.seed(42)
def make_net():
return nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU(), nn.Linear(128, 2))
q_net, target_net = make_net(), make_net()
target_net.load_state_dict(q_net.state_dict())
optimizer = torch.optim.Adam(q_net.parameters(), lr=1e-3)
buffer = deque(maxlen=50_000)
gamma, batch_size, target_every = 0.99, 64, 500
eps, step = 1.0, 0
s, _ = env.reset(seed=42)
for step in range(1, 30_001):
if random.random() < eps:
a = env.action_space.sample()
else:
with torch.no_grad():
a = int(q_net(torch.as_tensor(s, dtype=torch.float32)).argmax())
s2, r, terminated, truncated, _ = env.step(a)
buffer.append((s, a, r, s2, terminated))
s = s2
if terminated or truncated:
s, _ = env.reset()
eps = max(0.05, 1.0 - step / 10_000) # linear decay
if len(buffer) >= 1_000:
batch = random.sample(buffer, batch_size)
states, actions, rewards, next_states, dones = map(np.array, zip(*batch))
states = torch.as_tensor(states, dtype=torch.float32)
next_states = torch.as_tensor(next_states, dtype=torch.float32)
actions = torch.as_tensor(actions).unsqueeze(1)
rewards = torch.as_tensor(rewards, dtype=torch.float32)
dones = torch.as_tensor(dones, dtype=torch.float32)
q_sa = q_net(states).gather(1, actions).squeeze(1)
with torch.no_grad():
target = rewards + gamma * target_net(next_states).max(1).values * (1 - dones)
loss = nn.functional.smooth_l1_loss(q_sa, target)
optimizer.zero_grad()
loss.backward()
nn.utils.clip_grad_norm_(q_net.parameters(), 10.0)
optimizer.step()
if step % target_every == 0:
target_net.load_state_dict(q_net.state_dict()) # hard target update
Only terminated zeroes the future value; a time-limit truncated episode still bootstraps. For real projects, stable_baselines3.DQN("MlpPolicy", env).learn(100_000) gives a tested implementation.
Why It Works and Where It Struggles
Strength
Weakness
Learns from raw, high-dimensional inputs
Discrete actions only
Off-policy, reuses data
Overestimates Q (fixed by Double DQN)
One recipe across many games
Sample-hungry (millions of frames)
Sensitive to hyperparameters
Common Pitfalls
No target network or replay buffer (training diverges or oscillates).
Starting training before the buffer has diverse data.
Treating truncation as termination.
Using DQN for continuous actions (use DDPG, TD3 or SAC).
Interview Tips
Describe the network’s input/output and the TD loss.
Explain why replay and target networks are needed.
Name later improvements (Double, Dueling, PER, Rainbow).
15
What is experience replay in DQN, and why is it important?
KnownRevise
Experience replay stores the agent’s transitions (s, a, r, s′, done) in a large buffer and trains the Q-network on mini-batches sampled at random from that buffer instead of on the most recent transition. It matters for three reasons: it breaks temporal correlation (consecutive frames are nearly identical, and training on them in order violates the i.i.d. assumption of gradient descent, causing unstable updates and forgetting); it reuses data, so each costly interaction contributes to many updates (better sample efficiency); and it smooths the training distribution over many past policies instead of only the current one, reducing feedback loops where the policy’s own data reinforces its mistakes. It is possible because Q-learning is off-policy: old transitions from earlier policies are still valid training data. Prioritized experience replay samples transitions with large TD errors more often, with importance-sampling weights to correct the bias.
Read full answerHide full answer
Applies toPython 3.10+, NumPy, PyTorch 2.x
Design Choices
Choice
Typical value
Effect
Capacity
10⁵-10⁶ transitions
Larger = more diverse, but older (more off-policy) data
Warm-up before learning
1k-50k transitions
Avoid overfitting to the first few episodes
Batch size
32-256
Gradient noise vs compute
Replay ratio
Updates per environment step
Higher = more reuse, risk of overfitting
Sampling
Uniform, prioritized
PER speeds learning on informative transitions
n-step returns
Store n-step transitions
Faster credit assignment
Code Example: Uniform Replay Buffer (NumPy)
import numpy as np
class ReplayBuffer:
def __init__(self, capacity, obs_dim, seed=42):
self.capacity, self.idx, self.size = capacity, 0, 0
self.obs = np.zeros((capacity, obs_dim), dtype=np.float32)
self.next_obs = np.zeros((capacity, obs_dim), dtype=np.float32)
self.actions = np.zeros(capacity, dtype=np.int64)
self.rewards = np.zeros(capacity, dtype=np.float32)
self.dones = np.zeros(capacity, dtype=np.float32)
self.rng = np.random.default_rng(seed)
def add(self, s, a, r, s2, done):
i = self.idx
self.obs[i], self.actions[i], self.rewards[i], self.next_obs[i], self.dones[i] = s, a, r, s2, done
self.idx = (self.idx + 1) % self.capacity # circular: overwrite the oldest
self.size = min(self.size + 1, self.capacity)
def sample(self, batch_size):
ids = self.rng.integers(0, self.size, size=batch_size)
return self.obs[ids], self.actions[ids], self.rewards[ids], self.next_obs[ids], self.dones[ids]
buf = ReplayBuffer(capacity=5, obs_dim=4)
for t in range(7): # add 7 transitions to a buffer of 5
buf.add(np.full(4, t), t % 2, 1.0, np.full(4, t + 1), False)
print(buf.size, buf.idx) # 5 2
print(sorted(buf.obs[:, 0].astype(int).tolist())) # [2, 3, 4, 5, 6] oldest two were overwritten
s, a, r, s2, d = buf.sample(3)
print(s.shape, a.shape) # (3, 4) (3,)
Prioritized Experience Replay (Schaul et al., 2016)
Formula
Meaning
p_i = |δ_i| + ε
Priority from the TD error
P(i) = p_i^α / Σ_k p_k^α
Sampling probability (α ≈ 0.6)
w_i = (N · P(i))^(−β) / max_j w_j
Importance weight, β annealed to 1
High-error transitions are replayed more often, and the weights stop that preference from biasing the learned Q-values. A sum-tree keeps sampling at O(log N).
Common Pitfalls
Using replay with on-policy methods (PPO, A2C) without corrections.
A buffer so small that it only holds recent, correlated data.
Storing terminated and truncated together as one “done” flag.
Storing large image observations naively (store frames once and index stacks).
Interview Tips
Give the three reasons: decorrelation, data reuse, smoother distribution.
Tie it to off-policy learning.
Mention PER and its importance-sampling correction.
16
What are the main elements of the Proximal Policy Optimization (PPO) algorithm?
KnownRevise
PPO is an on-policy actor-critic algorithm whose core idea is to improve the policy as much as possible from a batch of collected data without moving it too far from the policy that collected it, using a clipped surrogate objective: L^CLIP = E[min(r_t(θ) Â_t, clip(r_t(θ), 1 − ε, 1 + ε) Â_t)], where r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t) is the probability ratio and Â_t the advantage estimate. Its main elements are: rollouts collected with the current policy (often from parallel environments); Generalized Advantage Estimation (GAE) to compute advantages from a learned value function; the clipped policy loss; a value-function loss (often clipped too); an entropy bonus for exploration; and several epochs of mini-batch SGD over each batch. It gets much of TRPO’s stability with only first-order optimization, which is why it is a default for continuous control, games and RLHF fine-tuning of language models.
The printed float32 values are 2.4, −0.8 and 1.0 up to rounding.
Code Example: PPO with Stable-Baselines3
import gymnasium as gym
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy
env = gym.make("CartPole-v1")
model = PPO("MlpPolicy", env, n_steps=2048, batch_size=64, n_epochs=10,
gamma=0.99, gae_lambda=0.95, clip_range=0.2, ent_coef=0.0, seed=42, verbose=0)
model.learn(total_timesteps=100_000)
mean_reward, std_reward = evaluate_policy(model, env, n_eval_episodes=10)
print(f"{mean_reward:.1f} +/- {std_reward:.1f}")
Common Pitfalls
Too many epochs or too large a learning rate: the policy drifts despite clipping (monitor approximate KL and clip fraction).
Not normalizing advantages or observations.
Mixing up the sign of the loss.
Treating truncated episodes as terminal in GAE.
Interview Tips
Write the clipped objective and explain the ratio.
List GAE, value loss, entropy bonus and multiple epochs.
Compare with TRPO (trust region via KL constraint) and mention PPO in RLHF.
17
How do actor-critic methods work in reinforcement learning?
KnownRevise
Actor-critic methods combine a policy-gradient learner and a value-based learner: the actor is a policy π_θ(a|s) that chooses actions, and the critic is a value function (V_w(s) or Q_w(s, a)) that evaluates them; the critic is trained by temporal-difference learning, and the actor is updated in the direction suggested by the critic, typically ∇θ log π_θ(a|s) · Â, where the advantage  = r + γ V_w(s′) − V_w(s) (the TD error) says whether the action turned out better or worse than expected. Using the critic’s estimate instead of the full Monte Carlo return (as REINFORCE does) greatly reduces variance and allows online, per-step or per-rollout updates, at the cost of some bias. The family includes A2C/A3C, PPO and TRPO (on-policy, with V critics and GAE), and DDPG, TD3 and SAC (off-policy, with Q critics for continuous actions).
w ← w + α_w δ ∇w V_w(s), δ = r + γV_w(s′) − V_w(s)
Advantage Â
Was the action better than average?
δ (one-step), n-step, or GAE
Variants
Method
Critic
On/off-policy
Actions
One-step actor-critic
V
On
Discrete/continuous
A2C / A3C
V, n-step
On
Both
PPO / TRPO
V, GAE
On
Both
DDPG / TD3
Q (TD3: two)
Off
Continuous
SAC
Two Q + entropy
Off
Continuous (and discrete variants)
Code Example: One-Step Actor-Critic on CartPole (PyTorch)
import gymnasium as gym
import torch
from torch import nn
env = gym.make("CartPole-v1")
torch.manual_seed(42)
actor = nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 2))
critic = nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 1))
actor_opt = torch.optim.Adam(actor.parameters(), lr=1e-3)
critic_opt = torch.optim.Adam(critic.parameters(), lr=5e-3)
gamma = 0.99
episode_lengths = []
for episode in range(500):
s, _ = env.reset(seed=42 + episode)
done, steps = False, 0
while not done:
s_t = torch.as_tensor(s, dtype=torch.float32)
dist = torch.distributions.Categorical(logits=actor(s_t))
a = dist.sample()
s2, r, terminated, truncated, _ = env.step(int(a))
v = critic(s_t).squeeze()
with torch.no_grad():
v_next = critic(torch.as_tensor(s2, dtype=torch.float32)).squeeze() * (not terminated)
td_target = r + gamma * v_next
delta = td_target - v # TD error = advantage estimate
critic_loss = delta.pow(2)
critic_opt.zero_grad()
critic_loss.backward()
critic_opt.step()
actor_loss = -dist.log_prob(a) * delta.detach() # push up actions that beat expectations
actor_opt.zero_grad()
actor_loss.backward()
actor_opt.step()
s, done, steps = s2, terminated or truncated, steps + 1
episode_lengths.append(steps)
print(sum(episode_lengths[-20:]) / 20) # average length of the last 20 episodes
Episode lengths usually rise well above the random-policy level (about 20 steps) as training goes on, though one-step actor-critic can be noisy; the exact average depends on the run. For robust training use PPO("MlpPolicy", env) or A2C("MlpPolicy", env) from Stable-Baselines3.
Actor-Critic vs Pure Methods
Method
Strength
Weakness
Value-based (DQN)
Sample-efficient, off-policy
Discrete actions; argmax over actions
Policy gradient (REINFORCE)
Any action space, stochastic policies
High variance
Actor-critic
Lower variance, continuous actions, online
Two networks to tune; critic bias
Common Pitfalls
Backpropagating the actor loss into the critic (detach the advantage).
Bootstrapping from terminal states.
Critic learning rate too low, so the actor follows a poor critic.
No entropy regularization, leading to premature convergence.
Interview Tips
Name the actor and critic roles and write both updates.
Explain variance reduction via the advantage/TD error.
Place A2C, PPO, DDPG, TD3 and SAC in the family.
18
What improvements does Double DQN make over standard DQN?
KnownRevise
Double DQN (van Hasselt, Guez and Silver, 2016) fixes DQN’s overestimation of Q-values by separating action selection from action evaluation in the target: standard DQN uses y = r + γ max_a′ Q_θ⁻(s′, a′), where the same target network both picks the best next action and scores it; Double DQN uses y = r + γ Q_θ⁻(s′, argmax_a′ Q_θ(s′, a′)), where the online network picks the action and the target network evaluates it. The max of noisy estimates is biased upward (it tends to pick actions whose value happens to be overestimated), and with bootstrapping that bias compounds through the value function, leading to over-optimistic values and worse policies. Decoupling the two roles largely removes the bias, at almost no cost — one extra forward pass, and the target network DQN already has — and it improved scores on many Atari games. It is part of Rainbow DQN, and the same idea appears as clipped double Q-learning in TD3 and SAC.
Read full answerHide full answer
Applies toPyTorch 2.x, NumPy
Why max Overestimates
If all actions have true value 0 and the estimates have independent noise, E[max_a Q̂(a)] > 0 even though max_a Q(a) = 0. For 10 actions with N(0, 1) noise, the expected maximum is about 1.54.
import numpy as np
rng = np.random.default_rng(42)
noisy_q = rng.normal(0.0, 1.0, size=(100_000, 10)) # true value of every action is 0
print(round(noisy_q.max(axis=1).mean(), 2)) # about 1.54, an upward bias from noise alone
import torch
online_next = torch.tensor([[3.0, 5.0, 4.0]])
target_next = torch.tensor([[6.0, 2.0, 4.0]])
rewards, dones, gamma = torch.tensor([1.0]), torch.tensor([0.0]), 0.9
dqn_target = rewards + gamma * target_next.max(dim=1).values * (1 - dones)
best_actions = online_next.argmax(dim=1, keepdim=True) # select with the online network
double_target = rewards + gamma * target_next.gather(1, best_actions).squeeze(1) * (1 - dones)
print([round(v, 4) for v in dqn_target.tolist()], [round(v, 4) for v in double_target.tolist()]) # [6.4] [2.8]
In a training loop the only change from DQN is these two lines; compute both inside torch.no_grad().
Related Improvements
Method
Fixes
Double DQN
Overestimation bias
Dueling DQN
Separate V(s) and advantage streams: better value estimates
Prioritized replay
Replays informative transitions more
n-step returns
Faster reward propagation
Distributional (C51, QR-DQN)
Learns the return distribution
Noisy nets
Learned exploration
Rainbow
All of the above combined
Clipped double Q (TD3, SAC)
min of two critics for continuous control
Common Pitfalls
Using the target network for both selection and evaluation and calling it Double DQN.
Forgetting torch.no_grad() for the online network’s argmax pass (harmless for gradients through argmax, but wasteful).
Expecting Double DQN to fix every instability; it targets overestimation only.
Interview Tips
Explain maximization bias with the noisy-max example.
Write both targets and show they differ only in who selects the action.
Mention TD3’s clipped double Q as the continuous-control version.
19
What role do target networks play in stabilizing deep RL training?
KnownRevise
A target network is a separate, slowly changing copy of the value network used only to compute the bootstrapped targets (for example y = r + γ max_a′ Q_θ⁻(s′, a′) in DQN), while the main network Q_θ is trained toward those targets. Without it, the same parameters produce both the prediction and the target, so every gradient step also moves the target — like chasing a moving goalpost — which leads to oscillation, feedback loops and divergence, especially when combined with function approximation and off-policy bootstrapping (the “deadly triad”). The target network holds the targets fixed for a while, turning learning into a sequence of more stable regression problems. It is updated either by a hard copy every C steps (DQN: every 10,000 steps on Atari) or by soft Polyak averaging every step, θ⁻ ← τ θ + (1 − τ) θ⁻ with small τ (DDPG, TD3 and SAC use τ = 0.005).
Read full answerHide full answer
Applies toPyTorch 2.x
Hard vs Soft Updates
Update
Rule
Used in
Trade-off
Hard
θ⁻ ← θ every C steps
DQN, Double DQN
Simple; targets jump every C steps
Soft (Polyak)
θ⁻ ← τθ + (1 − τ)θ⁻ every step
DDPG, TD3, SAC
Smooth; τ controls lag
A soft update with τ has an effective lag of about 1/τ steps (200 for τ = 0.005).
Code Example: Hard and Soft Updates (PyTorch)
import copy
import torch
from torch import nn
torch.manual_seed(42)
q_net = nn.Linear(4, 2)
target_net = copy.deepcopy(q_net)
for p in target_net.parameters():
p.requires_grad_(False) # the target network is never trained directly
def hard_update(target, online):
target.load_state_dict(online.state_dict())
@torch.no_grad()
def soft_update(target, online, tau):
for t_param, o_param in zip(target.parameters(), online.parameters()):
t_param.mul_(1 - tau).add_(tau * o_param)
# Soft update by hand: target 0, online 1, tau 0.005
t, o = nn.Linear(1, 1, bias=False), nn.Linear(1, 1, bias=False)
with torch.no_grad():
t.weight.fill_(0.0)
o.weight.fill_(1.0)
soft_update(t, o, 0.005)
print(round(t.weight.item(), 6)) # 0.005
soft_update(t, o, 0.005)
print(round(t.weight.item(), 6)) # 0.009975 = 0.005 + 0.005 * (1 - 0.005)
# Inside a DQN step, targets come from target_net with no gradient
s2, r, done = torch.randn(8, 4), torch.ones(8), torch.zeros(8)
with torch.no_grad():
y = r + 0.99 * target_net(s2).max(dim=1).values * (1 - done)
print(y.requires_grad) # False
Why It Helps
Without target network
With target network
Update raises Q(s, a) and similar Q(s′, ·), so the target rises too
Target fixed while Q(s, a) moves toward it
Overestimation feeds on itself
Feedback loop slowed
Oscillating or diverging losses
Closer to supervised regression
Common Pitfalls
Computing targets with gradients enabled (gradients flow into the target).
Updating too often (no stabilization) or too rarely (slow learning on stale targets).
Forgetting to copy the weights at the start, so the target network is random.
Using a target network for the actor and forgetting the critic’s (DDPG/TD3 use both).
Interview Tips
Explain the moving-target problem and the deadly triad.
Give hard and Polyak updates with typical C and τ.
Mention that Double DQN reuses the target network to decouple selection from evaluation.
20
How does the Asynchronous Advantage Actor-Critic (A3C) algorithm work?
KnownRevise
A3C (Mnih et al., 2016) trains an actor-critic network with many parallel workers: each worker has its own copy of the environment and of the network, collects a short rollout (for example 5-20 steps), computes n-step returns and advantages, calculates gradients of the actor loss (−log π(a|s) · advantage), the critic loss (squared error of V(s)) and an entropy bonus, and applies them asynchronously to a shared global network, then copies the global weights back and continues. Running different workers in different parts of the environment at the same time decorrelates the data, which takes the place of a replay buffer, and it ran efficiently on multi-core CPUs. Its synchronous version, A2C, waits for all workers and makes one batched update; it performs as well or better and uses GPUs more efficiently, so A2C (and later PPO) mostly replaced A3C in practice.
Code Example: The Worker’s Loss Computation (PyTorch)
import numpy as np
import torch
from torch import nn
def n_step_returns(rewards, bootstrap, gamma):
R, out = bootstrap, []
for r in reversed(rewards):
R = r + gamma * R
out.append(R)
return out[::-1]
returns = n_step_returns([1.0, 1.0, 1.0], bootstrap=5.0, gamma=0.9)
values = np.array([6.0, 5.0, 5.0])
print(np.round(returns, 3).tolist()) # [6.355, 5.95, 5.5]
print(np.round(np.array(returns) - values, 3).tolist()) # [0.355, 0.95, 0.5]
class ActorCritic(nn.Module):
def __init__(self, obs_dim, n_actions):
super().__init__()
self.body = nn.Sequential(nn.Linear(obs_dim, 128), nn.ReLU())
self.pi = nn.Linear(128, n_actions)
self.v = nn.Linear(128, 1)
def forward(self, x):
h = self.body(x)
return self.pi(h), self.v(h).squeeze(-1)
def worker_loss(model, states, actions, returns, value_coef=0.5, entropy_coef=0.01):
logits, values = model(states)
dist = torch.distributions.Categorical(logits=logits)
advantages = returns - values
actor_loss = -(dist.log_prob(actions) * advantages.detach()).mean() # no gradient into the critic here
critic_loss = advantages.pow(2).mean()
return actor_loss + value_coef * critic_loss - entropy_coef * dist.entropy().mean()
torch.manual_seed(42)
global_model = ActorCritic(4, 2)
global_model.share_memory() # A3C: workers update shared parameters
local_model = ActorCritic(4, 2)
local_model.load_state_dict(global_model.state_dict())
loss = worker_loss(local_model, torch.randn(3, 4), torch.tensor([0, 1, 0]), torch.tensor(returns, dtype=torch.float32))
loss.backward()
for g_param, l_param in zip(global_model.parameters(), local_model.parameters()):
g_param.grad = l_param.grad # hand local gradients to the global model
print(all(p.grad is not None for p in global_model.parameters())) # True
In full A3C, several processes started with torch.multiprocessing each run this loop and call a shared optimizer’s step() on the global model. For practical use, stable_baselines3.A2C("MlpPolicy", env) with vectorized environments gives the synchronous version.
A3C vs A2C vs PPO
Aspect
A3C
A2C
PPO
Updates
Asynchronous, per worker
Synchronous, batched
Synchronous, several epochs per batch
Hardware
Many CPU cores
GPU-friendly
GPU-friendly
Stale gradients
Yes
No
No
Data reuse
One update per rollout
One update per rollout
Multiple epochs (clipped)
Today
Mostly historical
Simple baseline
Default choice
Common Pitfalls
Letting actor-loss gradients flow into the critic through the advantage (detach it).
Forgetting the entropy bonus, so the policy collapses early.
Bootstrapping from V at terminal states.
Assuming asynchrony is what made A3C work; parallel data collection was the main benefit.
Interview Tips
Describe workers, n-step advantages, the three loss terms and asynchronous updates.
What is reward shaping, and how does it affect an RL agent?
KnownRevise
Reward shaping adds extra reward signals to the environment’s original reward to guide the agent toward useful behaviour sooner, typically when the true reward is sparse (for example, +1 only on reaching a goal) and random exploration would almost never find it. Done well, it speeds up learning dramatically by giving intermediate feedback (“getting closer is good”). Done badly, it changes the optimal policy: the agent learns to collect the shaping bonus instead of solving the task (reward hacking), such as circling a checkpoint forever to farm points. The safe form is potential-based reward shaping (Ng, Harada and Russell, 1999): add F(s, s′) = γΦ(s′) − Φ(s) for any potential function Φ over states; this provably leaves the optimal policy unchanged, because the shaping terms telescope along any trajectory and only shift values by Φ(s₀).
Over an episode, Σ_t γᵗ [γΦ(s_(t+1)) − Φ(s_t)] = γᵀΦ(s_T) − Φ(s₀), which telescopes. With Φ = 0 at terminal states, the shaping changes every return from s₀ by the same constant −Φ(s₀), so the ranking of policies, and therefore the optimal policy, does not change.
import gymnasium as gym
class DistanceShaping(gym.Wrapper):
"""Adds gamma * phi(s') - phi(s), with phi = -Manhattan distance to the goal on FrozenLake."""
def __init__(self, env, gamma=0.99, size=4):
super().__init__(env)
self.gamma, self.size = gamma, size
self.goal = (size - 1, size - 1)
def phi(self, s, terminal=False):
if terminal:
return 0.0 # potential is 0 at terminal states
row, col = divmod(int(s), self.size)
return -float(abs(row - self.goal[0]) + abs(col - self.goal[1]))
def reset(self, **kwargs):
obs, info = self.env.reset(**kwargs)
self.last_obs = obs
return obs, info
def step(self, action):
obs, reward, terminated, truncated, info = self.env.step(action)
shaping = self.gamma * self.phi(obs, terminated) - self.phi(self.last_obs)
self.last_obs = obs
info["shaping"] = shaping
return obs, reward + shaping, terminated, truncated, info
env = DistanceShaping(gym.make("FrozenLake-v1", is_slippery=False))
obs, _ = env.reset(seed=42)
obs, r, terminated, truncated, info = env.step(2) # move right: one step closer to the goal
print(obs, round(info["shaping"], 2)) # 1 1.05
Moving from cell 0 (distance 6) to cell 1 (distance 5) gives γΦ(s′) − Φ(s) = 0.99 × (−5) + 6 = 1.05. Moving away from the goal gives a negative bonus (0.99 × (−6) + 5 = −0.94). Over any loop the discounted bonuses telescope to a small constant, so walking back and forth never beats reaching the goal: the optimal policy is unchanged.
Common Pitfalls
Ad-hoc bonuses that can be farmed (agent loops near a checkpoint).
Step penalties so large that ending the episode early (falling in a hole) beats trying.
Forgetting Φ = 0 at terminal states, which breaks the policy-invariance guarantee.
Evaluating the agent on the shaped reward instead of the true task reward.
Interview Tips
Define shaping and why sparse rewards need it.
State the potential-based formula and the policy-invariance result.
Give a concrete reward-hacking example.
22
What are policy gradients, and how are they used to learn policies?
KnownRevise
Policy gradient methods learn a parameterized policy π_θ(a|s) directly by gradient ascent on the expected return J(θ) = E_(τ∼π_θ)[G(τ)], using the policy gradient theorem: ∇θ J(θ) = E[Σ_t ∇θ log π_θ(a_t|s_t) · Ψ_t], where Ψ_t is a measure of how good the action was — the return G_t (REINFORCE), the return minus a baseline G_t − b(s_t), or an advantage estimate Â_t (actor-critic). Intuitively, it increases the log-probability of actions that led to better-than-expected outcomes and decreases it for worse ones. No model of the environment is needed, and the “log-derivative trick” means the gradient only needs samples and the policy’s own log-probabilities. Policy gradients handle continuous and high-dimensional actions and stochastic policies naturally, and they are the basis of REINFORCE, A2C/A3C, TRPO, PPO and the RL stage of RLHF. Their main weakness is high variance, reduced with baselines, advantages, GAE and large batches.
Read full answerHide full answer
Applies toPyTorch 2.x, Gymnasium 1.x
The Log-Derivative Trick
∇θ E_(a∼π_θ)[f(a)] = E[f(a) ∇θ log π_θ(a)], because ∇θ π_θ = π_θ ∇θ log π_θ. For trajectories, the environment’s transition probabilities do not depend on θ, so they drop out of ∇θ log p(τ), leaving Σ_t ∇θ log π_θ(a_t|s_t).
Choices for Ψ_t
Ψ_t
Method
Variance
G(τ), total return
Basic REINFORCE
Highest
G_t, reward-to-go
REINFORCE (causality)
High
G_t − V(s_t)
REINFORCE with baseline
Lower
Q(s_t, a_t)
Actor-critic with Q critic
Lower, biased
Â_t (TD error, n-step, GAE)
A2C, PPO
Low, tunable bias
Code Example: Policy Gradient Step on a Bandit (PyTorch)
import torch
torch.manual_seed(42)
logits = torch.zeros(3, requires_grad=True) # policy over 3 actions, starts uniform
true_rewards = torch.tensor([1.0, 0.0, 2.0])
optimizer = torch.optim.SGD([logits], lr=0.1)
for step in range(300):
dist = torch.distributions.Categorical(logits=logits)
actions = dist.sample((64,))
rewards = true_rewards[actions] + 0.1 * torch.randn(64)
baseline = rewards.mean() # reduces variance, keeps the gradient unbiased in expectation
loss = -(dist.log_prob(actions) * (rewards - baseline)).mean()
optimizer.zero_grad()
loss.backward()
optimizer.step()
print(torch.softmax(logits, 0).argmax().item()) # 2: the policy concentrates on the best action
The loss is the negative of the policy-gradient objective, so minimizing it with SGD performs gradient ascent on expected reward. The same loss, with states as network inputs and returns or advantages as weights, gives REINFORCE and actor-critic methods.
Advantages and Disadvantages
Advantages
Disadvantages
Continuous and large action spaces
High-variance gradients
Stochastic policies (exploration, POMDPs, games)
Sample-inefficient (on-policy)
Smooth policy changes, convergence to local optimum
Local optima, sensitive step size
Works with any differentiable policy
Needs many episodes
Common Pitfalls
Wrong sign (descending on reward instead of ascending).
No baseline or advantage normalization, causing very noisy training.
Steps too large, collapsing the policy (motivates TRPO/PPO).
Reusing old trajectories without importance correction.
Interview Tips
Write the policy gradient theorem and explain the log-derivative trick.
Explain variance reduction with baselines and advantages.
Connect to REINFORCE, actor-critic, PPO and RLHF.
23
What are common challenges in designing reward functions for RL?
KnownRevise
Designing reward functions is hard because the agent optimizes exactly what you reward, not what you meant: the main challenges are sparse rewards (the signal is too rare to learn from), reward hacking or specification gaming (the agent finds loopholes that score well without doing the task), delayed rewards that make credit assignment difficult, balancing several objectives with weights, unintended side effects that the reward does not penalize, reward scale that destabilizes learning, and rewards that are hard to measure at all (helpfulness, comfort, safety). Classic examples: a boat-racing agent that circled to collect respawning targets instead of finishing the race (OpenAI’s CoastRunners example), and a simulated robot exploiting physics bugs. Mitigations include potential-based shaping, careful normalization, constraints instead of penalties (constrained RL), learning rewards from demonstrations or human preferences (inverse RL, RLHF), and thorough evaluation with metrics separate from the training reward.
Reward the outcome, red-team the reward, separate evaluation metrics
Delayed reward
Chess win after 60 moves
Value functions, n-step/TD(λ), discount choice
Multiple objectives
Speed vs energy vs comfort
Explicit weights tuned on held-out evaluation; constrained RL; Pareto analysis
Side effects
Robot knocks over a vase en route
Impact penalties, constraints, human oversight
Reward scale
Rewards of ±1000 destabilize value learning
Normalize/clip rewards, PopArt
Hard-to-measure goals
“Helpful” chatbot answers
Learned reward models from human preferences (RLHF), with KL penalty
Proxy metrics
Clicks instead of satisfaction
Combine proxies; long-term metrics; guardrails
Non-stationarity
Reward based on users who change
Monitoring and periodic retraining
Code Example: A Multi-Objective Reward with Logged Components
from dataclasses import dataclass
@dataclass
class RewardWeights:
progress: float = 1.0
energy: float = 0.05
collision: float = 10.0
def reward_fn(progress_m, energy_j, collided, w=RewardWeights()):
components = {
"progress": w.progress * progress_m,
"energy": -w.energy * energy_j,
"collision": -w.collision * float(collided),
}
return sum(components.values()), components # log components, not just the total
total, parts = reward_fn(progress_m=0.5, energy_j=2.0, collided=False)
print(round(total, 3), parts) # 0.4 {'progress': 0.5, 'energy': -0.1, 'collision': -0.0}
total, parts = reward_fn(progress_m=0.5, energy_j=2.0, collided=True)
print(round(total, 3)) # -9.6
Logging each component makes it possible to spot hacking (progress rising while collisions also rise) and to see which term dominates learning.
A Checklist for Reward Design
Question
Why
Would a policy that maximizes this reward do what I want?
Catch loopholes early
Can the agent get reward without the outcome?
Specification gaming
Are penalties so large that quitting is optimal?
Early-termination exploits
Is the scale reasonable (≈ −10 to 10 per step)?
Stable value learning
What metric will I use to evaluate, separate from the reward?
Detect hacking
Common Pitfalls
Rewarding actions (“press the button”) instead of outcomes.
Tuning reward weights on the same runs used to report results.
Assuming a learned reward model is correct far from its training data (RLHF needs a KL penalty to stay close).
Interview Tips
Name sparse rewards, hacking, delayed credit and multi-objective trade-offs.
Give a concrete specification-gaming example.
Offer mitigations, including learning rewards from humans.
24
Describe TRPO and how it differs from other policy gradient methods.
KnownRevise
Trust Region Policy Optimization (Schulman et al., 2015) is a policy gradient method that limits how much the policy can change in each update: it maximizes the surrogate objective E[(π_θ(a|s) / π_θ_old(a|s)) Â(s, a)] subject to a constraint on the average KL divergence between the old and new policies, E_s[KL(π_θ_old(·|s) ‖ π_θ(·|s))] ≤ δ (for example δ = 0.01). Vanilla policy gradient methods take steps of a fixed size in parameter space, but a small parameter change can cause a huge change in behaviour, collapsing performance and ruining the data future updates depend on. TRPO measures step size in policy space instead, which gives approximate monotonic improvement. It solves the constrained problem with a second-order method: approximate the KL with the Fisher information matrix, compute the natural gradient direction with conjugate gradient (using Fisher-vector products, never forming the matrix), then do a backtracking line search to make sure the constraint holds and the surrogate improves. PPO later replaced the hard constraint with a clipped objective that is simpler and usually as good.
Natural gradient + conjugate gradient + line search
First-order, several epochs
Data reuse
One gradient step per batch
One update per batch
Several epochs per batch
Guarantees
None on improvement
Approximate monotonic improvement
Empirical
Complexity
Low
High
Low
Works with shared actor-critic nets, dropout
Yes
Awkward
Yes
The Update
Step
What happens
1
Collect trajectories with π_old; estimate  with GAE
2
Compute the policy gradient g of the surrogate
3
Solve F x = g with conjugate gradient (F = Fisher matrix, via Fisher-vector products)
4
Step size β = √(2δ / (xᵀ F x)); proposed θ_new = θ_old + β x
5
Backtracking line search: shrink the step until KL ≤ δ and the surrogate improves
6
Fit the value function separately (regression on returns)
Code Example: Conjugate Gradient and the Step Size (NumPy)
import numpy as np
def conjugate_gradient(Avp, b, iters=10, tol=1e-10):
x = np.zeros_like(b)
r, p = b.copy(), b.copy()
rr = r @ r
for _ in range(iters):
Ap = Avp(p)
alpha = rr / (p @ Ap)
x += alpha * p
r -= alpha * Ap
rr_new = r @ r
if rr_new < tol:
break
p = r + (rr_new / rr) * p
rr = rr_new
return x
F = np.array([[4.0, 1.0],
[1.0, 3.0]]) # stand-in Fisher matrix (symmetric positive definite)
g = np.array([1.0, 2.0]) # policy gradient
x = conjugate_gradient(lambda v: F @ v, g) # natural gradient direction F^-1 g
print(np.round(x, 4)) # [0.0909 0.6364]
delta = 0.01
beta = np.sqrt(2 * delta / (x @ F @ x)) # largest step with 0.5 * step^T F step = delta
step = beta * x
print(round(0.5 * step @ F @ step, 6)) # 0.01 (quadratic KL estimate equals delta)
By hand, F⁻¹g solves 4x₁ + x₂ = 1 and x₁ + 3x₂ = 2: x₁ = 1/11 ≈ 0.0909, x₂ = 7/11 ≈ 0.6364. In real TRPO, F @ v is replaced by a Fisher-vector product computed with two automatic-differentiation passes through the KL.
Common Pitfalls
Forming the Fisher matrix explicitly (impossible for large networks).
Skipping the line search; the quadratic KL approximation can be wrong.
Choosing δ too large (unstable) or too small (slow learning).
Expecting TRPO to beat PPO by default; in practice PPO is usually preferred for its simplicity.
Interview Tips
State the constrained objective with the KL trust region.
Explain natural gradient, conjugate gradient and line search.
Contrast with vanilla PG (step size in parameter space) and PPO (clipping).
25
How do you scale reinforcement learning to high-dimensional state spaces?
KnownRevise
Tabular methods break down when states are high-dimensional (images, many sensors, text) because almost every state is new; RL scales by replacing tables with function approximation — neural networks that generalize across similar states — and by learning compact representations of the observations. In practice that means: convolutional encoders for images (DQN on Atari pixels), MLPs for vector states, transformers or recurrent networks for sequences and partial observability; preprocessing such as frame stacking, downsampling, grayscale conversion and observation normalization; representation learning with auxiliary or self-supervised losses (CURL, SPR, contrastive or reconstruction objectives) and data augmentation (DrQ, RAD); world models that learn a low-dimensional latent state and plan or train in it (Dreamer, MuZero); pretrained encoders from vision or language models; and engineering at scale — vectorized environments, distributed actors and learners (IMPALA, Ape-X), and large replay buffers.
Overfitting to visual details of training levels; use augmentation and many training levels.
Interview Tips
Lead with function approximation and representation learning.
Mention preprocessing, augmentation, world models and pretrained encoders.
Add the systems angle: vectorized and distributed training.
26
What strategies transfer knowledge across tasks in reinforcement learning?
KnownRevise
Transfer in reinforcement learning reuses something learned on source tasks — representations, policies, value functions, models, or data — to learn a target task faster or better. Common strategies are: fine-tuning a pretrained policy or encoder on the new task; feature/representation transfer (freeze or partially freeze a pretrained visual encoder); policy distillation and behaviour cloning from source-task policies or demonstrations as initialization; reward and value transfer such as successor features, which separate environment dynamics from the reward so a new reward can reuse the learned dynamics; multi-task and goal-conditioned learning (one policy conditioned on a task or goal, as in universal value functions and hindsight experience replay); meta-RL (learn to adapt quickly); sim-to-real transfer with domain randomization; and model transfer, reusing a learned world model with a new reward. Watch for negative transfer, where source knowledge hurts, and measure jump-start, asymptotic performance and time-to-threshold against training from scratch.
Read full answerHide full answer
Applies toPyTorch 2.x, Stable-Baselines3
Strategies
Strategy
Transfers
Good when
Fine-tuning
Whole policy/value net
Similar tasks, same action space
Frozen/partially frozen encoder
Representation
Same observation type, different goals
Policy distillation
Teacher policies → student
Combining several experts
Behaviour cloning + RL
Demonstrations
Expert data available
Successor features
Dynamics-related features
Same dynamics, new rewards
Goal-conditioned / multi-task
Shared policy across goals
Family of related tasks
Meta-RL
Learning-to-adapt
Many tasks from a distribution
Domain randomization
Robust policy
Sim-to-real
World-model reuse
Dynamics model
New reward, same environment
Code Example: Fine-Tuning with a Frozen Encoder (PyTorch)
import torch
from torch import nn
torch.manual_seed(42)
class Policy(nn.Module):
def __init__(self, obs_dim, n_actions):
super().__init__()
self.encoder = nn.Sequential(nn.Linear(obs_dim, 128), nn.ReLU(), nn.Linear(128, 128), nn.ReLU())
self.head = nn.Linear(128, n_actions)
def forward(self, x):
return self.head(self.encoder(x))
source = Policy(obs_dim=8, n_actions=4) # imagine this was trained on the source task
target = Policy(obs_dim=8, n_actions=6) # new task: same observations, different actions
target.encoder.load_state_dict(source.encoder.state_dict()) # transfer the representation
for p in target.encoder.parameters():
p.requires_grad_(False) # stage 1: train only the new head
trainable = sum(p.numel() for p in target.parameters() if p.requires_grad)
print(trainable) # 774 = 128 * 6 + 6
optimizer = torch.optim.Adam([p for p in target.parameters() if p.requires_grad], lr=3e-4)
# Stage 2 (later): unfreeze the encoder with a smaller learning rate
for p in target.encoder.parameters():
p.requires_grad_(True)
optimizer = torch.optim.Adam([
{"params": target.encoder.parameters(), "lr": 3e-5},
{"params": target.head.parameters(), "lr": 3e-4},
])
With Stable-Baselines3, the simplest transfer is model = PPO.load("source.zip", env=new_env) followed by model.learn(...), which fine-tunes all weights when observation and action spaces match.
Measuring Transfer
Metric
Question
Jump-start
Is initial performance better than from scratch?
Time to threshold
How much sooner is a target score reached?
Asymptotic performance
Is the final score better (or worse)?
Total reward / area under curve
Overall benefit over training
Common Pitfalls
Negative transfer from dissimilar tasks; always compare with training from scratch.
Catastrophic forgetting of the source task when that matters (use distillation or replay of source data).
Value functions transferred with a different reward scale.
Mismatched observation normalization statistics between source and target.
Interview Tips
List several strategies with what each transfers.
Mention successor features or goal-conditioned policies for reward changes, and domain randomization for sim-to-real.
Explain how to measure transfer and detect negative transfer.
27
How do you make RL agents generalize to unseen environments?
KnownRevise
RL agents often overfit to their training environments — memorizing specific levels, visual details or simulator quirks — so generalization has to be designed in: train on a broad distribution of environments (procedurally generated levels, many seeds, randomized physics and visuals), regularize and augment the observations, use architectures and representations that capture task-relevant structure, and always evaluate on held-out environments that the agent never trained on. Key techniques are domain randomization (vary friction, masses, lighting, textures, sensor noise), procedural content generation (Procgen, NetHack, MiniGrid), data augmentation of observations (random crops/shifts, colour jitter), regularization (L2, dropout, batch norm, entropy bonus, information bottlenecks), representation learning that ignores distractors, domain adaptation and meta-RL for quick adaptation, and robust/adversarial training against worst-case perturbations. Research on Procgen (Cobbe et al.) showed that agents need thousands of training levels before test performance approaches training performance.
import gymnasium as gym
import numpy as np
class RandomizedCartPole(gym.Wrapper):
"""Samples new pole length and cart mass at every reset."""
def __init__(self, env, length_range=(0.3, 0.8), masscart_range=(0.5, 2.0), seed=42):
super().__init__(env)
self.length_range, self.masscart_range = length_range, masscart_range
self.rng = np.random.default_rng(seed)
def reset(self, **kwargs):
u = self.env.unwrapped
u.length = float(self.rng.uniform(*self.length_range)) # half pole length
u.masscart = float(self.rng.uniform(*self.masscart_range))
u.total_mass = u.masspole + u.masscart # keep derived values consistent
u.polemass_length = u.masspole * u.length
return self.env.reset(**kwargs)
train_env = RandomizedCartPole(gym.make("CartPole-v1"))
obs, _ = train_env.reset(seed=42)
u = train_env.unwrapped
print(0.3 <= u.length <= 0.8, 0.5 <= u.masscart <= 2.0) # True True
# Evaluate on settings outside the training range (held out)
test_env = RandomizedCartPole(gym.make("CartPole-v1"), length_range=(0.9, 1.0), seed=7)
The training wrapper changes physics at every episode, so a policy has to work across a family of carts rather than one; the test wrapper uses lengths it never saw. CartPole’s internal attribute names (length, masscart, total_mass, polemass_length) are those of the current Gymnasium implementation; check them for your version.
Evaluation Protocol
Split
Contents
Train
Levels/seeds/parameters used for learning
Validation
Held-out levels for model selection
Test
Untouched levels, plus shifted conditions (out-of-distribution)
Report
Train vs test gap, mean ± std over several training seeds
Common Pitfalls
Evaluating on the training levels and calling it generalization.
Randomizing so widely that the task becomes unlearnable.
Reporting one training seed.
Ignoring the gap between simulator and real sensors (latency, noise).
Interview Tips
Say clearly that RL agents overfit to training environments.
Name domain randomization, procedural generation, augmentation and regularization.
Describe a held-out evaluation protocol.
28
How can RL agents overfit, and how do you prevent it?
KnownRevise
Overfitting in reinforcement learning means the agent performs well in the exact conditions it trained on but poorly on slightly different ones: it memorizes specific levels, start states, random seeds, simulator artefacts or visual details instead of learning the underlying skill. It shows up in several forms: environment overfitting (a policy that only works on its training levels or physics settings), overfitting to early data (the value network fits the first experiences in the replay buffer so strongly that it loses plasticity), critic overfitting in offline RL (overestimating values for actions absent from the dataset), and exploiting simulator bugs that do not exist in the real world. Prevention: train on diverse environments (many seeds and procedurally generated levels, domain randomization), augment observations, regularize networks (L2, dropout, layer norm, entropy bonus), limit updates per sample or periodically reset parts of the network, use conservative methods for offline RL, and evaluate on held-out environments with several training seeds.
Fix the sim, add penalties, validate in a different simulator
Hyperparameter overfitting
Tuned settings fail on new seeds
Tune on separate seeds, report variance
Code Example: Measuring the Generalization Gap
import gymnasium as gym
import numpy as np
def evaluate(policy, env_id, seeds, episodes_per_seed=1, **env_kwargs):
env = gym.make(env_id, **env_kwargs)
returns = []
for seed in seeds:
for ep in range(episodes_per_seed):
obs, _ = env.reset(seed=seed * 1000 + ep)
done, total = False, 0.0
while not done:
obs, r, terminated, truncated, _ = env.step(policy(obs))
total += r
done = terminated or truncated
returns.append(total)
env.close()
return float(np.mean(returns))
def random_policy(obs, rng=np.random.default_rng(42)):
return int(rng.integers(4))
train_seeds, test_seeds = range(0, 20), range(1000, 1020) # disjoint
map_kwargs = {"map_name": "8x8", "is_slippery": True}
train_score = evaluate(random_policy, "FrozenLake-v1", train_seeds, **map_kwargs)
test_score = evaluate(random_policy, "FrozenLake-v1", test_seeds, **map_kwargs)
print(f"train {train_score:.2f} test {test_score:.2f} gap {train_score - test_score:.2f}")
The function is the important part: replace random_policy with your trained agent and keep the train and test seeds (or levels) disjoint. FrozenLake’s map is fixed, so here seeds only change the random slips; with procedurally generated environments (Procgen, MiniGrid) seeds change the level layout, and the train-test gap measures overfitting directly.
Common Pitfalls
Reporting training-environment returns as the result.
Using one training seed and one evaluation seed.
Updating too many times per sample (high replay ratio) without resets or regularization.
Leaving exploration noise on during evaluation (or off during training).
Interview Tips
Describe several kinds of overfitting, not only memorizing levels.
Give remedies matched to each kind.
Stress disjoint train/test environments and multiple seeds.
29
How does REINFORCE update policies, and how does it handle variance?
KnownRevise
REINFORCE (Williams, 1992) is the basic Monte Carlo policy-gradient algorithm: run a full episode with the current stochastic policy π_θ, compute the return-to-go G_t from every step, and update the parameters by gradient ascent with θ ← θ + α Σ_t γᵗ G_t ∇θ log π_θ(a_t|s_t) (the γᵗ factor is often dropped in practice). Actions followed by high returns become more likely, actions followed by low returns less likely. The estimate is unbiased but has very high variance: returns sum many random rewards, and every action in an episode gets credit for rewards it may not have caused. Variance is reduced by using reward-to-go instead of the total episode return (actions cannot affect past rewards), subtracting a baseline b(s_t) — typically a learned value function V(s_t), which leaves the gradient unbiased — normalizing returns or advantages in each batch, averaging over many episodes per update, a smaller discount factor, and, moving beyond pure REINFORCE, actor-critic methods that replace G_t with bootstrapped advantage estimates.
Read full answerHide full answer
Applies toPyTorch 2.x, Gymnasium 1.x
Variance-Reduction Techniques
Technique
Change
Bias?
Reward-to-go
Use G_t from step t, not total return
No
Baseline b(s)
Weight by G_t − b(s_t)
No (b must not depend on the action)
Learned value baseline
b(s) = V_w(s)
No (actor)
Return/advantage normalization
Zero mean, unit std per batch
Slight, helps in practice
More episodes per update
Average over a batch
No
Lower γ
Shorter effective horizon
Yes (changes objective)
Actor-critic / GAE
Bootstrapped advantages
Yes, but much lower variance
Why a Baseline Is Unbiased
E_(a∼π)[b(s) ∇θ log π_θ(a|s)] = b(s) ∇θ Σ_a π_θ(a|s) = b(s) ∇θ 1 = 0. Subtracting b(s) therefore removes no signal on average but cancels much of the noise when b(s) is close to the expected return.
Code Example: REINFORCE with a Value Baseline on CartPole (PyTorch)
import gymnasium as gym
import torch
from torch import nn
env = gym.make("CartPole-v1")
torch.manual_seed(42)
policy = nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 2))
value = nn.Sequential(nn.Linear(4, 128), nn.ReLU(), nn.Linear(128, 1))
pi_opt = torch.optim.Adam(policy.parameters(), lr=1e-3)
v_opt = torch.optim.Adam(value.parameters(), lr=1e-3)
gamma = 0.99
def returns_to_go(rewards, gamma):
G, out = 0.0, []
for r in reversed(rewards):
G = r + gamma * G
out.append(G)
return torch.tensor(out[::-1], dtype=torch.float32)
for episode in range(600):
obs, _ = env.reset(seed=42 + episode)
states, log_probs, rewards = [], [], []
done = False
while not done:
s = torch.as_tensor(obs, dtype=torch.float32)
dist = torch.distributions.Categorical(logits=policy(s))
a = dist.sample()
obs, r, terminated, truncated, _ = env.step(int(a))
states.append(s)
log_probs.append(dist.log_prob(a))
rewards.append(r)
done = terminated or truncated
G = returns_to_go(rewards, gamma) # reward-to-go
S = torch.stack(states)
baseline = value(S).squeeze(-1)
advantages = G - baseline.detach() # baseline subtraction
advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8) # normalization
policy_loss = -(torch.stack(log_probs) * advantages).sum()
pi_opt.zero_grad()
policy_loss.backward()
pi_opt.step()
value_loss = (baseline - G).pow(2).mean() # fit the baseline to the returns
v_opt.zero_grad()
value_loss.backward()
v_opt.step()
print(len(rewards)) # length of the last episode
Episodes typically get much longer as training goes on (CartPole’s maximum is 500), but single-run REINFORCE is noisy, so the final length varies from run to run. Removing the baseline and normalization lines usually makes learning visibly slower and less stable.
Common Pitfalls
Using the total episode return for every step instead of reward-to-go.
A baseline that depends on the action (that introduces bias).
Not detaching the baseline in the policy loss.
Single-episode updates on long tasks (very noisy); batch several episodes.
Interview Tips
Write the REINFORCE update and explain it in words.
Explain why variance is high and why a baseline keeps the estimate unbiased.
Mention the step to actor-critic and GAE.
30
How are eligibility traces used in reinforcement learning?
KnownRevise
Eligibility traces are a short-term memory vector that records which states (or state-action pairs, or weights) were visited recently, so that when a TD error occurs, credit or blame is passed back to all of them at once in proportion to how recently and how often they were visited. Each step, every trace decays by γλ, and the trace of the current state is increased; then every value is updated by α · δ · e(s), where δ is the one-step TD error. This implements TD(λ), which interpolates between one-step TD (λ = 0, only the current state is updated) and Monte Carlo (λ = 1, credit reaches all earlier states as if the full return were used), with an efficient online, incremental update instead of waiting for the episode to end. Variants are accumulating traces (e += 1), replacing traces (e = 1), Dutch traces (true online TD(λ)), and control versions SARSA(λ) and Watkins’s Q(λ). In deep RL the same idea appears in the λ-return used by GAE and in Retrace/TD(λ) targets.
Read full answerHide full answer
Applies toPython 3.10+, NumPy
The Update (Tabular TD(λ), Accumulating Traces)
Step
Formula
TD error
δ_t = r(t+1) + γ V(s(t+1)) − V(s_t)
Decay all traces
e(s) ← γλ e(s)
Mark current state
e(s_t) ← e(s_t) + 1
Update all states
V(s) ← V(s) + α δ_t e(s)
Episode start
e ← 0
Worked Example
Chain s0 → s1 → s2 → terminal, rewards 0, 0, 1; γ = 1, λ = 0.5, α = 0.5, V = 0 everywhere. The first two TD errors are 0. At the last step δ = 1 and the traces are e(s2) = 1, e(s1) = 0.5, e(s0) = 0.25, so V becomes [0.125, 0.25, 0.5]. One-step TD (λ = 0) would only update V(s2) = 0.5 in that episode.
Code Example: TD(λ) on the Chain (NumPy)
import numpy as np
def td_lambda_episode(V, states, rewards, alpha, gamma, lam):
e = np.zeros_like(V)
for t, s in enumerate(states):
next_value = V[states[t + 1]] if t + 1 < len(states) else 0.0 # terminal after the last state
delta = rewards[t] + gamma * next_value - V[s]
e *= gamma * lam
e[s] += 1.0 # accumulating trace
V += alpha * delta * e # every traced state is updated
return V
states, rewards = [0, 1, 2], [0.0, 0.0, 1.0]
print(td_lambda_episode(np.zeros(3), states, rewards, alpha=0.5, gamma=1.0, lam=0.5)) # [0.125 0.25 0.5 ]
print(td_lambda_episode(np.zeros(3), states, rewards, alpha=0.5, gamma=1.0, lam=0.0)) # [0. 0. 0.5]
print(td_lambda_episode(np.zeros(3), states, rewards, alpha=0.5, gamma=1.0, lam=1.0)) # [0.5 0.5 0.5]
With λ = 1 every visited state moves halfway toward the episode’s return (1), which matches the Monte Carlo update; with λ = 0 only the last state learns.
Forward vs Backward View
View
Description
Forward
Update each state toward the λ-return G^λ_t = (1 − λ) Σ_n λⁿ⁻¹ G_t^(n) (needs future rewards)
Backward
Traces + TD errors, computed online (equivalent for offline updates; true online TD(λ) makes it exact online)
In Watkins’s Q(λ), forgetting to cut traces after an exploratory (non-greedy) action.
Using traces with experience replay naively (traces assume sequential on-policy data).
Accumulating traces with large α on states revisited many times (replacing traces are steadier).
Interview Tips
Explain traces as decaying records of recent visits.
Write δ, the trace decay and the update.
Connect λ to the TD-MC spectrum and to GAE.
31
How is hierarchical reinforcement learning used for complex tasks?
KnownRevise
Hierarchical reinforcement learning (HRL) splits a long, complex task into levels of decision-making: a high-level policy chooses subgoals or skills (“go to the kitchen”, “pick up the cup”) at a coarse time scale, and low-level policies carry them out with primitive actions over many steps. This attacks the main difficulties of long-horizon, sparse-reward problems: temporal abstraction shortens the effective horizon the high level must reason over, exploration improves because the agent explores in the space of meaningful skills rather than random joint torques, and reuse comes for free because a learned skill (open a door) can serve many tasks. The main frameworks are the options framework (Sutton, Precup and Singh: an option has an initiation set, an internal policy and a termination condition; learned end-to-end in Option-Critic), feudal / goal-conditioned hierarchies (FeUdal Networks, HIRO, HAC: the manager outputs goals and the worker is rewarded for reaching them), and skill discovery (DIAYN learns diverse skills without rewards). Challenges include non-stationarity between levels, designing or discovering good subgoals, and skills collapsing into trivial ones.
Code Example: A Two-Level Agent Skeleton (PyTorch)
import torch
from torch import nn
torch.manual_seed(42)
OBS, GOAL, ACT, K = 6, 2, 3, 10 # manager acts every K steps
class Manager(nn.Module): # pi_hi(goal | state)
def __init__(self):
super().__init__()
self.net = nn.Sequential(nn.Linear(OBS, 64), nn.ReLU(), nn.Linear(64, GOAL))
def forward(self, s):
return torch.tanh(self.net(s)) # goal = desired (x, y) offset in [-1, 1]
class Worker(nn.Module): # pi_lo(action | state, goal)
def __init__(self):
super().__init__()
self.net = nn.Sequential(nn.Linear(OBS + GOAL, 64), nn.ReLU(), nn.Linear(64, ACT))
def forward(self, s, g):
return torch.distributions.Categorical(logits=self.net(torch.cat([s, g], -1)))
def intrinsic_reward(state_xy, start_xy, goal_offset):
return -torch.norm(state_xy - (start_xy + goal_offset)).item() # worker: reach the subgoal
manager, worker = Manager(), Worker()
s = torch.randn(OBS)
goal, start = manager(s), s[:2].clone()
for t in range(25):
if t % K == 0 and t > 0:
goal, start = manager(s), s[:2].clone() # new subgoal every K steps
action = worker(s, goal).sample()
s = s + 0.1 * torch.randn(OBS) # stand-in for env.step(action)
r_lo = intrinsic_reward(s[:2], start, goal)
print(goal.shape, action.shape, isinstance(r_lo, float)) # torch.Size([2]) torch.Size([]) True
The manager is trained on the environment’s reward summed over its K-step windows; the worker is trained on the intrinsic reward for reaching the manager’s goal. Both can use standard algorithms (PPO, SAC, TD3).
Benefits and Challenges
Benefits
Challenges
Shorter effective horizon
Non-stationarity: the worker changes under the manager
Structured exploration
Choosing goal spaces and time scales
Reusable skills
Options collapsing (always one option, or very short ones)
Interpretability of subgoals
More components and hyperparameters
Common Pitfalls
Hand-picking a goal space that does not cover what the task needs.
Training both levels simultaneously without corrections (HIRO relabels old goals).
Using HRL when a flat agent with good reward shaping would do.
Interview Tips
Explain temporal abstraction and why it helps with long horizons and sparse rewards.
Describe options and goal-conditioned manager/worker designs.
Give a robotics or navigation example and mention the non-stationarity challenge.
32
What is inverse reinforcement learning?
KnownRevise
Inverse reinforcement learning (IRL) works backwards from standard RL: instead of being given a reward function and learning a policy, it observes an expert’s behaviour (demonstrations) and infers a reward function that explains it — the reward under which the expert’s behaviour would be (near-)optimal. The learned reward can then train a policy with ordinary RL, and because rewards capture why the expert acts rather than what it did, they often transfer better to new situations than simply copying actions (behaviour cloning). The core difficulty is that the problem is ill-posed: many rewards (including the trivial all-zero reward) explain the same behaviour. Methods resolve this differently: max-margin / apprenticeship learning (Abbeel and Ng: match expert feature expectations), Maximum Entropy IRL (Ziebart: prefer the reward under which the expert’s trajectories are most likely, with a max-entropy trajectory distribution), and adversarial methods (GAIL, AIRL) that train a discriminator between expert and agent behaviour. Learning reward models from human preference comparisons, as in RLHF, is a closely related idea.
Read full answerHide full answer
Applies toPython 3.10+, NumPy
Approaches
Method
Idea
Output
Behaviour cloning (baseline)
Supervised learning of actions
Policy only
Apprenticeship / max-margin
Reward w·φ(s) making expert better than others by a margin
Linear reward + policy
MaxEnt IRL
Maximize likelihood of demos under p(τ) ∝ exp(R(τ))
Reward
Deep MaxEnt IRL
Neural reward network
Nonlinear reward
GAIL
Discriminator as implicit reward; policy fools it
Policy (reward not reusable)
AIRL
Discriminator structured to recover a transferable reward
Reward + policy
Preference learning (RLHF)
Fit reward from human pairwise comparisons
Reward model
Code Example: MaxEnt IRL Gradient as Feature Matching (NumPy)
import numpy as np
# Linear reward R(s) = w . phi(s); MaxEnt IRL gradient = expert features - expected features under the model
phi = np.array([[1.0, 0.0], # state 0 features
[0.0, 1.0], # state 1
[1.0, 1.0]]) # state 2
expert_visits = np.array([0.1, 0.2, 0.7]) # expert spends most time in state 2
def model_visits(w):
# Toy stand-in for the soft-optimal policy's state visitation: softmax over state rewards
r = phi @ w
p = np.exp(r - r.max())
return p / p.sum()
w = np.zeros(2)
for _ in range(500):
grad = expert_visits @ phi - model_visits(w) @ phi # match feature expectations
w += 0.5 * grad
print(np.round(expert_visits @ phi, 3), np.round(model_visits(w) @ phi, 3)) # features now match closely
print(bool(phi[2] @ w > phi[0] @ w and phi[2] @ w > phi[1] @ w)) # True: state 2 gets the highest reward
In real MaxEnt IRL, model_visits comes from solving a soft-RL problem (soft value iteration in small MDPs, or a policy optimization loop in large ones) with the current reward, which is why IRL is expensive: every reward update needs an RL solve.
IRL vs Behaviour Cloning
Aspect
Behaviour cloning
IRL
Learns
Actions
Reward (then policy)
Needs environment interaction
No
Yes (usually)
Compounding errors off the expert’s path
Yes
Less, the policy learns to recover
Transfer to new dynamics
Poor
Better (reward is reused)
Cost
Low
High
Applications and Caveats
Applications: autonomous driving (learning human driving preferences), robotics from demonstrations, modelling pedestrian or customer behaviour, and preference-based reward models for language models. Caveats: reward ambiguity, assumed expert optimality, needing many demonstrations, and learned rewards that can be exploited by the policy.
Common Pitfalls
Ignoring reward ambiguity (the recovered reward is not “the” true reward).
Assuming demonstrations are optimal when humans are noisy or inconsistent.
Using GAIL when a reusable reward is needed.
Interview Tips
Define IRL as inferring a reward from demonstrations, and why that is ill-posed.
Name MaxEnt IRL, apprenticeship learning and GAIL/AIRL.
Contrast with behaviour cloning and connect to RLHF reward models.
33
What is partial observability in RL, and how can it be addressed?
KnownRevise
Partial observability means the agent’s observation does not contain the full state of the environment, so the Markov property fails for observations: two situations that look identical can require different actions, and the best action may depend on what happened before. Examples: a single video frame without velocity, a robot whose camera cannot see behind it, a card game with hidden hands, a recommendation system that cannot observe a user’s mood. The formal model is a POMDP: an MDP plus an observation function O(o | s′, a); the optimal agent acts on a belief state — a probability distribution over hidden states updated with Bayes’ rule. Practical remedies: stack recent observations (Atari frame stacking), give the policy memory (LSTM/GRU or transformer policies such as DRQN or recurrent PPO), maintain explicit beliefs with Bayes filters (Kalman or particle filters) when a model exists, add information-gathering actions, use privileged information during training only (asymmetric actor-critic), and engineer observations that include missing quantities when possible.
import numpy as np
# Hidden state: tiger behind left (0) or right (1) door. "Listen" is correct 85% of the time.
belief = np.array([0.5, 0.5])
p_hear_left = np.array([0.85, 0.15]) # P(hear "left" | tiger left), P(hear "left" | tiger right)
def update(belief, heard_left):
likelihood = p_hear_left if heard_left else 1 - p_hear_left
posterior = likelihood * belief
return posterior / posterior.sum()
belief = update(belief, heard_left=True)
print(np.round(belief, 3)) # [0.85 0.15]
belief = update(belief, heard_left=True)
print(np.round(belief, 3)) # [0.97 0.03]
After two “left” observations: 0.85² / (0.85² + 0.15²) = 0.7225 / 0.745 ≈ 0.970. A POMDP agent decides on this belief, for example opening a door only once the belief is confident enough.
Code Example: Frame Stacking and a Recurrent Policy
import gymnasium as gym
import torch
from torch import nn
env = gym.wrappers.FrameStackObservation(gym.make("CartPole-v1"), stack_size=4)
obs, _ = env.reset(seed=42)
print(obs.shape) # (4, 4): last 4 observations
class RecurrentPolicy(nn.Module):
def __init__(self, obs_dim, n_actions, hidden=64):
super().__init__()
self.gru = nn.GRU(obs_dim, hidden, batch_first=True)
self.head = nn.Linear(hidden, n_actions)
def forward(self, obs_seq, h=None):
out, h = self.gru(obs_seq, h) # h carries memory between calls during an episode
return self.head(out[:, -1]), h
torch.manual_seed(42)
policy = RecurrentPolicy(obs_dim=2, n_actions=2) # e.g. observations with velocities removed
logits, h = policy(torch.randn(1, 1, 2))
logits, h = policy(torch.randn(1, 1, 2), h) # next step reuses the hidden state
print(logits.shape, h.shape) # torch.Size([1, 2]) torch.Size([1, 1, 64])
For a ready-made option, sb3_contrib.RecurrentPPO("MlpLstmPolicy", env) trains an LSTM policy; reset the hidden state at episode boundaries.
Common Pitfalls
Treating observations as states and blaming the algorithm when it fails.
Not resetting recurrent state between episodes, or training on shuffled single transitions with an RNN.
Stacking too few frames for the memory the task needs.
Letting the actor use privileged simulator state that will not exist at deployment.
Interview Tips
Define POMDPs and belief states, with an example.
List practical remedies: stacking, recurrent policies, belief tracking.
Mention asymmetric actor-critic for sim-to-real.
34
Implement the epsilon-greedy action selection strategy in Python.
KnownRevise
Epsilon-greedy action selection picks a uniformly random action with probability ε (explore) and the action with the highest estimated value with probability 1 − ε (exploit). A good implementation also handles a few details: ties among best actions should be broken randomly (otherwise an all-zero Q-table always picks action 0), it should take a seeded random generator for reproducibility, it may support action masks for invalid actions, and it often comes with a decay schedule for ε (linear or exponential, from about 1.0 down to 0.01-0.05). Note that with probability ε the random action can still be the greedy one, so the greedy action is chosen with total probability 1 − ε + ε/|A|.
Read full answerHide full answer
Applies toPython 3.10+, NumPy, PyTorch 2.x
Code Example: Tabular ε-Greedy with Random Tie-Breaking and Masks
import numpy as np
def epsilon_greedy(q_values, epsilon, rng, mask=None):
"""Return an action index. mask: boolean array, True = action allowed."""
q = np.asarray(q_values, dtype=float)
allowed = np.ones_like(q, dtype=bool) if mask is None else np.asarray(mask, dtype=bool)
valid = np.flatnonzero(allowed)
if rng.random() < epsilon:
return int(rng.choice(valid)) # explore among valid actions
best_value = q[valid].max()
best = valid[q[valid] == best_value] # all actions tied for best
return int(rng.choice(best)) # random tie-break
rng = np.random.default_rng(42)
q = [1.0, 3.0, 3.0, 0.5]
greedy_picks = [epsilon_greedy(q, 0.0, rng) for _ in range(1000)]
print(sorted(set(greedy_picks))) # [1, 2] ties broken randomly
print(epsilon_greedy(q, 0.0, rng, mask=[True, False, False, True])) # 0 best among allowed actions
picks = np.array([epsilon_greedy([0.0, 1.0, 0.0, 0.0], 0.2, rng) for _ in range(100_000)])
print(round((picks == 1).mean(), 2)) # about 0.85 = 1 - 0.2 + 0.2 / 4
By hand: halfway through the linear schedule, 1.0 + 0.5 × (0.05 − 1.0) = 0.525; the exponential schedule at step 1,000 gives 0.05 + 0.95 × e⁻¹ ≈ 0.399.
Code Example: Batched ε-Greedy for a DQN (PyTorch)
np.argmax alone: always picks the first of tied actions (biased early exploration).
Forgetting to decay ε, or decaying it before the agent has seen any reward.
Leaving ε > 0 during evaluation (or using ε = 0 in environments that need exploration to avoid getting stuck).
Exploring invalid actions in environments with action masks.
Interview Tips
Write the function with random tie-breaking.
State the true probability of the greedy action, 1 − ε + ε/|A|.
Add a decay schedule and mention alternatives (softmax, UCB).
35
Write a Python script that simulates a simple MDP using a transition matrix.
KnownRevise
To simulate an MDP from a transition matrix, store the dynamics as an array P[s, a, s′] whose entries are the probabilities of moving from state s to s′ when taking action a (each P[s, a, :] must sum to 1), store rewards as R[s, a] (or R[s, a, s′]), and then repeatedly sample the next state with rng.choice(n_states, p=P[s, a]), add the discounted reward, and stop at a terminal state or a step limit. Wrapping this in a small class with reset and step methods that follow the Gymnasium convention makes it usable by RL code; the same arrays can be used to compute exact values with dynamic programming, which is a good way to check the simulation (sampled returns should average to the computed values).
Read full answerHide full answer
Applies toPython 3.10+, NumPy
Code Example: MDP Simulator with Exact Value Check
import numpy as np
class TabularMDP:
def __init__(self, P, R, terminal, gamma=0.9, max_steps=100, seed=42):
self.P, self.R = np.asarray(P, dtype=float), np.asarray(R, dtype=float)
assert np.allclose(self.P.sum(axis=2), 1.0), "each P[s, a, :] must sum to 1"
self.n_states, self.n_actions = self.R.shape
self.terminal, self.gamma, self.max_steps = set(terminal), gamma, max_steps
self.rng = np.random.default_rng(seed)
def reset(self, start=0):
self.state, self.t = start, 0
return self.state, {}
def step(self, action):
reward = self.R[self.state, action]
self.state = int(self.rng.choice(self.n_states, p=self.P[self.state, action]))
self.t += 1
terminated = self.state in self.terminal
truncated = self.t >= self.max_steps
return self.state, reward, terminated, truncated, {}
def policy_value(self, policy):
"""Exact V for a deterministic policy: solve (I - gamma P_pi) V = R_pi."""
idx = np.arange(self.n_states)
P_pi = self.P[idx, policy].copy()
R_pi = self.R[idx, policy].copy()
for s in self.terminal: # terminal: no reward, no future
P_pi[s], R_pi[s] = 0.0, 0.0
return np.linalg.solve(np.eye(self.n_states) - self.gamma * P_pi, R_pi)
# States: 0 = start, 1 = middle, 2 = goal (terminal). Actions: 0 = safe, 1 = risky
P = [
[[0.0, 1.0, 0.0], [0.5, 0.0, 0.5]], # from 0: safe -> 1; risky -> 0 or goal (50/50)
[[0.0, 0.0, 1.0], [0.0, 0.0, 1.0]], # from 1: both reach the goal
[[0.0, 0.0, 1.0], [0.0, 0.0, 1.0]], # goal is absorbing
]
R = [[-1.0, -1.0],
[5.0, 5.0],
[0.0, 0.0]]
mdp = TabularMDP(P, R, terminal=[2], gamma=0.9)
policy = np.array([0, 0, 0]) # always "safe"
print(np.round(mdp.policy_value(policy), 2)) # [3.5 5. 0. ]
returns = []
for episode in range(5_000):
s, _ = mdp.reset()
G, discount, done = 0.0, 1.0, False
while not done:
s, r, terminated, truncated, _ = mdp.step(policy[s])
G += discount * r
discount *= mdp.gamma
done = terminated or truncated
returns.append(G)
print(round(float(np.mean(returns)), 2)) # 3.5 (the safe path is deterministic)
By hand for the “safe” policy: from state 1 the agent earns 5 and reaches the goal, so V(1) = 5; from state 0 it earns −1 then V(1): V(0) = −1 + 0.9 × 5 = 3.5. Every simulated episode follows the same path, so the sample mean equals 3.5. Switching to the risky action at state 0 (policy = [1, 0, 0]) gives V(0) = −1 + 0.9 × 0.5 × V(0), so V(0) = −1 / 0.55 ≈ −1.82, and the simulated average then varies around that value.
Extensions
Extension
Change
R[s, a, s′]
Reward = R[s, a, next_state] after sampling
Stochastic policy
Sample a ~ π[s] before each step
Gymnasium compatibility
Subclass gym.Env with spaces.Discrete
Speed
Pre-compute cumulative probabilities and use searchsorted
Common Pitfalls
Rows of P that do not sum to 1 (rng.choice raises an error).
Counting rewards after reaching a terminal state.
Using np.random.seed globally instead of a Generator.
No step limit for policies that never reach a terminal state.
Interview Tips
Show the P[s, a, s′] layout and the validation check.
Sample next states with rng.choice(..., p=...).
Verify the simulation against the exact Bellman solution.
36
Code a Q-learning algorithm in Python to solve a grid-world problem.
KnownRevise
A grid-world Q-learning solution needs three parts: a small environment (states are grid cells, actions are up/down/left/right, rewards for reaching the goal and penalties for each step or for traps), a Q-table of shape (number of states × number of actions), and the training loop that picks ε-greedy actions and applies Q(s, a) ← Q(s, a) + α [r + γ max_a′ Q(s′, a′) − Q(s, a)], with no future term at terminal states. After training, the greedy policy argmax_a Q(s, a) gives an arrow per cell, which should trace a shortest safe path to the goal. Because the grid is tiny and deterministic, the result can be checked against value iteration or by counting steps.
Read full answerHide full answer
Applies toPython 3.10+, NumPy
Code Example: Grid World + Q-Learning
import numpy as np
GRID = [
"S..#",
".#..",
"...X",
"#..G",
] # S start, G goal (+10), X trap (-10), # wall, -1 per step
ROWS, COLS = len(GRID), len(GRID[0])
ACTIONS = [(-1, 0), (1, 0), (0, -1), (0, 1)] # up, down, left, right
ARROWS = "^v<>"
def find(ch):
for r, row in enumerate(GRID):
if ch in row:
return r, row.index(ch)
START = find("S")
def step(pos, a):
r, c = pos[0] + ACTIONS[a][0], pos[1] + ACTIONS[a][1]
if not (0 <= r < ROWS and 0 <= c < COLS) or GRID[r][c] == "#":
r, c = pos # bump into wall: stay
cell = GRID[r][c]
if cell == "G":
return (r, c), 10.0, True
if cell == "X":
return (r, c), -10.0, True
return (r, c), -1.0, False
def idx(pos):
return pos[0] * COLS + pos[1]
rng = np.random.default_rng(42)
Q = np.zeros((ROWS * COLS, len(ACTIONS)))
alpha, gamma = 0.5, 0.95
eps, eps_min, eps_decay = 1.0, 0.05, 0.99
for episode in range(1_000):
pos, done, steps = START, False, 0
while not done and steps < 100:
s = idx(pos)
if rng.random() < eps:
a = int(rng.integers(4))
else:
best = np.flatnonzero(Q[s] == Q[s].max())
a = int(rng.choice(best)) # random tie-break
pos, r, done = step(pos, a)
target = r + (0.0 if done else gamma * Q[idx(pos)].max())
Q[s, a] += alpha * (target - Q[s, a])
steps += 1
eps = max(eps_min, eps * eps_decay)
# Print the greedy policy
for r in range(ROWS):
print(" ".join(GRID[r][c] if GRID[r][c] in "#GX" else ARROWS[int(np.argmax(Q[idx((r, c))]))]
for c in range(COLS)))
# Follow the greedy policy from the start
pos, path, done = START, [START], False
while not done and len(path) < 20:
pos, _, done = step(pos, int(np.argmax(Q[idx(pos)])))
path.append(pos)
print(path[-1] == find("G"), len(path) - 1) # True 6 (a shortest path has 6 moves)
The shortest route from S at (0, 0) to G at (3, 3) needs 6 moves (3 down and 3 right, avoiding the walls and the trap at (2, 3)), for example down, down, right, right, down, right. With 1,000 episodes on this 16-cell grid, the greedy policy reliably finds such a path; several 6-move paths exist, so the printed arrows can differ from run to run with other seeds while the path length stays 6.
Design Notes
Choice
Reason
−1 per step
Encourages shortest paths
Terminal: no future term
Correct Bellman target
Random tie-breaking
Untrained Q-rows are all 0
ε decay 0.99 per episode
Explores early, exploits later
Step cap 100
Prevents endless wandering early on
Common Pitfalls
Bootstrapping from terminal states (values leak past the goal or trap).
Always choosing action 0 on ties.
Letting the agent walk through walls or off the grid.
Evaluating with ε > 0.
Interview Tips
Build the environment, Q-table and update clearly separated.
Explain the reward design (step cost, goal, trap) and terminal handling.
Verify the result (path length, compare with value iteration).
37
Implement value iteration for a given MDP in Python.
KnownRevise
Value iteration finds the optimal value function and policy of a known MDP by repeatedly applying the Bellman optimality backup to every state, V(s) ← max_a Σ_s′ P(s′|s, a)[R(s, a, s′) + γ V(s′)], until the largest change in a sweep falls below a threshold θ; the optimal policy is then greedy with respect to the final values, π(s) = argmax_a Q(s, a). With NumPy the backup for all states and actions is one tensor expression: Q = R + γ P·V. The algorithm converges because the backup is a γ-contraction; a stopping threshold of θ(1 − γ)/(2γ) on the change guarantees the greedy policy’s values are within θ of optimal. Each sweep costs O(|S|²|A|).
import numpy as np
def value_iteration(P, R, gamma=0.9, theta=1e-8, max_iters=10_000):
"""P: (S, A, S') transition probs. R: (S, A) expected rewards. Returns V, policy, iterations."""
V = np.zeros(P.shape[0])
for i in range(1, max_iters + 1):
Q = R + gamma * P @ V # (S, A): Bellman backup for every state-action
V_new = Q.max(axis=1)
if np.max(np.abs(V_new - V)) < theta:
return V_new, Q.argmax(axis=1), i
V = V_new
return V, Q.argmax(axis=1), max_iters
# Example: states 0, 1, 2 (2 = terminal goal). Actions: 0 = safe, 1 = risky
P = np.array([
[[0.0, 1.0, 0.0], [0.5, 0.0, 0.5]],
[[0.0, 0.0, 1.0], [0.0, 0.0, 1.0]],
[[0.0, 0.0, 1.0], [0.0, 0.0, 1.0]],
])
R = np.array([[-1.0, -1.0],
[5.0, 5.0],
[0.0, 0.0]])
V, policy, iters = value_iteration(P, R, gamma=0.9)
print(np.round(V, 2), policy) # [3.5 5. 0. ] [0 0 0]
By hand: V(2) = 0 (absorbing, no reward), V(1) = 5, and in state 0 the safe action gives −1 + 0.9 × 5 = 3.5 while the risky action gives −1 + 0.9 × (0.5 × 3.5 + 0.5 × 0) = 0.575, so the optimal value is 3.5 with the safe action. (For state 1 both actions tie, and argmax returns the first.)
Code Example: Value Iteration on FrozenLake’s Model (Gymnasium)
import gymnasium as gym
import numpy as np
env = gym.make("FrozenLake-v1", is_slippery=True)
model = env.unwrapped.P # model[s][a] = [(prob, next_s, reward, terminated), ...]
S, A = env.observation_space.n, env.action_space.n
P = np.zeros((S, A, S))
R = np.zeros((S, A))
for s in range(S):
for a in range(A):
for prob, s2, r, terminated in model[s][a]:
if not terminated:
P[s, a, s2] += prob # terminal transitions have no future value
R[s, a] += prob * r
gamma, theta = 0.99, 1e-10
V = np.zeros(S)
while True:
Q = R + gamma * P @ V
V_new = Q.max(axis=1)
if np.max(np.abs(V_new - V)) < theta:
break
V = V_new
policy = Q.argmax(axis=1)
print(np.round(V_new.reshape(4, 4), 3))
print(policy.reshape(4, 4)) # 0=left, 1=down, 2=right, 3=up
Leaving terminal transitions out of P makes their future value zero without special-casing terminal states. On the slippery map the optimal policy often points away from holes rather than straight at the goal, because each move only goes in the intended direction one third of the time.
Value Iteration vs Policy Iteration
Aspect
Value iteration
Policy iteration
Per iteration
One max-backup sweep
Full policy evaluation + improvement
Iterations
Many
Few
Cost per iteration
Low
High (solve a linear system)
Typical use
Simple, large-ish tabular MDPs
Small MDPs, exact solutions
Common Pitfalls
Shapes: P @ V must contract over the last axis (s′).
Counting future value after terminal transitions.
θ too large when γ is close to 1 (the policy may still be suboptimal).
Using in-place updates and thinking the result differs (Gauss-Seidel variants converge to the same values, often faster).
Interview Tips
Write the Bellman optimality backup and the stopping rule.
Vectorize with NumPy and extract the greedy policy.
Compare with policy iteration and mention the contraction argument.
38
Write a function to compute discounted rewards for a reward sequence.
KnownRevise
The discounted return from each time step t is G_t = r_t + γ r(t+1) + γ² r(t+2) + … to the end of the episode, and it is computed efficiently in one backward pass using the recursion G_t = r_t + γ G_(t+1) with G after the last step equal to 0 (or to a bootstrap value V(s_T) if the episode was cut off rather than finished). That gives O(T) time instead of the O(T²) of summing each tail separately. For rollouts that contain several episodes, the recursion must reset at episode boundaries using the done flags. The same backward pattern computes GAE advantages in PPO. In policy-gradient code, the returns are often normalized (zero mean, unit variance) afterwards to reduce variance.
Looping forward and summing tails (O(T²), and easy to get wrong).
Not resetting at episode boundaries in multi-episode buffers.
Treating time-limit truncation as termination (should bootstrap from V).
Integer arrays: np.zeros_like(rewards) on an int list silently truncates returns.
Interview Tips
Give the recursion and do the backward loop.
Handle done flags and bootstrapping.
Mention normalization and the link to GAE.
39
Develop a SARSA agent in Python for the Taxi-v3 environment.
KnownRevise
SARSA learns Q-values on-policy from the quintuple (s, a, r, s′, a′): the agent picks a′ with the same ε-greedy policy it is following and updates Q(s, a) ← Q(s, a) + α [r + γ Q(s′, a′) − Q(s, a)]. Taxi-v3 (now in Gymnasium; the old OpenAI Gym package is no longer maintained) is a good fit: 500 discrete states (taxi position × passenger location × destination), 6 actions (south, north, east, west, pickup, dropoff), −1 per step, +20 for a correct drop-off, −10 for illegal pickup/drop-off, and a 200-step time limit. A table of 500 × 6 Q-values, ε decayed over a few thousand episodes, α ≈ 0.1-0.5 and γ ≈ 0.99 typically learns a near-optimal policy whose evaluation return is about +8 per episode on average (20 minus the steps taken).
Read full answerHide full answer
Applies toPython 3.10+, NumPy, Gymnasium 1.x
Code Example: SARSA on Taxi-v3
import gymnasium as gym
import numpy as np
env = gym.make("Taxi-v3")
n_states, n_actions = env.observation_space.n, env.action_space.n
print(n_states, n_actions) # 500 6
rng = np.random.default_rng(42)
Q = np.zeros((n_states, n_actions))
alpha, gamma = 0.1, 0.99
eps, eps_min, eps_decay = 1.0, 0.01, 0.999
def choose(state, epsilon):
if rng.random() < epsilon:
return int(rng.integers(n_actions))
best = np.flatnonzero(Q[state] == Q[state].max())
return int(rng.choice(best))
for episode in range(10_000):
s, _ = env.reset(seed=int(rng.integers(1_000_000)))
a = choose(s, eps)
done = False
while not done:
s2, r, terminated, truncated, _ = env.step(a)
a2 = choose(s2, eps) # next action from the same policy
target = r if terminated else r + gamma * Q[s2, a2] # truncated still bootstraps
Q[s, a] += alpha * (target - Q[s, a])
s, a = s2, a2
done = terminated or truncated
eps = max(eps_min, eps * eps_decay)
def evaluate(episodes=100):
total, successes = 0.0, 0
for i in range(episodes):
s, _ = env.reset(seed=100_000 + i) # held-out seeds
done, ep_return = False, 0.0
while not done:
s, r, terminated, truncated, _ = env.step(int(np.argmax(Q[s])))
ep_return += r
done = terminated or truncated
total += ep_return
successes += int(terminated)
return total / episodes, successes / episodes
mean_return, success_rate = evaluate()
print(f"mean return {mean_return:.2f}, success rate {success_rate:.2f}")
After training, the greedy policy usually completes nearly every evaluation episode with a mean return in the high single digits (an optimal taxi averages around +8); the exact figure depends on the run.
Hyperparameters
Parameter
Value
Effect
α
0.1
Stable updates in a stochastic start-state setting
γ
0.99
Plans across the whole trip
ε
1.0 → 0.01, ×0.999 per episode
ε ≈ 0.37 after 1,000 episodes, reaches 0.01 near episode 4,600
Episodes
10,000
Enough for 500 × 6 table
SARSA vs Q-Learning on Taxi
Both converge here; Q-learning’s target uses max Q(s′, ·) and often learns a little faster, while SARSA’s on-policy target makes it account for its own exploration (more cautious in environments with large penalties near the optimal path).
Common Pitfalls
Using import gym and the old 4-value step API.
Choosing a′ greedily (that is Q-learning, not SARSA).
Treating the 200-step truncation as terminal.
Evaluating with ε > 0 or on the same seeds as training.
Interview Tips
Write the SARSA update and point out where a′ comes from.
Describe Taxi’s state, actions and rewards.
Report evaluation with a greedy policy on held-out seeds.
40
Build a neural network in PyTorch or TensorFlow that approximates a policy.
KnownRevise
A policy network maps a state to a distribution over actions: for discrete actions it outputs one logit per action and a categorical distribution (softmax) is built from them; for continuous actions it outputs the mean of a Gaussian, with a learned log standard deviation, often squashed by tanh to respect action bounds. The network should provide three things to the training algorithm: sampling (act) for exploration, log-probabilities of taken actions for the policy-gradient loss, and the entropy for exploration bonuses; for deterministic evaluation it returns the most likely action (argmax or the mean). Good defaults: two hidden layers of 64-256 units with tanh or ReLU, orthogonal initialization with a small gain (0.01) on the output layer so the initial policy is close to uniform, and observation normalization.
Read full answerHide full answer
Applies toPyTorch 2.x; Keras 3 version included
Code Example: Discrete and Continuous Policy Networks (PyTorch)
import torch
from torch import nn
from torch.distributions import Categorical, Normal
def layer_init(layer, gain=2 ** 0.5):
nn.init.orthogonal_(layer.weight, gain)
nn.init.zeros_(layer.bias)
return layer
class DiscretePolicy(nn.Module):
def __init__(self, obs_dim, n_actions, hidden=64):
super().__init__()
self.net = nn.Sequential(
layer_init(nn.Linear(obs_dim, hidden)), nn.Tanh(),
layer_init(nn.Linear(hidden, hidden)), nn.Tanh(),
layer_init(nn.Linear(hidden, n_actions), gain=0.01), # near-uniform initial policy
)
def distribution(self, obs):
return Categorical(logits=self.net(obs))
def act(self, obs, deterministic=False):
dist = self.distribution(obs)
action = dist.probs.argmax(-1) if deterministic else dist.sample()
return action, dist.log_prob(action), dist.entropy()
class GaussianPolicy(nn.Module):
def __init__(self, obs_dim, act_dim, hidden=64):
super().__init__()
self.mu = nn.Sequential(
layer_init(nn.Linear(obs_dim, hidden)), nn.Tanh(),
layer_init(nn.Linear(hidden, hidden)), nn.Tanh(),
layer_init(nn.Linear(hidden, act_dim), gain=0.01),
)
self.log_std = nn.Parameter(torch.zeros(act_dim)) # state-independent std
def act(self, obs, deterministic=False):
dist = Normal(self.mu(obs), self.log_std.exp())
action = dist.mean if deterministic else dist.sample()
return action, dist.log_prob(action).sum(-1), dist.entropy().sum(-1)
torch.manual_seed(42)
pi = DiscretePolicy(obs_dim=4, n_actions=2)
obs = torch.randn(5, 4)
a, logp, ent = pi.act(obs)
print(a.shape, logp.shape, ent.shape) # torch.Size([5]) torch.Size([5]) torch.Size([5])
print(torch.allclose(pi.distribution(obs).probs, torch.full((5, 2), 0.5), atol=0.05)) # True: near uniform
gpi = GaussianPolicy(obs_dim=3, act_dim=2)
a, logp, ent = gpi.act(torch.randn(5, 3))
print(a.shape, logp.shape) # torch.Size([5, 2]) torch.Size([5])
# One policy-gradient step: increase log-prob of actions with positive advantage
advantages = torch.tensor([1.0, -1.0, 0.5, 0.0, 2.0])
a, logp, ent = pi.act(obs)
loss = -(logp * advantages).mean() - 0.01 * ent.mean()
loss.backward()
print(all(p.grad is not None for p in pi.parameters())) # True
The small output gain keeps the initial logits near zero, so both CartPole actions start with probability close to 0.5; the policy then learns from the loss above (REINFORCE, A2C) or from PPO’s clipped version of it.
Keras 3 Equivalent (Discrete)
import keras
from keras import layers
policy = keras.Sequential([
keras.Input(shape=(4,)),
layers.Dense(64, activation="tanh"),
layers.Dense(64, activation="tanh"),
layers.Dense(2), # logits; sample with keras.random.categorical
])
print(policy.output_shape) # (None, 2)
Design Choices
Choice
Discrete
Continuous
Output
Logits per action
Mean (and log std)
Distribution
Categorical
Gaussian (tanh-squashed for SAC)
Deterministic action
argmax
Mean
Bounds
—
tanh squash or clip to action space
Invalid actions
Mask logits to −∞
—
Common Pitfalls
Applying softmax and then passing the probabilities as logits=.
Summing log-probs over action dimensions for Gaussian policies (forgetting it).
Clipping sampled actions but computing log-prob of the unclipped action without care.
Large output initialization, so the initial policy is nearly deterministic.
Interview Tips
Explain logits → categorical and mean/std → Gaussian.
Show act returning action, log-prob and entropy.
Mention initialization, normalization and action masking.
41
Write a Python implementation of the REINFORCE algorithm.
KnownRevise
REINFORCE is implemented in four steps per update: run one or more full episodes with the current stochastic policy while recording the log-probability of each chosen action and each reward; compute the discounted return-to-go G_t for every step with a backward pass; form the loss −Σ_t log π_θ(a_t|s_t) · G_t (usually with returns normalized or a baseline subtracted, to reduce variance); and take a gradient step on that loss. Minimizing the negative of the objective is gradient ascent on expected return. The implementation below uses PyTorch for the policy network and Gymnasium’s CartPole-v1, batches several episodes per update for steadier gradients, and evaluates greedily at the end. Expect noisy learning curves; REINFORCE is the simplest policy-gradient method, not the most efficient.
Sample N episodes with π_θ, storing log π_θ(a_t|s_t) and r_t
2
G_t = r_t + γ G_(t+1), backward per episode
3
Normalize G across the batch (or subtract a baseline)
4
loss = −mean over steps of [log π_θ(a_t|s_t) · G_t]
5
Backpropagate, optimizer step, repeat
Code Example (PyTorch + Gymnasium)
import gymnasium as gym
import numpy as np
import torch
from torch import nn
SEED, GAMMA, LR = 42, 0.99, 1e-2
EPISODES_PER_UPDATE, UPDATES = 8, 150
torch.manual_seed(SEED)
env = gym.make("CartPole-v1")
obs_dim, n_actions = env.observation_space.shape[0], env.action_space.n
policy = nn.Sequential(nn.Linear(obs_dim, 64), nn.Tanh(), nn.Linear(64, n_actions))
optimizer = torch.optim.Adam(policy.parameters(), lr=LR)
def returns_to_go(rewards, gamma):
G, out = 0.0, []
for r in reversed(rewards):
G = r + gamma * G
out.append(G)
return out[::-1]
def run_episode(seed):
obs, _ = env.reset(seed=seed)
log_probs, rewards, done = [], [], False
while not done:
logits = policy(torch.as_tensor(obs, dtype=torch.float32))
dist = torch.distributions.Categorical(logits=logits)
action = dist.sample()
obs, reward, terminated, truncated, _ = env.step(action.item())
log_probs.append(dist.log_prob(action))
rewards.append(reward)
done = terminated or truncated
return log_probs, rewards
history = []
for update in range(UPDATES):
all_log_probs, all_returns, lengths = [], [], []
for i in range(EPISODES_PER_UPDATE):
log_probs, rewards = run_episode(seed=SEED + update * EPISODES_PER_UPDATE + i)
all_log_probs += log_probs
all_returns += returns_to_go(rewards, GAMMA)
lengths.append(len(rewards))
returns = torch.tensor(all_returns, dtype=torch.float32)
returns = (returns - returns.mean()) / (returns.std() + 1e-8) # variance reduction
loss = -(torch.stack(all_log_probs) * returns).mean()
optimizer.zero_grad()
loss.backward()
optimizer.step()
history.append(np.mean(lengths))
if (update + 1) % 25 == 0:
print(f"update {update + 1:3d} mean episode length {history[-1]:.1f}")
def evaluate(episodes=10):
lengths = []
for i in range(episodes):
obs, _ = env.reset(seed=10_000 + i)
done, steps = False, 0
while not done:
with torch.no_grad():
action = policy(torch.as_tensor(obs, dtype=torch.float32)).argmax().item()
obs, _, terminated, truncated, _ = env.step(action)
steps += 1
done = terminated or truncated
lengths.append(steps)
return float(np.mean(lengths))
print("greedy evaluation length:", evaluate())
A random CartPole policy lasts about 20 steps; as REINFORCE trains, the printed mean episode length should rise substantially toward the 500-step cap, but the curve is noisy and depends on the seed, so judge it over several runs.
Detaching log_prob or computing it under torch.no_grad() (no gradient reaches the policy).
Using the episode’s total return for every step instead of return-to-go.
Forgetting the minus sign (gradient descent on reward).
Learning rate too high: the policy collapses to one action.
Interview Tips
Walk through collect → returns → loss → step.
Explain why the loss is −log π · G.
Mention baselines and normalization for variance.
42
Code an epsilon-decreasing exploration strategy for an RL agent.
KnownRevise
An epsilon-decreasing strategy starts with a high exploration rate (ε close to 1, nearly random actions) and lowers it over time toward a small floor (0.01-0.05), so the agent explores widely while its value estimates are poor and exploits more as they improve. The common schedules are linear (ε drops by a fixed amount per step until the floor), exponential (ε multiplied by a decay factor each episode, or ε = ε_min + (ε_0 − ε_min)·e^(−kt)), inverse-time (ε = ε_0 / (1 + kt), which satisfies the conditions for convergence of tabular methods), and step-wise schedules. Implement the schedule as a small class with a value(step) method so the agent, logging and tests share one definition, and choose the decay length relative to the total training budget (for DQN, typically the first 10% of steps).
Read full answerHide full answer
Applies toPython 3.10+, NumPy, Gymnasium 1.x
Schedules
Schedule
Formula
Notes
Linear
ε = max(ε_min, ε_0 − (ε_0 − ε_min) t / T)
DQN default; reaches the floor at step T
Exponential (per episode)
ε ← max(ε_min, ε × d)
Simple; d = 0.995-0.9999
Exponential (continuous)
ε = ε_min + (ε_0 − ε_min) e^(−kt)
Smooth approach to the floor
Inverse time
ε = ε_0 / (1 + kt)
GLIE-style: decays to 0 slowly
Step-wise
ε halves every N episodes
Easy to reason about
Code Example: Schedules and an Agent That Uses Them
import math
import gymnasium as gym
import numpy as np
class LinearDecay:
def __init__(self, start=1.0, end=0.05, decay_steps=10_000):
self.start, self.end, self.decay_steps = start, end, decay_steps
def value(self, step):
frac = min(step / self.decay_steps, 1.0)
return self.start + frac * (self.end - self.start)
class ExponentialDecay:
def __init__(self, start=1.0, end=0.01, rate=1e-3):
self.start, self.end, self.rate = start, end, rate
def value(self, step):
return self.end + (self.start - self.end) * math.exp(-self.rate * step)
class InverseTimeDecay:
def __init__(self, start=1.0, k=1e-3):
self.start, self.k = start, k
def value(self, step):
return self.start / (1 + self.k * step)
for sched in (LinearDecay(), ExponentialDecay(), InverseTimeDecay()):
print(type(sched).__name__, [round(sched.value(t), 3) for t in (0, 1_000, 5_000, 20_000)])
# LinearDecay [1.0, 0.905, 0.525, 0.05]
# ExponentialDecay [1.0, 0.374, 0.017, 0.01]
# InverseTimeDecay [1.0, 0.5, 0.167, 0.048]
class EpsilonGreedyQAgent:
def __init__(self, n_states, n_actions, schedule, alpha=0.5, gamma=0.95, seed=42):
self.Q = np.zeros((n_states, n_actions))
self.schedule, self.alpha, self.gamma = schedule, alpha, gamma
self.rng = np.random.default_rng(seed)
self.steps = 0
def act(self, s, greedy=False):
eps = 0.0 if greedy else self.schedule.value(self.steps)
if self.rng.random() < eps:
return int(self.rng.integers(self.Q.shape[1]))
best = np.flatnonzero(self.Q[s] == self.Q[s].max())
return int(self.rng.choice(best))
def learn(self, s, a, r, s2, terminated):
target = r + (0.0 if terminated else self.gamma * self.Q[s2].max())
self.Q[s, a] += self.alpha * (target - self.Q[s, a])
self.steps += 1 # schedule advances per environment step
env = gym.make("FrozenLake-v1", is_slippery=False)
agent = EpsilonGreedyQAgent(env.observation_space.n, env.action_space.n, LinearDecay(decay_steps=5_000))
for episode in range(1_000):
s, _ = env.reset(seed=episode)
done = False
while not done:
a = agent.act(s)
s2, r, terminated, truncated, _ = env.step(a)
agent.learn(s, a, r, s2, terminated)
s, done = s2, terminated or truncated
print(round(agent.schedule.value(agent.steps), 3)) # current epsilon after training
By hand: linear at step 1,000 is 1 − 0.1 × 0.95 = 0.905; exponential at 1,000 is 0.01 + 0.99e⁻¹ ≈ 0.374 and at 5,000 is 0.01 + 0.99e⁻⁵ ≈ 0.017; inverse-time at 5,000 is 1/6 ≈ 0.167 and at 20,000 is 1/21 ≈ 0.048.
Choosing a Schedule
Situation
Suggestion
DQN on a fixed step budget
Linear over the first ~10% of steps to 0.01-0.05
Small tabular problems
Exponential per episode, or inverse time
Sparse rewards
Slower decay, or add intrinsic exploration
Non-stationary environment
Keep a non-zero floor
Common Pitfalls
Decaying per episode with a decay rate tuned for per-step decay (decay far too slow or too fast).
Reaching the floor before the agent has ever seen a reward.
Forgetting to use ε = 0 for evaluation.
Resetting the schedule’s step counter when resuming training from a checkpoint.
Interview Tips
Show at least linear and exponential schedules with formulas.
Explain why high early exploration and a non-zero floor make sense.
Mention GLIE conditions for convergence in tabular methods.
43
Implement a policy gradient method with a neural network in PyTorch.
KnownRevise
A practical neural policy-gradient implementation is an advantage actor-critic (A2C-style) agent: a shared network body with a policy head that outputs action logits and a value head that estimates V(s); rollouts of fixed length are collected from several vectorized environments; advantages are computed with Generalized Advantage Estimation (GAE); and one combined loss — policy gradient −log π(a|s)·Â, plus a value regression term, minus an entropy bonus — is minimized with Adam and gradient clipping. Compared with plain REINFORCE, the critic baseline and GAE reduce variance, and vectorized environments provide more, less correlated data per update. The code below follows Gymnasium’s vector API, which automatically resets finished sub-environments.
import gymnasium as gym
import numpy as np
import torch
from torch import nn
SEED, N_ENVS, N_STEPS, UPDATES = 42, 8, 32, 400
GAMMA, LAM, LR = 0.99, 0.95, 7e-4
torch.manual_seed(SEED)
envs = gym.make_vec("CartPole-v1", num_envs=N_ENVS, vectorization_mode="sync")
obs, _ = envs.reset(seed=SEED)
class ActorCritic(nn.Module):
def __init__(self, obs_dim, n_actions):
super().__init__()
self.body = nn.Sequential(nn.Linear(obs_dim, 64), nn.Tanh(), nn.Linear(64, 64), nn.Tanh())
self.pi = nn.Linear(64, n_actions)
self.v = nn.Linear(64, 1)
def forward(self, x):
h = self.body(x)
return torch.distributions.Categorical(logits=self.pi(h)), self.v(h).squeeze(-1)
model = ActorCritic(envs.single_observation_space.shape[0], envs.single_action_space.n)
optimizer = torch.optim.Adam(model.parameters(), lr=LR)
episode_returns, running = [], np.zeros(N_ENVS)
for update in range(UPDATES):
log_probs, values, entropies, rewards, terminals = [], [], [], [], []
for _ in range(N_STEPS):
dist, value = model(torch.as_tensor(obs, dtype=torch.float32))
actions = dist.sample()
obs, r, terminated, truncated, _ = envs.step(actions.numpy())
log_probs.append(dist.log_prob(actions))
values.append(value)
entropies.append(dist.entropy())
rewards.append(torch.as_tensor(r, dtype=torch.float32))
terminals.append(torch.as_tensor(terminated | truncated, dtype=torch.float32))
running += r
for i in np.flatnonzero(terminated | truncated):
episode_returns.append(running[i])
running[i] = 0.0
with torch.no_grad():
_, next_value = model(torch.as_tensor(obs, dtype=torch.float32))
# GAE, backwards over the rollout
advantages = torch.zeros(N_STEPS, N_ENVS)
gae = torch.zeros(N_ENVS)
for t in reversed(range(N_STEPS)):
v_next = next_value if t == N_STEPS - 1 else values[t + 1].detach()
nonterminal = 1.0 - terminals[t]
delta = rewards[t] + GAMMA * v_next * nonterminal - values[t].detach()
gae = delta + GAMMA * LAM * nonterminal * gae
advantages[t] = gae
values_t = torch.stack(values)
returns = advantages + values_t.detach()
policy_loss = -(torch.stack(log_probs) * advantages).mean()
value_loss = (returns - values_t).pow(2).mean()
entropy = torch.stack(entropies).mean()
loss = policy_loss + 0.5 * value_loss - 0.01 * entropy
optimizer.zero_grad()
loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), 0.5)
optimizer.step()
if (update + 1) % 100 == 0 and episode_returns:
print(f"update {update + 1}: mean return of last 20 episodes {np.mean(episode_returns[-20:]):.1f}")
envs.close()
The mean episode return should climb well above the random-policy level of about 20 as training proceeds; the exact values depend on the seed. For simplicity this treats time-limit truncation like termination (no bootstrap). Also, Gymnasium 1.x vector environments reset a finished sub-environment on the following step call, so each episode boundary adds one dummy transition with reward 0; a production implementation (or Stable-Baselines3’s A2C/PPO) masks those steps out and bootstraps truncated episodes from the final observation’s value.
Key Pieces
Piece
Purpose
Vectorized envs
More data per update, decorrelated samples
Value head
Baseline/critic for advantages
GAE (λ = 0.95)
Bias-variance balance in advantages
Entropy bonus
Keeps exploring
Gradient clipping
Stability
Common Pitfalls
Letting the policy loss backpropagate into the value estimates (advantages must be detached).
Not masking the bootstrap at episode ends.
Too large a learning rate: performance collapses after improving.
Expecting identical numbers between runs; report several seeds.
Interview Tips
Describe the actor and critic heads and the three-term loss.
Explain GAE in one sentence.
Mention vectorized environments and PPO as the next step.
44
How would you use RL to optimize traffic signals in a simulated city?
KnownRevise
I would frame each intersection’s signal controller as an RL agent in a traffic simulator such as SUMO (via its TraCI API or the sumo-rl package, which provides Gymnasium and PettingZoo environments) or CityFlow: the state is the local traffic picture (queue lengths and vehicle counts per incoming lane, current phase and how long it has been active, optionally neighbours’ states), the action is choosing the next green phase (or keeping/switching the current one) every few seconds subject to minimum-green and yellow constraints, and the reward is negative delay — for example the reduction in cumulative waiting time, or negative queue length or pressure. Train with DQN or PPO first on a single intersection, then scale to a network using independent agents with parameter sharing or cooperative multi-agent methods (neighbour observations, MA2C, CoLight-style graph attention). Train on realistic and randomized demand, compare against fixed-time and actuated baselines and max-pressure control on held-out traffic scenarios, and enforce safety constraints in the action layer before any field pilot.
Queue length, vehicle count, approaching vehicles per lane; current phase (one-hot); time in phase
Detector-based features are deployable; avoid perfect information
Action
Next green phase every 5 s (or keep/switch)
Enforce min green, yellow and all-red in the environment
Reward
−Δ(total waiting time), −queue length, or −pressure
Pressure = upstream − downstream vehicles
Episode
1 h of simulated traffic
Several demand profiles
Agents
One per intersection
Parameter sharing for scale
Algorithms
Option
When
DQN / Double DQN
Single intersection, discrete phases
PPO with shared parameters
Many similar intersections
MA2C, CoLight (graph attention)
Coordinating neighbours, green waves
Max-pressure (non-RL)
Strong baseline to beat
Code Example: Single Intersection with sumo-rl and Stable-Baselines3
# Requires SUMO installed (SUMO_HOME set) and: pip install sumo-rl stable-baselines3
from stable_baselines3 import DQN
from sumo_rl import SumoEnvironment
env = SumoEnvironment(
net_file="single-intersection.net.xml", # road network
route_file="single-intersection.rou.xml", # traffic demand
single_agent=True,
use_gui=False,
num_seconds=3600, # one simulated hour per episode
delta_time=5, # agent acts every 5 s
yellow_time=3,
min_green=5,
reward_fn="diff-waiting-time", # reduction in cumulative waiting time
)
model = DQN("MlpPolicy", env, learning_rate=1e-3, buffer_size=50_000,
exploration_fraction=0.2, exploration_final_eps=0.02, seed=42, verbose=0)
model.learn(total_timesteps=100_000)
model.save("dqn_single_intersection")
The .net.xml and .rou.xml files come from your SUMO scenario (the sumo-rl repository ships example networks). For many intersections, sumo_rl.parallel_env(...) gives a PettingZoo multi-agent environment.
Evaluation
Metric
Why
Average travel time and delay
Main objective
Queue length, stops per vehicle
Congestion and comfort
Throughput
Network capacity
Worst-approach delay
Fairness to side streets
Emissions / fuel
Secondary objective
Robustness
Unseen demand, incidents, sensor failures
Deployment Concerns
Simulation-to-real gaps (driver behaviour, detector noise), safety rules enforced outside the agent, fallback to actuated control if detectors fail, explainability for traffic engineers, and gradual pilots at a few intersections with before/after studies.
Common Pitfalls
Rewards that starve low-volume approaches (add fairness or maximum-wait terms).
Training on one demand pattern only.
Using states that cannot be measured with real detectors.
Comparing only against fixed-time plans rather than actuated or max-pressure control.
Interview Tips
Define state, action and reward clearly, with safety constraints.
Start single-intersection, then scale with parameter sharing or MARL.
Describe evaluation against strong non-RL baselines and a careful rollout plan.
45
What should you consider when applying RL to real-world robotics?
KnownRevise
Real-world robotics breaks many assumptions of RL benchmarks: real interaction is slow, expensive and can damage hardware or hurt people, so the main considerations are sample efficiency, safety, the gap between simulation and reality, and reliable sensing and resets. In practice teams train mostly in simulation (MuJoCo, Isaac Sim/Isaac Lab, PyBullet) with domain randomization of physics, latency, sensor noise and visuals, then transfer to hardware with fine-tuning; they start from demonstrations (behaviour cloning, offline RL) rather than random policies; they design safe action spaces (joint limits, velocity and torque limits, safety controllers that override the policy); they handle partial observability and delays with observation history or recurrent policies; they use smooth rewards and action penalties to avoid jerky motions; they plan automatic resets; and they monitor wear, temperature and failures. Modern systems often combine RL for low-level skills (locomotion, dexterous manipulation) with classical control and learned or language-based planners.
Higher-level actions (end-effector deltas) are easier and safer
Control frequency
Policy at 20-100 Hz, low-level PD at 1 kHz
Separate policy and controller loops
Rewards
Sparse success signals
Shaping, demonstrations, HER for goals
Resets
Someone must reset the scene
Reset policies, reset-free RL, automation
Smoothness
Jerky actions wear motors
Action-rate penalties, low-pass filters
Generalization
New objects, lighting
Diverse training, augmentation
Evaluation
Real success rate
Many trials, confidence intervals
Code Example: A Safety and Smoothness Action Wrapper (Gymnasium)
import gymnasium as gym
import numpy as np
class SafeSmoothActions(gym.ActionWrapper):
"""Clips actions to safe limits and limits how fast they can change."""
def __init__(self, env, max_abs=0.5, max_delta=0.1):
super().__init__(env)
self.max_abs, self.max_delta = max_abs, max_delta
self.prev = np.zeros(env.action_space.shape, dtype=np.float32)
def reset(self, **kwargs):
self.prev[:] = 0.0
return self.env.reset(**kwargs)
def action(self, action):
a = np.clip(action, -self.max_abs, self.max_abs) # safe range
a = np.clip(a, self.prev - self.max_delta, self.prev + self.max_delta) # rate limit
self.prev = a.astype(np.float32)
return self.prev
env = SafeSmoothActions(gym.make("Pendulum-v1"), max_abs=1.0, max_delta=0.25)
obs, _ = env.reset(seed=42)
print(env.action(np.array([2.0])), env.action(np.array([2.0]))) # [0.25] [0.5]
The first commanded torque of 2.0 is first clipped to the safe limit of 1.0 and then rate-limited to 0.25; the next one can rise only to 0.5. The same wrapper runs in simulation and on the robot, so the policy learns under the limits it will face.
Typical Workflow
Stage
Activity
1
Build and calibrate the simulator; identify system parameters
2
Collect demonstrations; pretrain with behaviour cloning
3
Train with RL in simulation with domain randomization
4
Evaluate in simulation on held-out randomizations
5
Deploy on hardware with safety limits and supervision
6
Fine-tune with real data (offline-to-online RL), monitor
Common Pitfalls
Policies exploiting simulator bugs (contacts, friction) that fail on hardware.
Reporting a handful of real trials as a success rate.
Interview Tips
Lead with sample efficiency, safety and sim-to-real.
Give concrete practices: domain randomization, demonstrations, safety layers.
Mention control architecture (policy frequency vs low-level controllers).
46
How can reinforcement learning be used to build an autonomous trading agent?
KnownRevise
An RL trading agent is framed as an MDP in which the state contains market features (recent returns, volatility, order-book or technical indicators) plus the agent’s own position, cash and unrealized P&L; the action sets a target position (for example short, flat or long, or a continuous portfolio weight) or places orders; and the reward is the change in portfolio value net of transaction costs, often risk-adjusted (differential Sharpe ratio, drawdown penalties). Training uses a historical-data simulator (backtesting environment) with realistic costs, slippage and position limits, and algorithms such as PPO, SAC or DQN; tasks with clearer structure, such as optimal execution (splitting a large order to minimize impact) and market making, are where RL has the most credible results. The hard part is not the algorithm but non-stationary, low signal-to-noise data: overfitting, look-ahead bias and unrealistic simulators produce backtests that fail live. Rigorous walk-forward evaluation, conservative assumptions and strict risk controls outside the agent are essential.
Window of log returns, volatility, volume features; current position; time of day
Action
Discrete target position {−1, 0, +1} or continuous weight in [−1, 1]
Reward
position × next return − cost × |position change| (optionally risk-adjusted)
Episode
One trading day or a rolling window
Constraints
Position and leverage limits, stop-loss, trading hours
Code Example: A Minimal Trading Environment (Gymnasium)
import gymnasium as gym
import numpy as np
from gymnasium import spaces
class SingleAssetEnv(gym.Env):
"""Target position in {-1, 0, +1}; reward = position * next return - costs."""
def __init__(self, returns, window=10, cost=0.0005):
super().__init__()
self.returns, self.window, self.cost = np.asarray(returns, dtype=np.float32), window, cost
self.action_space = spaces.Discrete(3) # 0 short, 1 flat, 2 long
self.observation_space = spaces.Box(-np.inf, np.inf, shape=(window + 1,), dtype=np.float32)
def _obs(self):
past = self.returns[self.t - self.window:self.t] # only past data: no look-ahead
return np.append(past, self.position).astype(np.float32)
def reset(self, seed=None, options=None):
super().reset(seed=seed)
self.t, self.position = self.window, 0.0
return self._obs(), {}
def step(self, action):
new_position = float(action) - 1.0
trade_cost = self.cost * abs(new_position - self.position)
self.position = new_position
reward = self.position * float(self.returns[self.t]) - trade_cost # return realized after acting
self.t += 1
terminated = self.t >= len(self.returns)
obs = self._obs() if not terminated else np.zeros(self.window + 1, dtype=np.float32)
return obs, reward, terminated, False, {}
rng = np.random.default_rng(42)
env = SingleAssetEnv(rng.normal(0.0, 0.01, 500))
obs, _ = env.reset(seed=42)
obs, reward, terminated, truncated, _ = env.step(2) # go long
print(obs.shape, obs[-1]) # (11,) 1.0
With synthetic random-walk returns there is nothing to learn, which makes this a useful sanity check: a trained agent should not show a profit here, and if it does, the environment leaks information. With real data, train PPO("MlpPolicy", env) on one period and evaluate on later, untouched periods.
Evaluation Protocol
Practice
Why
Walk-forward splits (train past, test future)
No look-ahead
Realistic costs, slippage, latency
Backtests otherwise overstate profit
Benchmarks: buy-and-hold, simple rules
Show real added value
Risk metrics: Sharpe, max drawdown, turnover
Profit alone is misleading
Many seeds and periods
Results are noisy
Paper trading before capital
Catch live-data differences
Common Pitfalls
Look-ahead bias (features using future prices, normalizing with full-period statistics).
Ignoring transaction costs, leading to hyperactive policies.
Survivorship bias in the asset universe.
Overfitting by tuning on the test period.
Assuming the market does not react to the agent’s trades (impact).
Interview Tips
Define state, action and reward with costs and risk.
Emphasize walk-forward evaluation and realistic simulation.
Mention execution and market making as stronger RL use cases, and risk controls outside the agent.
47
How is reinforcement learning applied to personalization and recommendations?
KnownRevise
Reinforcement learning treats recommendation as sequential decision-making: the system (agent) chooses what to show a user (action) given what it knows about the user and context (state), observes the response — clicks, watch time, purchases, returns (reward) — and learns a policy that maximizes long-term value such as retention or satisfaction rather than the next click alone. In practice there is a spectrum. Contextual bandits (LinUCB, Thompson sampling) handle one-step decisions with exploration and are the most widely deployed form (news article selection, ranking of promotions, notification timing). Full RL models the effect of recommendations on future user state: slate recommendation (SlateQ at YouTube), session-based policies, notification and email cadence, and long-term engagement optimization (REINFORCE with off-policy correction for YouTube recommendations, Chen et al. 2019). Because live exploration is risky, these systems rely heavily on logged data, off-policy evaluation (inverse propensity scoring, doubly robust estimators) and careful A/B testing.
Read full answerHide full answer
Applies toPython 3.10+, NumPy
Formulation
Element
Example
State
User features, recent interactions, session context, time
Code Example: Thompson Sampling for Item Selection (NumPy)
import numpy as np
rng = np.random.default_rng(42)
true_ctr = np.array([0.04, 0.05, 0.07, 0.03]) # unknown click-through rates
alpha = np.ones(4) # Beta(1, 1) prior per item
beta = np.ones(4)
shown = np.zeros(4, dtype=int)
for user in range(20_000):
sampled = rng.beta(alpha, beta) # sample a plausible CTR for each item
item = int(np.argmax(sampled)) # show the item that looks best in this sample
click = rng.random() < true_ctr[item]
alpha[item] += click
beta[item] += 1 - click
shown[item] += 1
print(shown) # most impressions go to item 2 (CTR 0.07)
print(np.round(alpha / (alpha + beta), 3)) # posterior mean CTRs
Thompson sampling explores items it is uncertain about and shifts traffic to the best item as evidence accumulates. A contextual version replaces the per-item Beta posteriors with a Bayesian model of CTR given user features.
Off-Policy Evaluation (Before Going Live)
Estimator
Idea
IPS
Reweight logged rewards by π_new(a|x) / π_logged(a|x)
Self-normalized IPS
Lower variance
Direct method
Predict rewards with a model
Doubly robust
Combine model and IPS; unbiased if either is right
These need logged propensities (the probability with which the old policy chose each item), so log them from day one.
Risks
Feedback loops and popularity bias, filter bubbles, clickbait from short-term rewards, delayed and noisy rewards, fairness to item providers, and privacy. Use long-term metrics, diversity constraints and guardrail metrics in experiments.
Common Pitfalls
Optimizing clicks only and harming long-term satisfaction.
Not logging action propensities, which rules out off-policy evaluation.
Training on logged data without correcting for the old policy’s bias.
Exploring in ways that hurt user experience (cap exploration, explore within safe candidates).
Interview Tips
Map the problem to state, action and reward with a long-term objective.
Distinguish contextual bandits from full RL and say which is used more.
Mention off-policy evaluation and feedback-loop risks.
48
In what ways can reinforcement learning be used in healthcare?
KnownRevise
Healthcare has many sequential decisions — choosing and adjusting treatments over time — so RL is studied for dynamic treatment regimes: sepsis fluid and vasopressor dosing in intensive care (the “AI Clinician” work), mechanical ventilation and sedation weaning, insulin dosing and closed-loop glucose control, chemotherapy and radiotherapy scheduling, HIV and chronic disease therapy sequencing, and adaptive clinical trial designs. Other uses include personalized mobile-health interventions (just-in-time reminders, often with contextual bandits), resource allocation such as bed and staff scheduling, and robotic surgery assistance. Because experimenting on patients is unacceptable, almost all work uses offline RL on electronic health records, with conservative algorithms (CQL, BCQ, IQL), off-policy evaluation, and clinician review. The main obstacles are confounding and missing data in records, the difficulty of defining rewards (survival, quality of life, side effects), distribution shift between hospitals, interpretability, fairness across patient groups, and regulatory requirements; deployed systems are decision support, not autonomous agents.
Read full answerHide full answer
Applies toOffline RL concepts; Python tooling such as d3rlpy for offline RL
Applications
Area
Decision
Reward signal
Typical approach
Sepsis management
Fluids, vasopressor doses every 4 h
Survival, organ function
Offline RL on ICU records (MIMIC)
Diabetes
Insulin dose
Time in glucose range, no hypoglycaemia
Simulator (UVA/Padova) + safe RL
Ventilation
Settings and weaning timing
Successful extubation
Offline RL
Oncology
Dose and schedule
Tumour control vs toxicity
Model-based RL, simulations
Mobile health
When and what to prompt
Activity, adherence
Contextual bandits (micro-randomized trials)
Operations
Beds, staff, appointments
Wait times, utilization
RL with simulators
Why Offline RL and Its Risks
Challenge
Mitigation
No online exploration
Offline RL with conservative value estimates
Policies choose actions rare in data
Behaviour regularization, action support constraints
Clinical outcome definitions with experts; multiple objectives
Distribution shift between sites
External validation
Code Example: Off-Policy Evaluation with Weighted Importance Sampling (NumPy)
import numpy as np
# Logged trajectories: for each step, behaviour-policy prob of the logged action,
# target-policy prob of the same action, and the episode's outcome reward.
episodes = [
{"b": [0.5, 0.5], "pi": [0.9, 0.8], "reward": 1.0},
{"b": [0.5, 0.5], "pi": [0.1, 0.2], "reward": 0.0},
{"b": [0.25, 0.5], "pi": [0.5, 0.5], "reward": 1.0},
]
weights = np.array([np.prod(np.array(e["pi"]) / np.array(e["b"])) for e in episodes])
rewards = np.array([e["reward"] for e in episodes])
print(np.round(weights, 2)) # [2.88 0.08 2. ]
wis = (weights * rewards).sum() / weights.sum()
print(round(wis, 3)) # 0.984
print(round(rewards.mean(), 3)) # 0.667 (behaviour policy's average)
By hand: weights are (0.9/0.5)(0.8/0.5) = 2.88, (0.1/0.5)(0.2/0.5) = 0.08 and (0.5/0.25)(0.5/0.5) = 2; WIS = (2.88 + 2) / (2.88 + 0.08 + 2) = 4.88 / 4.96 ≈ 0.984. The target policy looks better than the logged one (0.667), but the estimate rests on just two heavily weighted episodes, which is exactly the situation in which off-policy estimates mislead.
Common Pitfalls
Trusting a high off-policy estimate based on a few heavily weighted episodes (check effective sample size).
Using rewards that ignore side effects or quality of life.
Ignoring that records reflect clinicians’ decisions and biases.
Presenting a research policy as ready for autonomous use.
Interview Tips
Give specific applications (sepsis, insulin, ventilation, mobile health).
Stress offline RL, off-policy evaluation and safety.
Discuss reward design, confounding, fairness and regulation.
49
How would you tune hyperparameters for a reinforcement learning model?
KnownRevise
RL hyperparameter tuning is harder than in supervised learning because results vary a lot between random seeds, each trial is an expensive training run, and the evaluation signal (episode return) is noisy — so tune systematically with a fixed budget, multiple seeds and a separate evaluation protocol. Start from published defaults for the algorithm and environment class (for example the RL Baselines3 Zoo configurations), identify the few parameters that matter most — learning rate, discount factor γ, batch/rollout size, entropy coefficient or exploration schedule, network size, and algorithm-specific ones such as PPO’s clip range and GAE λ or DQN’s replay size and target update interval — and search them with Bayesian optimization (Optuna’s TPE) using early stopping/pruning of bad trials. Score each configuration by the mean evaluation return over several seeds (or a robust statistic such as the interquartile mean), check the best configurations on fresh seeds, and report variability rather than one best run.
Code Example: Optuna + Stable-Baselines3 with Pruning
import gymnasium as gym
import numpy as np
import optuna
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy
def objective(trial):
params = {
"learning_rate": trial.suggest_float("learning_rate", 1e-5, 1e-3, log=True),
"gamma": trial.suggest_float("gamma", 0.95, 0.999),
"n_steps": trial.suggest_categorical("n_steps", [256, 512, 1024, 2048]),
"ent_coef": trial.suggest_float("ent_coef", 1e-8, 1e-2, log=True),
"clip_range": trial.suggest_float("clip_range", 0.1, 0.3),
"gae_lambda": trial.suggest_float("gae_lambda", 0.9, 0.99),
}
scores = []
for seed in (0, 1, 2): # several seeds per configuration
env = gym.make("CartPole-v1")
model = PPO("MlpPolicy", env, seed=seed, verbose=0, **params)
for chunk in range(4):
model.learn(total_timesteps=10_000, reset_num_timesteps=False)
mean_reward, _ = evaluate_policy(model, gym.make("CartPole-v1"), n_eval_episodes=5)
if seed == 0:
trial.report(mean_reward, step=chunk)
if trial.should_prune(): # stop clearly bad trials early
raise optuna.TrialPruned()
mean_reward, _ = evaluate_policy(model, gym.make("CartPole-v1"), n_eval_episodes=10)
scores.append(mean_reward)
return float(np.mean(scores))
study = optuna.create_study(direction="maximize",
sampler=optuna.samplers.TPESampler(seed=42),
pruner=optuna.pruners.MedianPruner(n_warmup_steps=1))
study.optimize(objective, n_trials=30)
print(study.best_params)
After the search, retrain the best configuration (and the runners-up) on new seeds and compare their mean ± standard deviation or confidence intervals; the top trial of a noisy search is often partly lucky.
Practical Workflow
Step
Action
1
Reproduce a known result with default hyperparameters
2
Fix evaluation: deterministic policy, held-out seeds, N episodes
3
Search 3-6 key parameters, log scale for rates
4
Prune early, run in parallel
5
Re-evaluate top configurations on fresh seeds
6
Report variability (IQM, confidence intervals across ≥ 5 seeds)
Common Pitfalls
Choosing the best of many single-seed runs (selection on noise).
Tuning on the evaluation seeds used for the final report.
Changing environment wrappers or reward scale between trials without noticing.
Ignoring normalization, which often matters more than any single parameter.
Interview Tips
Name the most sensitive parameters and their typical ranges.
Describe Optuna/TPE with pruning and multi-seed scoring.
Stress robust evaluation and reporting across seeds.
50
How would you design an RL agent to learn optimal strategies for a game?
KnownRevise
I would design the agent by first formalizing the game as an environment — state (or observation), legal actions, reward, episode end, and whether it is single-agent, two-player zero-sum or multi-player, perfect or imperfect information — then choose a method matched to that structure: DQN or PPO for single-player arcade-style games, self-play with tree search and a policy-value network (AlphaZero-style) for perfect-information board games, and self-play with population-based training or counterfactual-regret methods for imperfect-information games like poker. The environment should be fast and follow a standard interface (Gymnasium for single-agent, PettingZoo for multi-agent), with action masking for illegal moves. Rewards start with the true outcome (win/loss or score) and add careful potential-based shaping only if learning stalls. Training uses vectorized environments, curricula or self-play against past versions to avoid strategy cycling, and evaluation measures strength against fixed baselines (random, scripted, earlier checkpoints) with Elo ratings.
Code Example: Masked Legal Moves and a Self-Play Opponent Pool
import numpy as np
import torch
from torch import nn
ROWS, COLS = 6, 7
def legal_mask(board):
"""board: (6, 7) with 0 empty, 1/-1 players. A column is legal if its top cell is empty."""
return board[0] == 0
class PolicyValueNet(nn.Module):
def __init__(self):
super().__init__()
self.body = nn.Sequential(nn.Conv2d(2, 32, 3, padding=1), nn.ReLU(),
nn.Conv2d(32, 32, 3, padding=1), nn.ReLU(), nn.Flatten())
self.policy = nn.Linear(32 * ROWS * COLS, COLS)
self.value = nn.Sequential(nn.Linear(32 * ROWS * COLS, 1), nn.Tanh())
def forward(self, planes, mask):
h = self.body(planes)
logits = self.policy(h).masked_fill(~mask, float("-inf")) # illegal moves get probability 0
return torch.distributions.Categorical(logits=logits), self.value(h).squeeze(-1)
def to_planes(board, player):
return torch.tensor(np.stack([board == player, board == -player]), dtype=torch.float32).unsqueeze(0)
torch.manual_seed(42)
net = PolicyValueNet()
board = np.zeros((ROWS, COLS), dtype=int)
board[:, 3] = [1, -1, 1, -1, 1, -1] # column 3 is full
mask = torch.tensor(legal_mask(board)).unsqueeze(0)
dist, value = net(to_planes(board, player=1), mask)
print(dist.probs[0, 3].item(), mask.sum().item()) # 0.0 6
opponent_pool = [] # past checkpoints for self-play
opponent_pool.append({k: v.clone() for k, v in net.state_dict().items()})
rng = np.random.default_rng(42)
opponent_weights = opponent_pool[rng.integers(len(opponent_pool))] # sample an opponent each game
Masking with −∞ before the softmax gives illegal columns exactly zero probability, so the agent never wastes experience on invalid moves. Sampling opponents from a pool of past checkpoints keeps the agent from overfitting to its latest self and cycling between strategies.
Common Pitfalls
Penalizing illegal moves instead of masking them (slow learning).
Pure self-play against only the latest version (strategy cycling, forgetting).
Observations not from the current player’s perspective in two-player games.
Evaluating only against itself; use fixed external baselines and Elo.
Interview Tips
Start with the game’s structure (players, information, determinism).
Choose the algorithm family to match, and justify it.
Cover masking, self-play opponent pools, reward design and evaluation.
51
Propose an RL framework for energy management in a smart grid.
KnownRevise
I would build the energy management system as a constrained, possibly multi-agent RL problem trained in a grid simulator: agents control flexible resources — battery storage charge/discharge, EV charging rates, heat pumps and HVAC setpoints, flexible industrial loads, and dispatchable generation — and observe the state of the system (prices or tariffs, demand and renewable forecasts, battery state of charge, temperatures, network voltages); the reward combines negative operating cost, penalties for unmet demand and comfort violations, battery degradation, emissions, and network constraint violations. Training uses historical and synthetic scenarios in simulators such as CityLearn (building demand response), Grid2Op (transmission operation) or pandapower-based distribution models; algorithms are SAC or PPO for continuous control, with safety layers or constrained RL (Lagrangian methods) for hard limits. A centralized-training, decentralized-execution multi-agent design scales to many buildings or feeders. Deployment adds forecasting, a rule-based or optimization fallback (model predictive control), human oversight, and staged rollout, and the RL policy must beat an MPC baseline to be worth it.
SoC bounds, charger limits, voltage/thermal limits, comfort band
Time step
15 min to 1 h, episodes of one day to one year
Agents
One per building, shared critic with grid-level information during training
Architecture
Layer
Role
Forecasting
Load, PV, price forecasts feed the state
RL policy
Proposes setpoints
Safety layer
Projects actions into feasible region (SoC, power, voltage)
Fallback
MPC or rule-based controller if forecasts or policy fail
Monitoring
Cost, peak, comfort, constraint violations vs baseline
Code Example: Battery Arbitrage Environment with a Safety Projection (Gymnasium)
import gymnasium as gym
import numpy as np
from gymnasium import spaces
class BatteryEnv(gym.Env):
"""Hourly battery control against a price signal. Action in [-1, 1] = fraction of max power."""
def __init__(self, prices, capacity_kwh=10.0, max_kw=5.0, efficiency=0.95):
super().__init__()
self.prices = np.asarray(prices, dtype=np.float32)
self.capacity, self.max_kw, self.eff = capacity_kwh, max_kw, efficiency
self.action_space = spaces.Box(-1.0, 1.0, shape=(1,), dtype=np.float32)
self.observation_space = spaces.Box(-np.inf, np.inf, shape=(3,), dtype=np.float32)
def _obs(self):
hour = self.t % 24
return np.array([self.soc / self.capacity, self.prices[self.t], hour / 23], dtype=np.float32)
def reset(self, seed=None, options=None):
super().reset(seed=seed)
self.t, self.soc = 0, 0.5 * self.capacity
return self._obs(), {}
def step(self, action):
power = float(np.clip(action[0], -1, 1)) * self.max_kw # + charge, - discharge (kW for 1 h)
# Safety projection: never exceed the battery's energy limits
power = min(power, (self.capacity - self.soc) / self.eff)
power = max(power, -self.soc * self.eff)
self.soc += power * self.eff if power > 0 else power / self.eff
grid_energy = power # kWh bought (+) or sold (-)
reward = -self.prices[self.t] * grid_energy - 0.01 * abs(power) # cost + degradation proxy
self.t += 1
terminated = self.t >= len(self.prices)
obs = self._obs() if not terminated else np.zeros(3, dtype=np.float32)
return obs, float(reward), terminated, False, {"soc": self.soc}
prices = np.array([0.10] * 6 + [0.30] * 6 + [0.15] * 6 + [0.40] * 6) # cheap night, expensive evening
env = BatteryEnv(prices)
obs, _ = env.reset(seed=42)
_, r, *_ , info = env.step(np.array([1.0]))
print(round(info["soc"], 2), round(r, 3)) # 9.75 -0.55
By hand: starting at 5 kWh, a full-power charge is capped at (10 − 5)/0.95 ≈ 5.26 kW, so the requested 5 kW is allowed; the battery stores 5 × 0.95 = 4.75 kWh (SoC 9.75) and the reward is −0.10 × 5 − 0.01 × 5 = −0.55. A trained SAC agent (SAC("MlpPolicy", env)) should learn to charge in cheap hours and discharge in the expensive evening; compare it with the optimal schedule from a linear program on the same prices.
Evaluation
Metric
Baseline
Energy cost, peak demand, emissions
Rule-based (time-of-use) and MPC with perfect/forecast information
Comfort and EV service violations
Must not increase
Constraint violations
Zero tolerated in deployment
Robustness
Unseen weather, price spikes, forecast errors
Common Pitfalls
Ignoring battery degradation, so the agent cycles the battery constantly.
Training on perfect forecasts and deploying with noisy ones.
Relying on reward penalties alone for hard grid constraints (use a safety layer).
Not comparing with MPC, which is often hard to beat.
Interview Tips
Define state, action, reward and constraints concretely.
Address safety (projection, fallback) and multi-agent scaling.
Describe evaluation against rule-based and MPC baselines.
52
How would you set up an RL environment for chess?
KnownRevise
A chess RL environment needs a correct game engine for the rules (the python-chess library handles legal moves, check, castling, en passant, promotion and all draw rules), an observation encoding of the board from the perspective of the player to move, an action space with masking of illegal moves, a reward at the end of the game (+1 win, 0 draw, −1 loss), and a multi-agent or self-play interface so the agent can play against itself and against fixed opponents. Following AlphaZero, the observation is a stack of 8×8 planes (one per piece type and colour, plus planes for side to move, castling rights, repetition counts and the 50-move counter, often with a few past positions), and the action space is a fixed set of 4,672 move encodings (73 move types from each of 64 squares: queen-like moves, knight moves and underpromotions), of which only the legal ones are unmasked. Training typically uses self-play with Monte Carlo Tree Search guided by a policy-value network, or PPO self-play for a simpler baseline; evaluation measures Elo against engines such as Stockfish at fixed strengths.
Code Example: Board Encoding and Legal-Move Mask (python-chess + NumPy)
import chess
import numpy as np
PIECES = [chess.PAWN, chess.KNIGHT, chess.BISHOP, chess.ROOK, chess.QUEEN, chess.KING]
def encode(board):
"""12 piece planes from the side-to-move's perspective + 1 plane for castling rights."""
planes = np.zeros((13, 8, 8), dtype=np.float32)
me, them = board.turn, not board.turn
for i, piece in enumerate(PIECES):
for color, offset in ((me, 0), (them, 6)):
for sq in board.pieces(piece, color):
rank, file = divmod(sq, 8)
if me == chess.BLACK:
rank = 7 - rank # flip so the mover always plays "up"
planes[offset + i, rank, file] = 1.0
planes[12, 0, 0] = float(board.has_kingside_castling_rights(me))
planes[12, 0, 1] = float(board.has_queenside_castling_rights(me))
return planes
# Simple action space: index over all from-square x to-square pairs (64 * 64) + promotion handled by queen default
def move_to_index(move):
return move.from_square * 64 + move.to_square
def legal_mask(board):
mask = np.zeros(64 * 64, dtype=bool)
for move in board.legal_moves:
mask[move_to_index(move)] = True
return mask
board = chess.Board()
print(encode(board).shape, int(encode(board)[:12].sum())) # (13, 8, 8) 32
print(legal_mask(board).sum()) # 20 legal moves at the start
def reward_for(board, mover):
outcome = board.outcome(claim_draw=True)
if outcome is None:
return 0.0, False
if outcome.winner is None:
return 0.0, True
return (1.0 if outcome.winner == mover else -1.0), True
board = chess.Board()
for uci in ["f2f3", "e7e5", "g2g4", "d8h4"]: # fool's mate
board.push_uci(uci)
print(reward_for(board, chess.BLACK)) # (1.0, True)
This simple 4,096-index space folds all promotions into one index per square pair; a full implementation adds underpromotion indices (or uses AlphaZero’s 4,672 encoding). For a ready-made multi-agent version, PettingZoo’s chess_v6 environment provides observations and action masks in this style.
Training Approach
Approach
Notes
AlphaZero (MCTS + policy-value net)
Strongest; needs large compute for self-play
Supervised pretraining on master games
Fast start; then RL fine-tuning
PPO self-play with masks
Simpler baseline; weaker than search-based methods
Evaluation
Elo vs Stockfish levels, tournaments between checkpoints
Common Pitfalls
Hand-coding chess rules (bugs in castling, en passant, repetition).
Encoding the board in absolute colours so the network must learn two mirror-image games.
Penalizing illegal moves instead of masking them.
No move cap, leading to extremely long early games.
Interview Tips
Describe observation planes, the action encoding and masking.
Explain self-play with MCTS (AlphaZero) and evaluation by Elo.
Mention using an established rules library.
53
What are recent advances in multi-agent reinforcement learning?
KnownRevise
Multi-agent reinforcement learning (MARL) studies several agents learning at the same time in a shared environment, cooperatively, competitively or both; major advances of recent years include: centralized training with decentralized execution (CTDE) through value factorization (VDN, QMIX, QPLEX) and centralized critics (MADDPG, COMA), the surprisingly strong performance of simple on-policy methods with parameter sharing (MAPPO, IPPO); large-scale self-play and league training that reached top human level in StarCraft II (AlphaStar), Dota 2 (OpenAI Five) and Stratego (DeepNash); sequence-model approaches that treat agents as tokens (Multi-Agent Transformer); population-based methods for zero-shot coordination with unseen partners (Other-Play, Fictitious Co-Play); and combining language models with RL for negotiation and cooperation (Meta’s CICERO in Diplomacy). Active research areas include communication learning, credit assignment, scalability to hundreds of agents with mean-field and graph-based methods, robustness to adversarial agents, and LLM-based multi-agent systems. Benchmarks such as SMAC/SMACv2, Google Research Football, Melting Pot, Overcooked and PettingZoo environments drive progress.
Read full answerHide full answer
Applies toPettingZoo, PyTorch 2.x; libraries such as EPyMARL, BenchMARL, RLlib, JaxMARL
Key Advances
Area
Methods
Idea
Value factorization
VDN, QMIX, QPLEX
Team Q = monotonic combination of per-agent Qs; decentralized argmax
import torch
from torch import nn
class MonotonicMixer(nn.Module):
"""Combines per-agent Q-values into Q_tot with non-negative weights produced from the global state."""
def __init__(self, n_agents, state_dim, embed=32):
super().__init__()
self.n_agents, self.embed = n_agents, embed
self.hyper_w1 = nn.Linear(state_dim, n_agents * embed)
self.hyper_b1 = nn.Linear(state_dim, embed)
self.hyper_w2 = nn.Linear(state_dim, embed)
self.hyper_b2 = nn.Sequential(nn.Linear(state_dim, embed), nn.ReLU(), nn.Linear(embed, 1))
def forward(self, agent_qs, state):
b = agent_qs.shape[0]
w1 = torch.abs(self.hyper_w1(state)).view(b, self.n_agents, self.embed) # abs keeps it monotonic
hidden = torch.nn.functional.elu(torch.bmm(agent_qs.unsqueeze(1), w1).squeeze(1) + self.hyper_b1(state))
w2 = torch.abs(self.hyper_w2(state)).unsqueeze(-1)
return (torch.bmm(hidden.unsqueeze(1), w2).squeeze(-1) + self.hyper_b2(state)).squeeze(-1)
torch.manual_seed(42)
mixer = MonotonicMixer(n_agents=3, state_dim=10)
qs = torch.randn(4, 3, requires_grad=True)
q_tot = mixer(qs, torch.randn(4, 10))
q_tot.sum().backward()
print(q_tot.shape, bool((qs.grad >= 0).all())) # torch.Size([4]) True
Because ∂Q_tot/∂Q_i ≥ 0, each agent maximizing its own Q_i also maximizes the team value, so agents can act independently at execution time while training uses the global state.
Common Pitfalls
Treating other agents as part of a stationary environment.
Comparing methods with different tuning budgets (MAPPO results showed tuning matters a lot).
Self-play without opponent diversity.
Interview Tips
Name CTDE, value factorization and centralized critics.
Mention landmark systems (AlphaStar, OpenAI Five, CICERO) and MAPPO’s lesson.
Explain the core challenges of non-stationarity and credit assignment.
54
How does curriculum learning work in reinforcement learning?
KnownRevise
Curriculum learning in RL trains an agent on a sequence of tasks that starts easy and becomes harder, so the agent gets learning signal early and builds skills that transfer to the final, hard task, instead of facing a sparse-reward problem it almost never solves by chance. Curricula can vary the start state (begin near the goal and move away, “reverse curriculum”), the goal distance, environment parameters (terrain roughness, obstacle density, opponent strength), the reward density, or the task itself (walk, then run, then jump). They can be hand-designed with performance thresholds, or automatic: teacher algorithms pick tasks where the agent’s learning progress is highest (Teacher-Student curriculum learning), Prioritized Level Replay replays levels with high value error, goal generators propose goals of intermediate difficulty (GoalGAN), and unsupervised environment design (PAIRED, ACCEL) generates environments at the edge of the agent’s ability. Self-play is a natural curriculum because the opponent improves with the agent.
Read full answerHide full answer
Applies toPython 3.10+, NumPy, Gymnasium 1.x
Curriculum Types
Type
What changes
Example
Start-state curriculum
Distance from goal at reset
Reverse curriculum for robot assembly
Goal curriculum
Goal difficulty
GoalGAN: goals the agent solves 10-90% of the time
from collections import deque
import gymnasium as gym
import numpy as np
class DistanceCurriculum(gym.Wrapper):
"""Grid-style task whose start distance from the goal grows when the agent succeeds often enough."""
def __init__(self, env, levels=(1, 2, 4, 8), promote_at=0.8, window=50):
super().__init__(env)
self.levels, self.level = levels, 0
self.promote_at = promote_at
self.recent = deque(maxlen=window)
@property
def start_distance(self):
return self.levels[self.level]
def record(self, success):
self.recent.append(float(success))
full = len(self.recent) == self.recent.maxlen
if full and np.mean(self.recent) >= self.promote_at and self.level < len(self.levels) - 1:
self.level += 1 # promote to a harder level
self.recent.clear()
def reset(self, **kwargs):
options = dict(kwargs.pop("options", None) or {})
options["start_distance"] = self.start_distance # the wrapped env reads this option
return self.env.reset(options=options, **kwargs)
curriculum = DistanceCurriculum(gym.make("CartPole-v1")) # stand-in env; a real one would use the option
for _ in range(50):
curriculum.record(success=True)
print(curriculum.start_distance) # 2 promoted after 50 successes
for _ in range(50):
curriculum.record(success=False)
print(curriculum.start_distance) # 2 stays while performance is low
The wrapper passes the current difficulty to the environment through reset(options=...) and promotes the agent only after a full window of episodes above 80% success. Some curricula also demote when performance drops, and many mix in a share of easier tasks to avoid forgetting.
Design Guidelines
Guideline
Reason
Keep tasks at intermediate difficulty
Maximum learning signal
Promote on measured success, not on a fixed schedule
Adapts to the agent
Replay earlier tasks occasionally
Prevents forgetting
Always evaluate on the final target task
The curriculum is a means, not the goal
Common Pitfalls
Promoting too fast, so the agent never masters the basics.
Easy tasks that teach habits that hurt on the hard task.
Forgetting easy cases after moving on.
Reporting curriculum-level returns instead of target-task performance.
Interview Tips
Explain why curricula help with sparse rewards.
Give hand-designed and automatic examples (PLR, PAIRED, GoalGAN).
Mention self-play as an implicit curriculum.
55
Explain the concept of meta-reinforcement learning.
KnownRevise
Meta-reinforcement learning (“learning to learn” for RL) trains an agent on a distribution of related tasks so that it can adapt to a new task from the same distribution with very little experience — a few episodes or even within one episode — rather than learning from scratch. There are two main families. Gradient-based meta-RL (MAML, Finn et al. 2017) learns initial network parameters from which a few policy-gradient steps on a new task give good performance; the outer loop optimizes performance after adaptation. Context- or memory-based meta-RL (RL², Duan et al. 2016; L2RL, Wang et al.) uses a recurrent or transformer policy that receives past observations, actions and rewards, so the learning algorithm is encoded in its activations: it explores, infers the task, and exploits, without weight updates at test time. Inference-based methods such as PEARL learn a latent task variable from context with a probabilistic encoder, which supports efficient off-policy training and posterior sampling for exploration. Applications include robots adapting to new payloads or damage, personalized recommendations and fast adaptation in games; challenges are designing task distributions, meta-overfitting, and exploration for task identification.
A few gradient steps from a meta-learned initialization
MAML, ProMP
Generalizes beyond training tasks somewhat
Second-order gradients, high variance
Recurrent / context
Hidden-state updates from history
RL², L2RL, transformer agents (AdA)
Fast, no test-time gradients
Hard to train; limited to similar tasks
Inference-based
Latent task posterior
PEARL, VariBAD
Sample-efficient, principled exploration
Needs good task encoders
Setup
Level
Contents
Task distribution p(T)
E.g. goal positions, payloads, bandit arm probabilities
Inner loop
Adapt to one task (gradient steps or recurrent updates)
Outer loop
Update meta-parameters to maximize post-adaptation return
Meta-test
New tasks from p(T); measure return after k episodes
Code Example: RL²-Style Policy Input (PyTorch)
import torch
from torch import nn
torch.manual_seed(42)
N_ARMS, HIDDEN = 5, 64
class RL2Policy(nn.Module):
"""Recurrent policy whose input includes the previous action and reward, so it can infer the task."""
def __init__(self, n_actions, obs_dim=1):
super().__init__()
self.gru = nn.GRU(obs_dim + n_actions + 1, HIDDEN, batch_first=True)
self.pi = nn.Linear(HIDDEN, n_actions)
self.n_actions = n_actions
def forward(self, obs, prev_action, prev_reward, h=None):
a_onehot = nn.functional.one_hot(prev_action, self.n_actions).float()
x = torch.cat([obs, a_onehot, prev_reward.unsqueeze(-1)], dim=-1).unsqueeze(1)
out, h = self.gru(x, h)
return torch.distributions.Categorical(logits=self.pi(out[:, 0])), h
policy = RL2Policy(N_ARMS)
arm_probs = torch.rand(N_ARMS) # one sampled task: a new bandit
h, prev_a, prev_r = None, torch.zeros(1, dtype=torch.long), torch.zeros(1)
for t in range(10): # hidden state is NOT reset between pulls within a task
dist, h = policy(torch.zeros(1, 1), prev_a, prev_r, h)
prev_a = dist.sample()
prev_r = (torch.rand(1) < arm_probs[prev_a]).float()
print(h.shape) # torch.Size([1, 1, 64])
During meta-training, the policy is trained (for example with PPO) on many sampled bandits, with returns summed across all pulls of a task, so it learns an exploration-then-exploitation strategy in its hidden state. At meta-test time it faces a new bandit and adapts with no gradient updates. The hidden state is reset only when a new task is sampled.
Common Pitfalls
Resetting the recurrent state between episodes of the same task (it forgets what it inferred).
Narrow task distributions that make the meta-learner memorize instead of adapt.
Evaluating on tasks from the training set.
Comparing with baselines that were not given the same number of adaptation episodes.
Interview Tips
Define meta-RL as fast adaptation across a task distribution.
Contrast MAML (gradients) with RL² (memory) and PEARL (latent inference).
Give a concrete use case such as robot adaptation.
56
What are the challenges of safe RL in sensitive domains like healthcare or driving?
KnownRevise
Safe reinforcement learning aims to learn good policies while avoiding unacceptable outcomes during both training and deployment, and in domains like healthcare and autonomous driving that is hard because standard RL learns by trial and error, maximizes a proxy reward, and offers no built-in guarantees. The main challenges are: safe exploration (you cannot try dangerous actions on patients or roads, so learning must happen offline or in simulation); specifying safety (rewards miss rare but catastrophic events, and penalties can be traded off against reward); distribution shift (situations unseen in training, such as new patient populations or unusual road users, where the policy may behave unpredictably); rare events and long tails that need far more testing than average-case metrics show; offline evaluation of a new policy from logged data; sim-to-real gaps; verification and interpretability for clinicians, engineers and regulators; and accountability and regulation (medical device rules, automotive safety standards). Techniques include constrained MDPs with Lagrangian methods (CPO, PPO-Lagrangian), safety layers and shields that override unsafe actions, conservative offline RL, uncertainty estimation and abstention, risk-sensitive objectives (CVaR), formal verification for limited components, human-in-the-loop operation, and staged deployment.
Interpretable policies, explanations, decision support only
Accountability
Who is responsible for an error?
Human oversight, audit logs, regulation
Constrained RL Formulation
Maximize E[Σ γᵗ r_t] subject to E[Σ γᵗ c_t] ≤ d, where c_t is a safety cost. The Lagrangian method alternates policy updates on r − λc with a dual update λ ← max(0, λ + η(Ĵ_c − d)), so λ rises while the cost exceeds its limit.
Code Example: Lagrange Multiplier Update and a Safety Shield (Python)
def update_lambda(lam, measured_cost, cost_limit, lr=0.05):
return max(0.0, lam + lr * (measured_cost - cost_limit))
lam = 0.0
for cost in [30.0, 30.0, 20.0, 10.0]: # episode safety costs, limit 25
lam = update_lambda(lam, cost, cost_limit=25.0)
print(round(lam, 2), end=" ")
print() # 0.25 0.5 0.25 0.0
def shield(proposed_dose_mg, max_dose_mg=10.0, last_dose_mg=0.0, max_increase_mg=2.0):
"""Rule-based safety layer: clip to the maximum dose and to a maximum increase per step."""
safe = min(proposed_dose_mg, max_dose_mg, last_dose_mg + max_increase_mg)
return max(safe, 0.0)
print(shield(15.0, last_dose_mg=5.0)) # 7.0
By hand: λ = 0 + 0.05 × 5 = 0.25, then 0.5, then 0.5 + 0.05 × (−5) = 0.25, then max(0, 0.25 − 0.75) = 0. The shield clips a proposed 15 mg to the 5 + 2 = 7 mg allowed increase; it is illustrative only, as real dosing limits come from clinical protocols.
Deployment Practices
Practice
Purpose
Decision support, not autonomy
Human makes the final decision
Shadow mode
Compare recommendations with real decisions without acting
Staged rollout with kill switch
Limit exposure
Monitoring of safety metrics and drift
Detect problems early
External validation
Different sites, populations, conditions
Common Pitfalls
Encoding safety only as a reward penalty that can be outweighed.
Trusting simulation results without real-world validation.
Ignoring fairness: safe on average but unsafe for subgroups.
Treating rare-event performance as covered by average metrics.
Interview Tips
List safe exploration, specification, shift, rare events and evaluation.
Describe constrained MDPs and safety layers.
Emphasize human oversight and staged deployment.
57
Why does interpretability matter in RL, and how can it be achieved?
KnownRevise
Interpretability matters in reinforcement learning because RL agents make sequences of decisions, often with neural policies, in settings where people must trust, debug, audit or override them — and because RL agents are known to find unexpected strategies, including reward hacking, that are invisible in aggregate return curves. Understanding why an agent acts helps catch exploits and spurious correlations before deployment, satisfies regulators and domain experts (clinicians, operators, traders), supports safe human-agent collaboration, and speeds up debugging. It can be achieved at different levels: intrinsically interpretable policies (decision trees distilled from neural policies, as in VIPER; linear or rule-based policies; programmatic policies); post-hoc explanations of neural policies (saliency maps over observations, SHAP for state features, attention weights); value and reward decomposition (separate Q-heads per reward component, so you can see that “avoid collision” dominates a choice); outcome-based explanations (predicted future states, counterfactual “what if” rollouts); summaries of behaviour (representative trajectories, critical states where the action matters most); and mechanistic analysis of internal representations.
Code Example: Distilling a Policy into a Decision Tree (scikit-learn)
import gymnasium as gym
import numpy as np
from sklearn.tree import DecisionTreeClassifier, export_text
def teacher_policy(obs):
"""Stand-in for a trained neural policy on CartPole: push toward the falling side."""
x, x_dot, theta, theta_dot = obs
return int(theta + 0.5 * theta_dot > 0)
env = gym.make("CartPole-v1")
states, actions = [], []
for ep in range(30):
obs, _ = env.reset(seed=ep)
done = False
while not done:
a = teacher_policy(obs)
states.append(obs)
actions.append(a)
obs, _, terminated, truncated, _ = env.step(a)
done = terminated or truncated
X, y = np.array(states), np.array(actions)
tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X, y)
print(f"agreement with teacher: {tree.score(X, y):.3f}")
print(export_text(tree, feature_names=["x", "x_dot", "theta", "theta_dot"]))
The tree’s rules (thresholds on the pole angle and angular velocity) can be read and checked by a person. VIPER improves on this simple behaviour cloning by collecting states visited by the tree itself (DAgger-style) and weighting states by how much the action matters (Q-value gaps). Always check the tree’s return in the environment, not only its agreement with the teacher.
Code Example: Critical States from Q-Values
import numpy as np
q_values = np.array([[1.0, 1.1, 0.9], # actions barely matter here
[5.0, -3.0, 0.0], # wrong action is very costly here
[2.0, 2.0, 1.5]])
importance = q_values.max(axis=1) - q_values.min(axis=1)
print(np.round(importance, 3).tolist(), int(np.argmax(importance))) # [0.2, 8.0, 0.5] 1
States with a large spread are where explanations and human review add the most value.
Common Pitfalls
Treating saliency maps as proof of what the agent “knows”; validate with interventions.
Distilled policies that look simple but perform worse; report their performance.
Explaining average behaviour while rare critical states go unexamined.
Interview Tips
Give reasons: trust, debugging, regulation, reward-hacking detection.
Name intrinsic (trees, decomposition) and post-hoc (saliency, counterfactuals) methods.
Show a concrete technique such as policy distillation.
58
What ethical concerns arise when deploying reinforcement learning systems?
KnownRevise
Reinforcement learning raises ethical concerns beyond those of predictive models because RL systems act in the world, optimize objectives over time and learn from interaction: they can cause harm through their actions, pursue a mis-specified reward in ways people did not intend, change the behaviour of the people and markets they interact with, and experiment on users while exploring. Key concerns include: safety and harm (physical harm from robots or vehicles, clinical harm from treatment policies); reward mis-specification and manipulation (engagement-optimizing recommenders that promote addictive or extreme content, or nudge users against their interests); fairness (policies that serve some groups worse, often inherited from historical data or proxy rewards); consent and experimentation (exploration means some users receive deliberately suboptimal or untested treatments); transparency and accountability (who is responsible when an autonomous policy causes harm, and can decisions be explained and contested?); privacy (logging detailed interaction data); feedback loops and societal effects (algorithmic pricing that can tacitly collude, trading agents that amplify volatility); and dual use (autonomous weapons, surveillance). Responsible practice combines careful objective design, constraints, human oversight, audits, impact assessments, transparency to users, and staged, monitored deployment.
Read full answerHide full answer
Applies toRL system design and governance
Concerns and Mitigations
Concern
Example
Mitigation
Physical/clinical harm
Robot collision, dosing error
Safety constraints, shields, human oversight, testing
Exploration policy, ethics/IRB review where relevant
Who can override or switch it off?
Runbook, kill switch
How will harms be detected after launch?
Monitoring, incident reporting
Common Pitfalls
Treating ethics as a post-launch review instead of a design input.
Measuring success only by the optimized reward.
Assuming offline evaluation captures how people will adapt to the system.
Interview Tips
Explain why acting and optimizing over time makes RL ethics distinctive.
Cover reward mis-specification, fairness, exploration on people and accountability.
Offer concrete mitigations and governance practices.
59
How can the alignment problem be tackled in reinforcement learning?
KnownRevise
The alignment problem in RL is the gap between what we want an agent to do and what it actually optimizes: we specify a reward that is only a proxy for our goals, and a capable optimizer finds ways to score highly on the proxy that we did not intend (specification gaming, reward hacking), may learn goals that only coincidentally matched the reward during training (goal misgeneralization), and may pursue instrumentally useful behaviour such as avoiding correction. Approaches to narrowing the gap include: learning rewards from people rather than hand-coding them — inverse RL from demonstrations and preference-based RL, which became RLHF for language models (a reward model trained on human comparisons, then policy optimization such as PPO with a KL penalty to a reference model), plus AI-feedback variants such as Constitutional AI / RLAIF and direct methods such as DPO; regularization against over-optimization (KL constraints, early stopping, reward model ensembles, uncertainty penalties); scalable oversight (debate, recursive reward modelling, AI-assisted evaluation) so humans can supervise tasks they cannot judge directly; conservatism and impact limits (avoid side effects, constrained RL, corrigibility research); interpretability to inspect what objectives a model has learned; and red-teaming and evaluation for hacking, deception and misgeneralization under distribution shift.
Read full answerHide full answer
Applies toRL and RLHF concepts; PyTorch 2.x for the illustration
Failure Modes
Failure
Example
Specification gaming
Boat circles to collect points instead of finishing a race
Reward model over-optimization
Language model output scores higher on the reward model but humans rate it worse
Goal misgeneralization
Agent trained with coins at the level’s end learns “go right” rather than “get the coin” (CoinRun)
Side effects
Agent breaks things en route because the reward ignores them
Sycophancy
Chat model agrees with users because raters liked agreement
Approaches
Approach
Idea
Preference learning / RLHF
Reward model from pairwise human comparisons
KL-regularized optimization
Stay near a trusted reference policy
Reward model ensembles and uncertainty
Penalize disagreement to resist over-optimization
RLAIF / Constitutional AI
AI feedback guided by written principles
Direct preference optimization (DPO)
Optimize the policy on preferences without an explicit RL loop
Inspect learned goals; red-team for hacking and deception
Code Example: Bradley-Terry Reward Model Loss and KL-Penalized Reward (PyTorch)
import torch
import torch.nn.functional as F
# Reward model scores for (preferred, rejected) responses in a batch of 3 comparisons
r_chosen = torch.tensor([2.0, 0.5, 1.0])
r_rejected = torch.tensor([1.0, 0.5, 2.0])
loss = -F.logsigmoid(r_chosen - r_rejected).mean() # Bradley-Terry preference loss
print(round(loss.item(), 4)) # 0.7732
# RLHF-style reward: reward model score minus a KL penalty to the reference policy
beta = 0.1
rm_score = torch.tensor(3.0)
logp_policy, logp_reference = torch.tensor(-10.0), torch.tensor(-14.0)
shaped = rm_score - beta * (logp_policy - logp_reference) # per-sample KL estimate
print(round(shaped.item(), 2)) # 2.6
By hand: the three logsigmoid terms are log σ(1) ≈ −0.3133, log σ(0) ≈ −0.6931 and log σ(−1) ≈ −1.3133, so the loss is (0.3133 + 0.6931 + 1.3133) / 3 ≈ 0.7732. The third pair, where the reward model prefers the rejected answer, contributes most of the loss.
Common Pitfalls
Treating the reward model as ground truth (it is a learned, exploitable proxy).
Optimizing too long against a fixed reward model.
Assuming good training performance shows the intended goal was learned (test under distribution shift).
Interview Tips
Define the alignment gap with specification gaming and goal misgeneralization examples.
Explain RLHF with the reward model and KL penalty, and its limits.
Mention scalable oversight, interpretability and evaluation.
60
Why do fairness and bias matter in reinforcement learning?
KnownRevise
Fairness matters in reinforcement learning because RL systems allocate opportunities and resources over time — loans, prices, jobs shown, medical treatments, police or inspection resources, content exposure — and an agent that maximizes total reward can systematically serve some groups worse, either by inheriting historical bias from logged data and proxy rewards or by creating new bias through feedback loops. RL adds dynamics that one-shot fairness does not capture: decisions change future states and data (a group that gets fewer loans builds less credit history, so the agent learns even less about it), exploration may be spent unevenly, and short-term fairness constraints can have different long-term effects (Liu et al., 2018, “Delayed Impact of Fair Machine Learning”). Addressing it involves defining fairness for the application (equal exposure, equal outcomes, equal error rates, individual fairness), measuring per-group returns and outcomes over time, adding fairness constraints to the MDP (constrained RL), rewarding long-term group welfare, ensuring adequate exploration for all groups, correcting biased logged data in offline training, and auditing with simulations of long-term effects.
Read full answerHide full answer
Applies toRL design and evaluation; Python/NumPy for metrics
Sources of Bias in RL
Source
Example
Historical logged data
Offline policy learns past discriminatory lending decisions
Proxy rewards
Clicks reward sensational content from certain creators
Feedback loops
Less exposure → fewer interactions → even less exposure
Unequal exploration
Agent stops exploring options for a smaller group early
State representation
Features act as proxies for protected attributes
Simulator bias
Simulated users don’t reflect real populations
Approaches
Approach
How
Constrained RL
Maximize reward s.t. per-group metric gaps ≤ ε
Reward design
Include group welfare or exposure terms
Fair exploration
Guarantee minimum exploration per group/item
Data correction
Reweighting, causal adjustments for offline data
Long-term simulation
Study effects on group states over many steps
Auditing
Per-group returns, error rates, exposure over time
Code Example: Exposure Fairness in a Recommender Slate (NumPy)
import numpy as np
# Exposure each item gets from a ranking: position bias 1 / log2(rank + 1)
positions = np.arange(1, 6)
position_weight = 1 / np.log2(positions + 1)
item_group = np.array(["A", "A", "A", "B", "B"]) # ranking order: group A items on top
exposure = {g: position_weight[item_group == g].sum() for g in ("A", "B")}
share = {g: round(float(e / sum(exposure.values())), 3) for g, e in exposure.items()}
print(share) # {'A': 0.723, 'B': 0.277}
# Interleaved ranking: A, B, A, B, A
interleaved = np.array(["A", "B", "A", "B", "A"])
exposure2 = {g: position_weight[interleaved == g].sum() for g in ("A", "B")}
print({g: round(float(e / sum(exposure2.values())), 3) for g, e in exposure2.items()}) # {'A': 0.64, 'B': 0.36}
By hand: position weights are 1, 0.631, 0.5, 0.431 and 0.387 (sum 2.948); group A in the top three gets 2.131 / 2.948 ≈ 0.723; interleaving gives A (positions 1, 3, 5) 1.887 / 2.948 ≈ 0.64. An RL ranker can be constrained or rewarded so that each group’s exposure tracks its relevance over time rather than compounding early advantages.
Common Pitfalls
Measuring fairness only at one time step.
Removing protected attributes and assuming the policy is fair (proxies remain).
Constraints that look fair short term but harm the group long term.
Ignoring fairness to suppliers/creators as well as users.
Interview Tips
Explain how sequential decisions and feedback loops create bias.
Name fairness definitions and constrained RL as a remedy.
Cite delayed impact and stress long-term, per-group evaluation.
61
What role does reinforcement learning play in NLP?
KnownRevise
Reinforcement learning plays its largest role in NLP today in fine-tuning language models toward goals that are hard to express as supervised labels: Reinforcement Learning from Human Feedback (RLHF) trains a reward model on human preference comparisons and then optimizes the language model against it (InstructGPT, ChatGPT, Claude), and RL with verifiable rewards trains models on maths and coding tasks where answers can be checked automatically, which has driven recent reasoning models (for example DeepSeek-R1 with the GRPO algorithm). In these setups the language model is the policy, the prompt and text generated so far are the state, each token (or the full response) is the action, and the reward comes at the end. RL is also used to optimize non-differentiable sequence metrics such as BLEU or ROUGE (self-critical sequence training for summarization and captioning), for task-oriented dialogue policies, for learning when to call tools or retrieve documents, and in text-based games. Because RL fine-tuning can exploit a learned reward, it is paired with a KL penalty to a reference model, and simpler preference methods without an RL loop, such as DPO, are now widely used alongside it.
Read full answerHide full answer
Applies toPyTorch 2.x; Hugging Face TRL (PPO, DPO, GRPO trainers)
Uses of RL in NLP
Use
Reward
Method
RLHF for assistants
Reward model from human comparisons
PPO with KL penalty; DPO as RL-free alternative
Reasoning (maths, code)
Verifiable correctness, unit tests
GRPO, PPO
Summarization / translation
ROUGE, BLEU, learned quality metrics, human preferences
Self-critical sequence training, RLHF
Dialogue systems
Task success, user satisfaction
Policy learning over dialogue acts
Tool use and retrieval
Answer quality, cost
RL over tool calls
Text games / agents
Game score
DQN-style or LLM agents
How the RLHF Loop Works
Step
Detail
1. Supervised fine-tuning
Train on demonstrations of good responses
2. Reward model
Pairwise comparisons → Bradley-Terry loss
3. RL fine-tuning
Maximize RM score − β·KL(π ‖ π_ref) with PPO or GRPO
4. Evaluation
Human and automatic evaluations, safety tests
Code Example: Group-Relative Advantages (GRPO-Style) and KL-Penalized Reward
import numpy as np
# Four sampled answers to the same maths prompt, scored by a verifier (1 = correct)
rewards = np.array([1.0, 0.0, 1.0, 0.0])
advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-8) # compare answers within the group
print(np.round(advantages, 3).tolist()) # [1.0, -1.0, 1.0, -1.0]
# PPO-style RLHF reward for one response: reward-model score minus a KL penalty to the reference model
beta = 0.05
rm_score = 1.2
logp_policy = np.array([-1.0, -0.5, -2.0]) # per-token log-probs of the sampled response
logp_ref = np.array([-1.5, -0.7, -2.1])
kl_estimate = (logp_policy - logp_ref).sum() # 0.5 + 0.2 + 0.1 = 0.8
print(round(rm_score - beta * kl_estimate, 3)) # 1.16
GRPO removes the critic network by using the group mean as a baseline: correct answers get positive advantages and incorrect ones negative, and the policy gradient raises the probability of the tokens in better answers. The KL term keeps the model close to its starting point so it does not drift into text that fools the reward model.
Challenges
Challenge
Mitigation
Reward hacking (verbose, sycophantic outputs)
KL penalty, reward model ensembles, length control, better data
Huge action space (vocabulary)
Start from a pretrained policy; RL as fine-tuning only
Sparse, sequence-level reward
Per-token KL shaping, value models or group baselines
Cost of human feedback
AI feedback (RLAIF), verifiable rewards
Instability
Small learning rates, clipping, careful monitoring
Common Pitfalls
Optimizing against a reward model for too long (over-optimization).
Forgetting the KL penalty or reference model.
Using automatic metrics like BLEU as the only reward; models learn to game them.
Interview Tips
Explain RLHF step by step and the role of the KL penalty.
Mention verifiable rewards and GRPO for reasoning, and DPO as an RL-free alternative.
Map NLP generation onto states, actions and rewards.
62
How is reinforcement learning used to improve data centre energy efficiency?
KnownRevise
Data centres spend a large share of their energy on cooling, and RL is used to control cooling equipment — chillers, cooling towers, pumps, fans, water and air setpoints — in response to IT load, weather and equipment state, minimizing energy use (or PUE, power usage effectiveness) while keeping temperatures and other conditions within safe limits. The best-known case is Google and DeepMind: in 2016 they reported that a machine-learning system recommending cooling settings cut cooling energy by up to 40%, and in 2018 they moved to an autonomous system that directly adjusts the plant under safety constraints, reporting average savings of about 30% over its first months. Their approach used neural networks trained on sensor history to predict future energy use and temperatures, choosing actions that minimize predicted energy subject to constraints, with multiple safety layers: actions verified against constraints, confidence checks, a local control system that can override, and operators able to take over at any time. Other uses include workload scheduling and placement (shifting batch jobs to cooler times or greener power), server power management (DVFS, sleep states) and demand response with the grid.
Read full answerHide full answer
Applies toRL for building/HVAC control; Gymnasium-style simulators (e.g. Sinergym with EnergyPlus)
Problem Formulation
Element
Cooling control
State
IT load, outside temperature and humidity, supply/return temperatures, pump and fan speeds, chiller status, time
Actions
Setpoints: chilled water temperature, cooling tower fan speed, number of chillers, air handler setpoints
Reward
−energy (or −PUE) − penalty for temperature/humidity violations
Constraints
Server inlet temperature limits, equipment operating ranges, rate of change limits
Control interval
Minutes
Safety Architecture
Layer
Purpose
Action constraint checks
Reject unsafe setpoints before sending
Uncertainty/confidence estimates
Do nothing if the model is unsure
Rate limits
Change setpoints gradually
Local control fallback
Existing building management system takes over on failure
Human override
Operators can switch to manual
Continuous monitoring
Compare with baseline performance
Code Example: Reward with Comfort Constraints and a Safe Setpoint Filter
import numpy as np
def cooling_reward(energy_kwh, inlet_temps_c, limit_c=27.0, penalty=10.0):
violation = np.clip(np.asarray(inlet_temps_c) - limit_c, 0, None).sum()
return -energy_kwh - penalty * violation
def safe_setpoint(proposed_c, current_c, low=6.0, high=14.0, max_step=0.5):
"""Chilled-water setpoint: keep within equipment range and change at most max_step per interval."""
bounded = min(max(proposed_c, low), high)
return float(np.clip(bounded, current_c - max_step, current_c + max_step))
print(round(cooling_reward(120.0, [24.5, 26.0, 27.4]), 6)) # -124.0 (0.4 C over the limit costs 4)
print(safe_setpoint(16.0, current_c=10.0)) # 10.5
print(safe_setpoint(9.8, current_c=10.0)) # 9.8
By hand: one rack inlet is 0.4 °C over the 27 °C limit, so the penalty is 10 × 0.4 = 4 and the reward is −124. The filter turns a large proposed jump to 16 °C into a 0.5 °C step.
Lessons from Practice
Lesson
Detail
Start with recommendations
Operators apply suggestions before autonomy
Use models trained on historical sensor data
Offline/model-based learning before live control
Constrain hard
Safety limits enforced outside the learned model
Measure carefully
Normalize savings for weather and load
Common Pitfalls
Optimizing energy while letting server inlet temperatures drift.
Evaluating savings without adjusting for weather and IT load changes.
Exploring live with large setpoint changes.
Neglecting sensor failures and drift in the inputs.
Interview Tips
Describe the state, actions, reward and constraints of cooling control.
Cite the DeepMind/Google results as reported figures, with their safety layers.
Mention workload scheduling and grid demand response as other uses.
63
What are emerging trends in reinforcement learning in fintech?
KnownRevise
Reinforcement learning in financial technology is moving from academic trading experiments toward narrower, well-defined decision problems where simulators and rewards are clearer: optimal trade execution (splitting large orders to minimize market impact), market making (setting bid and ask quotes while controlling inventory risk), deep hedging of derivatives under transaction costs (Buehler et al., 2019, which learns hedging strategies with neural networks in realistic markets), and dynamic portfolio allocation and rebalancing with risk constraints. Other trends include contextual bandits and RL for personalization in digital banking (offers, credit-limit adjustments, collections strategies), offline RL that learns from historical decisions without live experimentation, risk-sensitive and constrained RL (CVaR objectives, drawdown limits), multi-agent market simulators (agent-based models such as ABIDES) for testing strategies and studying market effects, and LLM-based agents that combine language models with RL or tool use for research and operations. Throughout, the field is constrained by non-stationary, low signal-to-noise data, the risk of overfitting backtests, model-risk management and regulatory requirements for explainability and fairness.
Read full answerHide full answer
Applies toRL applications in finance (concepts); Python simulation tools
Trends
Area
RL formulation
Why RL fits
Optimal execution
Choose how much to trade each interval
Clear objective (implementation shortfall), simulators of the order book
Market making
Quote prices and sizes
Sequential inventory management
Deep hedging
Hedge positions given costs and frictions
Model-free, handles realistic frictions
Portfolio management
Rebalancing weights
Multi-period with costs and risk constraints
Personalization
Offers, credit lines, collections
Bandits/RL with long-term customer value
Fraud and risk ops
Alert prioritization, investigation allocation
Resource allocation over time
Market simulation
Multi-agent simulators
Test strategies, study market impact
Offline and risk-sensitive RL
Learn from logs with CVaR/drawdown constraints
Safer than live exploration
Code Example: Execution Cost of a Schedule (NumPy)
With quadratic impact, splitting the order evenly costs one fifth as much (5 × 0.1 × 200² = 20,000 vs 0.1 × 1,000² = 100,000). Real execution trades this impact against the risk of prices moving while you wait (the Almgren-Chriss trade-off), and RL agents learn schedules that also react to live order-book conditions.
Challenges
Challenge
Detail
Non-stationarity
Market regimes change
Overfitting backtests
Many strategies tried on one history
Simulator realism
Market impact and other participants’ reactions
Regulation and model risk
Explainability, validation, audit trails
Fairness
Credit and pricing decisions affect customers
Common Pitfalls
Presenting backtest profits without costs, slippage and walk-forward validation.
Assuming RL beats classical methods; compare with Almgren-Chriss, simple heuristics and MPC.
Ignoring regulatory requirements for decisions affecting customers.
Interview Tips
Name execution, market making, hedging and portfolio management as the credible areas.
Mention offline, risk-sensitive RL and market simulators.
Be candid about non-stationarity and backtest overfitting.
64
What are the challenges of deploying RL models in production?
KnownRevise
Deploying reinforcement learning is harder than deploying supervised models because an RL policy acts, changes the data it will later see, and was usually trained in a simulator or on logged data that differs from reality. The main challenges: offline evaluation (you must estimate how a new policy performs before it acts, using simulators or off-policy evaluation, both of which can be wrong); simulation-to-real and distribution shift (dynamics, users or markets differ from training and change over time); safety and constraints (exploration and mistakes have real costs); reward mis-specification (production metrics reveal that the proxy reward rewarded the wrong behaviour); delayed and noisy feedback (long-term outcomes such as retention take weeks to observe); non-reproducibility and instability (results vary by seed, and retraining can produce a very different policy); online learning risks (continual updates can degrade quickly); latency and infrastructure (real-time inference, logging states, actions and propensities); and explainability and governance. Successful deployments use conservative policies, safety layers and fallbacks, shadow mode and gradual rollout with A/B tests, strong monitoring, and logged propensities for future off-policy evaluation.
Bounded ε, exploration budget, randomized but logged
Logging
Store state, action, action probability, reward, model version
Latency
Small networks, batching, distilled policies
Code Example: Production Action Logging and a Guarded Policy Call
import json
import time
from dataclasses import asdict, dataclass
import numpy as np
@dataclass
class DecisionLog:
request_id: str
model_version: str
state: list
action: int
propensity: float # probability of the chosen action: needed for off-policy evaluation
fallback_used: bool
ts: float
def decide(policy_probs_fn, state, rng, fallback_action=0, allowed=None, model_version="v1"):
try:
probs = np.asarray(policy_probs_fn(state), dtype=float)
if allowed is not None:
probs = probs * allowed
if not np.isfinite(probs).all() or probs.sum() <= 0:
raise ValueError("invalid policy output")
probs = probs / probs.sum()
action = int(rng.choice(len(probs), p=probs))
log = DecisionLog("req-1", model_version, list(map(float, state)), action, float(probs[action]), False, time.time())
except Exception:
log = DecisionLog("req-1", model_version, list(map(float, state)), fallback_action, 1.0, True, time.time())
return log.action, json.dumps(asdict(log))
rng = np.random.default_rng(42)
action, record = decide(lambda s: [0.2, 0.8, 0.0], [0.5, 1.0], rng, allowed=np.array([1, 1, 0]))
print(action in (0, 1), json.loads(record)["fallback_used"]) # True False
action, record = decide(lambda s: [np.nan, 1.0], [0.5, 1.0], rng)
print(action, json.loads(record)["fallback_used"]) # 0 True
Every decision logs the action’s probability and the model version, so later policies can be evaluated off-policy from production data, and any invalid model output falls back to a known safe action.
Rollout Stages
Stage
Exposure
Offline evaluation
None
Shadow mode
Policy computes actions, the old system acts
Canary / small A/B
1-5% of traffic with guardrails
Gradual ramp
Increase if metrics hold
Full deployment + monitoring
Champion-challenger for future versions
Common Pitfalls
Not logging propensities, which blocks off-policy evaluation later.
Retraining on data generated by the policy itself without correcting for its bias.
No fallback when the model service fails.
Judging success on short-term metrics only.
Interview Tips
Contrast RL deployment with supervised deployment (actions change the data).
Cover offline evaluation, safety, shift and monitoring.
Describe a staged rollout with shadow mode and A/B tests.
65
What are common pitfalls when scaling reinforcement learning applications?
KnownRevise
Scaling RL — to bigger environments, more agents, more compute or production traffic — fails in recurring ways: the simulator becomes the bottleneck, so expensive GPUs sit idle waiting for environment steps; distributed actors generate stale, off-policy data that destabilizes on-policy algorithms; hyperparameters tuned at small scale (learning rate, batch size, entropy) stop working at large batch sizes; reward hacking and simulator exploits appear once the agent is strong enough to find them; results are dominated by seed variance, so small-scale wins do not replicate; and engineering problems (non-determinism, memory blow-ups from replay buffers, bugs in vectorized resets, logging overload) multiply. Other pitfalls are treating truncation as termination at scale, normalization statistics that differ between actors, evaluating only on training environments, and scaling compute instead of fixing a weak reward or representation. Remedies include profiling throughput (steps per second at each stage), GPU-accelerated or batched simulators, architectures designed for asynchrony (IMPALA with V-trace, Ape-X, SEED RL), careful scaling of batch size and learning rate, many seeds with robust statistics, and strong evaluation and monitoring.
Read full answerHide full answer
Applies toDistributed RL systems (RLlib, Sample Factory, EnvPool, Isaac Lab, JAX environments)
Pitfalls and Remedies
Pitfall
Symptom
Remedy
Simulator bottleneck
Low GPU utilization
Vectorized/GPU simulators (EnvPool, Isaac Lab, Brax), more CPU actors
Stale data in distributed actors
Instability, worse returns
Off-policy corrections (V-trace), sync more often, bound staleness
Hyperparameters don’t transfer
Collapse at large batch
Retune learning rate/batch; LR warm-up; normalize advantages
Reward hacking emerges
Return rises, task fails
Behavioural evaluation, reward audits
Seed variance
Unreproducible wins
≥ 5 seeds, IQM and confidence intervals (rliable)
Replay buffer memory
Out-of-memory crashes
Compressed storage, frame dedup, sharded buffers
Vectorized reset bugs
Wrong bootstrapping
Unit tests for done/truncation handling
Observation normalization drift
Policy works in training, fails in eval
Share/freeze normalization statistics, save with the model
import time
import gymnasium as gym
def steps_per_second(num_envs, mode, steps=2_000):
envs = gym.make_vec("CartPole-v1", num_envs=num_envs, vectorization_mode=mode)
envs.reset(seed=42)
start = time.perf_counter()
for _ in range(steps):
envs.step(envs.action_space.sample())
elapsed = time.perf_counter() - start
envs.close()
return steps * num_envs / elapsed
if __name__ == "__main__": # needed for the "async" mode's subprocesses
for n in (1, 8):
print(n, "envs sync:", int(steps_per_second(n, "sync")))
print(8, "envs async:", int(steps_per_second(8, "async")))
Measure before scaling: for cheap environments such as CartPole, synchronous vectorization in one process is often faster than subprocesses (inter-process overhead dominates), while expensive simulators benefit from async workers or GPU simulation. Numbers depend on the machine.
Scaling Checklist
Check
Why
Steps/s per component (env, inference, learner)
Find the bottleneck
Learner throughput vs data generation
Avoid stale or wasted data
Hyperparameters retuned at target scale
Small-scale settings can fail
Evaluation on held-out environments, many seeds
Real gains, not noise
Reproducible configs and versioned environments
Debuggable results
Common Pitfalls (Summary)
Scaling an algorithm that is still buggy at small scale.
Comparing methods at different compute budgets.
Ignoring the cost of inference latency in actors.
Interview Tips
Name throughput bottlenecks, stale data, hyperparameter transfer and seed variance.
Mention IMPALA/V-trace and GPU simulators.
Stress profiling and robust evaluation.
66
How do you monitor and manage a deployed reinforcement learning system?
KnownRevise
Monitoring a deployed RL system means watching four things continuously: the business or task outcome the policy is meant to improve (and guardrail metrics it must not harm), the policy’s behaviour (action distribution, entropy, how often the fallback or safety layer intervenes), the inputs (state distribution shift relative to training data, missing or broken sensors/features), and the learning signal (reward distribution, delayed outcomes, value estimates versus realized returns). Management means having the controls to act on what monitoring shows: versioned policies with a champion-challenger setup, safe rollback to a previous policy or rule-based fallback, kill switches, retraining pipelines with off-policy evaluation gates, bounded and logged exploration, alerting with clear owners, and periodic audits for reward hacking and fairness. Because RL policies influence the data they receive, also watch for feedback loops and evaluate new versions against logged propensities before rollout.
Read full answerHide full answer
Applies toRL MLOps; Python/NumPy for drift metrics
Feature drift (PSI, KS), missing values, sensor health
PSI > 0.2 on a key feature
Learning signal
Reward distribution, delayed outcome rates
Mean reward shifts 3σ
Value calibration
Predicted value vs realized return
Systematic overestimation
System
Latency, errors, throughput
p99 latency over budget
Fairness
Per-group outcomes and exposure
Gap exceeds threshold
Code Example: Action Distribution Drift (Population Stability Index)
import numpy as np
def psi(expected_counts, actual_counts, eps=1e-6):
e = np.asarray(expected_counts, dtype=float)
a = np.asarray(actual_counts, dtype=float)
e, a = e / e.sum() + eps, a / a.sum() + eps
return float(np.sum((a - e) * np.log(a / e)))
training_actions = [500, 300, 200] # action counts during validation of the policy
this_week = [480, 310, 210]
after_incident = [900, 50, 50]
print(round(psi(training_actions, this_week), 4)) # 0.0016 stable
print(round(psi(training_actions, after_incident), 3)) # 0.891 large shift: investigate
By hand for the stable week: shares move from (0.5, 0.3, 0.2) to (0.48, 0.31, 0.21), giving (−0.02)·ln(0.96) + 0.01·ln(1.0333) + 0.01·ln(1.05) ≈ 0.00082 + 0.00033 + 0.00049 ≈ 0.0016. A common rule of thumb treats PSI above 0.2 as a significant shift.
Management Practices
Practice
Purpose
Model registry and versioning
Know which policy acted when
Champion-challenger / A/B
Compare new policies safely
Off-policy evaluation gate before rollout
Catch regressions early
Rollback and fallback policy
Recover quickly
Retraining schedule + triggers
Handle drift
Incident runbooks and owners
Respond consistently
Periodic behaviour audits
Detect reward hacking, unfairness
Common Pitfalls
Monitoring only average reward and missing behaviour changes.
Retraining automatically on drifted data without evaluation gates.
No record of action propensities.
Alert thresholds without owners or runbooks.
Interview Tips
Organize monitoring into outcomes, behaviour, inputs and learning signal.
Give one concrete drift metric and alert.
Describe rollback, champion-challenger and retraining gates.
67
Discuss a recent RL research paper and its implications.
KnownRevise
A recent paper with broad implications is “Mastering Diverse Domains through World Models” by Danijar Hafner, Jurgis Pasukonis, Jimmy Ba and Timothy Lillicrap, which introduced DreamerV3 (preprint 2023, published in Nature in 2025). DreamerV3 is a model-based agent that learns a world model — a recurrent state-space model that encodes observations into compact latent states and predicts future latents, rewards and episode ends — and then trains an actor and critic entirely on imagined trajectories in that latent space. Its main contribution is robustness: with one fixed set of hyperparameters it performed strongly across more than 150 tasks spanning Atari, continuous control, DMLab, procedurally generated games and Minecraft, and it was reported as the first algorithm to collect diamonds in Minecraft from scratch, without human data or curricula. The techniques that made this possible include symlog transformations of inputs and rewards to handle very different scales, two-hot encoded discrete regression for rewards and values, percentile-based return normalization, KL balancing with free bits in the world model, and an architecture that improves predictably with model size. Implications: RL practitioners have historically retuned hyperparameters for each domain; DreamerV3 suggests general-purpose agents with fixed settings are achievable, that model-based RL can be both sample-efficient and general, and that world models are a promising path to agents that plan and learn from imagination, with open questions about compute cost, long-horizon prediction errors and real-world transfer.
Read full answerHide full answer
Applies toModel-based RL concepts; NumPy for the illustration
Rewards spanning six orders of magnitude become values between about −7 and 7 while remaining exactly invertible, so the same network and loss settings work whether a game gives rewards of 1 or 1,000: one reason a single hyperparameter set could cover so many domains.
Implications and Open Questions
Implication
Open question
Less per-task tuning
Compute cost of world-model training
Model-based RL as a general method
Long-horizon prediction errors
Imagination-based planning
Transfer to real robots with noisy sensors
Scaling laws in RL
How far scaling goes
Common Pitfalls (When Discussing a Paper)
Restating the abstract without the technical idea that made it work.
Overstating results (check exactly what was reported and under which conditions).
Ignoring limitations and follow-up work.
Interview Tips
Pick a paper you can explain in depth: problem, idea, results, limitations.
Connect it to practical implications for your work.
Be accurate about venues, dates and claims.
68
Explain a new RL-related technique from a major conference like NeurIPS or ICML.
KnownRevise
A good example is Direct Preference Optimization (DPO), presented at NeurIPS 2023 by Rafailov, Sharma, Mitchell, Ermon, Manning and Finn (“Direct Preference Optimization: Your Language Model is Secretly a Reward Model”), which received an outstanding paper runner-up award. It solves the same problem as RLHF — fine-tuning a model to match human preferences under a KL constraint to a reference model — without training a separate reward model or running a reinforcement learning loop. The key insight is that the KL-constrained RL objective has a closed-form optimal policy, π*(y|x) ∝ π_ref(y|x) exp(r(x, y)/β); rearranging gives the reward in terms of the policy, r(x, y) = β log(π(y|x)/π_ref(y|x)) + β log Z(x). Substituting that into the Bradley-Terry preference model makes the partition function Z(x) cancel for a pair of responses, leaving a simple classification-style loss on preferred (y_w) and rejected (y_l) responses: L = −log σ(β [log π(y_w|x)/π_ref(y_w|x) − log π(y_l|x)/π_ref(y_l|x)]). It is more stable and much cheaper than PPO-based RLHF and became widely used, along with follow-up variants (IPO, KTO, SimPO), although online RL methods remain competitive or better in some settings.
Read full answerHide full answer
Applies toPyTorch 2.x; Hugging Face TRL `DPOTrainer`
Comparison with PPO-Based RLHF
Aspect
RLHF with PPO
DPO
Reward model
Trained separately
Implicit in the policy
Sampling during training
Yes (on-policy generations)
No (fixed preference pairs)
Models in memory
Policy, reference, reward, value
Policy, reference
Stability
Sensitive to hyperparameters
Stable, supervised-style training
Exploration / online data
Yes
Offline (online variants exist)
Code Example: The DPO Loss (PyTorch)
import torch
import torch.nn.functional as F
def dpo_loss(logp_chosen, logp_rejected, ref_logp_chosen, ref_logp_rejected, beta=0.1):
"""Inputs: summed log-probs of full responses under the policy and the frozen reference."""
chosen_ratio = logp_chosen - ref_logp_chosen
rejected_ratio = logp_rejected - ref_logp_rejected
logits = beta * (chosen_ratio - rejected_ratio)
return -F.logsigmoid(logits).mean(), logits
# Policy equals the reference: logits 0, loss log 2
loss, logits = dpo_loss(torch.tensor([-20.0]), torch.tensor([-22.0]), torch.tensor([-20.0]), torch.tensor([-22.0]))
print(round(loss.item(), 4), logits.item()) # 0.6931 0.0
# Policy raised the chosen response by 5 nats and lowered the rejected one by 5 nats: logits = 0.1 * 10 = 1
loss, logits = dpo_loss(torch.tensor([-15.0]), torch.tensor([-27.0]), torch.tensor([-20.0]), torch.tensor([-22.0]))
print(round(loss.item(), 4), round(logits.item(), 4)) # 0.3133 1.0
By hand: with logits 0 the loss is −log σ(0) = log 2 ≈ 0.6931; with logits 1 it is −log σ(1) = log(1 + e⁻¹) ≈ 0.3133. The gradient raises the policy’s relative likelihood of preferred responses and lowers it for rejected ones, weighted more on pairs the model currently gets wrong.
Limitations
Limitation
Notes
Offline data only
Distribution shift from the data-generating model
Can over-optimize
Likelihood of both responses may drop; regularized variants exist
Sensitive to preference data quality
Noisy labels hurt directly
Common Pitfalls
Forgetting the reference model (the loss needs both ratios).
Using per-token averages where summed log-probs are expected, which changes the effective β.
Expecting DPO to explore: it only learns from the pairs given.
Interview Tips
State the problem (RLHF), the key derivation idea and the final loss.
Compare with PPO-based RLHF on cost and stability.
Mention limitations and follow-up methods.
69
How is adversarial robustness being addressed in current RL research?
KnownRevise
Adversarial robustness in RL addresses agents whose behaviour can be broken by small, deliberate changes — perturbed observations, manipulated rewards or poisoned training data, shifted dynamics, or adversarial opponents in multi-agent settings — and research has shown each of these is a real threat: tiny pixel perturbations can make Atari policies fail (Huang et al., 2017), “adversarial policies” that act strangely can defeat strong agents in multi-agent games without being good players (Gleave et al., 2020), and adversarial strategies have beaten superhuman Go programs such as KataGo (Wang et al., 2023). Defences being studied include: adversarial training against attacked observations; the state-adversarial MDP (SA-MDP) framework with policy-smoothness regularizers (Zhang et al., 2020) and alternating training against a learned adversary (ATLA); robust adversarial RL (RARL), which trains the protagonist against an adversary that applies disturbance forces (Pinto et al., 2017); robust MDPs that optimize for the worst case over an uncertainty set of dynamics; certified robustness methods that give provable bounds on actions or returns under bounded perturbations (randomized smoothing, interval bound propagation); population and league training so agents face diverse opponents; and defences against reward and data poisoning for offline and RLHF settings.
League/population training, diverse opponents, exploitability testing
Dynamics shift / disturbances
Wind, payload changes
RARL, robust MDPs, domain randomization
Reward/data poisoning
Corrupted logged rewards, poisoned preferences
Robust statistics, data auditing, anomaly detection
Backdoors (trojans)
Trigger pattern causes harmful action
Data provenance, trigger detection
Code Example: FGSM Attack on a Policy and a Smoothness Regularizer (PyTorch)
import torch
from torch import nn
torch.manual_seed(42)
policy = nn.Sequential(nn.Linear(4, 64), nn.Tanh(), nn.Linear(64, 2))
obs = torch.randn(16, 4)
def fgsm_attack(policy, obs, eps=0.05):
"""Perturb observations to reduce the probability of the action the policy currently prefers."""
obs_adv = obs.clone().requires_grad_(True)
logits = policy(obs_adv)
preferred = logits.argmax(dim=1)
loss = nn.functional.cross_entropy(logits, preferred) # increase this loss = move away from preferred action
loss.backward()
return (obs + eps * obs_adv.grad.sign()).detach()
obs_adv = fgsm_attack(policy, obs)
print(bool(((obs_adv - obs).abs() <= 0.05 + 1e-6).all())) # True: perturbation stays within eps
with torch.no_grad():
flipped = (policy(obs).argmax(1) != policy(obs_adv).argmax(1)).float().mean()
print(0.0 <= flipped.item() <= 1.0) # True (fraction of actions changed)
def smoothness_penalty(policy, obs, obs_adv):
"""SA-MDP-style regularizer: KL between action distributions on clean and perturbed states."""
p_clean = torch.log_softmax(policy(obs), dim=1)
p_adv = torch.log_softmax(policy(obs_adv), dim=1)
return nn.functional.kl_div(p_adv, p_clean, log_target=True, reduction="batchmean")
penalty = smoothness_penalty(policy, obs, obs_adv)
print(penalty.item() >= 0) # True
Adding this penalty (times a coefficient) to the usual RL loss encourages the policy to choose similar actions for nearby observations, which makes small perturbations less effective. The fraction of flipped actions for an untrained policy depends on the random weights; for trained agents, evaluations report return under attack at several budgets ε.
Evaluating Robustness
Evaluation
Report
Return vs perturbation budget ε
Curve, not one number
Against learned adversaries
Stronger than fixed attacks
Exploitability in games
Best-response win rate
Clean performance
Robustness should not destroy normal returns
Common Pitfalls
Testing only against weak, fixed attacks.
Reporting robustness while clean performance collapsed.
Assuming robustness to one perturbation type transfers to others.
Interview Tips
List threat types: observations, opponents, dynamics, rewards/data.
Name defences: adversarial training, SA-MDP, RARL, robust MDPs, certification.
Cite a concrete attack result, such as adversarial policies against KataGo.
70
Describe an end-to-end pipeline for training, validating and deploying RL commercially.
KnownRevise
An end-to-end commercial RL pipeline turns a business decision problem into a monitored, continuously improving policy through these stages: problem framing (decision, objective, constraints, and whether a simpler bandit or optimization method suffices), data and environment (logged interaction data with propensities, and a simulator calibrated to it), offline training and evaluation (algorithm selection, hyperparameter search over seeds, off-policy evaluation and simulator tests against strong baselines), pre-deployment validation (safety constraints, stress tests, fairness and robustness checks, review with stakeholders), staged deployment (shadow mode, canary A/B test, gradual ramp with guardrails and fallbacks), and operations (monitoring, drift detection, logging, retraining with evaluation gates, rollback). Every stage is versioned and reproducible — environment, data, code, configs, seeds and model artefacts — and the decision log (state, action, action probability, reward, model version) is designed in from the start because it powers later evaluation and retraining.
Read full answerHide full answer
Applies toGymnasium 1.x, Stable-Baselines3, MLflow (or similar) for tracking; any serving stack
Code Example: Train, Evaluate Against a Baseline, and Gate Promotion (Stable-Baselines3)
import gymnasium as gym
import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy
ENV_ID, SEEDS, EVAL_EPISODES = "CartPole-v1", [0, 1, 2], 20
def baseline_return(episodes=EVAL_EPISODES):
env, returns = gym.make(ENV_ID), []
for i in range(episodes):
obs, _ = env.reset(seed=10_000 + i) # held-out evaluation seeds
done, total = False, 0.0
while not done:
action = int(obs[2] + 0.5 * obs[3] > 0) # simple rule-based champion
obs, r, terminated, truncated, _ = env.step(action)
total += r
done = terminated or truncated
returns.append(total)
return float(np.mean(returns))
candidates = []
for seed in SEEDS:
model = PPO("MlpPolicy", gym.make(ENV_ID), seed=seed, verbose=0)
model.learn(total_timesteps=50_000)
eval_env = gym.make(ENV_ID)
eval_env.reset(seed=10_000)
mean_r, std_r = evaluate_policy(model, eval_env, n_eval_episodes=EVAL_EPISODES, deterministic=True)
candidates.append((mean_r, std_r, seed, model))
champion = baseline_return()
best_mean, best_std, best_seed, best_model = max(candidates, key=lambda c: c[0])
all_seeds_ok = all(c[0] >= champion for c in candidates) # robust across seeds, not one lucky run
promote = all_seeds_ok and best_mean > champion
print(f"champion {champion:.1f}, candidates {[round(c[0], 1) for c in candidates]}, promote={promote}")
if promote:
best_model.save(f"ppo_cartpole_seed{best_seed}") # then register the version with its metrics
The gate requires every training seed to match the current champion and the best to exceed it on held-out seeds; in a real system the same gate also checks off-policy estimates on production logs, guardrail metrics and safety tests before the policy reaches shadow mode.
Engineering Practices
Practice
Why
Version everything (env, data, code, config, seeds)
Reproducibility and audits
Same feature code for training and serving
Avoid training-serving skew
Experiment tracking (MLflow/W&B)
Compare runs and seeds
Automated tests for environment logic
Bugs in rewards and resets are common
Decision logging with propensities
Enables OPE and retraining
Fallback policy and kill switch
Safe failure
Common Pitfalls
Skipping the simple baseline (rules, bandits, MPC) that the RL policy must beat.
Trusting a simulator that was never validated against real logs.
Promoting the best single seed.
Retraining on policy-generated data without correcting for its bias.
Interview Tips
Walk through framing → data/simulator → training → offline evaluation → safety → staged rollout → operations.
Emphasize gates between stages and logging of propensities.
Mention baselines, multi-seed evaluation and rollback.