Skip to content
Machine Learning & Data Science

RNN interview questions and answers

47 questions, 47 with code. About 3 h 45 min to read them all.

By TestLab · Updated

Plan for an interview
47 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What are RNNs, and how do they differ from feedforward networks?

A Recurrent Neural Network (RNN) is a neural network for sequential data that processes inputs one step at a time and keeps a hidden state — a memory vector updated at each step from the current input and the previous state (h_t = tanh(W_hh h_(t−1) + W_xh x_t + b)) — using the same weights at every step. A feedforward network (MLP, and in this sense CNNs too) maps a fixed-size input to an output in a single pass with no loops and no memory: each input is processed independently. The key differences are that RNNs have a cycle (recurrence), share weights across time, handle variable-length sequences, and take order into account, while feedforward networks need fixed-size inputs and treat each example on its own. RNNs are trained with backpropagation through time, which brings vanishing and exploding gradient issues, so in practice the gated variants (LSTM, GRU) are used.

Read full answer

Applies toPyTorch 2.x (`nn.RNN`, `nn.Linear`)

Comparison

Aspect Feedforward network RNN
Connections Input → hidden → output, no cycles Hidden state feeds back to itself
Memory None between inputs Hidden state carries context
Input size Fixed Variable-length sequences
Order sensitivity Only through fixed input positions Built in
Weight sharing Separate weights per layer Same weights at every time step
Training Standard backprop Backpropagation through time
Gradient issues Mostly with depth Also over sequence length
Parallelism Fully parallel over inputs Sequential over time steps
Typical uses Tabular data, fixed-size images (CNNs) Text, speech, time series, event streams

Code Example: Same Data, Two Models (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

batch, T, F, H = 4, 6, 3, 8

# Feedforward: must flatten a fixed-length sequence into one vector
mlp = nn.Sequential(nn.Flatten(), nn.Linear(T * F, H), nn.ReLU(), nn.Linear(H, 1))
x = torch.randn(batch, T, F)
print(mlp(x).shape)                                  # torch.Size([4, 1])

try:
    mlp(torch.randn(batch, T + 2, F))                # longer sequence
except RuntimeError:
    print("MLP fails on a different length")         # weights are tied to input size T * F


class RNNModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.rnn = nn.RNN(F, H, batch_first=True)    # weights depend only on F and H
        self.head = nn.Linear(H, 1)

    def forward(self, x):
        _, h_n = self.rnn(x)
        return self.head(h_n[-1])


rnn_model = RNNModel()
print(rnn_model(x).shape, rnn_model(torch.randn(batch, T + 2, F)).shape)
# torch.Size([4, 1]) torch.Size([4, 1])

n_mlp = sum(p.numel() for p in mlp.parameters())
n_rnn = sum(p.numel() for p in rnn_model.parameters())
print(n_mlp, n_rnn)                                  # 161 113

The MLP’s first layer has T·F inputs, so its size is tied to the sequence length (18·8 + 8 + 8 + 1 = 161 parameters). The RNN’s parameters (3·8 + 8·8 + 8 + 8 = 104 in the cell, plus 9 in the head = 113) do not depend on T, so it accepts any length.

The Recurrence Unrolled

An RNN over T steps is equivalent to a T-layer feedforward network whose layers share weights and each receive one input x_t. That view explains both its strength (one set of weights for any length) and its training difficulty (a very deep network with repeated multiplication by the same matrix).

Common Pitfalls

  • Saying feedforward networks “cannot” handle sequences; with fixed windows (or CNNs/transformers) they can, but without a recurrent state.

  • Forgetting that vanilla RNNs struggle with long-range dependencies; mention LSTM/GRU.

  • Claiming RNNs are the default for sequences today; transformers dominate large-scale NLP.

Interview Tips

  • Define the hidden-state update and weight sharing in one sentence each.

  • Contrast memory, variable length and order handling with feedforward networks.

  • Mention BPTT, vanishing gradients, LSTM/GRU and where transformers fit.

2

What does a time step mean in an RNN?

A time step is one position in the input sequence and one application of the RNN cell: at step t the network reads input x_t, combines it with the hidden state h_(t−1) from the previous step, produces the new state h_t and optionally an output ŷ_t. “Time” is just the order index: it can be a moment in a time series (one hourly reading), one token in a sentence, one audio frame, one video frame or one event in a user session. The number of time steps is the sequence length T, which appears as the second dimension of the input tensor (batch, time steps, features). During training the network is unrolled over all T steps, which share the same weights, and backpropagation through time sends gradients back across them. Choosing the time-step granularity (characters vs words, 10 ms vs 1 s frames, hourly vs daily) and the window length T is a key modelling decision.

Read full answer

Applies toPyTorch 2.x (`batch_first=True`) and Keras 3 input shapes

What a Time Step Is in Different Data

Data One time step Features per step Typical T
Hourly energy load One hour Load, temperature, hour, holiday flag 168 (one week)
Sentence (words) One token Embedding vector 20-200
Characters One character Embedding or one-hot 100-1,000
Speech One 10 ms frame 40-80 mel filterbank values 100 per second
Video One frame CNN feature vector 16-64
Clickstream One event Item embedding + event type + time gap Session length

Code Example: Shapes and Windows

import numpy as np
import torch
from torch import nn

series = np.arange(10, dtype=np.float32)         # 10 readings: 0, 1, ..., 9
T = 4                                            # time steps per window


def make_windows(values, T):
    X = np.stack([values[i:i + T] for i in range(len(values) - T)])
    y = values[T:]                               # predict the next value
    return X[..., None], y                       # (samples, time steps, features)


X, y = make_windows(series, T)
print(X.shape, y.shape)                          # (6, 4, 1) (6,)
print(X[0, :, 0], y[0])                          # [0. 1. 2. 3.] 4.0

torch.manual_seed(42)
rnn = nn.RNN(input_size=1, hidden_size=8, batch_first=True)
out, h_n = rnn(torch.from_numpy(X))
print(out.shape)                                 # torch.Size([6, 4, 8])  one hidden state per time step
print(h_n.shape)                                 # torch.Size([1, 6, 8])  state after the last time step

Each window has 4 time steps with 1 feature; out has one hidden vector per time step, and h_n is the state after the fourth step, which a linear layer would map to the forecast.

Unrolling

For T = 3: h_1 = f(h_0, x_1), h_2 = f(h_1, x_2), h_3 = f(h_2, x_3). The same f (same weights) is used at each step, so the parameter count does not depend on T, but the computation and the BPTT memory do.

Choosing Granularity and Length

Decision Trade-off
Finer steps (characters, 10 ms) More detail, longer sequences, harder long-range learning
Coarser steps (words, hourly) Shorter sequences, possible loss of detail
Longer window T More context, more compute and vanishing-gradient pressure
Irregular events Add the time gap as a feature at each step

Common Pitfalls

  • Swapping the time and feature axes ((batch, features, time) instead of (batch, time, features)).

  • Forgetting PyTorch’s default (time, batch, features) when batch_first=False.

  • Treating irregularly spaced events as equally spaced steps.

  • Choosing T shorter than the dependency the task needs (e.g. 24 hours for weekly seasonality).

Interview Tips

  • Define a time step as one sequence position and one cell application.

  • Give the tensor shape and examples from several domains.

  • Discuss unrolling, weight sharing and how to pick granularity and window length.

3

What kinds of sequences are RNNs good at modelling?

RNNs are good at modelling ordered data where each element depends on what came before and the useful context is short to medium in length: natural-language text (characters, words, subwords), speech and audio frames, univariate and multivariate time series (sensors, energy, finance, health signals), event and clickstream sequences (user sessions, logs, transactions), handwriting and pen strokes, music, biological sequences (DNA, proteins) and control or trajectory data. They fit especially well when data arrives in a stream, when you need constant memory per step at inference (on-device keyword spotting, real-time sensors), when sequences have variable length, and when datasets are moderate. They are weaker for very long-range dependencies across thousands of steps and for large-scale pretraining, where transformers dominate, and for irregularly sampled data unless time gaps are given as inputs or a continuous-time variant is used.

Read full answer

Applies toRNN, LSTM and GRU models in PyTorch 2.x / Keras 3

Sequence Types and Typical Tasks

Sequence type Example task RNN pattern
Text Sentiment, NER, character-level generation Many-to-one, many-to-many, generative
Speech/audio Keyword spotting, ASR with CTC Many-to-many (frame labels)
Time series Forecasting load, anomaly detection Many-to-one / seq2seq
Clickstreams, sessions Next-item recommendation (GRU4Rec) Many-to-many next-step
Transactions Fraud detection over a card’s history Many-to-one, streaming state
Handwriting, strokes Online handwriting recognition Many-to-many with CTC
Music (MIDI events) Melody generation Generative next-event
DNA/protein Secondary structure, binding sites Bidirectional tagging
Video features Action recognition over CNN features Many-to-one
Control/robotics Policies with memory (partial observability) Recurrent policy

Code Example: The Same GRU Handles Variable-Length Sequences

import torch
from torch import nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_sequence

torch.manual_seed(42)

# Three user sessions of different lengths, each step = 5 features
sessions = [torch.randn(n, 5) for n in (7, 3, 12)]
lengths = torch.tensor([len(s) for s in sessions])
padded = pad_sequence(sessions, batch_first=True)          # (3, 12, 5)

gru = nn.GRU(5, 16, batch_first=True)
packed = pack_padded_sequence(padded, lengths, batch_first=True, enforce_sorted=False)
_, h_n = gru(packed)
print(padded.shape, h_n.shape)                             # torch.Size([3, 12, 5]) torch.Size([1, 3, 16])

# Streaming: constant memory per new event
h = torch.zeros(1, 1, 16)
for event in sessions[2]:                                  # events arrive one at a time
    _, h = gru(event.view(1, 1, 5), h)
print(torch.allclose(h[0, 0], h_n[0, 2], atol=1e-5))       # True: same final state as the batched pass

Packing lets one batch hold sequences of lengths 7, 3 and 12 without the padding affecting the results, and the streaming loop shows the same model updating a fixed-size state one event at a time.

Where Other Models Usually Win

Situation Better fit
Very long documents, large pretraining Transformers
Very long sequences with linear cost State-space models (Mamba, S4)
Short fixed windows, local patterns 1-D CNNs (TCN)
Tabular features with little order Gradient boosting
Irregular timestamps Time-aware RNNs, neural ODEs, or time gaps as inputs

Common Pitfalls

  • Using an RNN on data where order is meaningless (bag of features).

  • Ignoring irregular time gaps between events.

  • Expecting dependency learning over thousands of steps.

  • Not considering a simpler baseline (lagged features + gradient boosting) for time series.

Interview Tips

  • List several domains with a concrete task each.

  • Name the strengths (streaming, variable length, constant memory) and the weaknesses (long range, parallelism).

  • Say when you would choose a transformer, CNN or tree model instead.

4

How does the hidden state in an RNN work?

The hidden state h_t is the RNN’s memory: a fixed-size vector that summarizes everything the network has seen up to step t. At each step the cell combines the previous hidden state with the current input — in a vanilla RNN, h_t = tanh(W_hh h_(t−1) + W_xh x_t + b) — and the result is both passed on to the next step and used to make that step’s prediction (ŷ_t = W_hy h_t + b_y). It usually starts as zeros (or a learned or provided vector), the same weights update it at every step, and its size (the hidden size) sets how much information it can hold. Because it is a lossy, learned compression of the past, old information fades unless the network is designed to keep it, which is what the LSTM’s separate cell state and gates are for. In encoder-decoder models, the final hidden state is the summary handed to the decoder.

Read full answer

Applies toPyTorch 2.x (`nn.RNN`, `nn.RNNCell`); same ideas in Keras

What the Hidden State Does

Aspect Detail
Update h_t = f(h_(t−1), x_t) with shared weights
Initial value h_0 = zeros by default; can be learned or set (e.g. from an encoder or user profile)
Size Hidden size H; more capacity, more parameters (H² recurrent weights)
Output use Per-step predictions, or the last state for the whole sequence
Across batches Reset per sequence; carried (detached) in stateful/streaming setups
LSTM Two states: hidden h_t (output) and cell c_t (long-term memory)
Stacked layers Each layer has its own hidden state; the upper layer reads the lower layer’s states

Code Example: Stepping Through the Hidden State

import torch
from torch import nn

torch.manual_seed(42)

cell = nn.RNNCell(input_size=2, hidden_size=3)
x = torch.tensor([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])    # 3 time steps, 2 features
h = torch.zeros(1, 3)                                      # h_0

states = []
with torch.no_grad():
    for t in range(len(x)):
        manual = torch.tanh(x[t] @ cell.weight_ih.T + cell.bias_ih + h @ cell.weight_hh.T + cell.bias_hh)
        h = cell(x[t].unsqueeze(0), h)                     # same update, done by PyTorch
        assert torch.allclose(manual, h)
        states.append(h.clone())

print(len(states), states[-1].shape)                       # 3 torch.Size([1, 3])

# The full-sequence layer gives the same final state
rnn = nn.RNN(2, 3, batch_first=True)
rnn.load_state_dict({"weight_ih_l0": cell.weight_ih, "weight_hh_l0": cell.weight_hh,
                     "bias_ih_l0": cell.bias_ih, "bias_hh_l0": cell.bias_hh})
with torch.no_grad():
    output, h_n = rnn(x.unsqueeze(0))
print(torch.allclose(output[0, -1], states[-1][0]), torch.allclose(h_n[0], states[-1]))   # True True

# The state depends on order: same inputs, reversed order, different final state
with torch.no_grad():
    _, h_rev = rnn(x.flip(0).unsqueeze(0))
print(torch.allclose(h_rev, h_n))                          # False

output holds the hidden state at every step and h_n holds only the last one; for a one-layer, one-direction RNN, output[:, -1] equals h_n[0]. The last check shows the state encodes order, not just the set of inputs.

Intuition

Reading “the movie was not good”, the hidden state after “not” carries a negation signal that changes how “good” updates the state, so the final state points toward negative sentiment. The network learns these representations from the training objective; individual units are rarely cleanly interpretable.

Common Pitfalls

  • Confusing output (all steps, top layer) with h_n (last step, all layers).

  • Carrying the hidden state across unrelated sequences, or carrying it without detach() (memory grows).

  • Using the last padded step as the state for short sequences (pack or index by length).

  • Thinking a bigger hidden size alone fixes forgetting; the vanishing gradient is the real limit.

Interview Tips

  • Write the update equation and say the weights are shared across steps.

  • Explain the hidden state as a learned, lossy summary of the past.

  • Mention initial state options, LSTM’s two states, and output vs h_n.

5

What are the challenges of training vanilla RNNs?

Vanilla RNNs (h_t = tanh(W h_(t−1) + U x_t + b)) are hard to train mainly because the same recurrent matrix is applied at every step, so gradients flowing back through time are products of many Jacobians: they usually vanish (the network cannot learn long-range dependencies) and sometimes explode (loss spikes, NaN). Around that core problem sit practical challenges: training is sequential and slow on long sequences, memory grows with sequence length, results are sensitive to initialization and learning rate, saturating tanh units stop learning, variable-length batches need padding and masking, and the models overfit small datasets. The standard answers are gradient clipping, orthogonal initialization, gated cells (LSTM/GRU), truncated BPTT, normalization and dropout variants designed for recurrence, or moving to attention-based models.

Read full answer

Applies toPyTorch 2.x (`nn.RNN`, `nn.LSTM`, `nn.GRU`)

Challenges and Fixes

Challenge Why it happens Fix
Vanishing gradients Jacobian norms < 1 multiplied over time LSTM/GRU, orthogonal/identity init, attention, skip connections
Exploding gradients Jacobian norms > 1 Gradient norm clipping, lower LR, orthogonal init
Short effective memory Consequence of vanishing gradients Gated cells, attention
Slow training Step t depends on step t−1; no parallelism over time Larger batches, cuDNN kernels, shorter sequences, transformers
Memory on long sequences BPTT stores every step Truncated BPTT, gradient checkpointing
Sensitivity to hyperparameters Chaotic dynamics for large weights Careful LR, warm-up, init, normalization
Saturated units tanh near ±1 has derivative near 0 Input scaling, layer norm, smaller init
Variable lengths Padding changes the final state Packing or masking
Overfitting Many parameters, small sequence datasets Dropout between layers, variational dropout, weight decay, early stopping

Code Example: A Stable Training Setup (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)


class VanillaRNNClassifier(nn.Module):
    def __init__(self, n_features, hidden, n_classes):
        super().__init__()
        self.rnn = nn.RNN(n_features, hidden, batch_first=True, nonlinearity="tanh")
        self.norm = nn.LayerNorm(hidden)
        self.fc = nn.Linear(hidden, n_classes)
        for name, p in self.rnn.named_parameters():
            if "weight_hh" in name:
                nn.init.orthogonal_(p)          # singular values 1: neither shrinks nor grows at start
            elif "weight_ih" in name:
                nn.init.xavier_uniform_(p)
            else:
                nn.init.zeros_(p)

    def forward(self, x):
        _, h_n = self.rnn(x)
        return self.fc(self.norm(h_n[-1]))


model = VanillaRNNClassifier(n_features=8, hidden=64, n_classes=3)
w_hh = model.rnn.weight_hh_l0.detach()
print(torch.allclose(w_hh @ w_hh.T, torch.eye(64), atol=1e-5))   # True: orthogonal

x, y = torch.randn(32, 50, 8), torch.randint(0, 3, (32,))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loss = nn.functional.cross_entropy(model(x), y)
opt.zero_grad()
loss.backward()
norm = nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)  # guard against explosions
opt.step()
print(f"loss {loss.item():.3f}, grad norm before clipping {norm.item():.3f}")

Orthogonal recurrent weights keep the linear part of each step norm-preserving at the start, clipping caps any explosion, and layer norm keeps the classifier input in a stable range. None of these give a vanilla RNN long-term memory; for that, switch to an LSTM or GRU.

Common Pitfalls

  • Diagnosing poor results as “needs more epochs” when gradients to early steps are vanishing.

  • Applying standard dropout to the recurrent connection at every step (it destroys memory); use dropout between layers or variational dropout.

  • Using a ReLU RNN without careful initialization (activations can explode).

  • Comparing models trained on padded batches without masking.

Interview Tips

  • Lead with vanishing/exploding gradients and explain them with the Jacobian product.

  • Then list the practical issues: speed, memory, sensitivity, padding, overfitting.

  • Pair each challenge with its standard fix.

6

Why do activation functions matter in RNNs?

Activation functions matter even more in RNNs than in feedforward networks because the same nonlinearity is applied at every time step and its derivative is multiplied into the gradient at every step of backpropagation through time: the choice controls whether the hidden state stays bounded, whether gradients vanish or explode, and how the gates of LSTMs and GRUs behave. The standard choices are tanh for the hidden state and candidate content (bounded in (−1, 1), zero-centred, derivative at most 1) and sigmoid for gates (values in (0, 1) act as soft switches). ReLU in plain RNNs is unbounded, so with recurrent weights above 1 activations can grow without limit; it works only with careful setups such as identity initialization (IRNN) and clipping. Without a nonlinearity, an RNN collapses to a linear dynamical system.

Read full answer

Applies toPyTorch 2.x (`nn.RNN(nonlinearity="tanh"|"relu")`; LSTM/GRU gates fixed to sigmoid/tanh)

Where Each Activation Is Used

Activation Range Max derivative Role in RNNs
tanh (−1, 1) 1 (at 0) Hidden state (vanilla RNN), LSTM/GRU candidate, LSTM output squashing
sigmoid (0, 1) 0.25 (at 0) LSTM input/forget/output gates, GRU update/reset gates
ReLU [0, ∞) 1 Optional in vanilla RNNs (IRNN); needs care
Softmax probabilities — Output layer for classification / next token
Linear ℝ 1 Output layer for regression

Why tanh for the State

Property Effect
Bounded Hidden state cannot blow up across many steps
Zero-centred Positive and negative values; better-conditioned updates than sigmoid
Derivative ≤ 1 Prevents explosion from the activation, but contributes to vanishing when saturated

Code Example: Derivatives and Bounded vs Unbounded States

import torch
from torch import nn

z = torch.tensor([0.0, 2.0, 5.0], requires_grad=True)
torch.tanh(z).sum().backward()
print([round(v, 4) for v in z.grad.tolist()])      # [1.0, 0.0707, 0.0002]  tanh' = 1 - tanh^2

z2 = torch.tensor([0.0, 2.0, 5.0], requires_grad=True)
torch.sigmoid(z2).sum().backward()
print([round(v, 4) for v in z2.grad.tolist()])     # [0.25, 0.105, 0.0066]

# Same recurrent weights (scaled identity, factor 1.5) with tanh vs ReLU
torch.manual_seed(42)
x = torch.ones(1, 30, 1)
for act in ("tanh", "relu"):
    rnn = nn.RNN(1, 4, batch_first=True, nonlinearity=act)
    with torch.no_grad():
        rnn.weight_hh_l0.copy_(1.5 * torch.eye(4))
        rnn.weight_ih_l0.fill_(1.0)
        rnn.bias_ih_l0.zero_()
        rnn.bias_hh_l0.zero_()
        out, _ = rnn(x)
    print(act, f"{out.abs().max().item():.4g}")
# tanh 0.986   (bounded: settles at the fixed point of h = tanh(1.5h + 1))
# relu 3.835e+05

With ReLU each step computes h_t = 1.5·h_(t−1) + 1, so h_30 = 2·(1.5³⁰ − 1) ≈ 3.835 × 10⁵; with tanh the same weights keep the state below 1 (it settles near 0.986, where tanh is saturated, which is the price: small derivatives).

Gates Need Sigmoid

An LSTM’s c_t = f ⊙ c_(t−1) + i ⊙ g requires gate values between 0 and 1 to mean “fraction to keep/write”. Sigmoid gives exactly that; replacing it with tanh would allow negative “keep” values and flip memory signs.

Common Pitfalls

  • Using ReLU in a vanilla RNN with default initialization and no clipping.

  • Claiming tanh fixes vanishing gradients; it avoids explosion from the activation but still saturates.

  • Changing LSTM gate activations without a clear reason.

  • Using sigmoid for the hidden state (not zero-centred, derivative ≤ 0.25).

Interview Tips

  • Separate the roles: tanh for state and content, sigmoid for gates, softmax/linear for outputs.

  • Connect activation derivatives to the BPTT Jacobian product.

  • Mention ReLU RNNs with identity initialization as the exception, with their risks.

7

How does backpropagation through time (BPTT) work in RNNs?

Backpropagation through time is ordinary backpropagation applied to an RNN that has been unrolled over the sequence: the forward pass runs the same cell T times, storing every hidden state, and builds a deep network with T layers that share weights; the backward pass starts at the loss (summed over the steps that have outputs), sends the error back step by step through the recurrence, and, because the weights are shared, adds up each step’s gradient contribution to get the gradient for W, U and the biases. The gradient reaching hidden state h_t has two parts: from its own output at step t and from the next hidden state h_(t+1). That recursive term is a product of Jacobians over time, which is the source of vanishing and exploding gradients. In practice, frameworks do this automatically with autograd, and long sequences use truncated BPTT.

Read full answer

Applies toNumPy (manual derivation), PyTorch 2.x autograd

Equations (Vanilla RNN, Loss at Every Step)

Forward: a_t = W h_(t−1) + U x_t + b, h_t = tanh(a_t), ŷ_t = V h_t + c, L = Σ_t L_t(ŷ_t, y_t)

Backward, from t = T down to 1:

Quantity Formula
Error from the output δo_t = ∂L_t/∂ŷ_t
Error at h_t dh_t = Vᵀ δo_t + Wᵀ da_(t+1) (second term is 0 at t = T)
Through tanh da_t = dh_t ⊙ (1 − h_t²)
Accumulate shared weights dW += da_t h_(t−1)ᵀ, dU += da_t x_tᵀ, db += da_t, dV += δo_t h_tᵀ

Code Example: Manual BPTT Checked Against Autograd

import numpy as np
import torch

rng = np.random.default_rng(42)
T, IN, H = 5, 3, 4
x = rng.standard_normal((T, IN))
W = rng.standard_normal((H, H)) * 0.5
U = rng.standard_normal((H, IN)) * 0.5
b = np.zeros(H)
V = rng.standard_normal((1, H))
target = rng.standard_normal((T, 1))

# Forward pass: store all hidden states
hs = [np.zeros(H)]
for t in range(T):
    hs.append(np.tanh(W @ hs[-1] + U @ x[t] + b))
preds = np.array([V @ h for h in hs[1:]])                 # (T, 1)
loss = 0.5 * np.sum((preds - target) ** 2)

# Backward pass through time
dW, dU, db = np.zeros_like(W), np.zeros_like(U), np.zeros_like(b)
da_next = np.zeros(H)
for t in reversed(range(T)):
    dh = V.T @ (preds[t] - target[t]) + W.T @ da_next    # from the output + from the future
    da = dh * (1 - hs[t + 1] ** 2)                        # through tanh
    dW += np.outer(da, hs[t])                              # shared weights: accumulate
    dU += np.outer(da, x[t])
    db += da
    da_next = da

# Same computation with autograd
Wt, Ut, bt = (torch.tensor(m, requires_grad=True) for m in (W, U, b))
Vt, xt, yt = torch.tensor(V), torch.tensor(x), torch.tensor(target)
h = torch.zeros(H, dtype=torch.float64)
loss_t = torch.zeros((), dtype=torch.float64)
for t in range(T):
    h = torch.tanh(Wt @ h + Ut @ xt[t] + bt)
    loss_t = loss_t + 0.5 * ((Vt @ h - yt[t]) ** 2).sum()
loss_t.backward()

print(np.isclose(loss, loss_t.item()))                    # True
print(np.allclose(dW, Wt.grad.numpy()), np.allclose(dU, Ut.grad.numpy()), np.allclose(db, bt.grad.numpy()))
# True True True

The line dh = V.T @ ... + W.T @ da_next is BPTT in one line: each step’s error includes everything coming back from later steps, multiplied by Wᵀ and the tanh derivative each time.

Truncated BPTT

For long streams, process chunks of k steps, carry the hidden state forward but detach() it at chunk boundaries, so the backward pass covers at most k steps. Memory and update frequency improve; dependencies longer than k cannot be learned through gradients.

Common Pitfalls

  • Forgetting that shared weights’ gradients are sums over time steps.

  • Leaving out the recurrent term Wᵀ da_(t+1) and getting plain per-step backprop.

  • Not storing hidden states in the forward pass (they are needed for the tanh derivative and weight gradients).

  • Forgetting gradient clipping on long sequences.

Interview Tips

  • Describe unrolling, the backward recursion and gradient accumulation for shared weights.

  • Point to the repeated Wᵀ · tanh′ factor as the cause of vanishing/exploding gradients.

  • Mention truncated BPTT and its trade-off.

8

What are the limitations of BPTT, and how can they be mitigated?

Backpropagation through time (BPTT) trains an RNN by unrolling it over the sequence and backpropagating through every step, and that design causes its limitations: memory grows linearly with sequence length (all intermediate activations must be stored), computation is sequential and cannot be parallelized across time, gradients vanish or explode over long spans, and updates can only happen after the whole sequence is processed (no online learning). The common practical mitigation, truncated BPTT (backpropagate only k steps while carrying the hidden state forward), fixes memory and update frequency but introduces bias: the model cannot learn dependencies longer than k through gradients. Other mitigations are gradient clipping, gated cells (LSTM/GRU), gradient checkpointing (recompute activations to save memory), attention and skip connections, online approximations of real-time recurrent learning (UORO, e-prop), and, at the architectural level, models that train in parallel (transformers, state-space models).

Read full answer

Applies toPyTorch 2.x

Limitations and Mitigations

Limitation Cause Mitigation
Memory O(T) Activations of every step stored for backprop Truncated BPTT, gradient checkpointing, smaller batch
Sequential compute Step t needs h_(t−1) Larger batches; parallel-scan architectures (SSMs); transformers
Vanishing gradients Products of Jacobians < 1 LSTM/GRU, forget bias, attention, skip connections
Exploding gradients Products of Jacobians > 1 Gradient clipping, orthogonal init, lower LR
Truncation bias No gradient beyond k steps Larger k, overlapping windows, auxiliary losses, attention over longer context
No online updates Must wait for the end of the sequence Truncated BPTT; RTRL approximations (UORO, e-prop)
Long-sequence latency in training Thousands of steps per sequence Downsample, chunk, hierarchical RNNs

Code Example: Truncated BPTT with Carried State (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

SEQ_LEN, K, BATCH, FEATURES, HIDDEN = 1000, 100, 8, 4, 32
x = torch.randn(BATCH, SEQ_LEN, FEATURES)          # one long stream per batch row
y = torch.randn(BATCH, SEQ_LEN, 1)

lstm = nn.LSTM(FEATURES, HIDDEN, batch_first=True)
head = nn.Linear(HIDDEN, 1)
params = list(lstm.parameters()) + list(head.parameters())
opt = torch.optim.Adam(params, lr=1e-3)

state = None
updates = 0
for start in range(0, SEQ_LEN, K):
    chunk_x, chunk_y = x[:, start:start + K], y[:, start:start + K]
    if state is not None:
        state = tuple(s.detach() for s in state)   # keep the values, cut the gradient path
    out, state = lstm(chunk_x, state)
    loss = nn.functional.mse_loss(head(out), chunk_y)
    opt.zero_grad()
    loss.backward()                                # backprop through at most K = 100 steps
    nn.utils.clip_grad_norm_(params, 1.0)
    opt.step()
    updates += 1

print(updates)                                     # 10 parameter updates instead of 1

detach() is the key line: the hidden state still carries information from all previous chunks in the forward pass, but gradients stop at the chunk boundary, so memory stays proportional to K instead of the full 1,000 steps.

Gradient Checkpointing

torch.utils.checkpoint.checkpoint(fn, *inputs, use_reentrant=False) around segments of the unrolled computation stores only segment boundaries and recomputes the rest in the backward pass: memory drops from O(T) towards O(√T) for evenly sized segments, at the cost of extra compute.

Choosing K

Consideration Effect
Longest dependency you need gradients for K should be at least that long
GPU memory Upper bound on K × batch × hidden
Update frequency Smaller K = more frequent updates
Variance Overlapping windows reduce boundary effects

Common Pitfalls

  • Forgetting detach(), so the graph keeps growing across chunks and memory explodes (or an error about backpropagating through a freed graph appears).

  • Resetting the hidden state to zero at every chunk, which throws away context and makes training different from inference.

  • Shuffling chunks of the same stream, which breaks the carried state.

  • Assuming truncated BPTT learns dependencies longer than K.

Interview Tips

  • List the four core limitations: memory, sequential compute, unstable gradients, no online learning.

  • Explain truncated BPTT with detach() and its truncation bias.

  • Mention checkpointing, clipping, gating and parallel architectures as mitigations.

9

What is the vanishing gradient problem in RNNs, and why does it matter?

The vanishing gradient problem is the exponential shrinking of the training signal as backpropagation through time carries it from later time steps back to earlier ones: the gradient of the loss with respect to an early hidden state is a product of many per-step Jacobians, each involving the recurrent weight matrix and the activation derivative (tanh′ ≤ 1, sigmoid′ ≤ 0.25), so when those factors are smaller than 1 the product approaches zero. It matters because the network then cannot learn long-range dependencies: weights receive almost no signal about how early inputs affect later outputs, so a vanilla RNN effectively remembers only the last few to tens of steps. It is the main reason vanilla RNNs were replaced by LSTMs and GRUs (additive, gated memory paths), and it motivated attention, skip/residual connections, careful initialization and, ultimately, transformers.

Read full answer

Applies toPyTorch 2.x, NumPy

The Math in One Line

For h_t = tanh(W h_(t−1) + U x_t + b):

‖∂h_T/∂h_k‖ ≤ Π_(t=k+1..T) ‖diag(tanh′(a_t))‖ · ‖W‖ ≤ (γ · σ_max(W))^(T−k)

where γ ≤ 1 bounds tanh′. If γ · σ_max(W) < 1, the bound decays exponentially with the distance T − k (if it is above 1, gradients can explode instead).

Numbers

Per-step factor After 10 steps After 50 steps
0.9 0.349 0.00515
0.5 0.000977 8.9 × 10⁻¹⁶
0.25 (sigmoid′ max) 9.5 × 10⁻⁷ 7.9 × 10⁻³¹

Code Example: Measuring Gradient Decay Over Distance (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

for factor in (0.9, 0.5, 0.25):
    print(f"{factor}: 10 steps {factor ** 10:.3g}, 50 steps {factor ** 50:.3g}")
# 0.9: 10 steps 0.349, 50 steps 0.00515
# 0.5: 10 steps 0.000977, 50 steps 8.88e-16
# 0.25: 10 steps 9.54e-07, 50 steps 7.89e-31

T = 60
rnn = nn.RNN(input_size=4, hidden_size=32, batch_first=True)
x = torch.randn(1, T, 4, requires_grad=True)
out, _ = rnn(x)
out[0, -1].sum().backward()                    # loss uses only the final output

grad_norms = x.grad[0].norm(dim=1)             # sensitivity of the final output to each input step
for distance in (0, 5, 20, 59):
    print(f"input {distance:>2} steps before the end: grad norm {grad_norms[T - 1 - distance]:.2e}")

With typical initializations, the gradient norm drops by orders of magnitude as the distance grows, so the earliest inputs barely influence learning; the exact values depend on the random weights.

Why It Matters in Practice

Consequence Example
Long-range dependencies not learned Subject-verb agreement across a long clause; a sensor event 200 steps before a failure
Training dominated by recent steps Model behaves like a short sliding window
Slow or stalled learning Loss plateaus early

Remedies

Remedy How it helps
LSTM / GRU Additive gated state; gradient ≈ forget/update gate per step
Forget-gate bias ≈ 1 Keeps the memory path open at the start
Orthogonal / identity initialization Singular values ≈ 1
Attention, skip connections Short paths from outputs to distant inputs
ReLU RNNs with identity init (IRNN), layer norm Better-conditioned dynamics
Transformers / SSMs Different mechanisms for long-range dependencies

Common Pitfalls

  • Confusing vanishing gradients with overfitting or a bad learning rate; measure gradient norms by distance.

  • Expecting gradient clipping to help: clipping fixes exploding, not vanishing, gradients.

  • Assuming LSTMs remove the issue entirely for thousands of steps.

Interview Tips

  • Derive the product of Jacobians and the exponential bound.

  • Explain the consequence: no learning of long-range dependencies.

  • List the fixes and say why each works.

10

What is the exploding gradient problem, and how does it affect RNNs?

Exploding gradients happen when the gradient computed by backpropagation through time grows exponentially as it flows back over many steps, because it is multiplied at each step by the recurrent Jacobian (the recurrent weight matrix times the activation derivative) and those factors have norms above 1. The effect on training is sudden, huge parameter updates: the loss spikes or becomes NaN/inf, weights overflow, and training diverges or oscillates instead of converging. It is the mirror image of vanishing gradients, and it is most likely with long sequences, large recurrent weights, high learning rates and unsaturated activations. The standard fix is gradient clipping (rescale the gradient when its norm exceeds a threshold), supported by careful initialization (orthogonal recurrent weights), smaller learning rates, gated cells, normalization and truncated BPTT.

Read full answer

Applies toPyTorch 2.x (`clip_grad_norm_`, `clip_grad_value_`); Keras optimizers (`clipnorm`, `clipvalue`)

Why It Happens

If the largest singular value of the recurrent Jacobian is s > 1 along some direction, a gradient component in that direction grows roughly like s^(T−k) over T − k steps: 1.1⁵⁰ ≈ 117, 1.1¹⁰⁰ ≈ 13,781, 1.5¹⁰⁰ ≈ 4 × 10¹⁷.

Symptoms

Symptom Where you see it
Loss suddenly jumps or becomes NaN Training logs
Gradient norm spikes by orders of magnitude Logged total_norm
Weights become very large or inf Parameter statistics
Validation metric collapses after an update Evaluation

Code Example: Gradient Clipping (PyTorch)

import torch
from torch import nn

# 1. Clip by norm: rescale the whole gradient vector if its norm exceeds max_norm
p = nn.Parameter(torch.zeros(2))
p.grad = torch.tensor([3.0, 4.0])                       # norm 5
total = nn.utils.clip_grad_norm_([p], max_norm=1.0)
print(total.item(), p.grad.tolist())                    # 5.0 [0.6000000238418579, 0.800000011920929]

# 2. Clip by value: cap each component independently (changes the direction)
q = nn.Parameter(torch.zeros(2))
q.grad = torch.tensor([3.0, -4.0])
nn.utils.clip_grad_value_([q], clip_value=2.0)
print(q.grad.tolist())                                  # [2.0, -2.0]

# 3. In a training loop: clip after backward(), before step(), and log the norm
torch.manual_seed(42)
model = nn.LSTM(10, 64, batch_first=True)
head = nn.Linear(64, 1)
params = list(model.parameters()) + list(head.parameters())
opt = torch.optim.Adam(params, lr=1e-3)

x, y = torch.randn(16, 200, 10), torch.randn(16, 1)
out, _ = model(x)
loss = nn.functional.mse_loss(head(out[:, -1]), y)
opt.zero_grad()
loss.backward()
grad_norm = nn.utils.clip_grad_norm_(params, max_norm=1.0)   # returns the norm before clipping
opt.step()
print(f"gradient norm before clipping: {grad_norm.item():.3f}")

Clipping by norm keeps the gradient’s direction and only limits the step size, which is why it is the usual choice. The printed float values of [0.6, 0.8] show float32 rounding.

Fixes and Prevention

Technique Effect
Gradient norm clipping (max norm 0.5-5) Caps update size; the standard fix
Lower learning rate / warm-up Smaller steps early in training
Orthogonal recurrent initialization Singular values start at 1
Gated cells (LSTM/GRU) More controlled dynamics
Layer normalization in the cell Keeps activations in range
Truncated BPTT Fewer multiplied Jacobians per update
Mixed-precision loss scaling with inf checks Skips steps with overflowed gradients

Common Pitfalls

  • Clipping before backward() or after optimizer.step() (must be between them).

  • With mixed precision, clipping scaled gradients: call scaler.unscale_(optimizer) first.

  • Setting the threshold so low that every step is clipped (training slows to a crawl); log the norms to choose it.

  • Treating NaN from bad data (e.g. log(0) in the loss) as exploding gradients.

Interview Tips

  • Explain the exponential growth from repeated Jacobian products.

  • Describe the symptoms (loss spikes, NaN) and clipping by norm vs by value.

  • Mention prevention: initialization, learning rate, gating, normalization.

11

What are LSTMs, and how do they address vanishing gradients?

A Long Short-Term Memory (LSTM) network is a recurrent neural network whose units contain a memory cell and three gates (forget, input, output), introduced by Hochreiter and Schmidhuber in 1997 to learn dependencies across many time steps. Vanilla RNNs suffer from vanishing gradients because the gradient flowing back through time is multiplied at every step by the recurrent weight matrix and the derivative of tanh, so it shrinks exponentially. The LSTM adds a separate cell state updated additively: c_t = f_t ⊙ c(t−1) + i_t ⊙ g_t. Along this path, the gradient from c_t to c(t−1) is simply the forget gate f_t (element-wise), with no matrix multiplication or squashing; when the network learns f_t ≈ 1 for information it needs, the gradient passes through many steps almost unchanged (the “constant error carousel”). The gates are learned, so the network decides what to remember, update and forget.

Read full answer

Applies toPyTorch 2.x, NumPy

Why Vanilla RNN Gradients Vanish

For h_t = tanh(W h_(t−1) + U x_t), the gradient from step T back to step k is a product of T − k Jacobians:

∂h_T/∂h_k = Π_(t=k+1..T) diag(tanh′(a_t)) · W

Each factor tends to have norm below 1 (tanh′ ≤ 1, and usually much smaller when units are active), so the product shrinks exponentially with distance.

The LSTM’s Gradient Highway

Path Gradient factor per step Over 100 steps
Vanilla RNN (typical factor ≈ 0.5) ≈ 0.5 0.5¹⁰⁰ ≈ 7.9 × 10⁻³¹
LSTM cell state with f ≈ 0.99 f_t ≈ 0.99 0.99¹⁰⁰ ≈ 0.366
LSTM cell state with f = 1 1 1

Code Example: Gradient Reaching the First Step (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

T, H = 100, 32
print(f"0.5^100  = {0.5 ** 100:.1e}")     # 7.9e-31
print(f"0.99^100 = {0.99 ** 100:.3f}")    # 0.366


def grad_to_first_step(rnn, forget_bias=None):
    if forget_bias is not None:                         # LSTM: open the forget gate at the start
        rnn.bias_ih_l0.data[H:2 * H] = forget_bias
    x = torch.randn(1, T, 8, requires_grad=True)
    out, _ = rnn(x)
    out[0, -1].sum().backward()                         # loss depends only on the last step
    return x.grad[0, 0].norm().item(), x.grad[0, -1].norm().item()


rnn_first, rnn_last = grad_to_first_step(nn.RNN(8, H, batch_first=True))
lstm_first, lstm_last = grad_to_first_step(nn.LSTM(8, H, batch_first=True), forget_bias=3.0)
print(f"vanilla RNN: grad at step 1 / grad at step 100 = {rnn_first / rnn_last:.1e}")
print(f"LSTM       : grad at step 1 / grad at step 100 = {lstm_first / lstm_last:.1e}")

The ratio measures how much of the learning signal reaches the first input compared with the last. With default initialization the vanilla RNN’s ratio is typically many orders of magnitude smaller than the LSTM’s with an open forget gate (bias 3 gives σ(3) ≈ 0.95 at the start); the exact numbers depend on the random weights, so run it rather than trusting fixed values.

What LSTMs Do and Do Not Solve

Solved / helped Not solved
Vanishing gradients along the cell state Exploding gradients (still clip)
Learning dependencies over hundreds of steps Very long contexts (thousands of steps) remain hard
Selective memory via learned gates Sequential computation (no parallelism over time)

Common Pitfalls

  • Saying LSTMs eliminate the problem completely; gradients through h_t and the gates can still vanish, and the forget gate must stay near 1.

  • Forgetting gradient clipping for LSTMs.

  • Initializing the forget-gate bias at 0 for tasks with long dependencies.

  • Confusing vanishing gradients (a training problem) with limited memory capacity.

Interview Tips

  • Show the product of Jacobians for vanilla RNNs, then the additive cell update.

  • State that ∂c_t/∂c_(t−1) = f_t (diagonal), the “constant error carousel”.

  • Mention forget-gate bias initialization and that clipping is still needed.

12

Describe the gating mechanism of an LSTM cell.

An LSTM cell controls its memory with three gates, each a sigmoid layer that outputs values between 0 (block) and 1 (pass) for every unit, computed from the current input x_t and the previous hidden state h(t−1): the forget gate** f_t decides how much of the old cell state c(t−1) to keep, the input gate** i_t decides how much of a new candidate g_t = tanh(…) to write, and the output gate o_t decides how much of the (squashed) cell state to expose as the hidden state. The updates are c_t = f_t ⊙ c_(t−1) + i_t ⊙ g_t and h_t = o_t ⊙ tanh(c_t). Because the cell state is updated by element-wise multiplication and addition rather than by repeated matrix multiplication, information and gradients can flow across many steps when the forget gate stays near 1 — the key to the LSTM’s long-term memory.

Read full answer

Applies toPyTorch 2.x (`nn.LSTMCell`, gate order i, f, g, o); same mechanism in Keras

The Gates

Component Formula Range Role
Forget gate f_t = σ(W_f x_t + U_f h_(t−1) + b_f) (0, 1) Keep or erase old memory
Input gate i_t = σ(W_i x_t + U_i h_(t−1) + b_i) (0, 1) How much new content to write
Candidate g_t = tanh(W_g x_t + U_g h_(t−1) + b_g) (−1, 1) What new content to write
Cell update c_t = f_t ⊙ c_(t−1) + i_t ⊙ g_t unbounded Long-term memory
Output gate o_t = σ(W_o x_t + U_o h_(t−1) + b_o) (0, 1) How much memory to reveal
Hidden state h_t = o_t ⊙ tanh(c_t) (−1, 1) Short-term output

Code Example: An LSTM Cell by Hand, Checked Against PyTorch

import torch
from torch import nn

torch.manual_seed(42)

IN, H = 5, 4
cell = nn.LSTMCell(IN, H)
x = torch.randn(3, IN)
h0, c0 = torch.randn(3, H), torch.randn(3, H)


def lstm_step(x, h, c, cell):
    gates = x @ cell.weight_ih.T + cell.bias_ih + h @ cell.weight_hh.T + cell.bias_hh
    i, f, g, o = gates.chunk(4, dim=1)                 # PyTorch order: input, forget, cell, output
    i, f, g, o = torch.sigmoid(i), torch.sigmoid(f), torch.tanh(g), torch.sigmoid(o)
    c_new = f * c + i * g                              # forget part of the old memory, write new content
    h_new = o * torch.tanh(c_new)                      # expose a filtered view of the memory
    return h_new, c_new, (i, f, g, o)


with torch.no_grad():
    h1, c1, (i, f, g, o) = lstm_step(x, h0, c0, cell)
    h_ref, c_ref = cell(x, (h0, c0))

print(torch.allclose(h1, h_ref, atol=1e-6), torch.allclose(c1, c_ref, atol=1e-6))   # True True
print(bool(((f > 0) & (f < 1)).all()))                 # True: gates are soft switches in (0, 1)

# Extreme gate settings show their roles
with torch.no_grad():
    keep = 1.0 * c0 + 0.0 * g                          # f = 1, i = 0: memory copied unchanged
    reset = 0.0 * c0 + 1.0 * g                         # f = 0, i = 1: memory replaced
print(torch.equal(keep, c0), torch.equal(reset, g))    # True True

Intuition with an Example

Reading “The keys to the cabinet are on the table”: a unit can store “subject is plural” when it reads “keys” (input gate opens), keep it through “to the cabinet” (forget gate ≈ 1, input gate ≈ 0), and reveal it when the verb is predicted (output gate opens). When a new sentence starts, the forget gate can clear it.

Design Details

Detail Common choice
Forget-gate bias Initialize to about 1, so the cell remembers by default
Peephole connections Gates also see c_(t−1) (Gers & Schmidhuber); rarely used now
Coupled input-forget gate i = 1 − f (a step toward the GRU)
Cell state range Unbounded; tanh squashes it before output

Common Pitfalls

  • Mixing up gate order when loading weights (PyTorch: i, f, g, o; Keras: i, f, c, o).

  • Forgetting there are two biases in PyTorch (bias_ih and bias_hh) that add up.

  • Describing the candidate g as a gate: it is the content, with tanh, not sigmoid.

  • Claiming LSTMs fully solve long-term memory; very long dependencies are still hard.

Interview Tips

  • Write the six equations and explain each gate in one phrase.

  • Emphasize the additive cell update as the reason gradients survive.

  • Use an example (subject-verb agreement) to show the gates working.

13

What are the differences between LSTM and GRU networks?

LSTMs and GRUs are both gated recurrent units designed to fix the vanishing-gradient problem of vanilla RNNs, but the GRU is a simplified design: an LSTM keeps two states (a cell state c_t and a hidden state h_t) and uses three gates (input, forget, output), while a GRU keeps only one state h_t and uses two gates (update and reset), merging the LSTM’s forget and input gates into a single update gate and dropping the output gate. As a result a GRU has about 25% fewer parameters (3 weight blocks instead of 4) and runs a little faster, while an LSTM has slightly more control over what it stores and exposes. In practice their accuracy is usually similar and depends on the task; GRUs are a good default for smaller data and faster experiments, LSTMs are often preferred for very long dependencies or large data. Try both.

Read full answer

Applies toPyTorch 2.x (`nn.LSTM`, `nn.GRU`); same formulas in Keras

Equations

LSTM GRU
i = σ(W_i x + U_i h_(t−1) + b_i) z = σ(W_z x + U_z h_(t−1) + b_z) (update)
f = σ(W_f x + U_f h_(t−1) + b_f) r = σ(W_r x + U_r h_(t−1) + b_r) (reset)
g = tanh(W_g x + U_g h_(t−1) + b_g) ñ = tanh(W_n x + r ⊙ (U_n h_(t−1)) + b_n)
o = σ(W_o x + U_o h_(t−1) + b_o) h_t = (1 − z) ⊙ ñ + z ⊙ h_(t−1)
c_t = f ⊙ c_(t−1) + i ⊙ g
h_t = o ⊙ tanh(c_t)

(The GRU’s interpolation is written as in PyTorch; some papers swap z and 1 − z.)

Comparison

Aspect LSTM GRU
States Cell c_t + hidden h_t Hidden h_t only
Gates Input, forget, output (+ candidate) Update, reset (+ candidate)
Weight blocks 4 3
Parameters (same sizes) 4k 3k (~25% fewer)
Speed Slower Faster
Memory control Separate “write” (input) and “keep” (forget) Single update gate couples them
Exposure of memory Output gate controls what h shows Whole state is exposed
Typical use Long dependencies, large data Smaller data, faster training, edge

Code Example: Parameter Counts and Outputs (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

lstm = nn.LSTM(input_size=32, hidden_size=64, batch_first=True)
gru = nn.GRU(input_size=32, hidden_size=64, batch_first=True)


def n_params(m):
    return sum(p.numel() for p in m.parameters())


print(n_params(lstm))      # 25088 = 4 * (64*32 + 64*64 + 2*64)
print(n_params(gru))       # 18816 = 3 * (64*32 + 64*64 + 2*64)

x = torch.randn(8, 20, 32)
out_l, (h_l, c_l) = lstm(x)          # LSTM returns two states
out_g, h_g = gru(x)                  # GRU returns one
print(out_l.shape, h_l.shape, c_l.shape)   # torch.Size([8, 20, 64]) torch.Size([1, 8, 64]) torch.Size([1, 8, 64])
print(out_g.shape, h_g.shape)              # torch.Size([8, 20, 64]) torch.Size([1, 8, 64])

Swapping one for the other in code mostly means handling the extra cell state: (h, c) for LSTM vs h for GRU.

How to Choose

Situation Suggestion
Quick baseline, limited data or compute GRU
Very long sequences, plenty of data LSTM (try both)
Edge / mobile GRU (smaller, fewer operations)
Matching a published architecture Whatever the paper used
Need peak accuracy Tune both with the same budget, several seeds

Common Pitfalls

  • Declaring one universally better; results depend on data and tuning.

  • Comparing at equal hidden size only; also compare at equal parameter count.

  • Forgetting that LSTM state handling ((h, c)) differs when switching code.

  • Initializing LSTM forget-gate bias to 0 when long memory is needed (1 is a common choice).

Interview Tips

  • Name the gates and states of each and the merge of forget/input into update.

  • Quote the 4 vs 3 weight blocks (≈25% fewer parameters).

  • Recommend trying both and choosing on validation results.

14

How do attention mechanisms work with RNNs?

Attention lets an RNN decoder (or classifier) look directly at all hidden states of an encoder RNN instead of relying on one final state: at each step it computes a relevance score between its current query (usually the decoder hidden state) and every encoder state, turns the scores into weights with softmax, and takes the weighted sum of encoder states as a context vector that is combined with the decoder state to make the prediction. This removes the fixed-size bottleneck of basic seq2seq models, greatly improves long-sequence translation, gives shorter gradient paths to distant inputs, and produces interpretable alignment weights. The two classic variants are Bahdanau (additive) attention, score = vᵀ tanh(W₁s + W₂h), and Luong (multiplicative) attention, score = sᵀh or sᵀWh. For classification, attention pooling over an RNN’s outputs replaces “use the last hidden state”. Transformers later kept attention and dropped the recurrence.

Read full answer

Applies toPyTorch 2.x, NumPy; Bahdanau et al. (2015), Luong et al. (2015)

Steps at Decoder Step t

Step Formula
Scores eᵢ = score(s_t, hᵢ) for each encoder state hᵢ
Weights αᵢ = softmax(e)ᵢ (padding positions masked to −∞)
Context c_t = Σᵢ αᵢ hᵢ
Combine s̃_t = tanh(W_c [c_t; s_t]) → output layer

Score Functions

Name score(s, h) Notes
Dot (Luong) sᵀh Needs same dimension; no parameters
General (Luong) sᵀWh Learned bilinear form
Additive / concat (Bahdanau) vᵀ tanh(W₁s + W₂h) Works with different dimensions
Scaled dot (transformers) sᵀh / √d Keeps softmax from saturating

Worked Example: Dot Attention by Hand (NumPy)

import numpy as np

H = np.array([[1.0, 0.0],      # encoder state 1
              [0.0, 1.0],      # encoder state 2
              [1.0, 1.0]])     # encoder state 3
s = np.array([1.0, 0.0])       # current decoder state (query)

scores = H @ s                                   # [1, 0, 1]
weights = np.exp(scores) / np.exp(scores).sum()  # softmax
context = weights @ H

print(np.round(weights, 3))    # [0.422 0.155 0.422]
print(np.round(context, 3))    # [0.845 0.578]

States 1 and 3 match the query equally (score 1), so they share most of the weight: e/(2e + 1) ≈ 0.422 each, and state 2 gets 1/(2e + 1) ≈ 0.155. The context is the weighted average of the three states.

Code Example: Luong Attention Module with Masking (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)


class LuongAttention(nn.Module):
    def __init__(self, hidden):
        super().__init__()
        self.W = nn.Linear(hidden, hidden, bias=False)      # "general" score
        self.combine = nn.Linear(2 * hidden, hidden)

    def forward(self, query, keys, mask):
        # query: (B, H) decoder state; keys: (B, S, H) encoder outputs; mask: (B, S) True = real token
        scores = torch.bmm(self.W(keys), query.unsqueeze(-1)).squeeze(-1)   # (B, S)
        scores = scores.masked_fill(~mask, float("-inf"))
        weights = torch.softmax(scores, dim=-1)
        context = torch.bmm(weights.unsqueeze(1), keys).squeeze(1)          # (B, H)
        attentional = torch.tanh(self.combine(torch.cat([context, query], dim=-1)))
        return attentional, weights


encoder = nn.GRU(32, 64, batch_first=True)
src = torch.randn(2, 6, 32)
enc_out, h = encoder(src)
mask = torch.tensor([[True] * 6, [True] * 4 + [False] * 2])   # second sentence has 2 pad steps

attn = LuongAttention(64)
out, weights = attn(h[-1], enc_out, mask)
print(out.shape, weights.shape)          # torch.Size([2, 64]) torch.Size([2, 6])
print(weights[1, 4:].tolist())           # [0.0, 0.0]  padded positions get no weight
print(torch.allclose(weights.sum(dim=-1), torch.ones(2)))     # True

Benefits

Benefit Why
No fixed-vector bottleneck Decoder reads all encoder states
Better long-sequence quality Direct access to distant tokens
Better gradient flow Short paths from outputs to inputs
Interpretability Weight matrices show soft alignments

Common Pitfalls

  • Not masking padding, so attention spreads onto pad tokens.

  • Unscaled dot products with large hidden sizes, saturating the softmax.

  • Treating attention weights as complete explanations of the model.

  • Recomputing encoder projections at every decoder step (precompute W₂h once).

Interview Tips

  • Write the score → softmax → weighted sum steps and one score function.

  • Explain the bottleneck problem attention solves.

  • Connect to transformers: attention without recurrence, scaled dot product, multiple heads.

15

What are bidirectional RNNs, and when should you use them?

A bidirectional RNN runs two separate recurrent layers over the same sequence — one from the first step to the last (forward) and one from the last step to the first (backward) — and concatenates their hidden states at each position, so every output depends on both past and future context. Use them when the whole sequence is available before you need the output: text classification, named-entity recognition and part-of-speech tagging, offline speech recognition encoders, protein and DNA sequence labelling, and the encoder side of seq2seq models. Do not use them for tasks that must work left to right — language modelling and text generation, real-time streaming, or forecasting future values — because the backward pass would need inputs that do not exist yet (or would leak the answer). They roughly double parameters and computation.

Read full answer

Applies toPyTorch 2.x (`bidirectional=True`); Keras 3 (`layers.Bidirectional`)

How Outputs Are Arranged (PyTorch)

Tensor Shape (1 layer, batch_first=True) Meaning
output (B, T, 2H) [:, t, :H] forward state at t; [:, t, H:] backward state at t
h_n (2, B, H) h_n[0] forward state after the last step; h_n[1] backward state after the first step

Code Example (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

B, T, F, H = 4, 7, 10, 16
x = torch.randn(B, T, F)
bilstm = nn.LSTM(F, H, batch_first=True, bidirectional=True)
output, (h_n, c_n) = bilstm(x)

print(output.shape)                                   # torch.Size([4, 7, 32])
print(h_n.shape)                                      # torch.Size([2, 4, 16])

# Forward direction finishes at the last step; backward direction finishes at the first step
print(torch.allclose(output[:, -1, :H], h_n[0]))      # True
print(torch.allclose(output[:, 0, H:], h_n[1]))       # True

# Sentence representation for classification: both final states
sentence = torch.cat([h_n[0], h_n[1]], dim=-1)        # (B, 2H)

# Token-level tagging (e.g. NER): use every position of `output`
tagger = nn.Linear(2 * H, 9)                          # 9 BIO tags
print(sentence.shape, tagger(output).shape)           # torch.Size([4, 32]) torch.Size([4, 7, 9])

A common bug is using output[:, -1] (all 2H values) as the sentence vector: its backward half has only seen the last token. Use h_n (or pooling over output) instead.

Keras Equivalent

import keras
from keras import layers

model = keras.Sequential([
    keras.Input(shape=(None,), dtype="int32"),
    layers.Embedding(10_000, 64, mask_zero=True),
    layers.Bidirectional(layers.LSTM(32, return_sequences=True)),   # per-token outputs (tagging)
    layers.Bidirectional(layers.LSTM(32)),                          # sentence vector
    layers.Dense(1, activation="sigmoid"),
])
print(model.output_shape)          # (None, 1)

When to Use

Task Bidirectional? Reason
Sentiment / topic classification Yes Whole text available
NER, POS tagging, chunking Yes (often + CRF) Labels depend on both sides
Offline ASR encoder Yes Full utterance available
Seq2seq encoder Yes Source sentence is complete
Language model / generation No Future tokens unknown
Streaming ASR, live captions No (or limited look-ahead) Latency
Forecasting future values No Future inputs do not exist

Common Pitfalls

  • Using a bidirectional model on time series where inputs include the target period (leakage).

  • Taking output[:, -1] as a summary (backward half saw only one token).

  • Forgetting packing/masking: with padding, the backward direction starts on pad tokens.

  • Doubling cost for tasks where the forward direction alone is enough.

Interview Tips

  • Explain the two passes and concatenation, and that each position sees full context.

  • Give “whole sequence available” as the rule for when to use it.

  • Mention the output[:, -1] vs h_n detail; it shows hands-on experience.

16

How do you implement an RNN with TensorFlow or PyTorch?

Implementing an RNN in either framework follows the same steps: prepare data as tensors of shape (batch, time steps, features) — or integer token IDs plus an embedding layer for text — define a model with a recurrent layer (nn.RNN/GRU/LSTM in PyTorch, keras.layers.SimpleRNN/GRU/LSTM in Keras) followed by an output layer, choose a loss and optimizer, train with mini-batches and gradient clipping, and evaluate on held-out data. The main differences: PyTorch gives you an explicit training loop and returns (output, hidden) from the layer, where you choose which step to use; Keras hides the loop behind compile/fit and returns only the last step unless return_sequences=True. Both use fast cuDNN kernels on GPU for standard LSTM/GRU settings.

Read full answer

Applies toPyTorch 2.x; TensorFlow 2.16+ with Keras 3

Steps

Step PyTorch Keras
Input shape (batch, seq, features) with batch_first=True (batch, seq, features) by default
Recurrent layer nn.LSTM(in, hidden, batch_first=True) layers.LSTM(units)
Last step vs all steps Index output[:, -1] or use h_n return_sequences=False/True
Loss nn.BCEWithLogitsLoss, nn.CrossEntropyLoss "binary_crossentropy", "sparse_categorical_crossentropy"
Training Manual loop model.fit(...)
Clipping clip_grad_norm_ optimizer=Adam(clipnorm=1.0)

Shared Data: Does a Sequence Sum to a Positive Number?

import numpy as np

rng = np.random.default_rng(42)
X = rng.standard_normal((2000, 20, 1)).astype("float32")
y = (X.sum(axis=(1, 2)) > 0).astype("float32")
X_train, y_train, X_test, y_test = X[:1600], y[:1600], X[1600:], y[1600:]
print(X_train.shape, y_train.shape)        # (1600, 20, 1) (1600,)

PyTorch Implementation

import numpy as np
import torch
from torch import nn

rng = np.random.default_rng(42)
X = rng.standard_normal((2000, 20, 1)).astype("float32")
y = (X.sum(axis=(1, 2)) > 0).astype("float32")
X_train, y_train = torch.tensor(X[:1600]), torch.tensor(y[:1600])
X_test, y_test = torch.tensor(X[1600:]), torch.tensor(y[1600:])
torch.manual_seed(42)


class SumSignLSTM(nn.Module):
    def __init__(self, hidden=32):
        super().__init__()
        self.lstm = nn.LSTM(input_size=1, hidden_size=hidden, batch_first=True)
        self.fc = nn.Linear(hidden, 1)

    def forward(self, x):
        output, (h_n, c_n) = self.lstm(x)    # output: (B, T, H); h_n: (layers, B, H)
        return self.fc(h_n[-1]).squeeze(-1)  # logits from the last hidden state


model = SumSignLSTM()
optimizer = torch.optim.Adam(model.parameters(), lr=3e-3)
loss_fn = nn.BCEWithLogitsLoss()

for epoch in range(10):
    model.train()
    perm = torch.randperm(len(X_train))
    for i in range(0, len(X_train), 64):
        idx = perm[i:i + 64]
        loss = loss_fn(model(X_train[idx]), y_train[idx])
        optimizer.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()

model.eval()
with torch.no_grad():
    acc = ((model(X_test) > 0).float() == y_test).float().mean().item()
print(f"PyTorch test accuracy: {acc:.3f}")

Keras Implementation

import keras
import numpy as np
from keras import layers

rng = np.random.default_rng(42)
X = rng.standard_normal((2000, 20, 1)).astype("float32")
y = (X.sum(axis=(1, 2)) > 0).astype("float32")
keras.utils.set_random_seed(42)

model = keras.Sequential([
    keras.Input(shape=(20, 1)),
    layers.LSTM(32),                         # returns the last hidden state only
    layers.Dense(1, activation="sigmoid"),
])
model.compile(optimizer=keras.optimizers.Adam(learning_rate=3e-3, clipnorm=1.0),
              loss="binary_crossentropy", metrics=["accuracy"])
model.fit(X[:1600], y[:1600], validation_split=0.1, epochs=10, batch_size=64, verbose=0)
loss, acc = model.evaluate(X[1600:], y[1600:], verbose=0)
print(f"Keras test accuracy: {acc:.3f}")
print(model.output_shape)                    # (None, 1)

The task is learnable because an LSTM can accumulate the running sum; both versions should reach high accuracy, but the printed values depend on the run. The PyTorch model returns logits and uses BCEWithLogitsLoss, while the Keras model uses a sigmoid output with binary_crossentropy; both are the same objective.

Common Pitfalls

  • Shape mistakes: PyTorch RNNs default to (seq, batch, features) unless batch_first=True.

  • Applying a sigmoid before BCEWithLogitsLoss (double sigmoid).

  • In Keras, stacking LSTMs without return_sequences=True on the lower layer.

  • Taking output[:, -1] with padded variable-length batches (use packing or masking).

  • Non-default Keras LSTM options (e.g. recurrent_dropout) silently disabling the cuDNN kernel.

Interview Tips

  • State the input shape convention and what each framework returns.

  • Mention loss/output pairing (logits vs probabilities) and gradient clipping.

  • Show that the model-building steps are the same in both frameworks.

17

How would you preprocess text data for training an RNN?

Preprocessing text for an RNN turns raw strings into padded batches of integer IDs plus their true lengths: clean and normalize the text (Unicode normalization, lower-casing if case does not matter, handling of URLs, numbers and HTML), tokenize it (words, subwords or characters), build a vocabulary from the training split only with special tokens (<pad>, <unk>, and <sos>/<eos> for generation) and a frequency cut-off, map tokens to IDs, choose a maximum length from the length distribution and truncate, then pad each batch and keep the lengths so the model can mask or pack the padding. Subword tokenizers (SentencePiece, BPE, WordPiece) avoid most unknown words; if you use pretrained embeddings, the tokenization must match theirs. The same preprocessing code must run at training and at serving time.

Read full answer

Applies toPython 3.10+; PyTorch 2.x (`pad_sequence`, `pack_padded_sequence`); Keras 3 `TextVectorization` is the built-in equivalent

Pipeline

Step Choices Watch out for
Clean Strip HTML, normalize Unicode (NFKC), replace URLs/emails with tags Do not remove signal (emojis, “!!!” in sentiment)
Normalize Lower-case, unify quotes, optional number tags Case can matter (NER)
Tokenize Word regex, subword (SentencePiece/BPE), characters Must match pretrained embeddings
Vocabulary From training data; min_freq; size cap Never from validation/test
Special tokens <pad>=0, <unk>, <sos>, <eos> Padding index 0 for padding_idx/mask_zero
Numericalize token → ID, unknown → <unk> Track OOV rate
Length Truncate to max_len (e.g. 95th percentile); head or tail Keep the informative end
Pad + lengths Pad per batch; keep true lengths Use packing/masking in the model
Batch Bucket by length Less padding, faster training

Code Example (Python + PyTorch)

import re
import unicodedata
from collections import Counter

import numpy as np
import torch
from torch.nn.utils.rnn import pad_sequence

train_texts = ["The food was GREAT!", "Service was slow... very slow.", "Great food, great service"]
TOKEN_RE = re.compile(r"[a-z0-9']+")


def normalize(text):
    text = unicodedata.normalize("NFKC", text).lower()
    text = re.sub(r"https?://\S+", " <url> ", text)
    return text


def tokenize(text):
    return TOKEN_RE.findall(normalize(text))


def build_vocab(texts, min_freq=2, specials=("<pad>", "<unk>")):
    counts = Counter(tok for t in texts for tok in tokenize(t))
    vocab = {s: i for i, s in enumerate(specials)}
    for tok, c in sorted(counts.items(), key=lambda kv: (-kv[1], kv[0])):   # frequent first, then a-z
        if c >= min_freq:
            vocab[tok] = len(vocab)
    return vocab


def encode(text, vocab, max_len):
    ids = [vocab.get(tok, vocab["<unk>"]) for tok in tokenize(text)]
    return ids[:max_len]                       # truncate (keep the head)


vocab = build_vocab(train_texts)               # training data only
print(vocab)
# {'<pad>': 0, '<unk>': 1, 'great': 2, 'food': 3, 'service': 4, 'slow': 5, 'was': 6}

lengths_seen = [len(tokenize(t)) for t in train_texts]
max_len = int(np.percentile(lengths_seen, 95))
print(lengths_seen, max_len)                   # [4, 5, 4] 4

batch_texts = ["The service was terrible", "great food"]
encoded = [torch.tensor(encode(t, vocab, max_len)) for t in batch_texts]
lengths = torch.tensor([len(e) for e in encoded])
padded = pad_sequence(encoded, batch_first=True, padding_value=vocab["<pad>"])
print(padded.tolist(), lengths.tolist())
# [[1, 4, 6, 1], [2, 3, 0, 0]] [4, 2]

“the” and “terrible” map to <unk> (ID 1) because “the” appears only once in training and “terrible” never does. The padded batch and the lengths tensor go to an nn.Embedding(..., padding_idx=0) and pack_padded_sequence, so padding is ignored.

Subword Tokenization

For real projects, train a SentencePiece or BPE tokenizer on the training text (or use a pretrained one): rare words split into known pieces (“unbelievably” → “un”, “believ”, “ably”), so <unk> becomes rare and the vocabulary stays at 8k-32k pieces.

Common Pitfalls

  • Building the vocabulary or fitting the tokenizer on all data (leakage, inflated scores).

  • Different preprocessing in training and serving (e.g. lower-casing only in training).

  • Padding without masking/packing, or using index 0 for a real token.

  • Truncating reviews or documents from the wrong end.

  • Removing stop words or punctuation that carry meaning (“not”, “!”).

Interview Tips

  • Walk through clean → tokenize → vocab (train only) → IDs → truncate/pad → lengths.

  • Mention special tokens, OOV handling and subword tokenizers.

  • Stress identical preprocessing at serving time.

18

How would you use an RNN to generate text sequences?

Text generation with an RNN uses a language model: train an LSTM or GRU to predict the next token (character or subword) from all previous ones with cross-entropy on a corpus, then generate by feeding a prompt, sampling the next token from the model’s output distribution, appending it, and repeating while carrying the hidden state forward. The sampling strategy controls the output: greedy decoding is repetitive, temperature scaling (dividing logits by T before softmax) trades coherence (T < 1) for diversity (T > 1), and top-k or nucleus (top-p) sampling cut off unlikely tokens. Quality is measured with validation perplexity and by reading samples. Character-level LSTMs are a classic teaching example (Karpathy’s char-rnn); modern text generation uses transformers.

Read full answer

Applies toPyTorch 2.x, NumPy

Steps

Step Detail
Tokenize Characters (small vocab, long sequences) or subwords
Make training pairs Input text[i:i+L], target text[i+1:i+L+1]
Model Embedding → LSTM (2-3 layers) → linear over vocabulary
Train Cross-entropy, gradient clipping, truncated BPTT over chunks
Generate Prompt → loop: logits → temperature/top-k → sample → feed back
Evaluate Validation perplexity, sample reading, repetition rate

Code Example: Character-Level LSTM Generator (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

text = ("to be or not to be that is the question whether tis nobler in the mind to suffer "
        "the slings and arrows of outrageous fortune or to take arms against a sea of troubles ") * 20
chars = sorted(set(text))
stoi = {c: i for i, c in enumerate(chars)}
itos = {i: c for c, i in stoi.items()}
data = torch.tensor([stoi[c] for c in text])

SEQ_LEN, BATCH = 64, 32


class CharLSTM(nn.Module):
    def __init__(self, vocab, emb=32, hidden=128):
        super().__init__()
        self.embed = nn.Embedding(vocab, emb)
        self.lstm = nn.LSTM(emb, hidden, num_layers=2, batch_first=True, dropout=0.1)
        self.out = nn.Linear(hidden, vocab)

    def forward(self, x, state=None):
        y, state = self.lstm(self.embed(x), state)
        return self.out(y), state


model = CharLSTM(len(chars))
opt = torch.optim.Adam(model.parameters(), lr=3e-3)

for step in range(300):
    starts = torch.randint(0, len(data) - SEQ_LEN - 1, (BATCH,))
    x = torch.stack([data[s:s + SEQ_LEN] for s in starts])
    y = torch.stack([data[s + 1:s + SEQ_LEN + 1] for s in starts])     # shifted by one
    logits, _ = model(x)
    loss = nn.functional.cross_entropy(logits.reshape(-1, len(chars)), y.reshape(-1))
    opt.zero_grad()
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()


@torch.no_grad()
def generate(model, prompt, length=80, temperature=0.8, top_k=5):
    model.eval()
    ids = torch.tensor([[stoi[c] for c in prompt]])
    logits, state = model(ids)                      # warm up the hidden state on the prompt
    out = list(prompt)
    for _ in range(length):
        last = logits[0, -1] / temperature
        if top_k:
            kth = torch.topk(last, top_k).values[-1]
            last = last.masked_fill(last < kth, float("-inf"))
        next_id = torch.multinomial(torch.softmax(last, dim=-1), 1)
        out.append(itos[next_id.item()])
        logits, state = model(next_id.view(1, 1), state)   # feed one character, keep state
    return "".join(out)


print(f"final training loss: {loss.item():.3f}")
print(generate(model, "to be or "))

Because the corpus repeats one passage, a trained model reproduces fragments of it; on a real corpus (books, code, song lyrics) you would hold out a validation set and track perplexity.

Effect of Temperature

import numpy as np

logits = np.array([2.0, 1.0, 0.0])
for T in (0.5, 1.0, 2.0):
    p = np.exp(logits / T) / np.exp(logits / T).sum()
    print(T, [f"{v:.3f}" for v in p])
# 0.5 ['0.867', '0.117', '0.016']   sharper: safer, more repetitive
# 1.0 ['0.665', '0.245', '0.090']   the model's own distribution
# 2.0 ['0.506', '0.307', '0.186']   flatter: more diverse, more mistakes

Common Pitfalls

  • Greedy decoding, which loops on the same phrase.

  • Very high temperature, producing gibberish.

  • Re-running the model on the whole generated text every step instead of carrying the hidden state.

  • Evaluating only by loss on the training text (memorization looks perfect).

  • Generating from a model trained on too little data, which copies the corpus.

Interview Tips

  • Describe next-token training with shifted targets, then the sampling loop with state.

  • Explain temperature, top-k and top-p.

  • Mention perplexity and human reading as evaluation, and transformers as the modern choice.

19

What should you consider when initializing RNN weights?

RNN initialization matters more than in feed-forward networks because the same recurrent weight matrix is multiplied at every time step: if its largest singular values are well above 1, signals and gradients explode; if well below 1, they vanish. Common good practice is: orthogonal initialization for the hidden-to-hidden (recurrent) weights, which keeps all singular values at 1 so activations neither grow nor shrink at the start of training; Xavier/Glorot (uniform or normal) for the input-to-hidden weights of tanh/sigmoid cells; zero biases, except the LSTM forget-gate bias set to about 1 so the cell initially remembers (Jozefowicz et al., 2015); for ReLU RNNs, identity initialization of the recurrent matrix with small input weights (IRNN); and small normal or pretrained values for embeddings. Keras uses most of these by default (glorot_uniform kernel, orthogonal recurrent kernel, unit_forget_bias=True); PyTorch’s default is a uniform distribution in ±1/√hidden for all weights, so applying the scheme yourself can help on long sequences.

Read full answer

Applies toPyTorch 2.x (LSTM gate order i, f, g, o); Keras 3 defaults noted

Recommended Scheme

Parameter Initialization Reason
weight_hh (recurrent) Orthogonal, per gate block Singular values = 1: stable signal propagation over time
weight_ih (input) Xavier/Glorot uniform Keeps input variance balanced for tanh/sigmoid
Biases 0 Neutral start
LSTM forget-gate bias 1 (sometimes 1-2) Gate starts near σ(1) ≈ 0.73: remember by default
GRU Orthogonal recurrent, Xavier input, zero bias Same logic
ReLU vanilla RNN Identity recurrent matrix (IRNN), small input weights Starts as an “accumulator”
Embeddings N(0, 0.1²)-ish or pretrained Avoid large initial activations
Output layer Xavier; bias = log class priors for imbalanced classification Faster early training

Code Example: Initializing an LSTM (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)


def init_lstm(lstm: nn.LSTM, forget_bias: float = 1.0) -> None:
    H = lstm.hidden_size
    for name, param in lstm.named_parameters():
        if "weight_ih" in name:
            for gate in param.data.chunk(4, dim=0):            # i, f, g, o blocks
                nn.init.xavier_uniform_(gate)
        elif "weight_hh" in name:
            for gate in param.data.chunk(4, dim=0):
                nn.init.orthogonal_(gate)
        elif "bias_ih" in name:
            param.data.zero_()
            param.data[H:2 * H] = forget_bias                 # forget gate slice
        elif "bias_hh" in name:
            param.data.zero_()                                # both biases are added, so set only one


lstm = nn.LSTM(input_size=16, hidden_size=64, num_layers=2, batch_first=True)
init_lstm(lstm)

w_f = lstm.weight_hh_l0.data[64:128]                          # forget-gate recurrent block (64 x 64)
print(torch.allclose(w_f @ w_f.T, torch.eye(64), atol=1e-5))  # True: orthogonal
print(lstm.bias_ih_l0.data[64:128].unique().tolist())          # [1.0]
print((lstm.bias_ih_l0.data[64:128] + lstm.bias_hh_l0.data[64:128]).mean().item())   # 1.0


# Effect over time: repeated multiplication by the recurrent matrix
def norm_after(W, steps=100):
    h = torch.ones(64) / 8                                    # unit-norm starting vector
    for _ in range(steps):
        h = W @ h
    return h.norm().item()


Q = torch.empty(64, 64)
nn.init.orthogonal_(Q)
print(f"orthogonal: {norm_after(Q):.3f}")                     # 1.000: norm preserved
print(f"0.9 x orthogonal: {norm_after(0.9 * Q):.2e}")         # 2.66e-05: vanishes (0.9^100)
print(f"1.1 x orthogonal: {norm_after(1.1 * Q):.2e}")         # 1.38e+04: explodes (1.1^100)

The last three lines show the core issue in isolation (no nonlinearity): with all singular values equal to s, the state norm after 100 steps is s¹⁰⁰ — 1 for an orthogonal matrix, 0.9¹⁰⁰ ≈ 2.66 × 10⁻⁵ and 1.1¹⁰⁰ ≈ 1.38 × 10⁴ otherwise. Gradients through time behave the same way.

Keras Defaults

import keras

layer = keras.layers.LSTM(64)       # kernel_initializer="glorot_uniform",
                                    # recurrent_initializer="orthogonal",
                                    # bias_initializer="zeros", unit_forget_bias=True
print(layer.unit_forget_bias)       # True

Common Pitfalls

  • Setting the forget bias in both bias_ih and bias_hh in PyTorch, which makes the effective bias 2.

  • Applying one orthogonal initialization to the whole stacked 4H × H matrix instead of per gate block (the blocks are then not individually orthogonal).

  • Large random initial weights in a vanilla RNN, giving exploding activations at step one.

  • Re-initializing pretrained embeddings by accident in a loop over all parameters.

  • Treating initialization as a substitute for gating, normalization and gradient clipping; they work together.

Interview Tips

  • Explain why repeated multiplication makes the recurrent matrix’s singular values critical.

  • Give the standard recipe: orthogonal recurrent, Xavier input, zero bias, forget bias 1.

  • Mention IRNN for ReLU RNNs and Keras/PyTorch defaults.

20

How do you prevent overfitting when training an RNN?

Prevent RNN overfitting by limiting what the model can memorize and stopping before it memorizes: add dropout in the right places (on embeddings/inputs, between stacked layers, and, carefully, on recurrent connections with variational dropout or DropConnect on hidden-to-hidden weights as in AWD-LSTM), use weight decay (AdamW), stop early on a validation metric, reduce capacity (hidden size, layers) when data is small, use pretrained embeddings or models, enlarge or augment the data, and keep the validation split honest (chronological for time series, by user or document where samples are related). Gradient clipping is not a regularizer, but it keeps training stable so that these other methods can work.

Read full answer

Applies toPyTorch 2.x (Keras has equivalents: `dropout`, `recurrent_dropout`, `EarlyStopping`, `kernel_regularizer`)

Techniques

Technique How Notes
Dropout on inputs/embeddings nn.Dropout before the RNN Simple and effective
Dropout between layers nn.LSTM(..., num_layers=2, dropout=0.3) Applied to outputs of all but the last layer
Variational / recurrent dropout Same mask across time steps; Keras recurrent_dropout Standard dropout on recurrent connections hurts memory
Weight drop (DropConnect) Drop entries of weight_hh (AWD-LSTM) Strong for language models
Weight decay AdamW weight_decay=1e-4..1e-2 Penalizes large weights
Early stopping Stop when validation stops improving; restore best weights Cheap and essential
Smaller model Fewer units/layers First thing to try on small data
Pretrained embeddings / transfer GloVe, fastText, pretrained LMs Less to learn from scratch
Data augmentation Text: synonym replacement, back-translation; time series: jitter, scaling, window slicing Must preserve labels
Proper splits No overlap of users/documents/time between splits Otherwise overfitting is hidden

Code Example: Regularized LSTM and Early Stopping (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)


class RegularizedLSTM(nn.Module):
    def __init__(self, vocab_size, emb=100, hidden=128, n_classes=2):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, emb, padding_idx=0)
        self.emb_dropout = nn.Dropout(0.3)                      # input dropout
        self.lstm = nn.LSTM(emb, hidden, num_layers=2, batch_first=True,
                            dropout=0.3)                        # between-layer dropout
        self.out_dropout = nn.Dropout(0.3)
        self.fc = nn.Linear(hidden, n_classes)

    def forward(self, ids):
        _, (h, _) = self.lstm(self.emb_dropout(self.embed(ids)))
        return self.fc(self.out_dropout(h[-1]))


class EarlyStopping:
    def __init__(self, patience=2, min_delta=0.0):
        self.patience, self.min_delta = patience, min_delta
        self.best, self.best_epoch, self.wait = float("inf"), -1, 0

    def step(self, val_loss, epoch):
        """Return True when training should stop."""
        if val_loss < self.best - self.min_delta:
            self.best, self.best_epoch, self.wait = val_loss, epoch, 0
            return False
        self.wait += 1
        return self.wait >= self.patience


model = RegularizedLSTM(vocab_size=5000)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)
print(model(torch.randint(1, 5000, (4, 30))).shape)      # torch.Size([4, 2])

# Early stopping on a recorded validation-loss curve
val_losses = [0.90, 0.70, 0.65, 0.66, 0.67, 0.64, 0.70]
stopper = EarlyStopping(patience=2)
for epoch, val in enumerate(val_losses):
    if stopper.step(val, epoch):
        print(f"stopped at epoch {epoch}; best epoch {stopper.best_epoch} (val loss {stopper.best})")
        break
# stopped at epoch 4; best epoch 2 (val loss 0.65)

With patience 2, the loss fails to improve on 0.65 at epochs 3 and 4, so training stops at epoch 4 and the weights saved at epoch 2 are restored. (A larger patience would have reached the slightly better 0.64 at epoch 5, which is the usual trade-off when choosing patience.)

Diagnosing Overfitting

Symptom Meaning
Training loss falls, validation loss rises Classic overfitting: regularize or stop earlier
Both losses high Underfitting: more capacity or better features
Validation much better than test Leaky or unrepresentative validation split
Large gaps between seeds High variance: more data, regularization, ensembling

Common Pitfalls

  • Applying standard (new mask per step) dropout to recurrent connections, which damages long-term memory.

  • Using Keras recurrent_dropout without knowing it disables the fast cuDNN kernel.

  • Early stopping on the test set.

  • Random splits of time series or of samples from the same user, hiding overfitting.

  • Leaving dropout on at inference (forgetting model.eval()).

Interview Tips

  • List techniques by where they act: data, architecture, training procedure.

  • Mention variational dropout / weight drop as RNN-specific.

  • Explain how to read training vs validation curves.

21

What metrics are commonly used to evaluate RNN performance?

There is no RNN-specific metric: an RNN is evaluated with the metric of the task it solves. For sequence classification use accuracy, precision, recall, F1 and ROC-AUC/PR-AUC; for sequence labelling (NER, POS tagging) use token accuracy and, more importantly, entity-level F1; for language modelling use perplexity (exp of the average cross-entropy per token) or bits per character; for translation and summarization use BLEU, chrF, ROUGE (plus human evaluation); for speech recognition use word error rate (WER) and character error rate (CER); for time-series forecasting use MAE, RMSE, MAPE/sMAPE and MASE, plus quantile/pinball loss for probabilistic forecasts; and for next-item recommendation use Recall@k/Hit@k, MRR and NDCG. Alongside quality, production RNNs are also judged on latency, throughput and memory.

Read full answer

Applies toPython 3.10+, NumPy; libraries such as scikit-learn, seqeval, sacrebleu and jiwer implement these

Metrics by Task

Task Metrics Notes
Sequence classification Accuracy, F1 (macro), ROC-AUC, PR-AUC Use F1/PR-AUC for imbalanced classes
Sequence labelling Entity-level precision/recall/F1 seqeval; token accuracy is inflated by “O” tags
Language modelling Perplexity, bits per character Lower is better; compare only with the same tokenizer
Translation BLEU, chrF, COMET sacrebleu for comparable BLEU
Summarization ROUGE-1/2/L Add human or model-based evaluation
Speech recognition WER, CER Edit distance / reference length
Forecasting MAE, RMSE, MASE, sMAPE, pinball loss Compare to a naive baseline
Recommendation Recall@k, MRR@k, NDCG@k Time-based evaluation split
Efficiency Latency (p50/p95), throughput, memory Grows with sequence length

Code Example: Computing Common Sequence Metrics (Python)

import math

import numpy as np


def perplexity(mean_cross_entropy_nats):
    return math.exp(mean_cross_entropy_nats)


def edit_distance(ref, hyp):
    d = np.zeros((len(ref) + 1, len(hyp) + 1), dtype=int)
    d[:, 0] = np.arange(len(ref) + 1)
    d[0, :] = np.arange(len(hyp) + 1)
    for i in range(1, len(ref) + 1):
        for j in range(1, len(hyp) + 1):
            cost = 0 if ref[i - 1] == hyp[j - 1] else 1
            d[i, j] = min(d[i - 1, j] + 1, d[i, j - 1] + 1, d[i - 1, j - 1] + cost)
    return d[-1, -1]


def wer(reference, hypothesis):
    ref, hyp = reference.split(), hypothesis.split()
    return edit_distance(ref, hyp) / len(ref)


def mase(y_true, y_pred, y_train, season=1):
    naive_mae = np.mean(np.abs(y_train[season:] - y_train[:-season]))
    return np.mean(np.abs(y_true - y_pred)) / naive_mae


print(f"perplexity = {perplexity(2.0):.3f}")                                   # 7.389
print(f"WER = {wer('the cat sat on the mat', 'the cat sit on mat'):.3f}")      # 0.333
y_train = np.array([1.0, 2.0, 4.0, 3.0])
y_true, y_pred = np.array([3.0, 5.0, 4.0, 6.0]), np.array([3.5, 4.5, 4.5, 5.5])
print(f"MAE = {np.mean(np.abs(y_true - y_pred)):.3f}")                          # 0.500
print(f"MASE = {mase(y_true, y_pred, y_train):.3f}")                            # 0.375

Worked numbers: a mean cross-entropy of 2.0 nats gives perplexity e² ≈ 7.389 (the model is as uncertain as a uniform choice among about 7.4 tokens). The hypothesis has one substitution (sat → sit) and one deletion (the second “the”), so WER = 2/6. The naive one-step MAE on the training series is (1 + 2 + 1)/3 = 4/3, so MASE = 0.5 / (4/3) = 0.375 (below 1 means better than naive).

Choosing Well

  • Pick the metric that matches the business cost (missing a fraud vs. a false alarm).

  • Report at least one metric that is comparable with simple baselines.

  • For generation, automatic metrics are proxies; sample outputs and check them by hand.

  • Report efficiency metrics by sequence-length bucket.

Common Pitfalls

  • Accuracy on imbalanced data or token accuracy for NER.

  • Comparing perplexities across different tokenizers or vocabularies.

  • Computing BLEU with different tokenization than the published numbers (use sacrebleu).

  • MAPE when actual values can be zero or near zero.

  • Evaluating on padded positions.

Interview Tips

  • Say “the metric follows the task” and give two or three task/metric pairs.

  • Define perplexity and WER precisely.

  • Mention efficiency metrics, since RNN latency depends on sequence length.

22

How do you assess how different RNN architectures affect performance?

Assess RNN architectures with controlled experiments: hold the data splits, preprocessing, training budget and tuning effort fixed, change one architectural factor at a time (cell type RNN/GRU/LSTM, hidden size, depth, bidirectionality, attention, residual connections), train each variant with several random seeds, and compare them on the validation metric together with cost — parameters, training time, inference latency and memory. Report means with standard deviations or confidence intervals, check whether differences are larger than seed-to-seed noise (paired tests on the same splits), match parameter counts when comparing cell types, and include strong baselines (a simple non-recurrent model and, where relevant, a 1-D CNN or transformer). Finally, analyse where each architecture wins: by sequence length, by segment, or by error type.

Read full answer

Applies toPyTorch 2.x, NumPy

Experimental Design

Principle Practice
Same data Identical train/validation/test splits (chronological for time series)
One change at a time Ablations: + bidirectional, + attention, 1 → 2 layers
Fair tuning Same search budget per architecture (or tune the learning rate for each)
Seeds 3-10 seeds per configuration; report mean ± std
Size control Compare at similar parameter counts as well as at “best” settings
Cost Parameters, training time per epoch, latency, peak memory
Slices Metric by sequence length, class or segment
Final check Test set only for the selected configuration(s)

Code Example: Seeded Comparison of RNN, GRU and LSTM (PyTorch)

import time

import numpy as np
import torch
from torch import nn

rng = np.random.default_rng(42)

# Task: remember a value from the start of the sequence (long-range dependency)
SEQ_LEN, N = 50, 2000
X = rng.standard_normal((N, SEQ_LEN, 1)).astype(np.float32)
y = X[:, 0, 0] > 0                                   # label depends only on the first step
X, y = torch.tensor(X), torch.tensor(y, dtype=torch.long)
X_tr, y_tr, X_va, y_va = X[:1600], y[:1600], X[1600:], y[1600:]


class Classifier(nn.Module):
    def __init__(self, cell, hidden=32):
        super().__init__()
        self.rnn = {"rnn": nn.RNN, "gru": nn.GRU, "lstm": nn.LSTM}[cell](1, hidden, batch_first=True)
        self.fc = nn.Linear(hidden, 2)

    def forward(self, x):
        out, _ = self.rnn(x)
        return self.fc(out[:, -1])


def run(cell, seed, epochs=15):
    torch.manual_seed(seed)
    model = Classifier(cell)
    opt = torch.optim.Adam(model.parameters(), lr=3e-3)
    start = time.perf_counter()
    for _ in range(epochs):
        for i in range(0, len(X_tr), 64):
            opt.zero_grad()
            loss = nn.functional.cross_entropy(model(X_tr[i:i + 64]), y_tr[i:i + 64])
            loss.backward()
            nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            opt.step()
    train_time = time.perf_counter() - start
    with torch.no_grad():
        acc = (model(X_va).argmax(1) == y_va).float().mean().item()
    params = sum(p.numel() for p in model.parameters())
    return acc, train_time, params


for cell in ["rnn", "gru", "lstm"]:
    results = [run(cell, seed) for seed in (0, 1, 2)]
    accs = np.array([r[0] for r in results])
    print(f"{cell:>4}: val acc {accs.mean():.3f} ± {accs.std(ddof=1):.3f}   "
          f"params {results[0][2]}   train time {np.mean([r[1] for r in results]):.1f}s")
# Parameter counts are fixed by the architecture: rnn 1186, gru 3426, lstm 4546

This task needs memory over 50 steps, so it separates the cells: gated cells (GRU, LSTM) are expected to cope better than the vanilla RNN, whose gradients vanish. Check the accuracies across seeds rather than trusting one run, and note the GRU/LSTM cost 3-4× the parameters of the vanilla RNN at the same hidden size.

Parameter counts follow from the cell type: a vanilla RNN layer with input 1 and hidden 32 has 32·1 + 32·32 + 2·32 = 1,120 weights, a GRU has 3× and an LSTM 4× that, and the Linear(32, 2) head adds 66, giving 1,186, 3,426 and 4,546.

Reporting Template

Architecture Params Val metric (mean ± std, 3 seeds) Train time/epoch Latency (p95)
GRU, 1×64 … … … …
LSTM, 1×64 … … … …
LSTM, 2×64 + attention … … … …

Common Pitfalls

  • Comparing a tuned favourite against untuned alternatives.

  • Single-seed comparisons where the difference is within noise.

  • Different preprocessing or sequence lengths per model.

  • Ignoring cost: a 1% gain may not justify 3× latency.

  • Selecting on the test set.

Interview Tips

  • Describe a controlled ablation with seeds and fixed splits.

  • Mention parameter-matched comparisons and cost metrics.

  • Explain slicing results (e.g. by sequence length) to understand why one architecture wins.

23

How do you tune the hyperparameters of an RNN model?

Tune an RNN by deciding which hyperparameters matter most (learning rate first, then hidden size and number of layers, dropout, sequence length and batch size, cell type, and gradient-clipping threshold), defining sensible search ranges (log scale for learning rate and weight decay), choosing an efficient search method (random search or Bayesian optimization such as Optuna’s TPE, with early-stopping/pruning of bad trials via ASHA or Hyperband), and scoring each trial on a fixed validation set that respects the data’s structure (chronological for time series). Keep the test set untouched until the end, repeat the best configurations with several seeds, and track every trial (Optuna, Weights & Biases, MLflow) so results are reproducible.

Read full answer

Applies toPyTorch 2.x with Optuna 3.x/4.x (the same approach works with Keras Tuner or Ray Tune)

Hyperparameters and Typical Ranges

Hyperparameter Typical range Notes
Learning rate 1e-4 to 1e-2 (log scale) Most important; tune first
Hidden size 32 to 512 Larger = more capacity and cost
Layers 1 to 3 Gains flatten after 2-3
Dropout (between layers / embeddings) 0.0 to 0.5 Increase if overfitting
Weight decay 1e-6 to 1e-2 (log) AdamW
Batch size 32 to 256 Interacts with learning rate
Sequence / window length Task-dependent Longer = more context, slower
Gradient clipping norm 0.5 to 5 Stabilizes training
Cell type RNN / GRU / LSTM GRU is a good fast default
Bidirectional Yes / no Only if future context is available

Code Example: Optuna with Pruning (PyTorch)

import numpy as np
import optuna
import torch
from torch import nn

rng = np.random.default_rng(42)
torch.manual_seed(42)

# Synthetic task: predict the next value of a noisy sine wave from 30 steps
series = np.sin(np.arange(2000) * 0.1) + 0.1 * rng.standard_normal(2000)
X = np.stack([series[i:i + 30] for i in range(1900)])
y = series[30:1930]
X = torch.tensor(X, dtype=torch.float32).unsqueeze(-1)
y = torch.tensor(y, dtype=torch.float32)
X_tr, y_tr, X_va, y_va = X[:1500], y[:1500], X[1500:], y[1500:]   # chronological split


class Model(nn.Module):
    def __init__(self, cell, hidden, layers, dropout):
        super().__init__()
        rnn_cls = {"gru": nn.GRU, "lstm": nn.LSTM}[cell]
        self.rnn = rnn_cls(1, hidden, num_layers=layers, batch_first=True,
                           dropout=dropout if layers > 1 else 0.0)
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):
        out, _ = self.rnn(x)
        return self.head(out[:, -1]).squeeze(-1)


def objective(trial):
    torch.manual_seed(trial.number)
    params = {
        "cell": trial.suggest_categorical("cell", ["gru", "lstm"]),
        "hidden": trial.suggest_int("hidden", 16, 128, log=True),
        "layers": trial.suggest_int("layers", 1, 2),
        "dropout": trial.suggest_float("dropout", 0.0, 0.4),
    }
    lr = trial.suggest_float("lr", 1e-4, 1e-2, log=True)
    clip = trial.suggest_float("clip", 0.5, 5.0)
    model = Model(**params)
    opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=1e-4)
    for epoch in range(30):
        model.train()
        for i in range(0, len(X_tr), 64):
            opt.zero_grad()
            loss = nn.functional.mse_loss(model(X_tr[i:i + 64]), y_tr[i:i + 64])
            loss.backward()
            nn.utils.clip_grad_norm_(model.parameters(), clip)
            opt.step()
        model.eval()
        with torch.no_grad():
            val = nn.functional.mse_loss(model(X_va), y_va).item()
        trial.report(val, epoch)
        if trial.should_prune():                  # stop clearly bad trials early
            raise optuna.TrialPruned()
    return val


study = optuna.create_study(
    direction="minimize",
    sampler=optuna.samplers.TPESampler(seed=42),
    pruner=optuna.pruners.MedianPruner(n_warmup_steps=5),
)
study.optimize(objective, n_trials=20)
print("best params:", study.best_params)
print(f"best validation MSE: {study.best_value:.4f}")

The validation score of the best trial is optimistic because it was used for selection; retrain the chosen configuration with a few seeds and report the test score once.

Search Strategy

Method When
Manual / learning-rate range test First sanity checks; find a working learning rate
Random search Few dimensions matter; easy to parallelize
Bayesian (TPE, GP) Expensive trials; 20-100 trials
ASHA / Hyperband / median pruning Many trials; stop poor ones early
Grid search Only for 1-2 discrete choices

Common Pitfalls

  • Tuning on the test set, or reusing it for many decisions.

  • Random (shuffled) validation splits for time series.

  • Searching learning rate on a linear scale.

  • Judging configurations from a single seed when differences are small.

  • Changing data preprocessing between trials without recording it.

Interview Tips

  • Name the order of importance: learning rate, capacity (hidden/layers), regularization, sequence length.

  • Mention Bayesian search with pruning and a proper validation split.

  • Say you confirm the winner with multiple seeds and a single final test evaluation.

24

How can you visualize and interpret RNN models and their predictions?

RNN interpretation combines input attribution — which time steps or tokens drove a prediction — with inspection of the model’s internals. For attribution, the main tools are gradient-based saliency (gradient of the output with respect to each input step, often multiplied by the input), Integrated Gradients (gradients averaged along a path from a baseline, available in Captum), occlusion (mask one step or token and measure how the prediction changes) and attention weights when the model has attention (useful but not a full explanation). For internals, you can plot hidden-state heatmaps and PCA trajectories, look at gate activations (LSTM forget/input gates), find individual units that track features (quote marks, line length, sentiment), and train probing classifiers on hidden states. For forecasting models, compare predictions with actuals and plot error by horizon and by segment.

Read full answer

Applies toPyTorch 2.x; Captum for Integrated Gradients and related methods; Matplotlib for plots

Methods

Method Answers Cost
Gradient × input saliency Which steps the prediction is locally sensitive to 1 backward pass
Integrated Gradients Attribution relative to a baseline, satisfies completeness ~20-300 passes
Occlusion / leave-one-out How much removing a step changes the output 1 pass per step
Attention weights Where an attention model looked Free
Hidden-state plots How the internal state evolves 1 pass
Gate activations When the LSTM forgets or writes Step-by-step pass
Probing classifiers What information hidden states contain Train a small classifier

Code Example: Saliency and Occlusion for a Sequence Classifier (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

vocab = ["<pad>", "<unk>", "the", "food", "was", "not", "good", "very", "slow", "service"]
stoi = {w: i for i, w in enumerate(vocab)}


class SentimentLSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(len(vocab), 32, padding_idx=0)
        self.lstm = nn.LSTM(32, 64, batch_first=True)
        self.fc = nn.Linear(64, 2)

    def forward_embedded(self, emb):
        _, (h, _) = self.lstm(emb)
        return self.fc(h[-1])

    def forward(self, ids):
        return self.forward_embedded(self.embed(ids))


model = SentimentLSTM().eval()        # in practice: a trained model
tokens = "the food was not good".split()
ids = torch.tensor([[stoi[w] for w in tokens]])
target_class = 1                      # e.g. "positive"

# 1. Gradient x input saliency on the embeddings
emb = model.embed(ids).detach().requires_grad_(True)
score = model.forward_embedded(emb)[0, target_class]
score.backward()
saliency = (emb.grad * emb).sum(dim=-1).squeeze(0)          # one value per token


# 2. Occlusion: replace each token with <unk> and measure the probability change
@torch.no_grad()
def occlusion(model, ids, target):
    base = model(ids).softmax(-1)[0, target]
    changes = []
    for i in range(ids.shape[1]):
        masked = ids.clone()
        masked[0, i] = stoi["<unk>"]
        changes.append((base - model(masked).softmax(-1)[0, target]).item())
    return torch.tensor(changes)


occ = occlusion(model, ids, target_class)
for word, s, o in zip(tokens, saliency.tolist(), occ.tolist()):
    print(f"{word:>5}  saliency {s:+.4f}  occlusion {o:+.4f}")
print(saliency.shape, occ.shape)      # torch.Size([5]) torch.Size([5])

For a trained sentiment model, a useful sanity check is that “not” and “good” receive large attributions of opposite or interacting sign; for this untrained model the numbers are arbitrary. Captum’s LayerIntegratedGradients(model, model.embed) gives Integrated Gradients on the embedding layer with a padding baseline.

Visual Outputs

Plot Shows
Token heatmap (text coloured by attribution) Which words drove the decision
Attribution over time (line plot) Which time steps mattered for a forecast or alert
Attention matrix (target × source) Alignment in translation models
Hidden-state heatmap / PCA trajectory Internal dynamics
Error by horizon / segment Where a forecaster fails

Common Pitfalls

  • Treating attention weights as the explanation; they can disagree with gradient or occlusion attributions.

  • Interpreting raw gradients without a baseline (saturation hides important inputs).

  • Occluding with a token the model never saw in training, producing out-of-distribution effects; compare with several replacements.

  • Explaining an untrained or poorly performing model.

  • Drawing conclusions from one example; aggregate attributions over many.

Interview Tips

  • Separate “which inputs mattered” (attribution) from “what the model represents” (internals).

  • Name Integrated Gradients and occlusion, and the limits of attention as explanation.

  • Mention sanity checks (randomized-model comparison, aggregation over examples).

25

How would you use RNNs for a time-series forecasting task?

To forecast with an RNN, turn the series into supervised windows (the last L observations, plus known covariates such as calendar features, → the next H values), scale them with statistics from the training period only, split the data chronologically into train, validation and test, train an LSTM or GRU that reads the window and outputs the forecast (directly all H steps, or one step fed back recursively), and compare its errors against simple baselines such as the naive and seasonal-naive forecasts. Evaluate with MAE/RMSE and scale-free metrics such as MASE on a held-out final period or with rolling-origin backtesting. RNNs are worth it when there is enough data, nonlinear dynamics, many related series (one global model) or rich covariates; for short single series, ARIMA/ETS or gradient boosting on lag features are often as good.

Read full answer

Applies toPyTorch 2.x, NumPy; ideas also used by DeepAR and other global RNN forecasters

Workflow

Step Detail
1. Frame the problem Horizon H, input length L, frequency, covariates known in the future
2. Split by time Train → validation → test; never shuffle across time
3. Scale Fit the scaler on train only; per-series scaling for many series
4. Window X[t] = y[t−L+1 .. t], Y[t] = y[t+1 .. t+H]
5. Model LSTM/GRU → linear head with H outputs (direct multi-step)
6. Baselines Naive (last value), seasonal naive, ETS/ARIMA
7. Evaluate MAE, RMSE, MASE; backtest over several cut-off dates
8. Uncertainty Quantile loss or a distribution head (DeepAR-style)

Code Example (PyTorch)

import numpy as np
import torch
from torch import nn

torch.manual_seed(42)
rng = np.random.default_rng(42)

# Synthetic daily series: trend + weekly seasonality + noise
n = 730
t = np.arange(n)
series = 10 + 0.01 * t + 2 * np.sin(2 * np.pi * t / 7) + 0.5 * rng.standard_normal(n)

L, H = 28, 7                                    # 4 weeks in, 1 week out
train_end, val_end = 550, 640

mean, std = series[:train_end].mean(), series[:train_end].std()   # train statistics only
scaled = (series - mean) / std


def windows(x, start, end):
    X, Y = [], []
    for i in range(start, end - H + 1):
        if i - L < 0:
            continue
        X.append(x[i - L:i])
        Y.append(x[i:i + H])
    return (torch.tensor(np.array(X), dtype=torch.float32).unsqueeze(-1),
            torch.tensor(np.array(Y), dtype=torch.float32))


X_tr, Y_tr = windows(scaled, 0, train_end)
X_va, Y_va = windows(scaled, train_end, val_end)
X_te, Y_te = windows(scaled, val_end, n)


class LSTMForecaster(nn.Module):
    def __init__(self, hidden=64):
        super().__init__()
        self.lstm = nn.LSTM(1, hidden, num_layers=2, batch_first=True, dropout=0.1)
        self.head = nn.Linear(hidden, H)          # direct multi-step: all 7 days at once

    def forward(self, x):
        out, _ = self.lstm(x)
        return self.head(out[:, -1])


model = LSTMForecaster()
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
best_val, best_state = float("inf"), None
for epoch in range(80):
    model.train()
    opt.zero_grad()
    loss = nn.functional.l1_loss(model(X_tr), Y_tr)
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()
    model.eval()
    with torch.no_grad():
        val = nn.functional.l1_loss(model(X_va), Y_va).item()
    if val < best_val:
        best_val, best_state = val, {k: v.clone() for k, v in model.state_dict().items()}
model.load_state_dict(best_state)                 # keep the epoch with the best validation MAE

with torch.no_grad():
    pred = model(X_te).numpy() * std + mean       # back to original units
truth = Y_te.numpy() * std + mean
last_week = X_te[:, -7:, 0].numpy() * std + mean  # seasonal naive: repeat the last 7 days

mae_lstm = np.abs(pred - truth).mean()
mae_snaive = np.abs(last_week - truth).mean()
print(f"test MAE  LSTM: {mae_lstm:.3f}   seasonal naive: {mae_snaive:.3f}")
print(pred.shape)                                 # (84, 7): 84 forecast origins x 7 days

The model is selected on the validation period and judged once on the test period, in original units, against the seasonal-naive baseline; only beating that baseline justifies the extra complexity.

Multi-Step Strategies

Strategy How Trade-off
Recursive Predict 1 step, feed it back, repeat One model; errors compound
Direct (multi-output) Head outputs H values No compounding; ignores dependence between outputs
Seq2seq Encoder over history, decoder over horizon (with future covariates) Flexible; more complex

Common Pitfalls

  • Shuffling or randomly splitting time-series windows (future leaks into training).

  • Scaling with the full series’ mean and standard deviation.

  • Reporting only scaled-space errors; convert back to original units.

  • No naive or seasonal-naive baseline.

  • Using covariates that are not known at forecast time (e.g. the actual weather of tomorrow).

Interview Tips

  • Walk through windowing, chronological split, train-only scaling and baselines.

  • Explain direct vs recursive multi-step forecasting.

  • Mention backtesting and MASE, and when classical models are enough.

26

What should you consider when using RNNs for NLP tasks?

Using RNNs for NLP well depends mostly on decisions around the network: how text is tokenized (words, subwords or characters, and how unknown words are handled), how variable-length sentences are padded and masked or packed so padding does not affect results, which embeddings are used (random vs pretrained), which architecture fits the task (bidirectional for classification and tagging, unidirectional for generation and streaming, encoder-decoder with attention for translation), how long sequences are truncated, and how the model is regularized and evaluated. RNNs also have known limits for language: they process tokens sequentially (slow training on long texts), struggle with very long-range dependencies, and are usually outperformed by pretrained transformers, so a strong pretrained baseline should always be part of the comparison.

Read full answer

Applies toPyTorch 2.x (same considerations in TensorFlow/Keras)

Checklist

Area Consideration Typical choice
Tokenization Word vs subword vs character; casing; punctuation Subword (BPE/WordPiece/SentencePiece)
Vocabulary Size, <unk> handling, built on training data only 20k-50k subwords
Sequence length Truncation point; head vs tail of document Cover the 95th percentile of lengths
Padding Masking or packing so padding is ignored pack_padded_sequence / mask_zero=True
Embeddings Random vs pretrained (GloVe, fastText) vs contextual Pretrained when data is small
Direction Bidirectional for full-sentence tasks BiLSTM for classification/NER
Task head Many-to-one, many-to-many, encoder-decoder Matches the task
Regularization Dropout on embeddings and between layers, weight decay 0.2-0.5 dropout
Imbalance Class weights, resampling, threshold tuning Weighted loss
Evaluation Task-appropriate metric F1, entity F1, BLEU, perplexity
Baseline TF-IDF + logistic regression; pretrained transformer Always compare

Code Example: BiLSTM Text Classifier with Packing (PyTorch)

import torch
from torch import nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_sequence

torch.manual_seed(42)

vocab = {"<pad>": 0, "<unk>": 1, "the": 2, "movie": 3, "was": 4, "great": 5, "awful": 6, "plot": 7}


def encode(text):
    return torch.tensor([vocab.get(w, vocab["<unk>"]) for w in text.lower().split()])


class BiLSTMClassifier(nn.Module):
    def __init__(self, vocab_size, emb=50, hidden=64, n_classes=2):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, emb, padding_idx=0)
        self.emb_dropout = nn.Dropout(0.3)
        self.lstm = nn.LSTM(emb, hidden, batch_first=True, bidirectional=True)
        self.fc = nn.Linear(2 * hidden, n_classes)

    def forward(self, ids, lengths):
        x = self.emb_dropout(self.embed(ids))
        packed = pack_padded_sequence(x, lengths.cpu(), batch_first=True, enforce_sorted=False)
        _, (h, _) = self.lstm(packed)                 # h: (2, batch, hidden) for 1 bidirectional layer
        sentence = torch.cat([h[0], h[1]], dim=-1)    # last forward state + last backward state
        return self.fc(sentence)


texts = ["The movie was great", "awful plot", "the plot was surprisingly great"]
ids = [encode(t) for t in texts]
lengths = torch.tensor([len(x) for x in ids])
batch = pad_sequence(ids, batch_first=True)          # (3, 5), zeros as padding

model = BiLSTMClassifier(len(vocab))
print(batch.tolist())
# [[2, 3, 4, 5, 0], [6, 7, 0, 0, 0], [2, 7, 4, 1, 5]]   "surprisingly" -> <unk> (1)
print(model(batch, lengths).shape)                   # torch.Size([3, 2])

Packing makes the forward direction stop at each sentence’s last real token and the backward direction start there, so padding never influences the sentence vector.

Architecture by Task

Task Setup
Sentiment / topic classification BiLSTM/GRU → pooled or final states → softmax
Named-entity recognition / POS tagging BiLSTM over tokens → per-token labels (often + CRF layer)
Language modelling / generation Unidirectional LSTM predicting the next token
Translation / summarization Encoder-decoder with attention
Streaming (live captions, chat) Unidirectional, stateful

Common Pitfalls

  • Building the vocabulary or fitting tokenizers on validation/test data.

  • Ignoring padding (no masking/packing), so short sentences are dominated by pad tokens.

  • Taking output[:, -1] of a padded batch, which is a padding position for short sentences.

  • Truncating long documents from the wrong end (reviews often state the verdict at the end).

  • Using a bidirectional model for a task that must run left-to-right in real time.

  • Skipping a pretrained transformer baseline.

Interview Tips

  • Organize the answer: data (tokenization, padding), model (direction, embeddings, head), training (regularization, imbalance), evaluation.

  • Mention packing/masking explicitly; it is a frequent bug source.

  • Be honest that pretrained transformers usually win, and say when an RNN is still the right choice.

27

How can RNNs be used for speech recognition?

In speech recognition, RNNs turn a sequence of acoustic feature frames (for example 80 log-mel filterbank values every 10 ms) into a sequence of characters, subwords or words: a stack of (often bidirectional) LSTMs or GRUs encodes the audio frames into hidden states that capture context across time, and an output layer maps them to text units. Because audio frames and text have different lengths and no frame-level alignment is given, RNN systems are trained with one of three objectives: CTC (Connectionist Temporal Classification), which sums over all alignments using a blank symbol (DeepSpeech 2); attention-based encoder-decoder models, where a decoder RNN attends over encoder states (Listen, Attend and Spell); and the RNN-Transducer (RNN-T), which combines an audio encoder with a prediction network over previous tokens and supports streaming recognition, long used for on-device speech recognition. Modern state-of-the-art systems mostly use transformer or Conformer encoders (e.g. Whisper), but RNN-T decoding and streaming ideas remain widespread.

Read full answer

Applies toPyTorch 2.x, torchaudio 2.x

Approaches

Approach How alignment is handled Streaming? Examples
RNN + CTC Blank symbol; sum over all alignments Yes (unidirectional encoder) DeepSpeech 2
Attention encoder-decoder Decoder attends to all encoder states Hard (needs full utterance) LAS
RNN-Transducer Joint network over audio frames and previous tokens Yes On-device ASR
Hybrid HMM + RNN acoustic model Forced alignments from an HMM system Yes Older production systems

Code Example: BiLSTM + CTC Acoustic Model (PyTorch)

import torch
import torchaudio
from torch import nn

torch.manual_seed(42)

SAMPLE_RATE = 16_000
VOCAB = ["<blank>", " ", "'"] + [chr(c) for c in range(ord("a"), ord("z") + 1)]   # 29 symbols
char_to_id = {c: i for i, c in enumerate(VOCAB)}

features = nn.Sequential(
    torchaudio.transforms.MelSpectrogram(sample_rate=SAMPLE_RATE, n_fft=400, hop_length=160, n_mels=80),
    torchaudio.transforms.AmplitudeToDB(),
)


class CTCSpeechModel(nn.Module):
    def __init__(self, n_mels=80, hidden=256, n_tokens=len(VOCAB)):
        super().__init__()
        self.lstm = nn.LSTM(n_mels, hidden, num_layers=3, batch_first=True,
                            bidirectional=True, dropout=0.2)
        self.out = nn.Linear(2 * hidden, n_tokens)

    def forward(self, mels):                      # (batch, frames, n_mels)
        h, _ = self.lstm(mels)
        return self.out(h).log_softmax(dim=-1)    # (batch, frames, n_tokens)


def greedy_ctc_decode(ids, blank=0):
    """Collapse repeated symbols, then drop blanks."""
    out, prev = [], None
    for i in ids:
        if i != prev and i != blank:
            out.append(i)
        prev = i
    return out


# Two fake utterances of 1 s and 0.8 s with their transcripts
waves = [torch.randn(16_000), torch.randn(12_800)]
texts = ["hello world", "yes"]

mels = [features(w).transpose(0, 1) for w in waves]          # each: (frames, 80)
mel_lengths = torch.tensor([m.shape[0] for m in mels])
mels = nn.utils.rnn.pad_sequence(mels, batch_first=True)
targets = torch.tensor([char_to_id[c] for t in texts for c in t])
target_lengths = torch.tensor([len(t) for t in texts])

model = CTCSpeechModel()
log_probs = model(mels)                                       # (batch, frames, tokens)
ctc = nn.CTCLoss(blank=0, zero_infinity=True)
loss = ctc(log_probs.transpose(0, 1), targets, mel_lengths, target_lengths)   # CTC wants (T, B, C)
print(mels.shape)                       # torch.Size([2, 101, 80])
print(log_probs.shape)                  # torch.Size([2, 101, 29])
print(loss.item() > 0)                  # True

# Greedy CTC decoding on hand-made frame IDs: "a a _ a c c _" -> a, a, c
print(greedy_ctc_decode([3, 3, 0, 3, 5, 5, 0]))   # [3, 3, 5]

With a hop of 160 samples (10 ms), 1 s of audio gives 101 frames (the centred STFT adds one frame). The blank symbol lets CTC emit the same character twice in a row (“ll” in “hello”) by separating repeats with a blank.

Pipeline Around the Model

Stage Practice
Features 80-dim log-mel, per-utterance or global normalization; SpecAugment (time/frequency masking)
Subsampling Stride or frame stacking to cut frames by 2-4× before the RNN (faster, better CTC)
Decoding Beam search with an external n-gram or neural language model
Evaluation Word error rate (WER) and character error rate (CER)
Streaming Unidirectional LSTM or RNN-T; chunked processing with limited look-ahead

Common Pitfalls

  • Using a bidirectional encoder in a system that must stream (it needs future audio).

  • Input lengths shorter than target lengths after subsampling (CTC loss becomes infinite).

  • Forgetting to pass true (unpadded) input lengths to the CTC loss.

  • Reporting accuracy instead of WER/CER.

  • Training on clean audio and deploying in noise without augmentation.

Interview Tips

  • Explain the alignment problem and how CTC, attention and RNN-T solve it.

  • Mention log-mel features, SpecAugment and WER.

  • Note that Conformer/transformer encoders now lead, while RNN-T remains key for streaming.

28

How can RNNs be applied to video frame prediction?

Video frame prediction asks a model to generate the next frame(s) of a video from the previous ones, and recurrent models handle it by carrying a spatial memory through time: the most common choice is the ConvLSTM, an LSTM whose matrix multiplications are replaced by convolutions, so its hidden and cell states are feature maps (channels × height × width) that keep spatial layout while modelling motion over time. A typical model encodes each frame with convolutions, updates stacked ConvLSTM layers frame by frame, and decodes the hidden state into the predicted frame; for multi-step prediction it feeds its own predictions back in. Extensions such as PredRNN, PredRNN++, E3D-LSTM and MIM add spatiotemporal memory flows. Applications include precipitation nowcasting (the original ConvLSTM paper), robot planning, autonomous-driving perception, video compression and anomaly detection in surveillance; recent models also use transformers and diffusion, but ConvLSTM remains a strong, simple baseline.

Read full answer

Applies toPyTorch 2.x; ConvLSTM (Shi et al., 2015), PredRNN (Wang et al., 2017)

Approaches

Approach Idea
CNN encoder → LSTM → CNN decoder Flatten each frame to a vector; LSTM models the dynamics
ConvLSTM Convolutional gates; states are feature maps (keeps spatial structure)
PredRNN / PredRNN++ Memory flows both through time and up and down the layer stack
Stochastic models (SVG, SV2P) Latent variables to represent uncertain futures
Transformers / diffusion Strong recent alternatives for high-quality video

Code Example: ConvLSTM Next-Frame Predictor (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)


class ConvLSTMCell(nn.Module):
    def __init__(self, in_ch, hid_ch, kernel=3):
        super().__init__()
        self.hid_ch = hid_ch
        # One convolution computes all four gates from [input, hidden]
        self.conv = nn.Conv2d(in_ch + hid_ch, 4 * hid_ch, kernel, padding=kernel // 2)

    def forward(self, x, state):
        h, c = state
        i, f, g, o = torch.chunk(self.conv(torch.cat([x, h], dim=1)), 4, dim=1)
        c = torch.sigmoid(f) * c + torch.sigmoid(i) * torch.tanh(g)
        h = torch.sigmoid(o) * torch.tanh(c)
        return h, c


class FramePredictor(nn.Module):
    def __init__(self, channels=1, hidden=32):
        super().__init__()
        self.cell1 = ConvLSTMCell(channels, hidden)
        self.cell2 = ConvLSTMCell(hidden, hidden)
        self.decode = nn.Conv2d(hidden, channels, kernel_size=1)

    def forward(self, frames, future=0):
        # frames: (batch, time, channels, H, W)
        b, t, _, height, width = frames.shape
        zeros = frames.new_zeros(b, self.cell1.hid_ch, height, width)
        s1, s2 = (zeros, zeros), (zeros, zeros)
        preds = []
        x = frames[:, 0]
        for step in range(t + future):
            if step < t:
                x = frames[:, step]                    # observed frame
            s1 = self.cell1(x, s1)
            s2 = self.cell2(s1[0], s2)
            x = torch.sigmoid(self.decode(s2[0]))      # predicted next frame in [0, 1]
            preds.append(x)                            # after the observed part, x is fed back
        return torch.stack(preds, dim=1)


model = FramePredictor()
video = torch.rand(2, 10, 1, 64, 64)                    # 2 clips, 10 frames, 64x64 grayscale
pred = model(video[:, :-1], future=0)                   # predict frames 2..10 from frames 1..9
loss = nn.functional.mse_loss(pred, video[:, 1:])
print(pred.shape)                                       # torch.Size([2, 9, 1, 64, 64])

rollout = model(video, future=5)                        # observe 10 frames, then 5 steps on its own outputs
print(rollout[:, 10:].shape)                            # torch.Size([2, 5, 1, 64, 64]) predictions from fed-back frames
print(loss.item() > 0)                                  # True

A standard toy benchmark is Moving MNIST (digits bouncing in a 64×64 frame): the model sees 10 frames and predicts the next 10.

Training and Evaluation

Topic Practice
Loss MSE or L1; add SSIM or perceptual loss to reduce blur
Multi-step Scheduled sampling: gradually replace ground-truth inputs with predictions
Metrics MSE/MAE, PSNR, SSIM, LPIPS; FVD for realism
Data Normalize pixels; keep sequences in temporal order; split by video, not by frame
Compute ConvLSTM memory grows with resolution × time; downsample or use patch encoders

Common Pitfalls

  • Blurry predictions from pure MSE loss, which averages over possible futures.

  • Training only with ground-truth inputs (teacher forcing), then predictions drift when fed back.

  • Splitting frames of the same video between train and test.

  • Using a flat LSTM on high-resolution frames (loses spatial structure and has huge weight matrices).

Interview Tips

  • Explain ConvLSTM: convolutional gates and spatial hidden states.

  • Mention encoder-decoder rollout, scheduled sampling and blur from MSE.

  • Name an application (precipitation nowcasting) and metrics (PSNR, SSIM).

29

Give an example of using RNNs in a recommendation system.

The classic example is session-based recommendation with GRU4Rec: a user’s clicks in the current session (product A → B → C) are fed to a GRU one item at a time, and after each click the model scores every item in the catalogue as the likely next click; the top-k scores become the “recommended for you now” list. This works without long-term user profiles (useful for anonymous visitors), captures order and short-term intent (someone comparing phones after viewing a case), and updates the recommendation after every click by advancing the hidden state one step. Training uses each session’s next item as the target, with ranking losses (cross-entropy, BPR or TOP1 with sampled negatives), and evaluation uses Recall@k and MRR@k on later sessions. RNNs are also used to model a user’s long-term history, sequences of watched videos, playlists (next song), and time-aware purchase patterns; transformer models such as SASRec and BERT4Rec are now strong alternatives.

Read full answer

Applies toPyTorch 2.x; GRU4Rec (Hidasi et al., 2016) and follow-ups

How Session-Based Recommendation Works

Step Detail
Input Session as item IDs: [shoe_12, shoe_47, sock_3]
Embedding Each item → dense vector
GRU Hidden state summarizes the session so far
Scores h_t · item_embeddingᵀ (or a linear layer) for all items
Output Top-k items, filtered by business rules (in stock, not already bought)
Online update New click → one more GRU step → new top-k

Code Example: Model and Ranking Metrics (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)


class GRU4Rec(nn.Module):
    def __init__(self, n_items, dim=64):
        super().__init__()
        self.items = nn.Embedding(n_items, dim, padding_idx=0)
        self.gru = nn.GRU(dim, dim, batch_first=True)

    def forward(self, sessions):                     # (batch, steps) -> (batch, steps, n_items)
        h, _ = self.gru(self.items(sessions))
        return h @ self.items.weight.T               # tied weights: score = h · item embedding

    @torch.no_grad()
    def step(self, item_id, h=None):
        """Online use: advance the session by one click and return scores + new state."""
        out, h = self.gru(self.items(torch.tensor([[item_id]])), h)
        return (out[0, -1] @ self.items.weight.T), h


def recall_and_mrr_at_k(scores, targets, k):
    """scores: (n, n_items); targets: (n,). Rank 1 = best."""
    target_scores = scores.gather(1, targets.unsqueeze(1))
    rank = (scores > target_scores).sum(dim=1) + 1
    hit = rank <= k
    recall = hit.float().mean().item()
    mrr = torch.where(hit, 1.0 / rank.float(), torch.zeros_like(rank, dtype=torch.float)).mean().item()
    return recall, mrr


# Metric check on hand-made scores
scores = torch.tensor([[0.1, 0.9, 0.3, 0.2, 0.0],
                       [0.5, 0.1, 0.4, 0.8, 0.2]])
targets = torch.tensor([2, 0])                      # both targets are ranked 2nd
print(recall_and_mrr_at_k(scores, targets, k=1))   # (0.0, 0.0)
print(recall_and_mrr_at_k(scores, targets, k=2))   # (1.0, 0.5)

# Model shapes and online stepping
model = GRU4Rec(n_items=1000)
sessions = torch.randint(1, 1000, (8, 6))
print(model(sessions).shape)                        # torch.Size([8, 6, 1000])

h = None
for clicked in [17, 42, 42, 305]:                   # a live session
    item_scores, h = model.step(clicked, h)
item_scores[0] = float("-inf")                      # never recommend padding
print(item_scores.topk(5).indices.shape)            # torch.Size([5])

Training uses the same next-item shift as language modelling: inputs session[:, :-1], targets session[:, 1:], with cross-entropy over all items or a sampled loss for large catalogues.

Practical Considerations

Concern Practice
Large catalogue Sampled softmax / negative sampling, popularity-based negatives
Evaluation Time-based split (train on earlier sessions, test on later ones); Recall@20, MRR@20
Baselines Most popular, item-to-item co-occurrence (item-KNN), Markov chain
Cold-start items Content features (text, category) added to item embeddings
Business rules Remove out-of-stock and already-purchased items after scoring
Serving Keep the session hidden state in a cache keyed by session ID

Common Pitfalls

  • Random splits that leak future sessions into training.

  • Weak or missing baselines; item-KNN is often surprisingly hard to beat.

  • Evaluating with sampled negatives only, which inflates metrics.

  • Recommending items the user just viewed or bought without a rule.

  • Ignoring the popularity bias of the trained model.

Interview Tips

  • Describe GRU4Rec end to end: session → GRU → scores → top-k.

  • Name ranking metrics (Recall@k, MRR) and time-based evaluation.

  • Mention SASRec/BERT4Rec as transformer successors and item-KNN as the baseline to beat.

30

Implement a basic RNN to classify sequential data in Python.

A basic RNN sequence classifier reads each sequence step by step with a recurrent layer (h_t = tanh(W_x x_t + W_h h_(t−1) + b)), takes the final hidden state as a summary of the whole sequence, and passes it through a linear layer to produce class scores (many-to-one). Training is ordinary supervised learning: cross-entropy loss, backpropagation through time, an optimizer such as Adam, and gradient clipping to keep the vanilla RNN stable. The example below generates a synthetic two-class dataset (noisy sine waves vs. noisy square waves), trains a vanilla nn.RNN classifier in PyTorch, and reports test accuracy; swapping nn.RNN for nn.GRU or nn.LSTM is a one-line change and usually works better on longer sequences.

Read full answer

Applies toPyTorch 2.x, NumPy

Code Example (PyTorch)

import numpy as np
import torch
from torch import nn

torch.manual_seed(42)
rng = np.random.default_rng(42)

SEQ_LEN, N_PER_CLASS = 40, 600


def make_dataset():
    t = np.linspace(0, 4 * np.pi, SEQ_LEN)
    X, y = [], []
    for label in (0, 1):
        for _ in range(N_PER_CLASS):
            phase = rng.uniform(0, 2 * np.pi)
            freq = rng.uniform(0.8, 1.2)
            wave = np.sin(freq * t + phase)
            if label == 1:
                wave = np.sign(wave)                       # square wave
            X.append(wave + 0.3 * rng.standard_normal(SEQ_LEN))
            y.append(label)
    X = torch.tensor(np.array(X), dtype=torch.float32).unsqueeze(-1)   # (N, seq_len, 1)
    y = torch.tensor(y)
    perm = torch.randperm(len(y))
    return X[perm], y[perm]


X, y = make_dataset()
split = int(0.8 * len(y))
X_train, y_train, X_test, y_test = X[:split], y[:split], X[split:], y[split:]


class RNNClassifier(nn.Module):
    def __init__(self, n_features=1, hidden=32, n_classes=2):
        super().__init__()
        self.rnn = nn.RNN(n_features, hidden, batch_first=True, nonlinearity="tanh")
        self.fc = nn.Linear(hidden, n_classes)

    def forward(self, x):
        _, h_n = self.rnn(x)          # h_n: (num_layers, batch, hidden)
        return self.fc(h_n[-1])       # many-to-one: classify from the last hidden state


model = RNNClassifier()
optimizer = torch.optim.Adam(model.parameters(), lr=5e-3)
loss_fn = nn.CrossEntropyLoss()

for epoch in range(20):
    model.train()
    perm = torch.randperm(len(y_train))
    for i in range(0, len(y_train), 64):
        idx = perm[i:i + 64]
        loss = loss_fn(model(X_train[idx]), y_train[idx])
        optimizer.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)   # stabilizes vanilla RNNs
        optimizer.step()
    if epoch % 5 == 4:
        model.eval()
        with torch.no_grad():
            acc = (model(X_test).argmax(dim=1) == y_test).float().mean().item()
        print(f"epoch {epoch + 1:2d}  loss {loss.item():.3f}  test accuracy {acc:.3f}")

print(model(X_test[:3]).shape)        # torch.Size([3, 2])

The two classes differ in shape (smooth vs. flat with sharp jumps), which the RNN can learn from the sequence; the printed accuracy is not quoted because it depends on the run. A good habit is to also train a non-sequential baseline (e.g. logistic regression on summary features) to confirm the RNN adds value.

Building Blocks

Piece Role
Input shape (batch, seq_len, features) batch_first=True convention
nn.RNN Vanilla recurrence with tanh
h_n[-1] Final hidden state of the top layer
nn.Linear Class scores (logits)
CrossEntropyLoss Applies softmax internally; pass raw logits
clip_grad_norm_ Prevents exploding gradients

Variable-Length Sequences

Pad the batch, then use pack_padded_sequence(x, lengths, batch_first=True, enforce_sorted=False) before the RNN, so h_n is taken at each sequence’s real last step instead of after padding.

Common Pitfalls

  • Applying softmax before CrossEntropyLoss (double softmax).

  • Using the output at the last padded position instead of the true last step.

  • No gradient clipping with a vanilla RNN on long sequences.

  • Forgetting model.eval() and torch.no_grad() at evaluation.

  • Scaling features with statistics from the whole dataset before the split.

Interview Tips

  • State the recurrence equation and the many-to-one design.

  • Mention clipping and why GRU/LSTM are usually better.

  • Explain how variable-length sequences are handled.

31

Write TensorFlow/Keras code to build and train an LSTM on text.

A Keras LSTM text model has four stages: turn raw text into padded integer sequences (with TextVectorization or an already-tokenized dataset), embed the tokens with an Embedding layer that masks padding, run one or more (often bidirectional) LSTM layers, and finish with a dense output layer; it is compiled with an optimizer, a loss that matches the task (binary cross-entropy for two classes) and metrics, then trained with a validation split and early stopping. The example below trains a sentiment classifier on the IMDB movie-review dataset that ships with Keras, and shows how to put the text-vectorization step inside the exported model so it accepts raw strings.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3

Code Example: IMDB Sentiment with a Bidirectional LSTM

import keras
from keras import layers

keras.utils.set_random_seed(42)

VOCAB_SIZE, MAX_LEN = 20_000, 200

# IMDB reviews already converted to word indices (1 = start, 2 = unknown, 3+ = words by frequency)
(x_train, y_train), (x_test, y_test) = keras.datasets.imdb.load_data(num_words=VOCAB_SIZE)
x_train = keras.utils.pad_sequences(x_train, maxlen=MAX_LEN, padding="post", truncating="post")
x_test = keras.utils.pad_sequences(x_test, maxlen=MAX_LEN, padding="post", truncating="post")
print(x_train.shape, x_test.shape)          # (25000, 200) (25000, 200)

model = keras.Sequential([
    keras.Input(shape=(MAX_LEN,), dtype="int32"),
    layers.Embedding(VOCAB_SIZE, 128, mask_zero=True),      # mask the padding (index 0)
    layers.Bidirectional(layers.LSTM(64, return_sequences=True)),
    layers.Bidirectional(layers.LSTM(32)),
    layers.Dropout(0.3),
    layers.Dense(1, activation="sigmoid"),
])

model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3, clipnorm=1.0),
    loss="binary_crossentropy",
    metrics=["accuracy", keras.metrics.AUC(name="auc")],
)
model.summary()

history = model.fit(
    x_train, y_train,
    validation_split=0.2,
    epochs=5,
    batch_size=64,
    callbacks=[keras.callbacks.EarlyStopping(monitor="val_auc", mode="max", patience=1,
                                             restore_best_weights=True)],
)

test_loss, test_acc, test_auc = model.evaluate(x_test, y_test, verbose=0)
print(f"test accuracy = {test_acc:.3f}, test AUC = {test_auc:.3f}")

The scores depend on hardware and library versions, so they are not quoted here; compare them with a simple baseline such as TF-IDF + logistic regression on the same split.

Raw-Text Version with TextVectorization

import keras
import tensorflow as tf
from keras import layers

texts = tf.constant(["a wonderful, moving film", "boring and far too long",
                     "great acting and a clever plot", "I want my two hours back"])
labels = tf.constant([1, 0, 1, 0])

vectorizer = layers.TextVectorization(max_tokens=10_000, output_sequence_length=50)
vectorizer.adapt(texts)                       # learn the vocabulary from training text only
print(vectorizer.get_vocabulary()[:5])        # ['', '[UNK]', ...] then the most frequent words

inputs = keras.Input(shape=(1,), dtype="string")
x = vectorizer(inputs)
x = layers.Embedding(len(vectorizer.get_vocabulary()), 64, mask_zero=True)(x)
x = layers.LSTM(32)(x)
outputs = layers.Dense(1, activation="sigmoid")(x)
text_model = keras.Model(inputs, outputs)
text_model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
text_model.fit(tf.reshape(texts, (-1, 1)), labels, epochs=2, verbose=0)

print(text_model.predict(tf.constant([["what a clever film"]]), verbose=0).shape)   # (1, 1)

Because the vectorizer is part of the model, the saved .keras file accepts raw strings, and training and serving use exactly the same tokenization.

Layer Choices

Layer / setting Purpose
Embedding(..., mask_zero=True) Padding is skipped by later LSTM layers
Bidirectional(LSTM(...)) Uses left and right context (classification, tagging)
return_sequences=True Needed when stacking LSTMs or tagging every token
Dropout / recurrent_dropout Regularization (recurrent_dropout disables the fast cuDNN kernel)
clipnorm=1.0 Guards against exploding gradients
EarlyStopping(restore_best_weights=True) Stops when validation stops improving

Common Pitfalls

  • Fitting TextVectorization (or a tokenizer) on validation/test text.

  • Pre-padding vs post-padding mismatch between training and serving without masking.

  • Using recurrent_dropout and wondering why training is several times slower on GPU.

  • Very long MAX_LEN with small data, which slows training without helping.

  • Reporting only accuracy on imbalanced datasets; add AUC or F1.

Interview Tips

  • Walk through vectorize → embed (masking) → LSTM → dense → compile → fit with early stopping.

  • Mention bidirectional layers and gradient clipping.

  • Suggest a pretrained transformer (e.g. via KerasHub) as the stronger alternative for production NLP.

32

Build a PyTorch GRU network that predicts the next item in a sequence.

A next-item predictor embeds each item ID, runs a GRU over the sequence of past items, and maps the hidden state at every position to a score for every item in the catalogue; training minimizes cross-entropy between those scores and the item that actually came next, and prediction takes the top-k scores after the last observed item. This is the core of session-based recommenders (GRU4Rec) and of next-event prediction in logs or user journeys. The example below builds sequences, shifts them to create (input, target) pairs, trains with padding-aware loss, evaluates Hit@k and MRR on held-out sequences, and returns the top-k next items for a new sequence.

Read full answer

Applies toPyTorch 2.x

Code Example (PyTorch)

import numpy as np
import torch
import torch.nn.functional as F
from torch import nn

torch.manual_seed(42)
rng = np.random.default_rng(42)

N_ITEMS = 50          # item IDs 1..49, 0 = padding
SEQ_LEN = 12


def make_sequence():
    """Synthetic user journeys: mostly step +1 or +2 through the catalogue, sometimes a random jump."""
    seq = [int(rng.integers(1, N_ITEMS))]
    for _ in range(SEQ_LEN - 1):
        if rng.random() < 0.85:
            nxt = seq[-1] + int(rng.choice([1, 2], p=[0.7, 0.3]))
        else:
            nxt = int(rng.integers(1, N_ITEMS))
        seq.append((nxt - 1) % (N_ITEMS - 1) + 1)          # wrap around, never 0
    return seq


data = torch.tensor([make_sequence() for _ in range(3000)])
train, test = data[:2500], data[2500:]


class NextItemGRU(nn.Module):
    def __init__(self, n_items, emb=32, hidden=64):
        super().__init__()
        self.embed = nn.Embedding(n_items, emb, padding_idx=0)
        self.gru = nn.GRU(emb, hidden, batch_first=True)
        self.dropout = nn.Dropout(0.2)
        self.out = nn.Linear(hidden, n_items)

    def forward(self, items):                      # (batch, seq_len) -> (batch, seq_len, n_items)
        h, _ = self.gru(self.embed(items))
        return self.out(self.dropout(h))


model = NextItemGRU(N_ITEMS)
optimizer = torch.optim.Adam(model.parameters(), lr=3e-3)

for epoch in range(8):
    model.train()
    perm = torch.randperm(len(train))
    for i in range(0, len(train), 128):
        batch = train[perm[i:i + 128]]
        inputs, targets = batch[:, :-1], batch[:, 1:]          # predict item t+1 from items <= t
        logits = model(inputs)
        loss = F.cross_entropy(logits.reshape(-1, N_ITEMS), targets.reshape(-1), ignore_index=0)
        optimizer.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()


@torch.no_grad()
def evaluate(model, seqs, k=5):
    model.eval()
    logits = model(seqs[:, :-1])[:, -1]                        # score the last next-item only
    target = seqs[:, -1]
    logits[:, 0] = float("-inf")                               # never recommend padding
    rank = (logits > logits.gather(1, target.unsqueeze(1))).sum(dim=1) + 1
    hit = (rank <= k).float().mean().item()
    mrr = (1.0 / rank.float()).mean().item()
    return hit, mrr


@torch.no_grad()
def recommend(model, history, k=3):
    model.eval()
    logits = model(torch.tensor([history]))[0, -1]
    logits[0] = float("-inf")
    return logits.topk(k).indices.tolist()


hit5, mrr = evaluate(model, test)
print(f"Hit@5 = {hit5:.3f}, MRR = {mrr:.3f}")
print("next items after [10, 11, 12]:", recommend(model, [10, 11, 12]))

Because the synthetic journeys mostly step +1 or +2, a trained model should rank 13 and 14 near the top after [10, 11, 12]. Compare the metrics against simple baselines (most popular item, “same as last transition” Markov model) before trusting any GRU result on real data.

Design Choices

Choice Why
Predict at every position Uses all T−1 targets in each sequence for training
padding_idx=0 + ignore_index=0 Variable-length sequences can be padded safely
Hit@k and MRR Ranking quality matters more than exact top-1 accuracy
Masking already-seen or unavailable items Business rule applied at recommendation time
Sampled softmax / negative sampling Needed when the catalogue has millions of items

Common Pitfalls

  • Random train/test splits that put a user’s later sessions in training and earlier ones in test (leakage); split by time.

  • Evaluating with accuracy only; ranking metrics reflect what users see.

  • Full softmax over a huge catalogue (slow, memory-heavy).

  • Forgetting cold-start items that have no embedding yet.

  • No popularity baseline, so “improvements” are not measured against anything.

Interview Tips

  • Explain the shift-by-one input/target construction.

  • Mention Hit@k/Recall@k and MRR, and baselines.

  • Discuss scaling: sampled softmax, time-based splits, and transformers (SASRec) as the next step.

33

Build an attention-based RNN that translates English to French.

An English→French translator with attention has three parts: a bidirectional GRU encoder that turns the English tokens into a sequence of hidden states, an attention module that at each decoding step scores every encoder state against the decoder’s current state and builds a weighted context vector (Bahdanau additive attention), and a GRU decoder that combines the previous French token, its hidden state and the context to predict the next French token. Training uses teacher forcing and cross-entropy that ignores padding; translation uses greedy (or beam) decoding until <eos>. The example below is complete and runs on a toy parallel corpus; a real system needs a large corpus (e.g. Europarl, Tatoeba or WMT data), subword tokenization (SentencePiece/BPE) and evaluation with BLEU or chrF.

Read full answer

Applies toPyTorch 2.x; architecture from Bahdanau, Cho and Bengio (2015)

Architecture

Component Shape / formula
Encoder BiGRU over source embeddings → states H ∈ (src_len, 2·hid)
Bridge Linear(2·hid → hid) on the final forward/backward states → initial decoder state
Attention score e_i = vᵀ tanh(W_s s_(t−1) + W_h h_i), α = softmax(e) over source positions (padding masked)
Context c_t = Σ α_i h_i
Decoder GRU input = [embedding(y_(t−1)); c_t]; output layer over the French vocabulary

Code Example (PyTorch)

import random

import torch
import torch.nn.functional as F
from torch import nn

random.seed(42)
torch.manual_seed(42)

PAIRS = [
    ("i am cold", "j'ai froid"),
    ("i am hungry", "j'ai faim"),
    ("she is happy", "elle est heureuse"),
    ("he is tired", "il est fatigué"),
    ("we are friends", "nous sommes amis"),
    ("the cat sleeps", "le chat dort"),
    ("the dog eats", "le chien mange"),
    ("i love music", "j'aime la musique"),
]
PAD, SOS, EOS = 0, 1, 2


def build_vocab(sentences):
    vocab = {"<pad>": PAD, "<sos>": SOS, "<eos>": EOS}
    for s in sentences:
        for w in s.split():
            vocab.setdefault(w, len(vocab))
    return vocab


src_vocab = build_vocab(e for e, _ in PAIRS)
tgt_vocab = build_vocab(f for _, f in PAIRS)
tgt_itos = {i: w for w, i in tgt_vocab.items()}


def encode(sentence, vocab):
    return [vocab[w] for w in sentence.split()] + [EOS]


def batchify(pairs):
    src = [torch.tensor(encode(e, src_vocab)) for e, _ in pairs]
    tgt = [torch.tensor(encode(f, tgt_vocab)) for _, f in pairs]
    src = nn.utils.rnn.pad_sequence(src, batch_first=True, padding_value=PAD)
    tgt = nn.utils.rnn.pad_sequence(tgt, batch_first=True, padding_value=PAD)
    return src, tgt


class Encoder(nn.Module):
    def __init__(self, vocab_size, emb=64, hid=128):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, emb, padding_idx=PAD)
        self.gru = nn.GRU(emb, hid, batch_first=True, bidirectional=True)
        self.bridge = nn.Linear(2 * hid, hid)

    def forward(self, src):
        outputs, h = self.gru(self.embed(src))                  # outputs: (B, S, 2H)
        h0 = torch.tanh(self.bridge(torch.cat([h[0], h[1]], dim=-1)))
        return outputs, h0.unsqueeze(0)                          # decoder state: (1, B, H)


class BahdanauAttention(nn.Module):
    def __init__(self, hid=128):
        super().__init__()
        self.w_s = nn.Linear(hid, hid, bias=False)
        self.w_h = nn.Linear(2 * hid, hid, bias=False)
        self.v = nn.Linear(hid, 1, bias=False)

    def forward(self, s, enc_out, mask):
        scores = self.v(torch.tanh(self.w_s(s).unsqueeze(1) + self.w_h(enc_out))).squeeze(-1)
        scores = scores.masked_fill(~mask, float("-inf"))        # never attend to padding
        alpha = torch.softmax(scores, dim=-1)                    # (B, S)
        context = torch.bmm(alpha.unsqueeze(1), enc_out).squeeze(1)
        return context, alpha


class Decoder(nn.Module):
    def __init__(self, vocab_size, emb=64, hid=128):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, emb, padding_idx=PAD)
        self.attn = BahdanauAttention(hid)
        self.gru = nn.GRU(emb + 2 * hid, hid, batch_first=True)
        self.out = nn.Linear(hid + 2 * hid + emb, vocab_size)

    def forward(self, token, h, enc_out, mask):                  # one step; token: (B,)
        e = self.embed(token)
        context, alpha = self.attn(h[-1], enc_out, mask)
        out, h = self.gru(torch.cat([e, context], dim=-1).unsqueeze(1), h)
        logits = self.out(torch.cat([out.squeeze(1), context, e], dim=-1))
        return logits, h, alpha


class Translator(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = Encoder(len(src_vocab))
        self.decoder = Decoder(len(tgt_vocab))

    def forward(self, src, tgt, teacher_forcing=1.0):
        enc_out, h = self.encoder(src)
        mask = src != PAD
        token = torch.full((src.shape[0],), SOS, dtype=torch.long)
        all_logits = []
        for t in range(tgt.shape[1]):
            logits, h, _ = self.decoder(token, h, enc_out, mask)
            all_logits.append(logits)
            use_truth = random.random() < teacher_forcing
            token = tgt[:, t] if use_truth else logits.argmax(-1)
        return torch.stack(all_logits, dim=1)                    # (B, T, V)

    @torch.no_grad()
    def translate(self, sentence, max_len=10):
        src = torch.tensor([encode(sentence, src_vocab)])
        enc_out, h = self.encoder(src)
        mask = src != PAD
        token = torch.tensor([SOS])
        words, attention = [], []
        for _ in range(max_len):
            logits, h, alpha = self.decoder(token, h, enc_out, mask)
            token = logits.argmax(-1)
            if token.item() == EOS:
                break
            words.append(tgt_itos[token.item()])
            attention.append(alpha.squeeze(0))
        return " ".join(words), torch.stack(attention) if attention else None


model = Translator()
optimizer = torch.optim.Adam(model.parameters(), lr=3e-3)
src, tgt = batchify(PAIRS)

for epoch in range(300):
    model.train()
    optimizer.zero_grad()
    logits = model(src, tgt, teacher_forcing=0.9)
    loss = F.cross_entropy(logits.reshape(-1, logits.shape[-1]), tgt.reshape(-1), ignore_index=PAD)
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()

model.eval()
for english in ["i am hungry", "the cat sleeps", "we are friends"]:
    french, attn = model.translate(english)
    print(f"{english!r} -> {french!r}")
print(f"final training loss: {loss.item():.3f}")

On eight training pairs the model memorizes them, so it should reproduce the training translations; it cannot generalize to new sentences. The returned attention matrix (target words × source words) can be plotted as a heatmap to see which English word each French word attended to.

Scaling to a Real Translator

Need Practice
Data Hundreds of thousands to millions of sentence pairs
Tokenization SentencePiece/BPE subwords shared or per language
Batching Sort or bucket by length; mini-batches
Decoding Beam search (beam 4-5) with length normalization
Evaluation BLEU / chrF on a held-out test set, plus human review
Baseline Compare with a pretrained transformer (e.g. Marian/OPUS-MT en-fr), which will be far stronger

Common Pitfalls

  • Not masking padded source positions in attention.

  • Including padding in the loss (use ignore_index).

  • Evaluating on training sentences.

  • Word-level vocabularies on real data (huge vocab, many unknown words); use subwords.

  • Always using teacher forcing, so the model never learns to recover from its own errors.

Interview Tips

  • Explain the attention formula and why it removes the fixed-context bottleneck.

  • Describe training (teacher forcing) vs inference (greedy/beam) clearly.

  • Say how you would evaluate (BLEU/chrF) and what you would use in production (a pretrained transformer).

34

Write a function that visualizes an RNN's hidden state dynamics.

To visualize how an RNN’s hidden state evolves, run the network over a sequence, collect the hidden vector at every time step (a seq_len × hidden_size matrix), and plot it in complementary views: a heatmap of units over time (which units switch on when), the trajectory of the state projected to 2-D with PCA (how the state moves through its space), and the hidden-state norm and step-to-step change over time (where the input causes big updates). Plotting these next to the input sequence makes it easy to spot units that track specific features (a counter, a phase, a token type), saturation (values stuck near ±1 for tanh), and moments where the model “resets”. The function below works with any PyTorch nn.RNN, nn.GRU or nn.LSTM that uses batch_first=True.

Read full answer

Applies toPyTorch 2.x, NumPy, Matplotlib 3.8+

Code Example

import matplotlib

matplotlib.use("Agg")                       # render to a file without a display
import matplotlib.pyplot as plt  # noqa: E402
import numpy as np  # noqa: E402
import torch  # noqa: E402
from torch import nn  # noqa: E402

torch.manual_seed(42)


def plot_hidden_dynamics(rnn, sequence, path="hidden_dynamics.png", max_units=32):
    """Plot hidden-state dynamics of a batch_first RNN/GRU/LSTM for one sequence.

    sequence: tensor of shape (seq_len, n_features).
    Returns the hidden states as a (seq_len, hidden_size) NumPy array.
    """
    rnn.eval()
    with torch.no_grad():
        outputs, _ = rnn(sequence.unsqueeze(0))          # (1, seq_len, hidden * directions)
    states = outputs.squeeze(0).numpy()
    seq_len = states.shape[0]

    # PCA to 2-D via SVD of the centred states
    centred = states - states.mean(axis=0)
    _, s, vt = np.linalg.svd(centred, full_matrices=False)
    proj = centred @ vt[:2].T
    explained = (s[:2] ** 2).sum() / (s ** 2).sum()

    fig, axes = plt.subplots(2, 2, figsize=(12, 8))

    axes[0, 0].plot(sequence.numpy())
    axes[0, 0].set_title("Input features")
    axes[0, 0].set_xlabel("time step")

    im = axes[0, 1].imshow(states[:, :max_units].T, aspect="auto", cmap="RdBu_r", vmin=-1, vmax=1)
    axes[0, 1].set_title(f"Hidden units (first {min(max_units, states.shape[1])})")
    axes[0, 1].set_xlabel("time step")
    axes[0, 1].set_ylabel("unit")
    fig.colorbar(im, ax=axes[0, 1])

    sc = axes[1, 0].scatter(proj[:, 0], proj[:, 1], c=np.arange(seq_len), cmap="viridis", s=12)
    axes[1, 0].plot(proj[:, 0], proj[:, 1], lw=0.5, color="grey")
    axes[1, 0].set_title(f"PCA trajectory ({explained:.0%} variance in 2 PCs)")
    fig.colorbar(sc, ax=axes[1, 0], label="time step")

    norms = np.linalg.norm(states, axis=1)
    deltas = np.r_[0.0, np.linalg.norm(np.diff(states, axis=0), axis=1)]
    axes[1, 1].plot(norms, label="‖h_t‖")
    axes[1, 1].plot(deltas, label="‖h_t − h_(t−1)‖")
    axes[1, 1].set_title("State norm and update size")
    axes[1, 1].set_xlabel("time step")
    axes[1, 1].legend()

    fig.tight_layout()
    fig.savefig(path, dpi=120)
    plt.close(fig)
    return states


# Example: a GRU reading a sine wave whose frequency changes halfway
t = np.linspace(0, 8 * np.pi, 200)
signal = np.where(t < 4 * np.pi, np.sin(t), np.sin(3 * t))
seq = torch.tensor(np.stack([signal, np.cos(t)], axis=1), dtype=torch.float32)

gru = nn.GRU(input_size=2, hidden_size=48, batch_first=True)
states = plot_hidden_dynamics(gru, seq)
print(states.shape)          # (200, 48)

For an LSTM, outputs contains the hidden state h_t; to plot the cell state c_t as well, step through the sequence with nn.LSTMCell (or run the LSTM one step at a time) and record c at each step. Run the function on a trained model to see meaningful structure; an untrained network still shows how strongly inputs drive the state.

What to Look For

Pattern Possible meaning
Units saturated at ±1 for long stretches Gates or activations saturated; vanishing gradients likely
Loops in the PCA trajectory The model tracks a periodic phase
Sudden jumps in ‖h_t − h_(t−1)‖ Inputs that the model treats as important events (or anomalies)
A unit changing sign at a known event An interpretable “feature detector” unit
Steadily growing norm Possible instability with long inputs

Common Pitfalls

  • Plotting padded time steps of a padded batch; trim to the true length first.

  • Using a symmetric colour scale for ReLU-based states (use 0 to max instead).

  • Reading too much into one example; compare several sequences and a trained vs. untrained model.

  • Forgetting model.eval(), so dropout changes the states between runs.

Interview Tips

  • Mention the three complementary views: heatmap, low-dimensional trajectory, norm/change over time.

  • Explain how to get LSTM cell states (step-by-step with LSTMCell).

  • Connect the plots to diagnoses: saturation, periodic tracking, event detection.

35

What do recent transformer advances mean for the future of RNNs?

Transformers replaced RNNs as the default for most sequence tasks — language modelling, translation, speech recognition and many time-series problems — because they train in parallel across time steps, model long-range dependencies directly through attention, and scale predictably with data and compute; pretrained transformers (BERT, GPT, T5, Whisper) made this shift permanent for NLP. The future of RNNs is therefore narrower but not empty: (1) efficiency-critical inference, where a recurrent model’s constant memory per step beats a transformer’s growing key-value cache — streaming speech, edge devices, long-running sensors; (2) a new generation of recurrent architectures (Mamba and other state-space models, RWKV, xLSTM, minGRU) that train in parallel like transformers yet run as RNNs at inference, often in hybrids with a few attention layers; and (3) small-data or simple tasks where a small GRU is cheaper and good enough.

Read full answer

Applies toLandscape as of 2025-2026; Python example with standard arithmetic

Why Transformers Won

Factor RNN Transformer
Training parallelism over time No Yes
Long-range dependencies Through many steps (gradients fade) One attention step
Scaling with data/compute Plateaus earlier Scales well
Pretrained ecosystem Small Huge (Hugging Face)
Inference memory per sequence Constant Grows with length (KV cache)
Cost per new token Constant Grows with context length

Code Example: Inference Memory, Transformer vs RNN (Python)

def kv_cache_bytes(layers, d_model, seq_len, bytes_per_value=2):
    # Keys and values for every layer and every past token
    return 2 * layers * seq_len * d_model * bytes_per_value


def lstm_state_bytes(layers, hidden, bytes_per_value=2):
    # h and c for every layer, independent of sequence length
    return 2 * layers * hidden * bytes_per_value


layers, width = 12, 768
for seq_len in (1_024, 4_096, 32_768):
    kv = kv_cache_bytes(layers, width, seq_len)
    print(f"context {seq_len:>6}: transformer KV cache {kv / 2**20:8.1f} MiB")
print(f"LSTM state (any length): {lstm_state_bytes(layers, width) / 2**10:.1f} KiB")
# context   1024: transformer KV cache     36.0 MiB
# context   4096: transformer KV cache    144.0 MiB
# context  32768: transformer KV cache   1152.0 MiB
# LSTM state (any length): 36.0 KiB

Per sequence, a 12-layer, 768-wide transformer in fp16 needs 144 MiB of cache at 4,096 tokens, while a recurrent model of the same width keeps 36 KiB regardless of length. (Real transformers reduce this with grouped-query attention, sliding windows or cache compression.) This gap is the main reason recurrent ideas are coming back for long contexts and on-device use.

Likely Roles for RNN-Style Models

Area Why recurrence helps
Streaming ASR (RNN-Transducer) Low latency, frame-by-frame
Edge / IoT Tiny memory and compute
Long-context LMs (SSM or hybrid) Constant-memory inference
Control and RL with partial observability Step-by-step memory
Online time-series Update state as data arrives

Common Pitfalls

  • Choosing an LSTM for a large NLP task without comparing against a pretrained transformer.

  • Choosing a transformer for a microcontroller or a never-ending stream without considering memory.

  • Assuming new recurrent models match transformers on every task; exact recall from long contexts is still a weak point.

Interview Tips

  • Give a balanced answer: transformers dominate quality and training, RNNs win on inference efficiency.

  • Quote the KV-cache vs constant-state argument.

  • Name the new recurrent families (Mamba, RWKV, xLSTM) and hybrids.

36

How has transfer learning influenced RNN applications in NLP?

Transfer learning changed NLP from training each RNN from scratch on a small labelled dataset to pretraining on large unlabelled text and then adapting to the task, and RNNs were where this shift started. First came pretrained word embeddings (word2vec, GloVe, fastText) used as the embedding layer of LSTMs; then contextual embeddings from pretrained bidirectional LSTM language models (ELMo, 2018), added as features to task models, which improved many benchmarks at once; and ULMFiT (2018), which fine-tuned a whole AWD-LSTM language model with discriminative learning rates, slanted triangular schedules and gradual unfreezing, reaching strong text-classification results with only hundreds of labels. These ideas were quickly carried over to transformers (GPT, BERT), which now dominate; today RNNs are mostly used as light task heads on top of pretrained encoders or in settings where a small, streaming model is required.

Read full answer

Applies toPyTorch 2.x; history from 2013 (word2vec) to 2018 (ELMo, ULMFiT, BERT)

Timeline

Stage What transfers RNN role
Static embeddings (2013-2016) One vector per word Initialize the LSTM’s embedding layer
Contextual embeddings: ELMo (2018) Hidden states of a pretrained biLSTM LM Frozen features fed to task models
Fine-tuning: ULMFiT (2018) Entire LSTM language model Fine-tuned end to end
Transformers: GPT, BERT (2018+) Entire transformer RNN becomes an optional head or disappears

Code Example: Pretrained Embeddings, Freezing and Gradual Unfreezing (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

VOCAB, EMB, HID, CLASSES = 1000, 100, 128, 2
pretrained_vectors = torch.randn(VOCAB, EMB)        # stands in for GloVe/fastText rows aligned to the vocab


class LSTMClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding.from_pretrained(pretrained_vectors, freeze=True, padding_idx=0)
        self.lstm = nn.LSTM(EMB, HID, num_layers=2, batch_first=True, dropout=0.3)
        self.head = nn.Linear(HID, CLASSES)

    def forward(self, ids):
        _, (h, _) = self.lstm(self.embed(ids))
        return self.head(h[-1])


model = LSTMClassifier()


def trainable(m):
    return sum(p.numel() for p in m.parameters() if p.requires_grad)


print("stage 1 trainable:", trainable(model))       # embeddings frozen

# ULMFiT-style discriminative learning rates: lower layers change less
optimizer = torch.optim.AdamW([
    {"params": model.embed.parameters(), "lr": 1e-4},
    {"params": model.lstm.parameters(), "lr": 5e-4},
    {"params": model.head.parameters(), "lr": 2e-3},
])

# Gradual unfreezing: after the head and LSTM settle, release the embeddings too
model.embed.weight.requires_grad_(True)
print("stage 2 trainable:", trainable(model))       # now includes the 100,000 embedding weights

logits = model(torch.randint(1, VOCAB, (8, 40)))
print(logits.shape)                                  # torch.Size([8, 2])

Putting the embedding parameters in the optimizer from the start is harmless while they are frozen (they get no gradient); once unfrozen, they train with the smallest learning rate.

Impact on RNN Applications

Before transfer learning After
Thousands of labelled examples per task Hundreds can be enough (ULMFiT)
Random embeddings; OOV words lost Pretrained subword/character information
Separate model per task One pretrained LM adapted to many tasks
RNN = whole model RNN = head on BERT-style features, or replaced entirely

When RNNs Still Make Sense with Transfer Learning

  • A BiLSTM-CRF head on top of a pretrained encoder for sequence labelling (NER).

  • Small on-device models initialized from pretrained embeddings or distilled from a transformer.

  • Streaming text or speech where per-step cost must stay constant.

Common Pitfalls

  • Using pretrained embeddings with a different tokenizer or casing than the vocabulary they were trained with.

  • Fine-tuning everything with one high learning rate and destroying pretrained knowledge (catastrophic forgetting).

  • Comparing a from-scratch LSTM to a pretrained transformer and blaming the architecture rather than the pretraining.

Interview Tips

  • Tell the story: word2vec/GloVe → ELMo (biLSTM) → ULMFiT (fine-tuned LSTM LM) → BERT/GPT.

  • Name ULMFiT’s three techniques.

  • Explain where RNNs still fit on top of pretrained models.

37

What are some exciting research areas in RNNs and sequence modelling?

The most active research on recurrent and sequential models today asks how to get the efficiency of RNNs (constant memory and cost per step at inference) together with the parallel training and long-range recall of transformers. The main directions are linear recurrences and state-space models (S4, Mamba/Mamba-2, LRU) that train with parallel scans; modernized RNNs (RWKV, xLSTM, minGRU/minLSTM, Griffin/Hawk) that remove the nonlinearity between steps so training can be parallelized; hybrid architectures that mix a few attention layers with recurrent layers; long-context memory (compressive and retrieval memories, test-time-training layers); continuous-time models for irregular data (neural ODEs/CDEs, liquid time-constant networks); efficient on-device and streaming models; and theory about what recurrent models can and cannot remember (associative recall, state capacity).

Read full answer

Applies toResearch landscape as of 2025-2026; PyTorch 2.x example

Research Areas

Area Key idea Examples
State-space models (SSMs) Linear recurrence h_t = A h_(t−1) + B x_t computed by convolution or parallel scan S4, S5, Mamba, Mamba-2
Modern RNNs Gates depend only on the input, so the recurrence is linear in h and parallelizable RWKV, minGRU, xLSTM (mLSTM), LRU
Hybrids Mostly recurrent layers plus some attention for exact recall Griffin, Jamba, Zamba
Long-context memory Fixed-size state that stores more, or external retrieval Compressive memory, test-time training (TTT) layers
Continuous time Hidden state evolves by a learned ODE between irregular observations Neural CDEs, ODE-RNN, liquid networks
Streaming and edge Low-latency recognition and control on small devices RNN-T for speech, tiny GRUs
Understanding capacity Which tasks need attention; how much a fixed state can recall Associative-recall benchmarks, formal language studies
Biological plausibility Learning rules other than BPTT e-prop, real-time recurrent learning approximations

Code Example: Why Linear Recurrences Parallelize (PyTorch)

Modern RNNs and SSMs use recurrences of the form h_t = a_t · h_(t−1) + b_t, where a_t and b_t depend on the input but not on h. Then h_t has a closed form that can be computed for all t at once, instead of with a loop:

import torch

torch.manual_seed(42)

T, D = 50, 8
a = torch.rand(T, D, dtype=torch.float64) * 0.5 + 0.5   # decay gates in (0.5, 1), computed from inputs
b = torch.randn(T, D, dtype=torch.float64)               # input contributions


def sequential(a, b):
    h = torch.zeros(a.shape[1], dtype=a.dtype)
    out = []
    for t in range(a.shape[0]):           # classic RNN-style loop: T sequential steps
        h = a[t] * h + b[t]
        out.append(h)
    return torch.stack(out)


def parallel(a, b):
    # h_t = sum_k (prod_{j=k+1..t} a_j) b_k = A_t * sum_k b_k / A_k, with A_t = prod_{j<=t} a_j
    log_A = torch.cumsum(torch.log(a), dim=0)
    return torch.exp(log_A) * torch.cumsum(b * torch.exp(-log_A), dim=0)


print(torch.allclose(sequential(a, b), parallel(a, b), atol=1e-4))   # True

The parallel form uses cumulative sums, which GPUs compute in O(log T) depth with a scan; production implementations (e.g. Mamba’s kernels) use numerically safer scans than this direct formula, which can overflow for long sequences. At inference, the same model runs as a recurrence with constant memory per step, which is the main advantage over a transformer’s growing key-value cache.

Open Questions

Question Why it matters
Can a fixed-size state match attention on exact recall? Retrieval-heavy tasks still favour attention
How to train long sequences without BPTT memory cost? Truncation limits what is learned
How to handle irregular, multi-rate time series? Healthcare, finance, IoT data
How to make models interpretable over time? Safety-critical sequential decisions

Common Pitfalls

  • Claiming “RNNs are dead”: recurrent ideas are central to SSMs and efficient long-context models.

  • Comparing architectures at different parameter counts or training budgets.

  • Ignoring the numerical stability of parallel scans (log-space or chunked computation is needed).

Interview Tips

  • Frame the research around “RNN inference efficiency + transformer training parallelism”.

  • Name two or three model families (Mamba, RWKV, xLSTM) and their key idea.

  • Mention open problems such as associative recall with a fixed-size state.

38

What role do RNNs play in reinforcement learning and agent decisions?

In reinforcement learning, RNNs give an agent memory: when the current observation does not reveal the full state of the environment (a partially observable MDP), an LSTM or GRU summarizes the history of observations, actions and rewards into a hidden state that the policy or value function uses to decide. Typical uses are recurrent policies and value networks (DRQN, recurrent PPO/A2C, R2D2), world models that learn to predict the next observation and reward (the MDN-RNN in “World Models”, the recurrent state-space model in Dreamer), and meta-reinforcement learning (RL²), where the RNN’s hidden state adapts to a new task within an episode. The practical challenges are handling hidden states in experience replay, truncated backpropagation through time, and resetting state at episode boundaries.

Read full answer

Applies toPyTorch 2.x; algorithms DRQN (2015), R2D2 (2019), recurrent PPO, World Models (2018), DreamerV3 (2023)

Where RNNs Are Used

Use What the RNN does Examples
Partial observability Integrates observations over time (velocity from positions, hidden enemies) DRQN, recurrent PPO
Distributed value learning LSTM Q-network trained from replayed sequences R2D2
World models Predicts next latent state and reward; agent plans or trains “in imagination” World Models, Dreamer
Meta-RL Hidden state learns the task from rewards within an episode RL²
Multi-agent Each agent’s memory of its local history QMIX/MAPPO agents

Code Example: Recurrent Actor-Critic Step (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

OBS_DIM, N_ACTIONS, HIDDEN, N_ENVS = 8, 4, 64, 3


class RecurrentActorCritic(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(OBS_DIM + N_ACTIONS + 1, HIDDEN), nn.Tanh())
        self.lstm = nn.LSTMCell(HIDDEN, HIDDEN)
        self.policy = nn.Linear(HIDDEN, N_ACTIONS)
        self.value = nn.Linear(HIDDEN, 1)

    def forward(self, obs, prev_action, prev_reward, state, done):
        # Reset memory for environments whose previous episode just ended
        mask = (1.0 - done).unsqueeze(-1)
        h, c = state[0] * mask, state[1] * mask
        a = nn.functional.one_hot(prev_action, N_ACTIONS).float()
        x = self.encoder(torch.cat([obs, a, prev_reward.unsqueeze(-1)], dim=-1))
        h, c = self.lstm(x, (h, c))
        dist = torch.distributions.Categorical(logits=self.policy(h))
        return dist, self.value(h).squeeze(-1), (h, c)


net = RecurrentActorCritic()
state = (torch.zeros(N_ENVS, HIDDEN), torch.zeros(N_ENVS, HIDDEN))
prev_action = torch.zeros(N_ENVS, dtype=torch.long)
prev_reward = torch.zeros(N_ENVS)
done = torch.zeros(N_ENVS)

for step in range(5):
    obs = torch.randn(N_ENVS, OBS_DIM)                # partial observation from each env
    dist, value, state = net(obs, prev_action, prev_reward, state, done)
    prev_action = dist.sample()
    prev_reward = torch.randn(N_ENVS)                 # would come from env.step(...)
    done = (torch.rand(N_ENVS) < 0.2).float()         # some episodes end

print(prev_action.shape, value.shape, state[0].shape)
# torch.Size([3]) torch.Size([3]) torch.Size([3, 64])

Feeding the previous action and reward as inputs is standard in meta-RL and helps any recurrent agent; the done mask resets memory exactly where an episode ended.

Training Recurrent Agents

Issue Approach
Hidden state in replay Store sequences (e.g. 80 steps) and the initial hidden state; R2D2 recomputes with a burn-in of 20-40 steps
Long episodes Truncated BPTT over fixed-length chunks
On-policy (PPO) Keep rollout order; split into sequences, not shuffled single steps
Episode boundaries Mask states with done flags
Stability Gradient clipping, layer normalization, smaller learning rates

RNN vs Alternatives for Memory in RL

Option When
Frame stacking (last k observations) Short memory needs (velocity from 4 frames in Atari)
LSTM/GRU Moderate-length dependencies, streaming control
Transformers (GTrXL, decision transformers) Long dependencies, offline RL with large data
State-space world models Model-based RL (Dreamer)

Common Pitfalls

  • Shuffling individual transitions for a recurrent policy, which destroys the temporal order it needs.

  • Using zero hidden states for replayed sequences that started mid-episode without burn-in.

  • Forgetting to reset state on done, leaking memory across episodes.

  • Adding an RNN when frame stacking already gives enough information.

Interview Tips

  • Lead with partial observability and the POMDP framing.

  • Name DRQN/R2D2 for value-based and recurrent PPO for policy-based methods, and world models.

  • Explain hidden-state handling in replay (stored state + burn-in).

39

What are RNN applications in edge computing and IoT devices?

RNNs fit edge and IoT devices because they process streaming data one step at a time with a small, constant amount of memory (the hidden state), so a tiny GRU or LSTM can run continuously on a microcontroller, phone or gateway without sending raw data to the cloud. Typical applications are predictive maintenance (vibration, temperature and current sensors), anomaly detection on sensor streams, human activity recognition from accelerometer/gyroscope data on wearables, keyword spotting and wake-word detection, energy and battery forecasting in smart meters and EVs, gesture recognition, and health monitoring (ECG arrhythmia detection). Running at the edge reduces latency, bandwidth and cost and keeps data private, but it requires small, quantized models and careful engineering for limited memory, compute and power.

Read full answer

Applies toPyTorch 2.x for training; deployment through TensorFlow Lite / LiteRT (including TFLite Micro), ExecuTorch, ONNX Runtime Mobile, or vendor toolchains (e.g. STM32Cube.AI, Edge Impulse)

Applications

Application Input Model output
Predictive maintenance Vibration, temperature, current Remaining useful life, fault class
Sensor anomaly detection Any multivariate stream Anomaly score per step
Activity recognition IMU (accelerometer, gyroscope) Walking, running, falling…
Keyword spotting Audio features (MFCC) Wake word / command
Energy forecasting Meter readings, weather Next-hour load
Health monitoring ECG, PPG Arrhythmia alerts

Why RNNs Suit Streaming Devices

Property Benefit at the edge
One step at a time Process each sample as it arrives
Constant state size Memory does not grow with stream length
Small parameter count A GRU with 32-64 units has a few thousand to tens of thousands of weights
Natural for irregular duty cycles Save the state, sleep, resume

Code Example: Streaming Inference with a GRU Cell (PyTorch)

import io

import torch
from torch import nn

torch.manual_seed(42)

N_FEATURES, HIDDEN, N_CLASSES = 6, 32, 4      # e.g. 6 IMU channels -> 4 activities


class ActivityGRU(nn.Module):
    def __init__(self):
        super().__init__()
        self.gru = nn.GRU(N_FEATURES, HIDDEN, batch_first=True)
        self.head = nn.Linear(HIDDEN, N_CLASSES)

    def forward(self, x):                      # training: whole windows
        out, _ = self.gru(x)
        return self.head(out[:, -1])


model = ActivityGRU().eval()

# Deployment form: a GRUCell that consumes one sample at a time
cell = nn.GRUCell(N_FEATURES, HIDDEN)
cell.weight_ih.data.copy_(model.gru.weight_ih_l0.data)
cell.weight_hh.data.copy_(model.gru.weight_hh_l0.data)
cell.bias_ih.data.copy_(model.gru.bias_ih_l0.data)
cell.bias_hh.data.copy_(model.gru.bias_hh_l0.data)

window = torch.randn(1, 50, N_FEATURES)        # 50 samples from the sensor
with torch.no_grad():
    batch_logits = model(window)
    h = torch.zeros(1, HIDDEN)                 # the only state kept on the device
    for t in range(window.shape[1]):           # sample-by-sample, as data arrives
        h = cell(window[:, t], h)
    stream_logits = model.head(h)

print(torch.allclose(batch_logits, stream_logits, atol=1e-5))   # True: same result, O(1) memory

# Shrink for the device: int8 dynamic quantization
qmodel = torch.ao.quantization.quantize_dynamic(model, {nn.GRU, nn.Linear}, dtype=torch.qint8)


def size_kb(m):
    buf = io.BytesIO()
    torch.save(m.state_dict(), buf)
    return buf.getbuffer().nbytes / 1024


print(f"float32: {size_kb(model):.1f} KB, int8: {size_kb(qmodel):.1f} KB")

Microcontroller targets usually need full integer quantization through the target toolchain (e.g. TFLite Micro), not PyTorch dynamic quantization; the size comparison here only shows the order of savings.

Engineering Constraints

Constraint Technique
Memory (KB of RAM) Small hidden size, int8 weights, GRU instead of LSTM (3 vs 4 gate matrices)
Compute / latency Fewer time steps (downsample), fixed-point arithmetic, vendor kernels
Power Run inference on events; sleep between windows
Model updates Over-the-air updates with versioning and rollback; federated learning for privacy
Robustness Handle sensor dropouts, reset state after gaps, calibrate per device

Common Pitfalls

  • Training on windows but deploying step-by-step without checking that the outputs match.

  • Quantizing without measuring the accuracy drop on device-like data.

  • Forgetting to reset the hidden state after a long sensor gap or reboot.

  • Choosing a model that fits flash but not RAM (activations and state need RAM).

  • Ignoring device-to-device sensor differences.

Interview Tips

  • Lead with “constant memory, step-by-step processing” as the reason RNNs fit edge devices.

  • Name two or three concrete applications with their inputs.

  • Mention quantization, small GRUs and on-device toolchains (TFLite Micro, ExecuTorch).

40

How do sequence-to-sequence models work, and where are they used?

A sequence-to-sequence (seq2seq) model maps an input sequence to an output sequence of possibly different length with two networks: an encoder that reads the input and summarizes it, and a decoder that generates the output one token at a time, conditioned on that summary and on the tokens it has already produced. In the classic RNN version, the encoder LSTM/GRU’s final hidden state initializes the decoder; because one fixed-size vector is a bottleneck for long inputs, attention (Bahdanau, Luong) lets the decoder look at all encoder states at each step. Training uses teacher forcing (feeding the true previous token) with cross-entropy loss; inference uses greedy or beam search until an end-of-sequence token. Applications include machine translation, summarization, dialogue, speech recognition, text-to-speech, code generation and time-series multi-step forecasting. Today most seq2seq systems use transformer encoder-decoders (T5, BART, Whisper), but the structure is the same.

Read full answer

Applies toPyTorch 2.x; RNN seq2seq (Sutskever et al., 2014) and attention (Bahdanau et al., 2015)

Components

Part Role
Embedding + encoder RNN Turns source tokens into hidden states
Context Final encoder state (basic) or attention-weighted states (with attention)
Decoder RNN Predicts the next target token from the previous token and its state
<sos> / <eos> tokens Start decoding / stop decoding
Teacher forcing Feed ground-truth previous tokens during training
Search Greedy, beam search, sampling at inference

Code Example: GRU Encoder-Decoder (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

SRC_VOCAB, TGT_VOCAB, EMB, HID = 50, 60, 32, 64
SOS, EOS = 1, 2


class Encoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(SRC_VOCAB, EMB, padding_idx=0)
        self.gru = nn.GRU(EMB, HID, batch_first=True)

    def forward(self, src):
        outputs, h = self.gru(self.embed(src))
        return outputs, h                       # outputs used by attention variants


class Decoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = nn.Embedding(TGT_VOCAB, EMB, padding_idx=0)
        self.gru = nn.GRU(EMB, HID, batch_first=True)
        self.out = nn.Linear(HID, TGT_VOCAB)

    def forward(self, tokens, h):
        out, h = self.gru(self.embed(tokens), h)
        return self.out(out), h


class Seq2Seq(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder, self.decoder = Encoder(), Decoder()

    def forward(self, src, tgt_in):
        _, h = self.encoder(src)
        logits, _ = self.decoder(tgt_in, h)     # teacher forcing: whole shifted target at once
        return logits

    @torch.no_grad()
    def greedy_decode(self, src, max_len=10):
        _, h = self.encoder(src)
        token = torch.full((src.shape[0], 1), SOS, dtype=torch.long)
        result = []
        for _ in range(max_len):
            logits, h = self.decoder(token, h)
            token = logits.argmax(-1)           # (batch, 1)
            result.append(token)
            if (token == EOS).all():
                break
        return torch.cat(result, dim=1)


model = Seq2Seq()
src = torch.randint(3, SRC_VOCAB, (4, 12))          # batch of 4 source sentences, length 12
tgt = torch.randint(3, TGT_VOCAB, (4, 9))           # targets of length 9
tgt_in = torch.cat([torch.full((4, 1), SOS), tgt[:, :-1]], dim=1)   # shift right

logits = model(src, tgt_in)
loss = nn.functional.cross_entropy(logits.reshape(-1, TGT_VOCAB), tgt.reshape(-1), ignore_index=0)
print(logits.shape)                    # torch.Size([4, 9, 60])
print(loss.item() > 0)                 # True
print(model.greedy_decode(src).shape)  # torch.Size([4, 10]) (fewer columns if all reach <eos>)

The untrained model’s outputs are random; the code shows the data flow: shifted targets for teacher forcing in training, and a token-by-token loop at inference.

Applications

Application Input → Output
Machine translation Sentence in language A → sentence in language B
Summarization Document → summary
Dialogue / chatbots Conversation history → reply
Speech recognition Audio frames → text
Text-to-speech Text → spectrogram frames
Code / query generation Natural language → SQL or code
Multi-step forecasting Past values → next H values

Common Pitfalls

  • The fixed-vector bottleneck without attention: quality drops sharply on long inputs.

  • Exposure bias: trained only with teacher forcing, the model never sees its own mistakes (scheduled sampling or sequence-level training help).

  • Forgetting to mask padding in the loss (ignore_index).

  • Greedy decoding producing repetitive output; use beam search or length penalties.

  • Evaluating translation with accuracy instead of BLEU/chrF or human judgement.

Interview Tips

  • Draw encoder → context → decoder and explain teacher forcing vs inference.

  • Explain why attention was added.

  • Mention that transformers replaced RNNs in most seq2seq systems but kept the encoder-decoder idea.

41

How do CNNs compare with RNNs for processing sequence data?

Both can model sequences, but they work differently: an RNN processes one time step at a time and carries a hidden state, so in principle it sees the whole history, but it cannot parallelize across time and suffers from vanishing gradients on long sequences; a 1-D CNN slides learned filters over the sequence, so it processes all time steps in parallel and trains fast and stably, but each output only sees a fixed receptive field, which must be widened with deeper stacks, larger kernels or dilation. Temporal convolutional networks (TCNs), which use causal, dilated convolutions with residual connections, often match or beat LSTMs on sequence benchmarks with faster training. In practice, CNNs are strong for local patterns (n-grams in text, motifs in sensor signals, audio features), RNNs remain good for streaming and small stateful models, and transformers now dominate tasks needing long-range dependencies.

Read full answer

Applies toPyTorch 2.x (`nn.Conv1d`, `nn.LSTM`, `nn.GRU`); the same trade-offs hold in TensorFlow/Keras

Comparison

Aspect RNN (LSTM/GRU) 1-D CNN / TCN Transformer
Computation over time Sequential Parallel Parallel
Context Unbounded in principle; weak for very long gaps Fixed receptive field (grows with depth/dilation) Full sequence via attention
Training stability Vanishing/exploding gradients; needs gating, clipping Stable Stable with normalization
Training speed on GPU Slow for long sequences Fast Fast, O(n²) memory in length
Streaming inference Natural: O(1) state per step Needs a buffer of the last receptive-field inputs Needs a growing cache
Variable length Packing/masking Padding + masking or global pooling Masking
Good at Stateful streams, small models Local patterns, audio, sensors Long-range dependencies, NLP

Code Example: Same Input, Two Models (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

batch, seq_len, n_features, hidden = 16, 100, 8, 32
x = torch.randn(batch, seq_len, n_features)

lstm = nn.LSTM(n_features, hidden, batch_first=True)

# Two conv layers; the second is dilated. Receptive field = 1 + 2*1 + 2*2 = 7 time steps.
cnn = nn.Sequential(
    nn.Conv1d(n_features, hidden, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.Conv1d(hidden, hidden, kernel_size=3, padding=2, dilation=2),
)

lstm_out, _ = lstm(x)                         # (batch, seq_len, hidden)
cnn_out = cnn(x.transpose(1, 2))              # Conv1d expects (batch, channels, seq_len)

print(lstm_out.shape)                         # torch.Size([16, 100, 32])
print(cnn_out.shape)                          # torch.Size([16, 32, 100])


def n_params(m):
    return sum(p.numel() for p in m.parameters())


print(n_params(lstm))                         # 5376 = 4 * (32*(8+32) + 2*32)
print(n_params(cnn))                          # 3904 = (8*32*3 + 32) + (32*32*3 + 32)

The LSTM’s last output depends on all 100 steps; each CNN output depends on 7 neighbouring steps. Adding layers with dilations 1, 2, 4, 8, … grows the CNN’s receptive field exponentially with depth.

Receptive Field of Dilated Convolutions

For a stack of layers with kernel size k and dilations d₁, d₂, …, the receptive field is 1 + Σ (k − 1)·dᵢ. With k = 3 and dilations 1, 2, 4, 8, 16, 32, it is 1 + 2·63 = 127 time steps from only six layers.

Choosing

Situation Prefer
Real-time stream, tiny device, one step at a time GRU/LSTM
Local patterns, fast training, fixed-length windows 1-D CNN / TCN
Long documents, translation, large data Transformer
Very long sequences with limited compute TCN or state-space models (S4, Mamba)
Text classification baseline CNN over embeddings or a small pretrained transformer

Common Pitfalls

  • Forgetting that nn.Conv1d expects (batch, channels, length) while batch_first RNNs expect (batch, length, features).

  • Using non-causal padding in a forecasting CNN, letting it see future values.

  • Assuming an RNN actually uses all of its theoretical context; measure performance by dependency length.

  • Comparing models with very different parameter counts.

Interview Tips

  • Contrast sequential vs parallel computation and unbounded vs fixed receptive field.

  • Explain dilation and the receptive-field formula.

  • Mention TCNs and transformers as the modern alternatives, and RNNs’ streaming advantage.

42

How would you add external memory mechanisms to RNNs?

External memory gives an RNN a separate, addressable memory matrix (N slots × M values) that a recurrent controller reads from and writes to at every step, so information can be stored for many steps without being squeezed through a fixed-size hidden state. The standard design, from Neural Turing Machines (NTM) and Differentiable Neural Computers (DNC), uses differentiable addressing: the controller (an LSTM) emits a key, the memory computes content-based attention (cosine similarity between the key and each slot, sharpened by a strength parameter and normalized with softmax), reads a weighted sum of slots, and writes by erasing and adding vectors with the same soft weights, so the whole system trains end-to-end with backpropagation. Related approaches include Memory Networks / key-value memories (read-only memory of facts with attention), attention over the RNN’s own past hidden states (the simplest “memory”), and retrieval-augmented models that read from an external database.

Read full answer

Applies toPyTorch 2.x; ideas from NTM (Graves et al., 2014), Memory Networks (Weston et al., 2015), DNC (Graves et al., 2016)

Design Options

Approach Memory Read Write Use
Attention over past hidden states Encoder outputs Soft attention Implicit (append each step) Seq2seq, long documents
Memory Networks / key-value memory Facts or embeddings Soft attention (multi-hop) None at inference QA, dialogue with a knowledge base
NTM N × M matrix Content + location addressing Erase + add Algorithmic tasks (copy, sort)
DNC N × M matrix + usage + temporal links Content, forward/backward links Allocation by usage Graph and reasoning tasks
Retrieval augmentation External index (vector DB) Nearest neighbours Index updates Large knowledge sources

Code Example: LSTM Controller with Content-Addressed Memory (PyTorch)

import torch
import torch.nn.functional as F
from torch import nn

torch.manual_seed(42)


class MemoryAugmentedLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, n_slots=32, slot_size=16):
        super().__init__()
        self.n_slots, self.slot_size = n_slots, slot_size
        self.controller = nn.LSTMCell(input_size + slot_size, hidden_size)
        self.read_key = nn.Linear(hidden_size, slot_size)
        self.read_beta = nn.Linear(hidden_size, 1)       # key strength (sharpness)
        self.write_key = nn.Linear(hidden_size, slot_size)
        self.write_beta = nn.Linear(hidden_size, 1)
        self.erase = nn.Linear(hidden_size, slot_size)
        self.add = nn.Linear(hidden_size, slot_size)
        self.out = nn.Linear(hidden_size + slot_size, input_size)

    def address(self, memory, key, beta):
        # Content-based addressing: softmax(beta * cosine(key, slot))
        sim = F.cosine_similarity(memory, key.unsqueeze(1), dim=-1)    # (batch, n_slots)
        return torch.softmax(F.softplus(beta) * sim, dim=-1)

    def forward(self, x):                                  # x: (batch, seq_len, input_size)
        batch, steps, _ = x.shape
        h = x.new_zeros(batch, self.controller.hidden_size)
        c = torch.zeros_like(h)
        memory = x.new_full((batch, self.n_slots, self.slot_size), 1e-3)
        read = x.new_zeros(batch, self.slot_size)
        outputs = []
        for t in range(steps):
            h, c = self.controller(torch.cat([x[:, t], read], dim=-1), (h, c))
            # Write: erase then add, weighted by the write attention
            w_w = self.address(memory, self.write_key(h), self.write_beta(h)).unsqueeze(-1)
            erase = torch.sigmoid(self.erase(h)).unsqueeze(1)
            add = torch.tanh(self.add(h)).unsqueeze(1)
            memory = memory * (1 - w_w * erase) + w_w * add
            # Read: weighted sum of slots
            w_r = self.address(memory, self.read_key(h), self.read_beta(h))
            read = torch.bmm(w_r.unsqueeze(1), memory).squeeze(1)
            outputs.append(self.out(torch.cat([h, read], dim=-1)))
        return torch.stack(outputs, dim=1), memory


model = MemoryAugmentedLSTM(input_size=8, hidden_size=64)
x = torch.randn(4, 20, 8)
y, memory = model(x)
print(y.shape)        # torch.Size([4, 20, 8])
print(memory.shape)   # torch.Size([4, 32, 16])

This is a simplified NTM (content addressing only, one read and one write head). A full NTM adds location-based shifts and interpolation; a DNC adds memory allocation and temporal links. A common sanity test is the copy task: show a random bit sequence, then ask the model to reproduce it after a delay longer than an LSTM can reliably hold.

Practical Advice

Concern Recommendation
Training stability Gradient clipping, small learning rate, curriculum on sequence length
Memory initialization Small constant or learned values (not zeros, which make cosine similarity undefined)
Cost O(N × M) per step; keep N moderate or use sparse/top-k addressing
Simpler alternative first Attention over encoder states or a transformer often solves the same problem

Common Pitfalls

  • Initializing memory to all zeros (cosine similarity divides by zero norms).

  • Expecting NTM-style models to train as easily as LSTMs; they are sensitive to hyperparameters.

  • Using external memory where plain attention is enough.

  • Forgetting that memory must be reset between independent sequences.

Interview Tips

  • Explain the controller + memory + read/write heads structure.

  • Describe content-based addressing (cosine similarity + softmax) and erase/add writes.

  • Mention attention and retrieval augmentation as the practical modern forms of memory.

43

How can RNNs be used for anomaly detection in sequential data?

RNNs detect anomalies in sequences by learning what normal behaviour looks like and flagging points or windows the model explains badly. Two approaches dominate: prediction-based, where an LSTM/GRU trained on normal data forecasts the next value(s) and the prediction error (residual) becomes the anomaly score; and reconstruction-based, where an LSTM autoencoder compresses each window and reconstructs it, and a large reconstruction error signals an anomaly. In both, the threshold is set on errors from held-out normal data (for example the 99th percentile, or mean + k·std), sometimes with smoothing and a minimum anomaly length to cut false alarms. RNNs suit this because they capture temporal context: a value that is normal at one moment can be abnormal after a particular history.

Read full answer

Applies toPyTorch 2.x; univariate or multivariate time series (sensors, transactions, logs, network traffic)

Approaches

Approach Score Good for
Next-step forecasting (LSTM/GRU) ‘ y_t − ŷ_t ‘ or error normalized by rolling std Point anomalies, real-time streams
LSTM autoencoder Reconstruction error per window Contextual and collective (pattern) anomalies
Probabilistic forecasting Negative log-likelihood under a predicted distribution Heteroscedastic data
Sequence classification Probability of “anomalous” When labelled anomalies exist

Code Example: Forecasting-Based Detection (PyTorch)

import numpy as np
import torch
from torch import nn

torch.manual_seed(42)
rng = np.random.default_rng(42)

# Synthetic normal signal: sine wave + small noise
t = np.arange(1200)
series = np.sin(2 * np.pi * t / 50) + 0.05 * rng.standard_normal(len(t))
series[1000] += 2.5                           # injected point anomaly in the test region

WINDOW = 30


def make_windows(x):
    X = np.stack([x[i:i + WINDOW] for i in range(len(x) - WINDOW)])
    y = x[WINDOW:]
    return torch.tensor(X, dtype=torch.float32).unsqueeze(-1), torch.tensor(y, dtype=torch.float32)


class Forecaster(nn.Module):
    def __init__(self, hidden=32):
        super().__init__()
        self.lstm = nn.LSTM(1, hidden, batch_first=True)
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):
        out, _ = self.lstm(x)
        return self.head(out[:, -1]).squeeze(-1)


train_X, train_y = make_windows(series[:700])          # normal data only
val_X, val_y = make_windows(series[700:900])           # normal data for the threshold
test_X, test_y = make_windows(series[900:])            # contains the anomaly

model = Forecaster()
opt = torch.optim.Adam(model.parameters(), lr=1e-2)
loss_fn = nn.MSELoss()
for epoch in range(60):
    opt.zero_grad()
    loss = loss_fn(model(train_X), train_y)
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()

model.eval()
with torch.no_grad():
    val_err = (model(val_X) - val_y).abs().numpy()
    test_err = (model(test_X) - test_y).abs().numpy()

threshold = np.quantile(val_err, 0.99)                 # set on normal validation data
flagged = np.where(test_err > threshold)[0] + 900 + WINDOW   # map back to series index
print(f"threshold = {threshold:.3f}")
print("flagged indices:", flagged[:10])

With this setup the spike at index 1000 should produce one of the largest errors; the following few steps may also be flagged because the spike is inside their input window. Points after a true anomaly being flagged is normal behaviour for forecasting detectors, which is why alerts are usually grouped into events.

LSTM Autoencoder Skeleton

import torch
from torch import nn


class LSTMAutoencoder(nn.Module):
    def __init__(self, n_features=1, latent=16):
        super().__init__()
        self.encoder = nn.LSTM(n_features, latent, batch_first=True)
        self.decoder = nn.LSTM(latent, latent, batch_first=True)
        self.out = nn.Linear(latent, n_features)

    def forward(self, x):                                      # x: (batch, window, features)
        _, (h, _) = self.encoder(x)
        repeated = h[-1].unsqueeze(1).repeat(1, x.shape[1], 1)  # latent code at every step
        dec, _ = self.decoder(repeated)
        return self.out(dec)


ae = LSTMAutoencoder()
windows = torch.randn(8, 30, 1)
score = (ae(windows) - windows).pow(2).mean(dim=(1, 2))        # one anomaly score per window
print(score.shape)                                              # torch.Size([8])

Evaluation and Operations

Topic Practice
Labels Usually few; evaluate with precision@k, event-level recall, or time-to-detect on labelled incidents
Threshold Quantile of validation error; adjust to the alert budget
Seasonality Include time-of-day/week features or train per regime
Drift Retrain on recent normal data; a drifting baseline raises errors everywhere
Multivariate Score per channel and combined; report which channels contributed

Common Pitfalls

  • Training on data that already contains anomalies, so the model learns to predict them.

  • Setting the threshold on the test data.

  • Judging with point-wise accuracy on highly imbalanced data; use precision/recall at the alert level.

  • Ignoring the alarms that follow a real anomaly in forecasting detectors.

  • Scaling with statistics computed on the whole series (leakage).

Interview Tips

  • Name the two approaches (forecast residual vs reconstruction error) and how the threshold is chosen.

  • Explain why sequence context matters (contextual anomalies).

  • Discuss evaluation with few labels and alert fatigue.

44

How are RNNs applied in multi-agent systems, and what makes it complex?

In multi-agent systems, RNNs (usually GRUs or LSTMs) give each agent a memory of its own observation and action history, which is essential because agents typically see only part of the environment (partial observability) and must infer the hidden state of the world and of the other agents from what happened over time. They appear in three main places: recurrent agent policies or Q-networks (DRQN-style agents in QMIX, VDN and MAPPO, often with parameter sharing across agents), learned communication, where messages produced by one agent’s recurrent network feed into others (CommNet, DIAL, TarMAC), and modelling other agents, predicting teammates’ or opponents’ next actions from their past behaviour. The complexity comes from non-stationarity (other agents keep learning), credit assignment across agents, training recurrent policies from replay (hidden states at sampled time steps), scaling with the number of agents, and communication limits.

Read full answer

Applies toMulti-agent reinforcement learning (MARL) with PyTorch 2.x; frameworks such as PettingZoo, EPyMARL, RLlib

Where RNNs Fit

Role How the RNN is used Examples
Agent policy / Q-network GRU over each agent’s observation history QMIX, VDN, MAPPO with recurrent actors
Communication Hidden states produce messages; received messages are inputs at the next step CommNet, DIAL, TarMAC
Opponent / teammate modelling Predict other agents’ actions or goals from their trajectories Theory-of-mind style models
Centralized critic Recurrent critic over joint history during training only Centralized training, decentralized execution (CTDE)

Code Example: Shared Recurrent Policy for N Agents (PyTorch)

import torch
from torch import nn

torch.manual_seed(42)

N_AGENTS, OBS_DIM, N_ACTIONS, HIDDEN = 3, 10, 5, 64


class SharedRecurrentAgent(nn.Module):
    """One GRU policy shared by all agents; an agent-ID one-hot lets behaviour differ."""

    def __init__(self):
        super().__init__()
        self.encoder = nn.Linear(OBS_DIM + N_AGENTS + N_ACTIONS, HIDDEN)
        self.gru = nn.GRUCell(HIDDEN, HIDDEN)
        self.q_head = nn.Linear(HIDDEN, N_ACTIONS)

    def init_hidden(self, batch):
        return torch.zeros(batch * N_AGENTS, HIDDEN)

    def forward(self, obs, last_action, h):
        # obs: (batch, agents, obs_dim); last_action: (batch, agents) int; h: (batch*agents, hidden)
        batch = obs.shape[0]
        agent_id = torch.eye(N_AGENTS).expand(batch, -1, -1)
        last_a = nn.functional.one_hot(last_action, N_ACTIONS).float()
        x = torch.cat([obs, agent_id, last_a], dim=-1).reshape(batch * N_AGENTS, -1)
        h = self.gru(torch.relu(self.encoder(x)), h)
        q = self.q_head(h).reshape(batch, N_AGENTS, N_ACTIONS)
        return q, h


agent = SharedRecurrentAgent()
batch = 4
h = agent.init_hidden(batch)
last_action = torch.zeros(batch, N_AGENTS, dtype=torch.long)

for t in range(6):                                     # roll out 6 environment steps
    obs = torch.randn(batch, N_AGENTS, OBS_DIM)        # each agent sees only its own observation
    q, h = agent(obs, last_action, h)
    last_action = q.argmax(dim=-1)                     # greedy decentralized actions

print(q.shape)            # torch.Size([4, 3, 5])   Q-values per environment, agent, action
print(h.shape)            # torch.Size([12, 64])    one hidden state per agent per environment
print(last_action.shape)  # torch.Size([4, 3])

Each agent acts only on its own observation and memory (decentralized execution); during training, a mixer (QMIX) or centralized critic (MAPPO) combines agents’ values using global state.

Complexities

Problem Why it happens Common mitigation
Non-stationarity Other agents’ policies change during training CTDE, centralized critics, slower target updates
Credit assignment One team reward for many agents’ actions Value decomposition (VDN, QMIX), counterfactual baselines (COMA)
Recurrent replay Stored hidden states become stale as weights change Store full episodes or sequences; burn-in steps (R2D2)
Scaling with agents Joint action space grows exponentially Parameter sharing, mean-field approximations, attention over neighbours
Communication Bandwidth, delays, learned protocols hard to interpret Discrete or gated messages, attention-based targeting
Exploration Coordinated exploration is rare by chance Shared intrinsic rewards, role-based policies
Evaluation Results vary with opponents and seeds Many seeds, cross-play against held-out agents

Common Pitfalls

  • Forgetting to reset hidden states at episode boundaries for each environment in a batch.

  • Training recurrent agents on single transitions from replay without sequences or burn-in.

  • Giving agents global state at execution time when the deployment only provides local observations.

  • Sharing parameters without an agent-ID input, so agents cannot specialize.

Interview Tips

  • Lead with partial observability: memory is why RNNs matter here.

  • Name CTDE, parameter sharing and value decomposition (QMIX).

  • Mention non-stationarity and recurrent replay as the hardest training issues.

45

How do you deploy an RNN model to production, and what are the challenges?

Deploying an RNN means turning a trained checkpoint into a fast, versioned service: freeze and package the model together with its preprocessing (tokenizer or scaler), export it to a serving format (TorchScript, ONNX or a TensorFlow SavedModel), optimize it (dynamic quantization, batching, length bucketing), serve it behind an API or in a batch/streaming job (TorchServe, Triton, TF Serving, FastAPI, Spark), and roll it out gradually with monitoring and rollback. The challenges specific to RNNs are sequential computation (latency grows with sequence length and does not parallelize across time steps), variable-length inputs (padding, masking and batching), stateful streaming (keeping the hidden state per user or device), training/serving skew in preprocessing, and resource limits on CPUs and edge devices.

Read full answer

Applies toPyTorch 2.x (TorchScript, ONNX export, `torch.ao.quantization`); TensorFlow 2.16+ has equivalent tools (SavedModel, TF Serving, TFLite)

Deployment Steps

Step Tooling Notes
1. Freeze artifact Checkpoint + vocab/scaler + config One versioned bundle
2. Export TorchScript (torch.jit.script/trace), ONNX (torch.onnx.export), SavedModel Removes Python training code from serving
3. Optimize Dynamic int8 quantization of nn.LSTM/nn.Linear, ONNX Runtime, length bucketing CPU speed and memory
4. Serve TorchServe, Triton, TF Serving, FastAPI; Spark/Beam for batch Dynamic batching on GPUs
5. Validate Same outputs as the training model on a golden set; latency tests by length Catch export bugs
6. Roll out Shadow → canary → full, with rollback Compare on live traffic
7. Monitor Latency per length bucket, drift, quality See RNN monitoring

Code Example: Export, Quantize and Verify (PyTorch)

import io

import torch
from torch import nn

torch.manual_seed(42)


class Forecaster(nn.Module):
    def __init__(self, n_features=4, hidden=64):
        super().__init__()
        self.lstm = nn.LSTM(n_features, hidden, num_layers=2, batch_first=True)
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):                 # x: (batch, seq_len, n_features)
        out, _ = self.lstm(x)
        return self.head(out[:, -1])      # predict from the last time step


model = Forecaster().eval()
x = torch.randn(8, 30, 4)

# 1. Dynamic quantization: int8 weights for LSTM and Linear, float activations
qmodel = torch.ao.quantization.quantize_dynamic(model, {nn.LSTM, nn.Linear}, dtype=torch.qint8)


def size_kb(m):
    buf = io.BytesIO()
    torch.save(m.state_dict(), buf)
    return buf.getbuffer().nbytes / 1024


print(f"float32 size: {size_kb(model):.0f} KB, int8 size: {size_kb(qmodel):.0f} KB")

# 2. TorchScript export of the float model (no Python class needed to serve it)
scripted = torch.jit.trace(model, x)
scripted.save("forecaster.pt")
restored = torch.jit.load("forecaster.pt")

with torch.no_grad():
    ref = model(x)
    print(torch.allclose(ref, restored(x), atol=1e-6))           # True: export is faithful
    print(f"max abs difference after int8 quantization: {(ref - qmodel(x)).abs().max().item():.4f}")
print(restored(x).shape)                                         # torch.Size([8, 1])

Quantization roughly divides the weight size by four; always check the accuracy impact on a validation set, not only the raw difference shown here. Tracing records one execution path, which is fine for this model; use torch.jit.script when forward has data-dependent control flow.

Serving Variable-Length Batches

import torch
from torch.nn.utils.rnn import pack_padded_sequence, pad_sequence

seqs = [torch.randn(n, 4) for n in (12, 30, 7)]               # three requests of different lengths
lengths = torch.tensor([len(s) for s in seqs])
padded = pad_sequence(seqs, batch_first=True)                  # (3, 30, 4)
packed = pack_padded_sequence(padded, lengths, batch_first=True, enforce_sorted=False)
lstm = torch.nn.LSTM(4, 16, batch_first=True)
_, (h, _) = lstm(packed)
print(h[-1].shape)                                             # torch.Size([3, 16]): last real step of each

Grouping requests of similar length (length bucketing) reduces wasted computation on padding.

Main Challenges and Mitigations

Challenge Mitigation
Latency grows with sequence length Cap max_len, truncate old history, smaller hidden size, GRU instead of LSTM, quantization
Poor GPU parallelism over time Batch many requests; consider a 1-D CNN or transformer if latency is critical
Variable-length inputs Padding + packing/masking; length bucketing
Stateful streaming Store (h, c) per session with TTL; reset on gaps; version-tagged keys
Preprocessing skew Ship tokenizer/scaler in the same artifact; test end-to-end with raw inputs
Export issues (custom ops, dynamic shapes) Test ONNX with dynamic axes for batch and sequence; compare outputs
Edge hardware limits TFLite / ExecuTorch / ONNX Runtime Mobile, int8, pruning

Common Pitfalls

  • Validating only the Python model and not the exported one.

  • Forgetting model.eval() (dropout active at serving time).

  • Padding inputs but taking output[:, -1] from padded positions instead of the last real step.

  • Unbounded per-session hidden-state storage.

  • Retraining with new preprocessing while the service keeps the old version.

Interview Tips

  • Walk through package → export → optimize → serve → roll out → monitor.

  • Name RNN-specific issues: sequential latency, variable lengths, stateful serving.

  • Mention verifying exported outputs against the original model.

46

How do you monitor and maintain an RNN model in production?

Monitoring an RNN in production means watching four layers continuously — the service (latency, throughput, errors, memory), the inputs (drift in sequence length, vocabulary/out-of-vocabulary rate, feature statistics, missing time steps), the outputs (prediction distribution, confidence) and, once labels arrive, the actual quality metric — and maintaining it means acting on those signals with scheduled or triggered retraining, versioned rollouts (shadow, canary) and rollback. RNN-specific risks need extra checks: latency grows with sequence length, stateful streaming models can carry a corrupted or stale hidden state, the vocabulary or normalization statistics must match the training run exactly, and long sequences can be silently truncated by a max_len setting.

Read full answer

Applies toPyTorch 2.x / TensorFlow 2.16+ models served online or in batch; any monitoring stack (Prometheus/Grafana, Evidently, WhyLabs, cloud model monitors)

What to Monitor

Layer Signal Why it matters for RNNs
Service p50/p95/p99 latency per sequence-length bucket Cost grows with time steps
Service Throughput, error rate, GPU/CPU memory Long sequences can exhaust memory
Input Sequence-length distribution, truncation rate Truncated inputs lose context
Input OOV / unknown-token rate New words or item IDs the vocab does not know
Input Feature mean/std, missing or irregular time steps Normalization mismatch, sensor gaps
Output Prediction distribution, mean confidence, entropy Early warning before labels arrive
Quality Task metric on delayed labels (accuracy, F1, MAE) The real target
State Hidden-state norm / resets per session (stateful models) Detect exploding or stale state

Code Example: Input Drift Checks (Python)

import numpy as np

rng = np.random.default_rng(42)


def psi(expected, actual, bins):
    """Population Stability Index between two samples over fixed bins."""
    e_counts, _ = np.histogram(expected, bins=bins)
    a_counts, _ = np.histogram(actual, bins=bins)
    e = np.clip(e_counts / e_counts.sum(), 1e-6, None)
    a = np.clip(a_counts / a_counts.sum(), 1e-6, None)
    return float(np.sum((a - e) * np.log(a / e)))


def oov_rate(token_lists, vocab):
    tokens = [t for seq in token_lists for t in seq]
    return sum(t not in vocab for t in tokens) / max(len(tokens), 1)


# Reference window (training data) vs. live window (last 24 h): synthetic sequence lengths
train_lengths = rng.poisson(lam=40, size=5000)
live_lengths = rng.poisson(lam=55, size=1000)        # users now send longer sequences
bins = np.arange(0, 121, 10)

length_psi = psi(train_lengths, live_lengths, bins)
truncation_rate = float(np.mean(live_lengths > 64))   # model was trained with max_len=64

vocab = {"<pad>", "<unk>", "the", "order", "late", "refund", "delivery"}
live_tokens = [["the", "order", "late"], ["refund", "crypto", "wallet"], ["delivery", "drone"]]

print(f"length PSI      = {length_psi:.3f}")
print(f"truncation rate = {truncation_rate:.1%}")
print(f"OOV rate        = {oov_rate(live_tokens, vocab):.1%}")

alerts = []
if length_psi > 0.2:
    alerts.append("sequence-length drift")
if truncation_rate > 0.05:
    alerts.append("too many truncated inputs")
if oov_rate(live_tokens, vocab) > 0.1:
    alerts.append("vocabulary drift")
print("alerts:", alerts)

The thresholds (PSI > 0.2, truncation > 5%, OOV > 10%) are common starting points; tune them on your own history. In this synthetic example the live lengths were shifted on purpose, so the length checks are expected to fire, and 3 of the 9 live tokens are unknown (33%).

Maintenance Loop

Step Practice
Log Store inputs (or summaries), predictions, model version and latency per request
Join labels Attach delayed ground truth (e.g. churn after 30 days) to logged predictions
Evaluate Daily/weekly metric by segment and by sequence-length bucket
Trigger Retrain on schedule or when drift/quality alerts fire
Validate Compare candidate vs. current on the same recent holdout; check latency
Release Shadow → canary → full rollout, with automatic rollback on metric or latency regression
Version Tie weights, vocabulary, scaler and max_len together in one versioned artifact

Stateful (Streaming) RNNs

  • Reset the hidden state at session boundaries and after gaps longer than a threshold.

  • Persist per-session state with a TTL (e.g. Redis) and include the model version in the key, so a new model never reads an old model’s state.

  • Alert on unusually large hidden-state norms; they often indicate bad inputs.

Common Pitfalls

  • Monitoring only service health and never the model’s inputs and outputs.

  • Retraining with a new vocabulary but serving with the old tokenizer (or the reverse).

  • Averaging latency over all requests, hiding slow long-sequence requests.

  • Evaluating only on random samples instead of recent data, which hides drift.

  • Silent truncation of long sequences.

Interview Tips

  • Structure the answer by layer: service, input, output, quality.

  • Name RNN-specific checks: sequence length, truncation, OOV rate, hidden-state handling.

  • Describe the retraining and rollout loop, not just the dashboards.

47

What is model versioning, and why does it matter for deployed RNNs?

Model versioning means giving every trained model a unique, immutable identifier and recording everything needed to reproduce and serve it: the weights, the exact preprocessing artifacts, the code version, the hyperparameters, the training data snapshot and the evaluation results. It matters for any production model because it enables rollback, reproducibility, A/B comparison and audits, and it matters even more for RNNs because the weights are tightly coupled to things outside the weights file: the vocabulary/tokenizer (token IDs index the embedding table), normalization statistics, max_len and padding side, the hidden-state shape (layers, hidden size, bidirectional), and, for stateful streaming models, the stored hidden states themselves. Serving weights with the wrong vocabulary does not crash; it silently produces wrong predictions.

Read full answer

Applies toPyTorch 2.x (same ideas for TensorFlow/Keras); registries such as MLflow Model Registry, SageMaker, Vertex AI, W&B; data versioning with DVC or lakeFS

What Belongs in an RNN Model Version

Artifact Why it must be versioned with the weights
state_dict / SavedModel The model itself
Vocabulary / tokenizer files Token → ID mapping must match the embedding rows
Scaler / normalization stats Time-series inputs must be scaled exactly as in training
Config (hidden_size, num_layers, bidirectional, max_len, padding side) Needed to rebuild the model and preprocess inputs
Code commit and library versions Behaviour of layers and preprocessing can change
Data snapshot / query + date range Reproduce and debug training
Metrics and evaluation report Decide promotion and compare versions
Content hash Detect corrupted or mismatched files

Code Example: A Self-Describing Checkpoint (PyTorch)

import hashlib
import io
import json

import torch
from torch import nn

torch.manual_seed(42)


class TextClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_size, num_classes):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(embed_dim, hidden_size, batch_first=True)
        self.head = nn.Linear(hidden_size, num_classes)

    def forward(self, ids):
        _, (h, _) = self.lstm(self.embed(ids))
        return self.head(h[-1])


vocab = {"<pad>": 0, "<unk>": 1, "late": 2, "refund": 3, "great": 4, "service": 5}
config = {"vocab_size": len(vocab), "embed_dim": 16, "hidden_size": 32, "num_classes": 2}
model = TextClassifier(**config)


def save_version(model, vocab, config, version, path):
    buffer = io.BytesIO()
    torch.save(model.state_dict(), buffer)
    weights = buffer.getvalue()
    bundle = {
        "version": version,
        "config": config,
        "preprocessing": {"vocab": vocab, "max_len": 64, "padding": "post", "lowercase": True},
        "weights_sha256": hashlib.sha256(weights).hexdigest(),
        "state_dict": model.state_dict(),
        "code_commit": "abc1234",          # filled in by CI in practice
    }
    torch.save(bundle, path)
    return bundle["weights_sha256"]


def load_version(path):
    bundle = torch.load(path, weights_only=False)   # only load files you trust
    model = TextClassifier(**bundle["config"])
    model.load_state_dict(bundle["state_dict"])
    buffer = io.BytesIO()
    torch.save(model.state_dict(), buffer)
    if hashlib.sha256(buffer.getvalue()).hexdigest() != bundle["weights_sha256"]:
        raise ValueError("weights hash mismatch")
    if len(bundle["preprocessing"]["vocab"]) != model.embed.num_embeddings:
        raise ValueError("vocabulary does not match the embedding table")
    return model.eval(), bundle["preprocessing"], bundle["version"]


save_version(model, vocab, config, "sentiment-lstm-1.3.0", "sentiment-lstm-1.3.0.pt")
loaded, prep, version = load_version("sentiment-lstm-1.3.0.pt")
print(version, json.dumps({"max_len": prep["max_len"], "vocab_size": len(prep["vocab"])}))
# sentiment-lstm-1.3.0 {"max_len": 64, "vocab_size": 6}

Bundling the vocabulary and config with the weights means the serving code cannot pick up a mismatched tokenizer by accident. A registry (MLflow, SageMaker, Vertex AI) adds stages (staging, production), lineage and access control on top.

Versioning Scheme

Change Version bump (semantic) Example
Retrained on new data, same interface Patch: 1.3.0 → 1.3.1 Weekly retrain
New vocabulary or features, same API Minor: 1.3 → 1.4 Added tokens; clients unaffected
Input/output contract changes Major: 1.x → 2.0 New label set, different input format

Why It Matters for Deployed RNNs

  • Rollback: a bad retrain can be reverted in minutes when the previous bundle is intact.

  • Stateful serving: stored hidden states are only valid for the version that produced them; keys like session:{id}:v1.3.0 prevent a new model from reading old states.

  • Comparisons: A/B tests and shadow deployments need two clearly identified versions.

  • Audit and debugging: “which model made this prediction?” requires the version in every log line.

Common Pitfalls

  • Versioning the weights file but not the tokenizer, scaler or max_len.

  • Overwriting model_latest.pt in place, which makes rollback impossible.

  • Not logging the model version with each prediction.

  • Loading untrusted checkpoints with weights_only=False (pickle can execute code).

  • Changing the label order between versions without bumping the major version.

Interview Tips

  • Define versioning as “weights + preprocessing + config + code + data + metrics”.

  • Stress the RNN-specific coupling: vocabulary and hidden-state shape.

  • Mention a registry, semantic versions and per-prediction version logging.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 47 RNN answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects