What are RNNs, and how do they differ from feedforward networks?
A Recurrent Neural Network (RNN) is a neural network for sequential data that processes inputs one step at a time and keeps a hidden state — a memory vector updated at each step from the current input and the previous state (h_t = tanh(W_hh h_(t−1) + W_xh x_t + b)) — using the same weights at every step. A feedforward network (MLP, and in this sense CNNs too) maps a fixed-size input to an output in a single pass with no loops and no memory: each input is processed independently. The key differences are that RNNs have a cycle (recurrence), share weights across time, handle variable-length sequences, and take order into account, while feedforward networks need fixed-size inputs and treat each example on its own. RNNs are trained with backpropagation through time, which brings vanishing and exploding gradient issues, so in practice the gated variants (LSTM, GRU) are used.
Read full answerHide full answer
Applies toPyTorch 2.x (`nn.RNN`, `nn.Linear`)
Comparison
| Aspect | Feedforward network | RNN |
|---|---|---|
| Connections | Input → hidden → output, no cycles | Hidden state feeds back to itself |
| Memory | None between inputs | Hidden state carries context |
| Input size | Fixed | Variable-length sequences |
| Order sensitivity | Only through fixed input positions | Built in |
| Weight sharing | Separate weights per layer | Same weights at every time step |
| Training | Standard backprop | Backpropagation through time |
| Gradient issues | Mostly with depth | Also over sequence length |
| Parallelism | Fully parallel over inputs | Sequential over time steps |
| Typical uses | Tabular data, fixed-size images (CNNs) | Text, speech, time series, event streams |
Code Example: Same Data, Two Models (PyTorch)
import torch
from torch import nn
torch.manual_seed(42)
batch, T, F, H = 4, 6, 3, 8
# Feedforward: must flatten a fixed-length sequence into one vector
mlp = nn.Sequential(nn.Flatten(), nn.Linear(T * F, H), nn.ReLU(), nn.Linear(H, 1))
x = torch.randn(batch, T, F)
print(mlp(x).shape) # torch.Size([4, 1])
try:
mlp(torch.randn(batch, T + 2, F)) # longer sequence
except RuntimeError:
print("MLP fails on a different length") # weights are tied to input size T * F
class RNNModel(nn.Module):
def __init__(self):
super().__init__()
self.rnn = nn.RNN(F, H, batch_first=True) # weights depend only on F and H
self.head = nn.Linear(H, 1)
def forward(self, x):
_, h_n = self.rnn(x)
return self.head(h_n[-1])
rnn_model = RNNModel()
print(rnn_model(x).shape, rnn_model(torch.randn(batch, T + 2, F)).shape)
# torch.Size([4, 1]) torch.Size([4, 1])
n_mlp = sum(p.numel() for p in mlp.parameters())
n_rnn = sum(p.numel() for p in rnn_model.parameters())
print(n_mlp, n_rnn) # 161 113
The MLP’s first layer has T·F inputs, so its size is tied to the sequence length (18·8 + 8 + 8 + 1 = 161 parameters). The RNN’s parameters (3·8 + 8·8 + 8 + 8 = 104 in the cell, plus 9 in the head = 113) do not depend on T, so it accepts any length.
The Recurrence Unrolled
An RNN over T steps is equivalent to a T-layer feedforward network whose layers share weights and each receive one input x_t. That view explains both its strength (one set of weights for any length) and its training difficulty (a very deep network with repeated multiplication by the same matrix).
Common Pitfalls
-
Saying feedforward networks “cannot” handle sequences; with fixed windows (or CNNs/transformers) they can, but without a recurrent state.
-
Forgetting that vanilla RNNs struggle with long-range dependencies; mention LSTM/GRU.
-
Claiming RNNs are the default for sequences today; transformers dominate large-scale NLP.
Interview Tips
-
Define the hidden-state update and weight sharing in one sentence each.
-
Contrast memory, variable length and order handling with feedforward networks.
-
Mention BPTT, vanishing gradients, LSTM/GRU and where transformers fit.