Skip to content
Machine Learning & Data Science

PyTorch interview questions and answers

50 questions, 48 with code. About 2 h 36 min to read them all.

By TestLab · Updated

Plan for an interview
50 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is PyTorch and how does it differ from TensorFlow?

PyTorch is an open-source deep learning framework, originally from Meta AI and now governed by the PyTorch Foundation under the Linux Foundation, that provides GPU-accelerated tensors, automatic differentiation (autograd) and neural network building blocks with a Pythonic, define-by-run (eager) style: the computation graph is built as the code runs, so you debug models with ordinary Python tools. TensorFlow (with its Keras API) also runs eagerly by default since 2.x and offers tf.function graphs; the practical differences today are ecosystem and deployment tooling rather than core capability. PyTorch dominates research and much of industry training (Hugging Face models, most papers), while TensorFlow keeps strengths in TF Lite / LiteRT for mobile, TF Serving and TPU pipelines.

Read full answer

Applies toPyTorch 2.x, TensorFlow 2.x / Keras 3

Comparison

Aspect PyTorch TensorFlow / Keras
Execution Eager by default; torch.compile for graph speed-ups Eager by default; tf.function for graphs
API style Python classes (nn.Module), explicit training loops High-level model.fit (Keras) plus low-level ops
Debugging Standard Python debugger, prints Easy in eager mode; harder inside tf.function
Research adoption Dominant Smaller share of new papers
Deployment TorchServe, torch.export, ExecuTorch, ONNX TF Serving, TF Lite / LiteRT, TF.js
Distributed DDP, FSDP, torch.distributed tf.distribute strategies
Hardware CUDA, ROCm, Apple MPS, XLA (TPU via PyTorch/XLA) CUDA, TPU (native)
Ecosystem Hugging Face, Lightning, torchvision, PyG Keras, TFX, TF Hub

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"

model = nn.Sequential(nn.Linear(3, 8), nn.ReLU(), nn.Linear(8, 1)).to(device)
x = torch.randn(4, 3, device=device)
y = model(x)                  # runs immediately: you can print or set breakpoints here
print(y.shape)                # torch.Size([4, 1])
loss = y.pow(2).mean()
loss.backward()               # gradients for every parameter
print(model[0].weight.grad.shape)   # torch.Size([8, 3])

When to Choose Which

  • PyTorch: research, custom architectures, LLM work, Hugging Face-based projects.

  • TensorFlow: existing TF infrastructure, TF Lite mobile deployment, TFX pipelines.

  • Keras 3 can run on top of JAX, TensorFlow or PyTorch, which blurs the line further.

Common Pitfalls in Answers

  • Saying TensorFlow only uses static graphs (true only of TF 1.x).

  • Claiming one framework is strictly faster; performance depends on the model and compiler.

Interview Tips

  • Lead with eager execution, Pythonic design and ecosystem.

  • Mention torch.compile and deployment options to show current knowledge.

2

What are tensors in PyTorch?

A PyTorch tensor (torch.Tensor) is a multi-dimensional array, similar to a NumPy array, with three extra abilities: it can live on a GPU or other accelerator (device), it can record operations for automatic differentiation (requires_grad=True), and it is the input and output type of every PyTorch model. Each tensor has a shape, a dtype (for example float32, int64, bfloat16) and a device, and supports broadcasting, indexing and hundreds of mathematical operations.

Read full answer

Applies toPyTorch 2.x

Key Attributes

Attribute Example Meaning
shape / size() torch.Size([2, 3]) Dimensions
dtype torch.float32 Element type
device cpu, cuda:0, mps Where the data lives
requires_grad True Track operations for autograd
grad Tensor or None Gradient after backward()
ndim 2 Number of dimensions

Code Example

import torch

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"

a = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
print(a.shape, a.dtype, a.device)          # torch.Size([2, 3]) torch.float32 cpu

z = torch.zeros(2, 3)
r = torch.randn(2, 3)                      # standard normal
i = torch.arange(6).reshape(2, 3)          # int64 by default
print(i.dtype)                             # torch.int64

b = a + torch.tensor([10.0, 20.0, 30.0])   # broadcasting the row across both rows
print(b)                                   # tensor([[11., 22., 33.], [14., 25., 36.]])
print(a @ a.T)                             # matrix product: tensor([[14., 32.], [32., 77.]])
print(a[:, 1])                             # tensor([2., 5.])

a_dev = a.to(device)                       # move to the GPU if available
w = torch.ones(3, requires_grad=True)
out = (a @ w).sum()                        # 6 + 15 = 21
out.backward()
print(w.grad)                              # tensor([5., 7., 9.]): column sums of a

Tensors vs NumPy Arrays

Aspect Tensor NumPy array
GPU support Yes No (CPU only)
Autograd Yes No
Default float float32 float64
Conversion torch.from_numpy(x) (shares memory) t.numpy() (CPU tensors)

Common Pitfalls

  • Mixing devices in one operation (CPU tensor + CUDA tensor raises an error).

  • Integer tensors where floats are needed (torch.arange gives int64).

  • In-place operations (add_) on tensors that autograd needs.

Interview Tips

  • Name shape, dtype, device and requires_grad.

  • Show broadcasting and moving to a device.

3

What is the difference between a Tensor and a Variable in PyTorch?

In modern PyTorch there is no practical difference: Variable was a wrapper used before PyTorch 0.4 (2018) to add autograd features (.data, .grad, .grad_fn) to tensors, and since 0.4 tensors themselves support autograd, so torch.autograd.Variable(t) simply returns a tensor and is deprecated. Today you create a tensor with requires_grad=True (or use nn.Parameter inside modules) to track gradients, and torch.no_grad() or .detach() to stop tracking.

Read full answer

Applies toPyTorch 2.x (history from 0.3 and earlier)

Then and Now

Task Before 0.4 PyTorch 0.4 and later
Track gradients Variable(t, requires_grad=True) torch.tensor(..., requires_grad=True) or t.requires_grad_()
Get the raw data v.data t.detach() (safer than .data)
Disable tracking volatile=True with torch.no_grad(): or torch.inference_mode()
Scalar value v.data[0] t.item()
Learnable weight Variable in a module nn.Parameter

Code Example

import torch
import torch.nn as nn
from torch.autograd import Variable

x = torch.tensor([2.0, 3.0], requires_grad=True)   # the modern way
y = (x ** 2).sum()
y.backward()
print(x.grad)                         # tensor([4., 6.])

v = Variable(torch.ones(2))           # legacy call still works but just returns a Tensor
print(type(v) is torch.Tensor)        # True

w = nn.Parameter(torch.randn(3))      # a Tensor subclass registered by nn.Module
print(isinstance(w, torch.Tensor), w.requires_grad)   # True True

with torch.no_grad():
    z = x * 2
print(z.requires_grad)                # False

Common Pitfalls

  • Copying old tutorials that wrap everything in Variable.

  • Using .data to modify tensors, which can silently corrupt gradients; prefer .detach() or torch.no_grad().

Interview Tips

  • Say Variable and Tensor merged in 0.4; Variable is deprecated.

  • Show requires_grad=True, nn.Parameter and torch.no_grad().

4

How do you convert a NumPy array to a PyTorch tensor?

Use torch.from_numpy(arr) to create a tensor that shares memory with the NumPy array (no copy, so changes to one show in the other), or torch.tensor(arr) / torch.as_tensor(arr, dtype=..., device=...) when you want a copy or a different dtype or device. Watch the dtype: NumPy defaults to float64, while PyTorch models expect float32, so convert with .float() or dtype=torch.float32. Going back, t.numpy() works for CPU tensors that don’t require gradients; otherwise use t.detach().cpu().numpy().

Read full answer

Applies toPyTorch 2.x, NumPy 2.x

Options

Function Copies data? Notes
torch.from_numpy(a) No (shares memory) CPU only; keeps NumPy’s dtype
torch.as_tensor(a) Only if needed (dtype/device change) Flexible, avoids copies when possible
torch.tensor(a) Always Safest for independent data
t.numpy() No (shares memory) CPU tensor, no grad
t.detach().cpu().numpy() Copies if on GPU General way back

Code Example

import numpy as np
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
a = np.array([[1.0, 2.0], [3.0, 4.0]])          # float64

shared = torch.from_numpy(a)
copied = torch.tensor(a, dtype=torch.float32)
print(shared.dtype, copied.dtype)                # torch.float64 torch.float32

a[0, 0] = 100.0
print(shared[0, 0].item(), copied[0, 0].item())  # 100.0 1.0: only the shared tensor changed

x = torch.as_tensor(a, dtype=torch.float32, device=device)   # ready for a model
w = torch.ones(2, requires_grad=True, device=device)
out = x @ w
back = out.detach().cpu().numpy()                # safe conversion back to NumPy
print(type(back), back.dtype)                    # <class 'numpy.ndarray'> float32

Common Pitfalls

  • Feeding float64 tensors to float32 models (dtype mismatch errors).

  • Surprise mutations because from_numpy shares memory.

  • Calling .numpy() on a CUDA tensor or one that requires grad.

  • Non-writable or negatively strided arrays (for example a[::-1]), which need .copy() first.

Interview Tips

  • Contrast from_numpy (shared memory) with torch.tensor (copy).

  • Mention float64 vs float32 and detach().cpu().numpy().

5

What is the purpose of the .grad attribute in PyTorch tensors?

The .grad attribute stores the gradient of a scalar (usually the loss) with respect to that tensor, filled in by loss.backward() for leaf tensors with requires_grad=True, such as model parameters; optimizers read .grad to update the parameters. It starts as None, and gradients accumulate (add up) across backward calls, which is why training loops call optimizer.zero_grad() before each backward pass. Non-leaf (intermediate) tensors don’t keep .grad unless you call .retain_grad() on them.

Read full answer

Applies toPyTorch 2.x

Key Facts

Fact Detail
Populated by backward() on a scalar (or backward(gradient=...))
Exists for Leaf tensors with requires_grad=True
Initial value None
Accumulates Yes: new gradients are added to existing ones
Reset optimizer.zero_grad() (sets to None by default) or t.grad = None
Intermediate tensors Use .retain_grad() to keep their gradient

Code Example

import torch

w = torch.tensor([1.0, 2.0], requires_grad=True)
x = torch.tensor([3.0, 4.0])

loss = (w * x).sum()            # d(loss)/dw = x
loss.backward()
print(w.grad)                   # tensor([3., 4.])

loss = (w * x).sum()
loss.backward()                 # accumulates
print(w.grad)                   # tensor([6., 8.])

w.grad = None                   # reset (what zero_grad does by default)
h = w * 2                       # non-leaf tensor
h.retain_grad()
(h ** 2).sum().backward()       # d/dh = 2h = [4, 8]; d/dw = 2 * 2h = [8, 16]
print(h.grad, w.grad)           # tensor([4., 8.]) tensor([ 8., 16.])

with torch.no_grad():           # manual SGD step using .grad
    w -= 0.1 * w.grad
print(w)                        # tensor([0.2000, 0.4000], requires_grad=True)

Common Pitfalls

  • Forgetting zero_grad(), so gradients from earlier batches add up.

  • Expecting .grad on intermediate tensors without retain_grad().

  • Updating parameters outside torch.no_grad(), which records the update in the graph.

Interview Tips

  • Explain accumulation and why zero_grad() exists.

  • Mention leaf vs non-leaf tensors.

6

What is CUDA and how does PyTorch use it?

CUDA is NVIDIA’s parallel computing platform and programming model for running general computations on NVIDIA GPUs; PyTorch uses it, together with libraries such as cuDNN and cuBLAS, to run tensor operations on thousands of GPU cores, which speeds up deep learning training and inference by large factors compared with CPUs. In PyTorch you move tensors and models to the GPU with .to("cuda"), check availability with torch.cuda.is_available(), and the same code runs on CPU or GPU; GPU operations run asynchronously, and torch.cuda also manages memory, streams and multiple devices.

Read full answer

Applies toPyTorch 2.x with CUDA 12.x builds

Key Concepts

Concept In PyTorch
Device selection device = "cuda" if torch.cuda.is_available() else "cpu"
Moving data tensor.to(device), model.to(device)
Multiple GPUs cuda:0, cuda:1; torch.cuda.device_count()
Libraries cuDNN (convolutions), cuBLAS (matrix math), NCCL (multi-GPU communication)
Async execution Kernels queue up; torch.cuda.synchronize() before timing
Memory Caching allocator; torch.cuda.memory_allocated(), empty_cache()
Mixed precision torch.autocast("cuda", dtype=torch.bfloat16) uses Tensor Cores

Code Example

import time

import torch

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"
print("device:", device)
if device == "cuda":
    print(torch.cuda.get_device_name(0), "| GPUs:", torch.cuda.device_count())

a = torch.randn(2048, 2048, device=device)
b = torch.randn(2048, 2048, device=device)

start = time.perf_counter()
c = a @ b
if device == "cuda":
    torch.cuda.synchronize()             # wait for the asynchronous kernel before stopping the clock
print(f"matmul took {time.perf_counter() - start:.4f}s on {device}", c.shape)

model = torch.nn.Linear(2048, 10).to(device)
print(next(model.parameters()).device)   # cuda:0 on a GPU machine, cpu otherwise

Other Accelerators

  • AMD GPUs through ROCm (same cuda device string in ROCm builds).

  • Apple Silicon through mps.

  • TPUs through PyTorch/XLA.

Common Pitfalls

  • Device mismatch errors from mixing CPU and GPU tensors.

  • Timing GPU code without synchronizing.

  • Frequent .item() or .cpu() calls in loops, which force synchronization.

  • Installing a CPU-only PyTorch build by accident.

Interview Tips

  • Define CUDA, then show the device pattern.

  • Mention cuDNN, asynchronous execution and mixed precision.

7

How does automatic differentiation work in PyTorch with Autograd?

Autograd records every operation on tensors with requires_grad=True into a dynamic computation graph as the forward pass runs: each result tensor gets a grad_fn pointing to the function that created it; calling backward() on a scalar then walks this graph in reverse, applying the chain rule (reverse-mode differentiation) to compute gradients and accumulating them into the .grad of the leaf tensors. Because the graph is rebuilt on every forward pass, Python control flow (if statements, loops) works naturally, and the graph is freed after backward unless you pass retain_graph=True.

Read full answer

Applies toPyTorch 2.x

How It Works

  1. Forward: operations create tensors with grad_fn (for example MulBackward0, AddBackward0).

  2. Graph: a directed acyclic graph from leaves (inputs, parameters) to the output.

  3. Backward: starting with d(out)/d(out) = 1, each node multiplies the incoming gradient by its local derivative (vector-Jacobian products) and passes it on.

  4. Accumulate: leaf tensors receive gradients in .grad.

  5. Free: intermediate buffers are released.

Code Example

import torch

x = torch.tensor(2.0, requires_grad=True)
y = torch.tensor(3.0, requires_grad=True)

z = x * y + x ** 2            # z = 6 + 4 = 10
print(z.grad_fn)              # <AddBackward0 ...>
z.backward()
print(x.grad, y.grad)         # dz/dx = y + 2x = 7; dz/dy = x = 2 -> tensor(7.) tensor(2.)

# Control flow is fine: the graph follows the path that actually ran
def f(t):
    return t * 3 if t.item() > 0 else t ** 2

a = torch.tensor(-4.0, requires_grad=True)
f(a).backward()
print(a.grad)                 # derivative of t^2 at -4 = -8 -> tensor(-8.)

# Higher-order gradients with torch.autograd.grad
u = torch.tensor(1.5, requires_grad=True)
(g,) = torch.autograd.grad(u ** 3, u, create_graph=True)   # 3u^2 = 6.75
(h,) = torch.autograd.grad(g, u)                            # 6u = 9
print(g.item(), h.item())     # 6.75 9.0

Controls

Tool Purpose
requires_grad=True Track a tensor
torch.no_grad() / torch.inference_mode() Disable tracking (evaluation)
.detach() Cut a tensor from the graph
retain_graph=True Call backward twice on the same graph
create_graph=True Higher-order derivatives
torch.autograd.Function Custom forward/backward

Common Pitfalls

  • Calling backward() on a non-scalar without a gradient argument.

  • In-place operations that overwrite values needed for backward.

  • Storing losses (not loss.item()) in a list, which keeps graphs alive.

Interview Tips

  • Describe the dynamic graph, grad_fn and reverse-mode chain rule.

  • Work through a two-variable example by hand.

8

What are the steps to create a neural network model in PyTorch?

Creating a neural network in PyTorch follows a fixed recipe: prepare data as tensors wrapped in a Dataset and DataLoader; define the model as a subclass of nn.Module with layers created in __init__ and the computation written in forward; pick a loss function and an optimizer; run the training loop (zero gradients, forward pass, loss, backward(), optimizer step); then evaluate with model.eval() and torch.no_grad(), and save the state_dict.

Read full answer

Applies toPyTorch 2.x

Steps

  1. Data: tensors → TensorDataset or a custom Dataset → DataLoader(batch_size, shuffle=True).

  2. Model: class Net(nn.Module) with layers in __init__, logic in forward.

  3. Device: model.to(device), and move each batch to the same device.

  4. Loss: nn.CrossEntropyLoss (classification, raw logits), nn.MSELoss (regression).

  5. Optimizer: torch.optim.AdamW(model.parameters(), lr=1e-3).

  6. Train: loop over epochs and batches.

  7. Validate: model.eval(), torch.no_grad(), compute metrics.

  8. Save: torch.save(model.state_dict(), "model.pt").

Code Example

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset, random_split

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Synthetic data: 3 classes, 20 features
X = torch.randn(1200, 20)
y = (X[:, 0] + X[:, 1] > 0).long() + (X[:, 2] > 1).long()
train_ds, val_ds = random_split(TensorDataset(X, y), [1000, 200], generator=torch.Generator().manual_seed(42))
train_dl = DataLoader(train_ds, batch_size=64, shuffle=True)
val_dl = DataLoader(val_ds, batch_size=256)

# 2. Model
class Net(nn.Module):
    def __init__(self, in_features=20, hidden=64, n_classes=3):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(in_features, hidden), nn.ReLU(), nn.Dropout(0.1),
            nn.Linear(hidden, n_classes),
        )

    def forward(self, x):
        return self.layers(x)               # raw logits

model = Net().to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)

for epoch in range(10):
    model.train()
    for xb, yb in train_dl:
        xb, yb = xb.to(device), yb.to(device)
        optimizer.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
    model.eval()
    correct = 0
    with torch.no_grad():
        for xb, yb in val_dl:
            correct += (model(xb.to(device)).argmax(1) == yb.to(device)).sum().item()
    print(f"epoch {epoch + 1}: train loss {loss.item():.3f}, val acc {correct / len(val_ds):.3f}")

torch.save(model.state_dict(), "model.pt")

Common Pitfalls

  • Applying softmax before CrossEntropyLoss (it expects logits).

  • Forgetting model.eval(), so dropout stays active during validation.

  • Moving the model to the GPU but not the batches.

Interview Tips

  • Recite the steps in order and write the training loop from memory.

  • Mention train/eval modes and saving the state_dict.

9

What is nn.Sequential in PyTorch and how does it differ from subclassing nn.Module?

nn.Sequential is a container module that chains layers so the output of each becomes the input of the next, letting you define simple feed-forward models in one expression without writing a forward method; subclassing nn.Module means writing your own __init__ and forward, which you need for anything that isn’t a straight chain: skip connections, multiple inputs or outputs, branching, loops or conditional logic. nn.Sequential is itself an nn.Module, so the two combine: custom modules often use Sequential blocks inside them.

Read full answer

Applies toPyTorch 2.x

Comparison

Aspect nn.Sequential Custom nn.Module
Definition List of layers __init__ + forward
Data flow Strictly linear Anything: residuals, branches, multiple inputs
Code size Minimal More code
Named layers Optional via OrderedDict Attribute names
Debugging Index into layers (model[0]) Print or break inside forward
Typical use MLPs, simple CNN blocks, heads ResNets, transformers, anything complex

Code Example

from collections import OrderedDict

import torch
import torch.nn as nn

torch.manual_seed(42)

seq = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 2))

named = nn.Sequential(OrderedDict([
    ("fc1", nn.Linear(10, 32)),
    ("act", nn.ReLU()),
    ("out", nn.Linear(32, 2)),
]))

class ResidualMLP(nn.Module):
    """Needs a custom forward because of the skip connection."""

    def __init__(self, dim=10):
        super().__init__()
        self.block = nn.Sequential(nn.Linear(dim, dim), nn.ReLU(), nn.Linear(dim, dim))
        self.head = nn.Linear(dim, 2)

    def forward(self, x):
        x = x + self.block(x)             # residual connection
        return self.head(torch.relu(x))

x = torch.randn(4, 10)
print(seq(x).shape, named(x).shape, ResidualMLP()(x).shape)   # all torch.Size([4, 2])
print(named.fc1)                          # Linear(in_features=10, out_features=32, bias=True)
print(sum(p.numel() for p in seq.parameters()))   # 10*32 + 32 + 32*2 + 2 = 418

Common Pitfalls

  • Forcing a non-linear architecture into Sequential with awkward wrapper modules.

  • Storing layers in a plain Python list inside a custom module (use nn.ModuleList, or parameters won’t register).

  • Passing several inputs to a Sequential (it forwards only one).

Interview Tips

  • Say Sequential is for linear chains; custom modules for everything else.

  • Show a residual block as the classic reason to subclass.

10

How do you implement custom layers in PyTorch?

A custom layer in PyTorch is a subclass of nn.Module: declare learnable weights as nn.Parameter (or sub-modules) in __init__, initialize them, and write the computation in forward using tensor operations, which autograd differentiates automatically. Use register_buffer for non-learnable state that should move with the model and be saved (like running statistics), and implement a torch.autograd.Function with explicit forward and backward only when you need a custom gradient.

Read full answer

Applies toPyTorch 2.x

Building Blocks

Need Tool
Learnable weights nn.Parameter(tensor)
Non-learnable saved state self.register_buffer("name", tensor)
Sub-layers Assign modules as attributes; nn.ModuleList / nn.ModuleDict for collections
Initialization nn.init.* functions, often in a reset_parameters method
Custom gradient torch.autograd.Function with forward / backward static methods
Extra info when printed Override extra_repr

Code Example

import math

import torch
import torch.nn as nn

torch.manual_seed(42)

class ScaledLinear(nn.Module):
    """y = scale * (x @ W^T) + b, with a learnable scalar scale."""

    def __init__(self, in_features, out_features):
        super().__init__()
        self.weight = nn.Parameter(torch.empty(out_features, in_features))
        self.bias = nn.Parameter(torch.zeros(out_features))
        self.scale = nn.Parameter(torch.ones(1))
        self.register_buffer("calls", torch.zeros((), dtype=torch.long))   # saved, not trained
        self.reset_parameters()

    def reset_parameters(self):
        nn.init.kaiming_uniform_(self.weight, a=math.sqrt(5))   # same scheme as nn.Linear

    def forward(self, x):
        self.calls += 1
        return self.scale * (x @ self.weight.T) + self.bias

    def extra_repr(self):
        return f"in_features={self.weight.shape[1]}, out_features={self.weight.shape[0]}"

class ClipGrad(torch.autograd.Function):
    """Identity in the forward pass, gradient clipped to [-1, 1] in the backward pass."""

    @staticmethod
    def forward(ctx, x):
        return x.clone()

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output.clamp(-1.0, 1.0)

layer = ScaledLinear(4, 3)
x = torch.randn(2, 4)
out = layer(x)
print(layer, out.shape)                              # ScaledLinear(in_features=4, out_features=3) torch.Size([2, 3])
print([name for name, _ in layer.named_parameters()])  # ['weight', 'bias', 'scale']
print(list(layer.state_dict().keys()))               # ['weight', 'bias', 'scale', 'calls']

t = torch.tensor([1.0, 2.0], requires_grad=True)
(ClipGrad.apply(t) * 10).sum().backward()
print(t.grad)                                        # tensor([1., 1.]): 10 clipped to 1

Common Pitfalls

  • Plain tensors instead of nn.Parameter, so weights never train or move with .to(device).

  • Python lists of layers instead of nn.ModuleList.

  • Creating new tensors in forward on the CPU instead of x.device.

Interview Tips

  • Show nn.Parameter, register_buffer and forward.

  • Mention autograd.Function for custom gradients.

11

What is the role of the forward method in a PyTorch Module?

The forward method defines the computation a module performs on its inputs: how data flows through the layers created in __init__ to produce the output; you call the module itself (model(x)), not forward directly, because nn.Module.__call__ wraps forward with hooks and other bookkeeping. Autograd records whatever operations forward runs, so you never write a backward method for ordinary modules, and forward can contain any Python logic: branches, loops, multiple inputs and outputs.

Read full answer

Applies toPyTorch 2.x

Key Points

Point Detail
Purpose Maps inputs to outputs using the module’s layers and parameters
How it’s invoked model(x) → __call__ → hooks → forward(x) → hooks
Backward Derived automatically by autograd
Flexibility Any Python control flow, multiple arguments, dict or tuple outputs
Mode-dependent behaviour Dropout/BatchNorm check self.training, set by train() / eval()

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)

class TwoHeadNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.body = nn.Sequential(nn.Linear(8, 16), nn.ReLU())
        self.cls_head = nn.Linear(16, 3)
        self.reg_head = nn.Linear(16, 1)
        self.dropout = nn.Dropout(0.5)

    def forward(self, x, return_features=False):
        h = self.dropout(self.body(x))                # behaves differently in train vs eval mode
        out = {"logits": self.cls_head(h), "value": self.reg_head(h).squeeze(-1)}
        if return_features:
            out["features"] = h
        return out

model = TwoHeadNet()
calls = []
model.register_forward_hook(lambda module, inputs, output: calls.append("hook ran"))

x = torch.randn(5, 8)
out = model(x, return_features=True)               # call the module, not model.forward(x)
print({k: tuple(v.shape) for k, v in out.items()})  # {'logits': (5, 3), 'value': (5,), 'features': (5, 16)}
print(calls)                                        # ['hook ran']

model.forward(x)                                    # bypasses hooks
print(len(calls))                                   # still 1

Common Pitfalls

  • Calling model.forward(x) directly, which skips hooks (used by profilers, quantization and some libraries).

  • Creating layers inside forward, so their weights are re-created and never trained.

  • Forgetting that forward behaviour depends on model.train() / model.eval().

Interview Tips

  • Explain __call__ vs forward and hooks.

  • Note that backward comes free from autograd.

12

What are optimizers in PyTorch and how do you use them?

Optimizers in torch.optim update a model’s parameters from their gradients: you create one with model.parameters() and hyperparameters such as the learning rate, then in each training step call optimizer.zero_grad(), compute the loss, call loss.backward() to fill .grad, and call optimizer.step() to apply the update rule. Common choices are SGD with momentum (strong for vision with tuning), Adam, and AdamW (Adam with decoupled weight decay, the default for transformers); parameter groups let different parts of the model use different settings, and the optimizer’s state_dict should be saved with checkpoints.

Read full answer

Applies toPyTorch 2.x

Common Optimizers

Optimizer Update idea Typical use
SGD(lr, momentum=0.9, nesterov=True) Gradient step with velocity CNNs, well-tuned pipelines
Adam(lr=1e-3) Per-parameter adaptive steps from gradient moments Default for many tasks
AdamW(lr, weight_decay=0.01) Adam with decoupled weight decay Transformers, fine-tuning
RMSprop Adaptive by squared-gradient average RNNs, some RL
Adagrad Accumulated squared gradients Sparse features
LBFGS Quasi-Newton, needs a closure Small, full-batch problems

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"

X = torch.randn(256, 10, device=device)
y = X @ torch.randn(10, 1, device=device) + 0.1 * torch.randn(256, 1, device=device)

model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 1)).to(device)

optimizer = torch.optim.AdamW([
    {"params": model[0].parameters(), "lr": 1e-3},
    {"params": model[2].parameters(), "lr": 5e-3},          # a different LR for the output layer
], weight_decay=0.01)
loss_fn = nn.MSELoss()

for step in range(200):
    optimizer.zero_grad()
    loss = loss_fn(model(X), y)
    loss.backward()
    optimizer.step()
    if step % 50 == 0:
        print(step, round(loss.item(), 4))

checkpoint = {"model": model.state_dict(), "optimizer": optimizer.state_dict()}
torch.save(checkpoint, "checkpoint.pt")
print([group["lr"] for group in optimizer.param_groups])   # [0.001, 0.005]

Common Pitfalls

  • Forgetting zero_grad(), so gradients accumulate.

  • Creating the optimizer before moving the model to the GPU is fine, but recreating the model afterwards leaves the optimizer pointing at old parameters.

  • Using Adam(weight_decay=...) expecting AdamW behaviour.

  • Not saving optimizer state, so resumed training behaves differently.

Interview Tips

  • Write the zero_grad / backward / step sequence.

  • Contrast SGD+momentum, Adam and AdamW, and mention parameter groups.

13

What is the purpose of zero_grad() in PyTorch and when do you call it?

optimizer.zero_grad() (or model.zero_grad()) clears the .grad of every parameter before a new backward pass, because PyTorch accumulates gradients: each backward() adds to whatever is already stored. You call it once per optimization step, before loss.backward(), so each update uses only the current batch’s gradients. Accumulation is deliberate: it lets you simulate a larger batch by calling backward() on several micro-batches and zeroing only after the step. Since PyTorch 2.0, zero_grad() sets gradients to None by default (set_to_none=True), which saves memory.

Read full answer

Applies toPyTorch 2.x

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)
model = nn.Linear(3, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
x, y = torch.randn(8, 3), torch.randn(8, 1)
loss_fn = nn.MSELoss()

loss_fn(model(x), y).backward()
g1 = model.weight.grad.clone()
loss_fn(model(x), y).backward()                     # no zero_grad: gradients add up
print(torch.allclose(model.weight.grad, 2 * g1))    # True

optimizer.zero_grad()
print(model.weight.grad)                            # None (set_to_none=True by default)

# Gradient accumulation: effective batch = 4 micro-batches
accum_steps = 4
optimizer.zero_grad()
for i in range(accum_steps):
    xb, yb = torch.randn(8, 3), torch.randn(8, 1)
    loss = loss_fn(model(xb), yb) / accum_steps     # scale so the sum is an average
    loss.backward()
optimizer.step()
optimizer.zero_grad()

Where It Goes in the Loop

Order Call
1 optimizer.zero_grad()
2 outputs = model(inputs)
3 loss = loss_fn(outputs, targets)
4 loss.backward()
5 optimizer.step()

Calling zero_grad() right after step() instead of before the forward pass works equally well; what matters is once per update.

Common Pitfalls

  • Forgetting it: training becomes unstable because gradients grow every step.

  • Calling it between backward() and step(), which wipes the gradients before they are used.

  • Not dividing the loss by the number of accumulation steps.

Interview Tips

  • Explain accumulation as the reason and gradient accumulation as the use case.

  • Mention set_to_none=True.

14

How do you implement learning rate scheduling in PyTorch?

Learning rate scheduling in PyTorch uses the classes in torch.optim.lr_scheduler: wrap the optimizer in a scheduler such as StepLR, CosineAnnealingLR, OneCycleLR or ReduceLROnPlateau, and call scheduler.step() after optimizer.step() (per epoch for most schedulers, per batch for OneCycleLR and warm-up schedules, or with the validation loss for ReduceLROnPlateau). Schedules typically warm up briefly, then decay, which stabilizes early training and lets the model settle into a better minimum later.

Read full answer

Applies toPyTorch 2.x

Common Schedulers

Scheduler Behaviour Step when
StepLR(step_size, gamma) Multiply LR by gamma every step_size epochs Per epoch
MultiStepLR(milestones, gamma) Decay at listed epochs Per epoch
ExponentialLR(gamma) Multiply every epoch Per epoch
CosineAnnealingLR(T_max) Cosine decay to eta_min Per epoch (or per batch)
OneCycleLR(max_lr, total_steps) Warm up then anneal Per batch
LinearLR + SequentialLR Linear warm-up followed by another schedule Per batch or epoch
ReduceLROnPlateau(patience) Decay when a metric stops improving Per epoch with the metric
LambdaLR(fn) Any custom multiplier As designed

Code Example

import torch
import torch.nn as nn
from torch.optim.lr_scheduler import CosineAnnealingLR, LinearLR, ReduceLROnPlateau, SequentialLR, StepLR

torch.manual_seed(42)
model = nn.Linear(10, 1)

opt = torch.optim.SGD(model.parameters(), lr=0.1)
step_sched = StepLR(opt, step_size=3, gamma=0.5)
lrs = []
for epoch in range(8):
    opt.step()                              # training for one epoch would go here
    step_sched.step()
    lrs.append(round(step_sched.get_last_lr()[0], 4))
print(lrs)   # [0.1, 0.1, 0.05, 0.05, 0.05, 0.025, 0.025, 0.025]

opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
warmup = LinearLR(opt, start_factor=0.1, total_iters=5)          # 5 warm-up steps
cosine = CosineAnnealingLR(opt, T_max=45, eta_min=1e-5)
sched = SequentialLR(opt, schedulers=[warmup, cosine], milestones=[5])
for step in range(50):
    opt.step()
    sched.step()
print(f"after warm-up + cosine: {sched.get_last_lr()[0]:.2e}")   # close to eta_min

opt = torch.optim.Adam(model.parameters(), lr=1e-3)
plateau = ReduceLROnPlateau(opt, mode="min", factor=0.1, patience=2)
for val_loss in [1.0, 0.9, 0.9, 0.9, 0.9]:
    plateau.step(val_loss)                  # pass the metric
print(round(opt.param_groups[0]["lr"], 6))  # reduced to 0.0001 after 3 epochs without improvement

Common Pitfalls

  • Calling scheduler.step() before optimizer.step() (PyTorch warns; the first LR value is skipped).

  • Stepping a per-batch schedule once per epoch (or the reverse).

  • Forgetting to save and restore the scheduler’s state_dict when resuming.

Interview Tips

  • Name StepLR, cosine, OneCycle and ReduceLROnPlateau, and when each steps.

  • Mention warm-up for transformers.

15

How does backpropagation work in PyTorch?

Backpropagation in PyTorch happens when you call loss.backward(): autograd traverses the computation graph recorded during the forward pass in reverse order, and at each operation multiplies the incoming gradient by that operation’s local derivative (the chain rule, computed as vector-Jacobian products), until every parameter with requires_grad=True has its gradient accumulated in .grad. The optimizer then uses those gradients in optimizer.step(). You never write the backward pass yourself for standard layers; PyTorch has the derivative of each operation built in.

Read full answer

Applies toPyTorch 2.x

The Process

  1. Forward pass: compute predictions; autograd records the operations and keeps tensors needed for derivatives.

  2. Loss: reduce to a scalar.

  3. loss.backward(): start from d(loss)/d(loss) = 1, propagate backward through each grad_fn.

  4. Accumulate: each parameter’s .grad receives d(loss)/d(parameter).

  5. Update: optimizer.step(); then zero_grad() before the next iteration.

  6. Graph freed: saved intermediate tensors are released after backward.

Hand-Checked Example

For y_hat = w·x + b, loss = (y_hat − y)², with x = 2, y = 7, w = 1, b = 0:

  • y_hat = 2, error = −5, loss = 25.

  • d(loss)/dw = 2·(y_hat − y)·x = 2·(−5)·2 = −20.

  • d(loss)/db = 2·(y_hat − y) = −10.

import torch

x, y = torch.tensor(2.0), torch.tensor(7.0)
w = torch.tensor(1.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)

y_hat = w * x + b
loss = (y_hat - y) ** 2
loss.backward()
print(loss.item(), w.grad.item(), b.grad.item())     # 25.0 -20.0 -10.0

with torch.no_grad():                                 # one gradient descent step, lr = 0.01
    w -= 0.01 * w.grad
    b -= 0.01 * b.grad
print(round(w.item(), 2), round(b.item(), 2))         # 1.2 0.1

Practical Points

Topic Note
Vanishing / exploding gradients Use good initialization, normalization, residual connections, gradient clipping
Memory Activations are stored for backward; activation checkpointing trades compute for memory
Non-scalar outputs out.backward(gradient=...) or reduce to a scalar first
Inspecting gradients param.grad, hooks (register_hook), gradient norms

Common Pitfalls

  • Missing zero_grad(), which mixes gradients across batches.

  • Breaking the graph with .detach(), .item() or NumPy conversions in the middle of the model.

  • In-place operations on tensors needed for backward.

Interview Tips

  • Explain the chain rule over the recorded graph and work a tiny example.

  • Mention vanishing/exploding gradients and their fixes.

16

How does gradient clipping work in PyTorch and why is it needed?

Gradient clipping limits the size of gradients before the optimizer step so that one unusually large gradient cannot make a huge, destabilizing update; in PyTorch you call torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm) after loss.backward() and before optimizer.step(), which rescales all gradients together whenever their combined L2 norm exceeds max_norm, or clip_grad_value_ to clamp each element to a range. It is standard for RNNs and LSTMs (exploding gradients through time), transformers (often max_norm = 1.0) and reinforcement learning, and the returned total norm is a useful training diagnostic.

Read full answer

Applies toPyTorch 2.x

Norm vs Value Clipping

Method Function Effect
Norm clipping clip_grad_norm_(params, max_norm) If total norm > max_norm, scale all gradients by max_norm / total_norm; keeps the direction
Value clipping clip_grad_value_(params, clip_value) Clamp each element to [−c, c]; can change the direction

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)
model = nn.Linear(2, 1, bias=False)
with torch.no_grad():
    model.weight.copy_(torch.tensor([[0.0, 0.0]]))

x = torch.tensor([[3.0, 4.0]])
y = torch.tensor([[50.0]])
loss = ((model(x) - y) ** 2).mean()
loss.backward()
print(model.weight.grad)                       # 2 * (0 - 50) * [3, 4] = tensor([[-300., -400.]])

total = nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
print(round(total.item(), 1))                  # 500.0: the norm before clipping
print(model.weight.grad)                       # scaled to norm 1: tensor([[-0.6000, -0.8000]])

# In a training loop (with mixed precision, unscale first)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
scaler = torch.amp.GradScaler("cuda", enabled=torch.cuda.is_available())
optimizer.zero_grad()
loss = ((model(x) - y) ** 2).mean()
scaler.scale(loss).backward()
scaler.unscale_(optimizer)                     # clip the real gradients, not the scaled ones
nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
scaler.step(optimizer)
scaler.update()

Choosing max_norm

  • Log the total norm for a while; set max_norm around a typical value so only spikes are clipped.

  • Common defaults: 1.0 for transformers, 0.5–5 for RNNs, 0.5–10 in RL.

  • If gradients are clipped on nearly every step, the learning rate is probably too high.

Common Pitfalls

  • Clipping before backward() or after step().

  • Clipping scaled gradients under mixed precision without unscale_.

  • Using clipping to hide a bug such as a bad loss scale or learning rate.

Interview Tips

  • Explain norm clipping as rescaling that preserves direction.

  • Give the order: backward → clip → step.

17

How do you check if your PyTorch model is using the GPU?

Check three things: that a GPU is available (torch.cuda.is_available()), that the model’s parameters are on it (next(model.parameters()).device shows cuda:0), and that every input batch is moved to the same device (x.device); then confirm real use with torch.cuda.memory_allocated(), the PyTorch profiler, or nvidia-smi showing your process and GPU utilization. A model on the GPU with data on the CPU raises a device mismatch error, while a training loop starved by a slow data loader shows the GPU allocated but mostly idle.

Read full answer

Applies toPyTorch 2.x (CUDA; similar checks for `mps` on Apple Silicon)

Checks

Question How to check
Is a GPU available? torch.cuda.is_available(), torch.cuda.device_count(), torch.cuda.get_device_name(0)
Is the model on it? next(model.parameters()).device, or all(p.is_cuda for p in model.parameters())
Is the data on it? x.device inside the loop
Is memory being used? torch.cuda.memory_allocated(), torch.cuda.max_memory_allocated()
Is the GPU busy? nvidia-smi (utilization %), PyTorch profiler
Was PyTorch built with CUDA? torch.version.cuda (None for CPU-only builds)

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)
print("CUDA available:", torch.cuda.is_available(), "| built with CUDA:", torch.version.cuda)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = nn.Sequential(nn.Linear(512, 512), nn.ReLU(), nn.Linear(512, 10)).to(device)
print("model on:", next(model.parameters()).device)

x = torch.randn(64, 512).to(device)
print("batch on:", x.device)

if device.type == "cuda":
    before = torch.cuda.memory_allocated()
    out = model(x)
    torch.cuda.synchronize()
    print("name:", torch.cuda.get_device_name(0))
    print(f"memory allocated: {torch.cuda.memory_allocated() / 1e6:.1f} MB (was {before / 1e6:.1f} MB)")
else:
    out = model(x)
print(out.device)

Profiling Utilization

import torch
from torch.profiler import ProfilerActivity, profile

model = torch.nn.Linear(1024, 1024)
x = torch.randn(256, 1024)
activities = [ProfilerActivity.CPU]
if torch.cuda.is_available():
    model, x = model.cuda(), x.cuda()
    activities.append(ProfilerActivity.CUDA)

with profile(activities=activities) as prof:
    for _ in range(10):
        model(x)
print(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=5))

Common Pitfalls

  • Moving the model but forgetting inputs and targets.

  • A CPU-only PyTorch install (torch.version.cuda is None).

  • Low GPU utilization from a slow DataLoader (raise num_workers, use pin_memory=True).

Interview Tips

  • Check availability, parameters’ device, data device and utilization.

  • Mention nvidia-smi and the profiler.

18

How do you monitor and reduce overfitting in a PyTorch model?

Monitor overfitting by tracking training and validation loss and metrics every epoch (on a held-out split, with model.eval() and torch.no_grad()) and watching for the validation loss to rise while training loss keeps falling; reduce it with more or augmented data, regularization (weight decay via AdamW, dropout, label smoothing), early stopping on validation loss with the best checkpoint restored, a smaller or better-normalized model, and transfer learning from pre-trained weights.

Read full answer

Applies toPyTorch 2.x

Techniques

Technique PyTorch implementation
Validation monitoring Separate DataLoader; log both curves (TensorBoard, W&B)
Early stopping Track best val loss; save state_dict; stop after patience epochs
Weight decay torch.optim.AdamW(..., weight_decay=0.01)
Dropout nn.Dropout(p) between layers
Data augmentation torchvision.transforms.v2 (flips, crops, color jitter)
Label smoothing nn.CrossEntropyLoss(label_smoothing=0.1)
Smaller model / fewer epochs Reduce width or depth
Batch norm / layer norm nn.BatchNorm1d/2d, nn.LayerNorm
Transfer learning Pre-trained backbone, fine-tune with low LR

Code Example

import copy

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset

torch.manual_seed(42)
X = torch.randn(600, 30)
y = (X[:, :3].sum(dim=1) > 0).long()
train_dl = DataLoader(TensorDataset(X[:400], y[:400]), batch_size=32, shuffle=True)
val_dl = DataLoader(TensorDataset(X[400:], y[400:]), batch_size=200)

model = nn.Sequential(nn.Linear(30, 256), nn.ReLU(), nn.Dropout(0.3), nn.Linear(256, 2))
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.05)
loss_fn = nn.CrossEntropyLoss(label_smoothing=0.05)

best_loss, best_state, patience, bad_epochs = float("inf"), None, 5, 0
for epoch in range(100):
    model.train()
    train_loss = 0.0
    for xb, yb in train_dl:
        optimizer.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        train_loss += loss.item() * len(xb)
    model.eval()
    with torch.no_grad():
        val_loss = sum(loss_fn(model(xb), yb).item() * len(xb) for xb, yb in val_dl) / 200
    print(f"epoch {epoch + 1:3d}  train {train_loss / 400:.3f}  val {val_loss:.3f}")
    if val_loss < best_loss - 1e-4:
        best_loss, best_state, bad_epochs = val_loss, copy.deepcopy(model.state_dict()), 0
    else:
        bad_epochs += 1
        if bad_epochs >= patience:
            print("early stopping")
            break

model.load_state_dict(best_state)     # restore the best epoch

Reading the Curves

Pattern Meaning
Both losses fall together Still learning
Train falls, val rises Overfitting: regularize or stop
Both high and flat Underfitting: bigger model, more epochs, higher LR
Val noisy Validation set too small

Common Pitfalls

  • Evaluating in train mode (dropout on).

  • Tuning on the test set.

  • Keeping the last epoch’s weights instead of the best.

Interview Tips

  • Describe the train/val curve gap and early stopping.

  • List several regularizers with their PyTorch calls.

19

What is batch normalization and how does it affect training convergence?

Batch normalization normalizes each feature (or channel) over the current mini-batch to zero mean and unit variance, then applies a learnable scale (γ) and shift (β); during training it uses batch statistics and keeps running averages, and in evaluation mode it uses those running averages instead. It usually speeds up and stabilizes convergence: it allows higher learning rates, makes training less sensitive to initialization, smooths the loss landscape, and adds a mild regularizing noise. In PyTorch you use nn.BatchNorm1d (features), nn.BatchNorm2d (CNN channels), and switch modes with model.train() / model.eval().

Read full answer

Applies toPyTorch 2.x

The Computation

For a feature with batch values x₁…x_m:

  • μ = mean(x), σ² = var(x) (biased, over the batch)

  • x̂ = (x − μ) / √(σ² + ε)

  • y = γ·x̂ + β

  • Running stats: running_mean ← (1 − momentum)·running_mean + momentum·μ (momentum = 0.1 by default)

Effects on Convergence

Effect Why
Higher usable learning rates Normalized activations keep gradients well-scaled
Less sensitivity to initialization Each layer’s inputs are re-centred
Faster convergence Better-conditioned optimization (smoother loss landscape)
Mild regularization Batch statistics add noise
Weaker with tiny batches Noisy statistics; prefer GroupNorm or LayerNorm

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)
bn = nn.BatchNorm1d(3)
x = torch.randn(64, 3) * torch.tensor([10.0, 0.1, 3.0]) + torch.tensor([5.0, -2.0, 0.0])

bn.train()
out = bn(x)
print(out.mean(dim=0).abs().max().item() < 1e-5)            # True: mean ~0 per feature
print(out.std(dim=0, unbiased=False).round(decimals=3))     # about 1 for each feature
print(bn.running_mean.round(decimals=2))                    # 10% of the batch means after one step

bn.eval()
single = bn(x[:1])                     # works with batch size 1 in eval mode (uses running stats)
print(single.shape)

cnn_block = nn.Sequential(
    nn.Conv2d(3, 16, kernel_size=3, padding=1, bias=False),   # bias is redundant before BN
    nn.BatchNorm2d(16),
    nn.ReLU(),
)
print(cnn_block(torch.randn(8, 3, 32, 32)).shape)          # torch.Size([8, 16, 32, 32])

Common Pitfalls

  • Forgetting model.eval() at inference, which uses batch statistics and gives unstable predictions.

  • Very small batches (BN statistics become noisy).

  • Multi-GPU training without nn.SyncBatchNorm when per-GPU batches are small.

  • Freezing a pre-trained backbone but leaving BN layers in train mode, which shifts running stats.

Interview Tips

  • Write the normalization formula and explain train vs eval behaviour.

  • Mention LayerNorm as the transformer alternative.

20

How does PyTorch handle weight initialization for neural networks?

PyTorch initializes every built-in layer automatically when it is created: nn.Linear and nn.Conv* use Kaiming (He) uniform initialization with a = √5, which works out to a uniform distribution in ±1/√fan_in for the weights, and biases are drawn uniformly from ±1/√fan_in; normalization layers start with weight 1 and bias 0, and embeddings use a standard normal. You can override this with the functions in torch.nn.init (xavier_uniform_, kaiming_normal_, orthogonal_, zeros_), usually via model.apply(init_fn), matching the scheme to the activation: Kaiming for ReLU-family networks, Xavier/Glorot for tanh or sigmoid, small normal (std 0.02) for transformers.

Read full answer

Applies toPyTorch 2.x

Default and Common Schemes

Layer / scheme Default or formula Use with
nn.Linear, nn.Conv* weights Kaiming uniform (a = √5) → U(−1/√fan_in, 1/√fan_in) General default
Their biases U(−1/√fan_in, 1/√fan_in)
nn.BatchNorm*, nn.LayerNorm weight = 1, bias = 0
nn.Embedding N(0, 1)
Xavier / Glorot Var = 2 / (fan_in + fan_out) tanh, sigmoid
Kaiming / He Var = 2 / fan_in ReLU, LeakyReLU
Orthogonal Orthogonal matrix RNNs, RL policies
Normal(0, 0.02) GPT/BERT-style Transformers

Code Example

import math

import torch
import torch.nn as nn

torch.manual_seed(42)

layer = nn.Linear(100, 50)
bound = 1 / math.sqrt(100)
print(layer.weight.abs().max().item() <= bound)    # True: default weights within ±0.1

def init_weights(module):
    if isinstance(module, nn.Linear):
        nn.init.kaiming_normal_(module.weight, nonlinearity="relu")
        nn.init.zeros_(module.bias)

model = nn.Sequential(nn.Linear(512, 512), nn.ReLU(), nn.Linear(512, 512), nn.ReLU(), nn.Linear(512, 10))
model.apply(init_weights)                          # applies recursively to every sub-module

x = torch.randn(1024, 512)
with torch.no_grad():
    h = x
    for i, m in enumerate(model):
        h = m(h)
        if isinstance(m, nn.ReLU):
            print(f"after layer {i}: activation std {h.std().item():.2f}")
# With Kaiming init the activation scale stays roughly constant through the ReLU layers

Why It Matters

  • Too-small weights make activations and gradients shrink layer by layer (vanishing).

  • Too-large weights make them grow (exploding) or saturate tanh/sigmoid.

  • Good schemes keep the variance of activations roughly constant across layers.

Common Pitfalls

  • Re-initializing pre-trained weights by calling apply on a whole fine-tuned model.

  • Using Xavier with deep ReLU networks (activations shrink).

  • Zero-initializing all weights, which makes every neuron learn the same thing.

Interview Tips

  • State the default (Kaiming uniform with a = √5) and how to override it with nn.init and apply.

  • Match initialization to the activation function.

21

What common issues occur when training PyTorch models and how do you fix them?

The most common PyTorch training problems are a loss that won’t decrease (wrong loss inputs, learning rate off, missing zero_grad(), frozen or detached parameters), NaN or exploding losses (learning rate too high, log of zero, no gradient clipping, fp16 overflow), device and dtype errors (CPU vs CUDA tensors, float64 inputs, wrong label dtype), shape mismatches, out-of-memory errors, overfitting, and evaluation mistakes such as forgetting model.eval(). A reliable troubleshooting method is to overfit a single small batch first, then check shapes, gradients and data step by step.

Read full answer

Applies toPyTorch 2.x

Symptoms and Fixes

Symptom Likely cause Fix
Loss flat from the start LR too low/high, labels wrong, softmax before CrossEntropyLoss Overfit one batch; pass raw logits; check labels
Loss becomes NaN LR too high, log(0), division by zero, fp16 overflow Lower LR, clip gradients, use stable losses (BCEWithLogitsLoss), bf16 or GradScaler
Expected all tensors to be on the same device Model and data on different devices .to(device) for both
expected scalar type Float but found Double NumPy float64 data .float()
Expected Long for targets Class labels as floats .long()
mat1 and mat2 shapes cannot be multiplied Wrong in_features / missing flatten Print shapes; nn.Flatten()
CUDA out of memory Batch too big, storing graphs Smaller batch, loss.item(), AMP, gradient checkpointing
Validation worse than expected Dropout/BN in train mode model.eval() + torch.no_grad()
Training slow, GPU idle Data loading bottleneck num_workers, pin_memory=True
Gradients are None Graph broken by .detach(), .item(), NumPy Keep operations in torch

The “Overfit One Batch” Test

import torch
import torch.nn as nn

torch.manual_seed(42)
model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 5))
optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)
loss_fn = nn.CrossEntropyLoss()

xb = torch.randn(16, 20)
yb = torch.randint(0, 5, (16,))          # class indices, dtype int64
for step in range(200):
    optimizer.zero_grad()
    loss = loss_fn(model(xb), yb)        # raw logits, no softmax
    loss.backward()
    optimizer.step()
print(f"loss on one batch after 200 steps: {loss.item():.4f}")   # should approach 0

for name, p in model.named_parameters():                       # check every parameter gets gradient
    print(name, None if p.grad is None else round(p.grad.norm().item(), 4))

If the model can’t drive the loss near zero on 16 examples, the bug is in the model, loss or optimizer, not in the data size.

Useful Debugging Tools

  • torch.autograd.set_detect_anomaly(True): finds the operation that produced NaNs (slow; debug only).

  • Hooks or p.grad.norm() logging to spot vanishing or exploding gradients.

  • torch.cuda.memory_summary() for memory issues.

  • TensorBoard or Weights & Biases for curves.

Common Pitfalls

  • Changing several things at once while debugging.

  • Not fixing seeds, so results can’t be compared.

  • Ignoring warnings (for example about scheduler order or deprecated arguments).

Interview Tips

  • Give a structured list: data, model, loss, optimizer, device, evaluation.

  • Mention the overfit-one-batch test and anomaly detection.

22

How do you create a DataLoader in PyTorch for a custom dataset?

To load a custom dataset, subclass torch.utils.data.Dataset and implement __len__ (number of samples) and __getitem__(idx) (return one sample, usually a tuple of tensors, applying any transform), then wrap it in a DataLoader that handles batching, shuffling, parallel loading with num_workers, and collation into batch tensors. For variable-length samples, pass a custom collate_fn; for very large or streaming data, use an IterableDataset.

Read full answer

Applies toPyTorch 2.x

Key DataLoader Arguments

Argument Purpose
batch_size Samples per batch
shuffle=True Reshuffle every epoch (training only)
num_workers Parallel loading processes
pin_memory=True Faster host-to-GPU copies
collate_fn How to combine samples into a batch
drop_last Drop the final incomplete batch
persistent_workers=True Keep workers alive between epochs
generator / worker_init_fn Reproducible shuffling and worker seeds

Code Example

import torch
from torch.nn.utils.rnn import pad_sequence
from torch.utils.data import DataLoader, Dataset

torch.manual_seed(42)

class SensorDataset(Dataset):
    """Each sample is a variable-length sequence of readings with a class label."""

    def __init__(self, n_samples=100, transform=None):
        lengths = torch.randint(5, 15, (n_samples,))
        self.sequences = [torch.randn(int(n), 3) for n in lengths]
        self.labels = torch.randint(0, 2, (n_samples,))
        self.transform = transform

    def __len__(self):
        return len(self.sequences)

    def __getitem__(self, idx):
        seq = self.sequences[idx]
        if self.transform:
            seq = self.transform(seq)
        return seq, self.labels[idx]

def normalize(seq):
    return (seq - seq.mean(dim=0)) / (seq.std(dim=0) + 1e-6)

def pad_collate(batch):
    seqs, labels = zip(*batch)
    lengths = torch.tensor([len(s) for s in seqs])
    padded = pad_sequence(seqs, batch_first=True)        # (batch, max_len, 3), zero-padded
    return padded, lengths, torch.stack(labels)

ds = SensorDataset(transform=normalize)
loader = DataLoader(ds, batch_size=8, shuffle=True, collate_fn=pad_collate,
                    num_workers=0, generator=torch.Generator().manual_seed(42))

padded, lengths, labels = next(iter(loader))
print(len(ds), padded.shape[0], padded.shape[2], lengths.tolist()[:3], labels.shape)
# 100 8 3 [three lengths between 5 and 14] torch.Size([8])

Map-Style vs Iterable-Style

Type Implement Use when
Dataset (map-style) __len__, __getitem__ Random access possible (files, arrays)
IterableDataset __iter__ Streams, huge files, databases

Common Pitfalls

  • Loading all files into memory in __init__ for huge datasets (load lazily in __getitem__).

  • num_workers > 0 with non-picklable objects or no if __name__ == "__main__": guard on Windows/macOS.

  • Shuffling validation data (harmless but makes debugging harder).

  • Returning NumPy float64 arrays instead of float32 tensors.

Interview Tips

  • Write __len__ and __getitem__, then the DataLoader.

  • Mention collate_fn for variable-length data and num_workers / pin_memory for speed.

23

What are transforms in torchvision used for?

Torchvision transforms convert and prepare images (and, in the transforms.v2 API, bounding boxes, masks and videos) for models: they turn PIL images or arrays into tensors, resize and crop them to the model’s input size, normalize pixel values with the dataset’s mean and standard deviation, and apply random data augmentation (flips, crops, color jitter, rotations) during training. You chain them with Compose and pass them to a dataset, typically with a random augmentation pipeline for training and a deterministic one for validation; pre-trained models also ship their exact preprocessing as weights.transforms().

Read full answer

Applies totorchvision 0.17+ (`torchvision.transforms.v2`), PyTorch 2.x

Common Transforms

Category Examples
Conversion ToImage(), ToDtype(torch.float32, scale=True)
Geometry Resize, CenterCrop, RandomResizedCrop, RandomHorizontalFlip, RandomRotation
Color ColorJitter, RandomGrayscale, GaussianBlur
Normalization Normalize(mean, std)
Policies AutoAugment, RandAugment, TrivialAugmentWide
Regularization RandomErasing; CutMix / MixUp (applied to batches)

Code Example

import torch
from torchvision.transforms import v2

torch.manual_seed(42)
IMAGENET_MEAN, IMAGENET_STD = [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]

train_tf = v2.Compose([
    v2.ToImage(),
    v2.RandomResizedCrop(224, scale=(0.6, 1.0), antialias=True),
    v2.RandomHorizontalFlip(p=0.5),
    v2.ColorJitter(brightness=0.2, contrast=0.2),
    v2.ToDtype(torch.float32, scale=True),     # uint8 [0, 255] -> float [0, 1]
    v2.Normalize(IMAGENET_MEAN, IMAGENET_STD),
])

val_tf = v2.Compose([
    v2.ToImage(),
    v2.Resize(256, antialias=True),
    v2.CenterCrop(224),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize(IMAGENET_MEAN, IMAGENET_STD),
])

fake_image = torch.randint(0, 256, (3, 300, 400), dtype=torch.uint8)   # stands in for a photo
print(train_tf(fake_image).shape, val_tf(fake_image).shape)           # both torch.Size([3, 224, 224])

# With a dataset: datasets.ImageFolder("data/train", transform=train_tf)
# With pre-trained weights: from torchvision.models import ResNet50_Weights; ResNet50_Weights.DEFAULT.transforms()

Common Pitfalls

  • Random augmentation in the validation pipeline.

  • Normalizing with the wrong statistics for a pre-trained model.

  • Geometric transforms on images without the same transform on masks or boxes (v2 handles them together).

  • Augmentations that change the label meaning (flipping digits or text).

Interview Tips

  • Separate preprocessing (resize, normalize) from augmentation (random).

  • Mention transforms.v2 and weights.transforms().

24

How do you prepare time-series data in PyTorch for RNNs?

Preparing time-series data for an RNN means turning one long series into supervised windows: split the data by time (train before validation before test), fit scaling on the training part only, slice sliding windows of seq_len past steps as inputs with the next value(s) as targets, and shape them as (batch, seq_len, features) for batch_first=True layers. A Dataset that creates windows on the fly avoids storing copies, the training loader can shuffle windows (each window is self-contained), and variable-length sequences use padding plus pack_padded_sequence.

Read full answer

Applies toPyTorch 2.x

Steps

  1. Order and clean: sort by time, handle missing values (forward-fill, interpolation, a missing-indicator feature).

  2. Split chronologically: never shuffle before splitting.

  3. Scale: fit mean/std on the training range; apply to all.

  4. Window: input x[t − seq_len : t], target x[t] (or x[t : t + horizon]).

  5. Shape: (batch, seq_len, n_features) with batch_first=True.

  6. Load: DataLoader(shuffle=True) for training windows, shuffle=False for evaluation.

Code Example

import math

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, Dataset

torch.manual_seed(42)
t = torch.arange(1000, dtype=torch.float32)
series = torch.sin(2 * math.pi * t / 50) + 0.1 * torch.randn(1000)    # synthetic signal

train_end, val_end = 700, 850
mean, std = series[:train_end].mean(), series[:train_end].std()      # training stats only
scaled = (series - mean) / std

class WindowDataset(Dataset):
    def __init__(self, data, seq_len=30, horizon=1):
        self.data, self.seq_len, self.horizon = data, seq_len, horizon

    def __len__(self):
        return len(self.data) - self.seq_len - self.horizon + 1

    def __getitem__(self, i):
        x = self.data[i:i + self.seq_len].unsqueeze(-1)                # (seq_len, 1)
        y = self.data[i + self.seq_len:i + self.seq_len + self.horizon]
        return x, y

seq_len = 30
train_ds = WindowDataset(scaled[:train_end], seq_len)
val_ds = WindowDataset(scaled[train_end - seq_len:val_end], seq_len)  # context overlaps, targets don't
train_dl = DataLoader(train_ds, batch_size=64, shuffle=True)
val_dl = DataLoader(val_ds, batch_size=64)

class LSTMForecaster(nn.Module):
    def __init__(self, hidden=32):
        super().__init__()
        self.lstm = nn.LSTM(input_size=1, hidden_size=hidden, batch_first=True)
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):
        out, _ = self.lstm(x)               # (batch, seq_len, hidden)
        return self.head(out[:, -1])        # last time step -> (batch, 1)

model = LSTMForecaster()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)
loss_fn = nn.MSELoss()
for epoch in range(5):
    model.train()
    for xb, yb in train_dl:
        optimizer.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
    model.eval()
    with torch.no_grad():
        val = sum(loss_fn(model(xb), yb).item() * len(xb) for xb, yb in val_dl) / len(val_ds)
    print(f"epoch {epoch + 1}: val MSE {val:.4f}")

xb, yb = next(iter(train_dl))
print(xb.shape, yb.shape)                   # torch.Size([64, 30, 1]) torch.Size([64, 1])

Common Pitfalls

  • Shuffling before the chronological split (leaks the future).

  • Fitting scalers on the whole series.

  • Forgetting the feature dimension (LSTM expects 3-D input).

  • No naive baseline (last value, seasonal naive) to compare against.

Interview Tips

  • Describe windowing, chronological splits and training-only scaling.

  • Give the expected tensor shape.

25

What is data augmentation and how do you implement it in PyTorch?

Data augmentation creates new, label-preserving variations of training examples on the fly, such as flipped, cropped or color-shifted images, noisy audio, or token-masked text, so the model sees more diverse data, learns invariances and overfits less. In PyTorch, images use torchvision.transforms.v2 pipelines passed to a dataset (randomness is re-sampled every time a sample is loaded), batch-level methods such as MixUp and CutMix run inside the training loop or a collate function, and other data types use torchaudio transforms, custom functions or libraries such as Albumentations; augmentation is applied only to training data.

Read full answer

Applies toPyTorch 2.x, torchvision 0.17+

Augmentations by Data Type

Data Examples
Images Random crop, flip, rotation, color jitter, RandAugment, RandomErasing
Batch-level (images) MixUp, CutMix
Audio Time shift, noise, SpecAugment (time/frequency masking)
Text Synonym replacement, back-translation, token masking
Tabular / time series Gaussian noise, jittering, window slicing, scaling

Code Example

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, Dataset
from torchvision.transforms import v2

torch.manual_seed(42)

train_tf = v2.Compose([
    v2.RandomResizedCrop(32, scale=(0.8, 1.0), antialias=True),
    v2.RandomHorizontalFlip(),
    v2.RandAugment(num_ops=2, magnitude=9),
    v2.ToDtype(torch.float32, scale=True),
    v2.RandomErasing(p=0.25),
])

class ImageTensorDataset(Dataset):
    def __init__(self, images, labels, transform=None):
        self.images, self.labels, self.transform = images, labels, transform

    def __len__(self):
        return len(self.images)

    def __getitem__(self, i):
        img = self.images[i]
        return (self.transform(img) if self.transform else img), self.labels[i]

images = torch.randint(0, 256, (256, 3, 32, 32), dtype=torch.uint8)   # stands in for CIFAR-style data
labels = torch.randint(0, 10, (256,))
loader = DataLoader(ImageTensorDataset(images, labels, train_tf), batch_size=32, shuffle=True)

cutmix_or_mixup = v2.RandomChoice([v2.CutMix(num_classes=10), v2.MixUp(num_classes=10)])
model = nn.Sequential(nn.Conv2d(3, 16, 3, padding=1), nn.ReLU(), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(16, 10))
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)

for xb, yb in loader:
    xb, yb = cutmix_or_mixup(xb, yb)             # yb becomes soft labels of shape (batch, 10)
    optimizer.zero_grad()
    loss = nn.functional.cross_entropy(model(xb), yb)   # cross_entropy accepts probability targets
    loss.backward()
    optimizer.step()
print(xb.shape, yb.shape)                        # torch.Size([32, 3, 32, 32]) torch.Size([32, 10])

Choosing Augmentations

  • Keep labels valid: vertical flips are fine for satellite images, wrong for digits like 6 and 9.

  • Start mild, increase strength if the model still overfits.

  • Validate with a deterministic pipeline.

Common Pitfalls

  • Augmenting the validation or test set.

  • Augmenting once offline and reusing the same copies (loses the per-epoch randomness).

  • Geometric transforms that don’t update masks or bounding boxes.

Interview Tips

  • Explain why augmentation regularizes and give examples per data type.

  • Mention MixUp/CutMix and that augmentation is training-only.

26

How do you use multiple GPUs for distributed training in PyTorch?

The standard way to train on multiple GPUs in PyTorch is DistributedDataParallel (DDP): launch one process per GPU with torchrun, initialize a process group (NCCL backend), wrap the model in DDP, and give each process a different shard of the data with DistributedSampler; each process computes gradients on its batch and DDP all-reduces (averages) them so every model copy stays identical. For models too large for one GPU, Fully Sharded Data Parallel (FSDP, now FSDP2’s fully_shard) shards parameters, gradients and optimizer state across GPUs, and tensor or pipeline parallelism splits individual layers or stages. nn.DataParallel still exists but is slower and not recommended.

Read full answer

Applies toPyTorch 2.x

Options

Approach What is split Use when
DDP Data (each GPU has a full model copy) Model fits on one GPU; the default choice
FSDP / FSDP2 Data + model states sharded Large models (billions of parameters)
Tensor parallelism Individual layers across GPUs Very large layers (LLMs)
Pipeline parallelism Layer groups across GPUs Very deep models
nn.DataParallel Data, single process Legacy; avoid

Code Example (save as train_ddp.py)

import os

import torch
import torch.distributed as dist
import torch.nn as nn
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, DistributedSampler, TensorDataset

def main():
    backend = "nccl" if torch.cuda.is_available() else "gloo"
    dist.init_process_group(backend)
    rank, local_rank = dist.get_rank(), int(os.environ["LOCAL_RANK"])
    if torch.cuda.is_available():
        torch.cuda.set_device(local_rank)
        device = torch.device("cuda", local_rank)
    else:
        device = torch.device("cpu")
    torch.manual_seed(42)

    X, y = torch.randn(4096, 32), torch.randint(0, 4, (4096,))
    dataset = TensorDataset(X, y)
    sampler = DistributedSampler(dataset, shuffle=True, seed=42)     # each rank gets a different shard
    loader = DataLoader(dataset, batch_size=64, sampler=sampler, pin_memory=True)

    model = nn.Sequential(nn.Linear(32, 128), nn.ReLU(), nn.Linear(128, 4)).to(device)
    model = DDP(model, device_ids=[local_rank] if device.type == "cuda" else None)
    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
    loss_fn = nn.CrossEntropyLoss()

    for epoch in range(3):
        sampler.set_epoch(epoch)                    # different shuffle every epoch
        for xb, yb in loader:
            xb, yb = xb.to(device), yb.to(device)
            optimizer.zero_grad()
            loss = loss_fn(model(xb), yb)
            loss.backward()                         # gradients are all-reduced here
            optimizer.step()
        if rank == 0:
            print(f"epoch {epoch + 1} loss {loss.item():.4f}")
            torch.save(model.module.state_dict(), "model.pt")   # save from one rank only

    dist.destroy_process_group()

if __name__ == "__main__":
    main()

Run it on one machine with 4 GPUs: torchrun --nproc_per_node=4 train_ddp.py. For several machines add --nnodes, --node_rank and a rendezvous address.

Practical Notes

  • The effective batch size is per-GPU batch × number of GPUs; scale the learning rate or use warm-up.

  • Combine with mixed precision (torch.autocast) for more speed.

  • Use SyncBatchNorm if per-GPU batches are small.

  • Log and save only from rank 0.

Common Pitfalls

  • Forgetting sampler.set_epoch(epoch), so every epoch uses the same order.

  • Using shuffle=True in the DataLoader together with a DistributedSampler.

  • Saving model.state_dict() with the module. prefix instead of model.module.state_dict().

Interview Tips

  • Explain DDP’s one-process-per-GPU design and gradient all-reduce.

  • Mention FSDP for models that don’t fit on one GPU.

27

What is transfer learning and how do you implement it in PyTorch?

Transfer learning reuses a model pre-trained on a large dataset (for example a ResNet trained on ImageNet or a BERT trained on large text corpora) as the starting point for a new task, so you need far less data and compute than training from scratch. In PyTorch you load pre-trained weights (torchvision weights=..., timm, or Hugging Face from_pretrained), replace the final layer to match your number of classes, and either freeze the backbone and train only the new head (feature extraction) or fine-tune all layers with a small learning rate, often after first training the head.

Read full answer

Applies toPyTorch 2.x, torchvision 0.17+

Two Strategies

Strategy What trains When
Feature extraction Only the new head Small dataset, similar domain
Fine-tuning All or upper layers, low LR More data, or a different domain
Gradual unfreezing Head first, then deeper layers Balance between the two

Code Example

import torch
import torch.nn as nn
from torchvision.models import ResNet18_Weights, resnet18

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"
num_classes = 5

weights = ResNet18_Weights.DEFAULT
model = resnet18(weights=weights)              # downloads ImageNet weights the first time
preprocess = weights.transforms()              # the exact preprocessing these weights expect

for param in model.parameters():               # 1) freeze the backbone
    param.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, num_classes)   # new head trains by default
model = model.to(device)

head_optimizer = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print("trainable parameters (head only):", trainable)     # 512 * 5 + 5 = 2565

# 2) later: unfreeze the last block and fine-tune with discriminative learning rates
for param in model.layer4.parameters():
    param.requires_grad = True
finetune_optimizer = torch.optim.AdamW([
    {"params": model.layer4.parameters(), "lr": 1e-4},
    {"params": model.fc.parameters(), "lr": 1e-3},
], weight_decay=0.01)

x = torch.randn(4, 3, 224, 224, device=device)
model.train()
loss = nn.functional.cross_entropy(model(x), torch.randint(0, num_classes, (4,), device=device))
finetune_optimizer.zero_grad()
loss.backward()
finetune_optimizer.step()
print(model(x).shape)                          # torch.Size([4, 5])

Tips

  • Use the pre-trained model’s normalization and input size (weights.transforms()).

  • Keep BatchNorm layers in eval mode when the backbone is frozen and batches are small.

  • Fine-tune with a learning rate 10–100× smaller than for training from scratch.

Common Pitfalls

  • Creating the optimizer over all parameters, including frozen ones (harmless but confusing), or forgetting to add newly unfrozen parameters.

  • Wrong preprocessing, which quietly ruins accuracy.

  • A high learning rate that destroys the pre-trained features.

Interview Tips

  • Contrast feature extraction and fine-tuning and when to use each.

  • Show replacing the head and freezing with requires_grad = False.

28

How do RNNs, LSTMs and GRUs compare in PyTorch?

All three process sequences step by step with a hidden state, and PyTorch provides them as nn.RNN, nn.LSTM and nn.GRU with the same interface; the difference is gating: a plain RNN updates its hidden state with one tanh layer and struggles with long-range dependencies because of vanishing gradients, an LSTM adds a separate cell state controlled by input, forget and output gates to keep information over long spans, and a GRU merges this into two gates (reset and update) with no separate cell state, giving similar accuracy to LSTMs on many tasks with fewer parameters. In PyTorch the main API difference is that LSTM returns and expects a tuple (h, c), while RNN and GRU use only h.

Read full answer

Applies toPyTorch 2.x

Comparison

Aspect nn.RNN nn.LSTM nn.GRU
Gates None Input, forget, output (+ candidate) Reset, update (+ candidate)
States h h and c h
Weight blocks per layer 1 4 3
Long-term memory Poor Best-known for it Good
Speed Fastest Slowest In between
Typical use Short sequences, teaching Long dependencies, default choice Smaller models, similar accuracy

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)
batch, seq_len, n_in, hidden = 4, 12, 8, 16
x = torch.randn(batch, seq_len, n_in)

rnn = nn.RNN(n_in, hidden, batch_first=True)
lstm = nn.LSTM(n_in, hidden, batch_first=True)
gru = nn.GRU(n_in, hidden, batch_first=True)

out_r, h_r = rnn(x)
out_l, (h_l, c_l) = lstm(x)            # LSTM returns (h, c)
out_g, h_g = gru(x)
print(out_r.shape, out_l.shape, out_g.shape)   # each torch.Size([4, 12, 16])
print(h_l.shape, c_l.shape)                    # torch.Size([1, 4, 16]) twice

def count(m):
    return sum(p.numel() for p in m.parameters())

# Per weight block: hidden * (n_in + hidden) weights + 2 * hidden biases = 16 * 24 + 32 = 416
print(count(rnn), count(lstm), count(gru))     # 416, 4 * 416 = 1664, 3 * 416 = 1248

Practical Notes

  • num_layers, dropout (between layers) and bidirectional=True work the same for all three.

  • Use pack_padded_sequence for batches of variable-length sequences.

  • For many sequence tasks today, transformers or temporal convolutions outperform all three, but RNNs remain useful for streaming and small models.

Common Pitfalls

  • Forgetting batch_first=True (default input is (seq_len, batch, features)).

  • Unpacking an LSTM’s state as a single tensor.

  • Training long sequences without gradient clipping.

Interview Tips

  • Explain gates and the vanishing gradient problem.

  • Note the LSTM (h, c) tuple and the 4 vs 3 weight blocks.

29

What is TorchScript and how does it help deploy PyTorch models?

TorchScript is a way to turn a PyTorch model into a serialized, statically typed program that can run without Python, for example from C++ through LibTorch, by either tracing (torch.jit.trace records the operations run on example inputs) or scripting (torch.jit.script compiles the Python code, including control flow). It made deployment to C++ servers and mobile possible and allowed some graph optimizations. It is now in maintenance mode: for new work PyTorch recommends torch.export (with AOTInductor for compiled artifacts or ExecuTorch for mobile and edge devices), torch.compile for speed in Python, or ONNX export, so you mainly meet TorchScript in existing systems.

Read full answer

Applies toPyTorch 2.x

Tracing vs Scripting

Aspect torch.jit.trace torch.jit.script
How Runs the model on example input and records ops Compiles the Python source
Control flow Frozen to the path taken by the example Preserved (if/loops)
Python support Any code that runs A subset of Python with type annotations
Risk Silently wrong for data-dependent branches Compilation errors on unsupported code

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)

class Gate(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(4, 2)

    def forward(self, x):
        if x.sum() > 0:                    # data-dependent branch
            return self.fc(x)
        return -self.fc(x)

model = Gate().eval()
pos, neg = torch.ones(1, 4), -torch.ones(1, 4)

scripted = torch.jit.script(model)         # keeps the if statement
traced = torch.jit.trace(model, pos)       # records only the "positive" path (emits a TracerWarning)

print(torch.allclose(scripted(neg), model(neg)))   # True
print(torch.allclose(traced(neg), model(neg)))     # False: the trace ignores the branch

scripted.save("gate_scripted.pt")
loaded = torch.jit.load("gate_scripted.pt")        # can also be loaded in C++ with torch::jit::load
print(torch.allclose(loaded(pos), model(pos)))     # True

# Current recommended path for new deployments
exported = torch.export.export(nn.Linear(4, 2).eval(), (pos,))
print(type(exported).__name__)                     # ExportedProgram

Deployment Options Today

Option Use
torch.export + AOTInductor Compiled artifacts for servers, no Python at runtime
ExecuTorch Mobile and edge devices
ONNX (torch.onnx.export(..., dynamo=True)) ONNX Runtime, TensorRT, other runtimes
TorchScript Existing C++/LibTorch systems
TorchServe / Triton / FastAPI Serving infrastructure

Common Pitfalls

  • Tracing models with data-dependent control flow.

  • Assuming TorchScript speeds up every model (often it doesn’t much).

  • Starting new projects on TorchScript instead of torch.export.

Interview Tips

  • Explain trace vs script with the control-flow example.

  • Mention that torch.export is the modern replacement.

30

How do you implement a PyTorch DataLoader for a CSV dataset?

To load a CSV in PyTorch, read it with pandas, separate the feature columns from the target, preprocess (encode categories, fill missing values, scale numeric columns with statistics fitted on the training split only), convert to float32 feature tensors and the right target dtype (int64 class indices for classification, float32 for regression), and wrap them in a Dataset whose __getitem__ returns one (features, target) pair; then pass it to DataLoader with a batch size and shuffle=True for training.

Read full answer

Applies toPyTorch 2.x, pandas 2.x

Code Example

import numpy as np
import pandas as pd
import torch
from torch.utils.data import DataLoader, Dataset

rng = np.random.default_rng(42)
n = 500
pd.DataFrame({
    "age": rng.integers(18, 70, n),
    "income": rng.normal(50_000, 15_000, n).round(2),
    "city": rng.choice(["Pune", "Delhi", "Chennai"], n),
    "label": rng.integers(0, 2, n),
}).to_csv("customers.csv", index=False)            # sample file for the example

class CSVDataset(Dataset):
    def __init__(self, df, target, numeric, categorical, stats=None, categories=None):
        df = df.copy()
        if stats is None:                           # fit on the training split only
            stats = {c: (df[c].mean(), df[c].std()) for c in numeric}
        if categories is None:
            categories = {c: sorted(df[c].dropna().unique()) for c in categorical}
        self.stats, self.categories = stats, categories

        num = np.stack([((df[c].fillna(stats[c][0]) - stats[c][0]) / stats[c][1]).to_numpy()
                        for c in numeric], axis=1)
        cat = [pd.get_dummies(pd.Categorical(df[c], categories=categories[c])).to_numpy()
               for c in categorical]
        self.X = torch.tensor(np.hstack([num] + cat), dtype=torch.float32)
        self.y = torch.tensor(df[target].to_numpy(), dtype=torch.long)

    def __len__(self):
        return len(self.y)

    def __getitem__(self, idx):
        return self.X[idx], self.y[idx]

df = pd.read_csv("customers.csv")
train_df = df.sample(frac=0.8, random_state=42)
val_df = df.drop(train_df.index)

cols = {"target": "label", "numeric": ["age", "income"], "categorical": ["city"]}
train_ds = CSVDataset(train_df, **cols)
val_ds = CSVDataset(val_df, **cols, stats=train_ds.stats, categories=train_ds.categories)

train_dl = DataLoader(train_ds, batch_size=32, shuffle=True)
val_dl = DataLoader(val_ds, batch_size=128)
xb, yb = next(iter(train_dl))
print(len(train_ds), len(val_ds), xb.shape, xb.dtype, yb.dtype)
# 400 100 torch.Size([32, 5]) torch.float32 torch.int64  (2 numeric + 3 one-hot city columns)

Large CSV Files

  • Read in chunks (pd.read_csv(..., chunksize=...)) inside an IterableDataset.

  • Convert once to Parquet or NumPy memmaps for faster repeated loading.

  • Use num_workers for heavy per-row preprocessing.

Common Pitfalls

  • Scaling with statistics from the whole file (leakage into validation).

  • float64 features or float labels with CrossEntropyLoss.

  • Category columns whose one-hot columns differ between splits (fix the category list from training).

Interview Tips

  • Show preprocessing fitted on training data and reused for validation.

  • Mention dtypes and chunked loading for big files.

31

How do you slice, index, concatenate and transpose tensors in PyTorch?

PyTorch tensors support NumPy-style slicing (t[1:, ::2]), integer and boolean (mask) indexing, torch.gather and index_select for index-based selection, torch.cat (join along an existing dimension) and torch.stack (join along a new dimension), and reshaping or reordering with view / reshape, transpose, permute, unsqueeze and squeeze. Slicing and transpose return views that share memory, while advanced indexing returns copies, and view requires contiguous memory, so after transpose you use reshape or .contiguous().

Read full answer

Applies toPyTorch 2.x

Code Example

import torch

t = torch.arange(12).reshape(3, 4)
print(t)
# tensor([[ 0,  1,  2,  3],
#         [ 4,  5,  6,  7],
#         [ 8,  9, 10, 11]])

# Slicing (views)
print(t[1])            # tensor([4, 5, 6, 7])
print(t[:, 2])         # tensor([ 2,  6, 10])
print(t[1:, ::2])      # tensor([[ 4,  6], [ 8, 10]])
print(t[-1, -1])       # tensor(11)

# Indexing (copies)
print(t[[0, 2]])                         # rows 0 and 2
print(t[t % 2 == 0])                     # tensor([ 0,  2,  4,  6,  8, 10])
print(t.index_select(1, torch.tensor([0, 3])))   # columns 0 and 3: tensor([[0, 3], [4, 7], [8, 11]])
idx = torch.tensor([[3], [0], [1]])
print(torch.gather(t, 1, idx).squeeze(1))        # one element per row: tensor([3, 4, 9])

# Concatenating and stacking
a, b = torch.zeros(2, 3), torch.ones(2, 3)
print(torch.cat([a, b], dim=0).shape)    # torch.Size([4, 3])
print(torch.cat([a, b], dim=1).shape)    # torch.Size([2, 6])
print(torch.stack([a, b]).shape)         # torch.Size([2, 2, 3]): new leading dimension

# Transposing and reshaping
print(t.T.shape)                         # torch.Size([4, 3])
x = torch.randn(8, 32, 32, 3)            # (batch, height, width, channels)
print(x.permute(0, 3, 1, 2).shape)       # torch.Size([8, 3, 32, 32]): channels first
print(t.transpose(0, 1).is_contiguous()) # False
print(t.transpose(0, 1).reshape(-1)[:4]) # tensor([0, 4, 8, 1]): reshape copies when needed
print(t.unsqueeze(0).shape, t.unsqueeze(0).squeeze(0).shape)   # torch.Size([1, 3, 4]) torch.Size([3, 4])

# Views share memory
row = t[0]
row[0] = 100
print(t[0, 0].item())                    # 100

Quick Reference

Operation Function Returns
Basic slicing t[a:b, c] View
Mask / list indexing t[mask], t[[0, 2]] Copy
Join existing dim torch.cat New tensor
Join new dim torch.stack New tensor
Swap two dims transpose(d0, d1), .T (2-D) View (non-contiguous)
Reorder all dims permute(...) View
Change shape view (contiguous only), reshape View or copy
Add / remove size-1 dims unsqueeze, squeeze View

Common Pitfalls

  • view after transpose raises an error; use reshape or .contiguous().view(...).

  • Modifying a slice changes the original tensor.

  • Confusing cat and stack shapes.

Interview Tips

  • Distinguish views from copies.

  • Know permute for channels-last to channels-first conversion.

32

How do you build a simple feedforward neural network for MNIST in PyTorch?

A feedforward network for MNIST flattens each 28×28 grayscale image into a 784-value vector, passes it through a couple of fully connected layers with ReLU (for example 784 → 256 → 128 → 10), and outputs 10 raw logits trained with nn.CrossEntropyLoss; load the data with torchvision.datasets.MNIST, normalize with the dataset mean (0.1307) and standard deviation (0.3081), and train for a few epochs with Adam. Such a model typically reaches about 97–98% test accuracy; CNNs push this above 99%.

Read full answer

Applies toPyTorch 2.x, torchvision 0.17+

Code Example

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"

tf = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])
train_ds = datasets.MNIST("data", train=True, download=True, transform=tf)
test_ds = datasets.MNIST("data", train=False, download=True, transform=tf)
train_dl = DataLoader(train_ds, batch_size=128, shuffle=True, num_workers=2)
test_dl = DataLoader(test_ds, batch_size=1000)

class MLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Flatten(),                       # (batch, 1, 28, 28) -> (batch, 784)
            nn.Linear(784, 256), nn.ReLU(), nn.Dropout(0.2),
            nn.Linear(256, 128), nn.ReLU(), nn.Dropout(0.2),
            nn.Linear(128, 10),                 # logits
        )

    def forward(self, x):
        return self.net(x)

model = MLP().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()

def evaluate():
    model.eval()
    correct = 0
    with torch.no_grad():
        for xb, yb in test_dl:
            xb, yb = xb.to(device), yb.to(device)
            correct += (model(xb).argmax(dim=1) == yb).sum().item()
    return correct / len(test_ds)

if __name__ == "__main__":
    for epoch in range(5):
        model.train()
        total = 0.0
        for xb, yb in train_dl:
            xb, yb = xb.to(device), yb.to(device)
            optimizer.zero_grad()
            loss = loss_fn(model(xb), yb)
            loss.backward()
            optimizer.step()
            total += loss.item() * len(xb)
        print(f"epoch {epoch + 1}: loss {total / len(train_ds):.4f}, test acc {evaluate():.4f}")
    torch.save(model.state_dict(), "mnist_mlp.pt")

Design Choices

Choice Reason
nn.Flatten MLPs need vectors, not images
Normalization (0.1307, 0.3081) Zero-mean, unit-variance inputs train faster
Dropout 0.2 Mild regularization
Logits + CrossEntropyLoss Numerically stable softmax inside the loss
Parameters 784·256 + 256 + 256·128 + 128 + 128·10 + 10 = 235,146

Common Pitfalls

  • Applying softmax before CrossEntropyLoss.

  • Forgetting model.eval() during testing (dropout active).

  • num_workers > 0 without the if __name__ == "__main__": guard on macOS/Windows.

Interview Tips

  • Write the flatten → linear → ReLU → logits structure and the training loop.

  • Mention expected accuracy and why a CNN does better.

33

How do you manually compute gradients for linear regression in PyTorch?

For linear regression ŷ = Xw + b with mean squared error L = (1/n)·Σ(ŷ − y)², the gradients are ∂L/∂w = (2/n)·Xᵀ(ŷ − y) and ∂L/∂b = (2/n)·Σ(ŷ − y); you can compute them directly with tensor operations, update the parameters with gradient descent, and verify the result against autograd with torch.allclose. Writing the gradients by hand shows what loss.backward() does for you.

Read full answer

Applies toPyTorch 2.x

Code Example

import torch

torch.manual_seed(42)
n, d = 200, 3
X = torch.randn(n, d)
true_w, true_b = torch.tensor([2.0, -1.0, 0.5]), 4.0
y = X @ true_w + true_b + 0.1 * torch.randn(n)

def manual_gradients(X, y, w, b):
    """Return (loss, dL/dw, dL/db) for mean squared error."""
    n = X.shape[0]
    residual = X @ w + b - y                 # (n,)
    loss = (residual ** 2).mean()
    grad_w = (2.0 / n) * (X.T @ residual)    # (d,)
    grad_b = (2.0 / n) * residual.sum()      # scalar
    return loss, grad_w, grad_b

# Check against autograd at a random point
w = torch.randn(d, requires_grad=True)
b = torch.zeros((), requires_grad=True)
loss_auto = ((X @ w + b - y) ** 2).mean()
loss_auto.backward()
_, gw, gb = manual_gradients(X, y, w.detach(), b.detach())
print(torch.allclose(gw, w.grad, atol=1e-5), torch.allclose(gb, b.grad, atol=1e-5))   # True True

# Gradient descent using only the manual gradients
w_m, b_m, lr = torch.zeros(d), torch.tensor(0.0), 0.1
for step in range(300):
    loss, gw, gb = manual_gradients(X, y, w_m, b_m)
    w_m -= lr * gw
    b_m -= lr * gb
print("learned w:", w_m.round(decimals=2).tolist(), "b:", round(b_m.item(), 2))
# close to [2.0, -1.0, 0.5] and 4.0
print("final loss:", round(loss.item(), 4))          # near the noise variance, about 0.01

Derivation

  • Residual r = Xw + b − y, loss L = (1/n)·rᵀr.

  • dL/dr = (2/n)·r.

  • r depends on w through X, so dL/dw = Xᵀ·(2/n)·r.

  • r depends on b with derivative 1 for each sample, so dL/db = (2/n)·Σ r.

Common Pitfalls

  • Forgetting the 1/n (or using a sum loss and a mean gradient), so the learning rate behaves differently.

  • Shape errors: y as (n, 1) with predictions as (n,) broadcasts to (n, n).

  • Updating parameters that require grad without torch.no_grad() (not an issue here, since the manual tensors don’t track gradients).

Interview Tips

  • Derive the gradient with the chain rule and verify it against autograd.

  • Mention the closed-form solution (normal equations) as an alternative.

34

How do you implement a CNN for image classification in PyTorch?

A CNN for image classification stacks convolution blocks (nn.Conv2d → nn.BatchNorm2d → nn.ReLU → nn.MaxPool2d) that learn local features and shrink the spatial size, followed by global average pooling and a linear layer that outputs one logit per class; you train it with nn.CrossEntropyLoss, an optimizer such as AdamW or SGD with momentum, and data augmentation, using inputs shaped (batch, channels, height, width). On CIFAR-10, a small network like the one below typically reaches roughly 80–85% accuracy after a few dozen epochs; deeper residual networks go well beyond that.

Read full answer

Applies toPyTorch 2.x, torchvision 0.17+

Code Example

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torchvision import datasets
from torchvision.transforms import v2

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"

def conv_block(c_in, c_out):
    return nn.Sequential(
        nn.Conv2d(c_in, c_out, kernel_size=3, padding=1, bias=False),
        nn.BatchNorm2d(c_out), nn.ReLU(inplace=True),
        nn.Conv2d(c_out, c_out, kernel_size=3, padding=1, bias=False),
        nn.BatchNorm2d(c_out), nn.ReLU(inplace=True),
        nn.MaxPool2d(2),                       # halves height and width
    )

class SmallCNN(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.features = nn.Sequential(conv_block(3, 32), conv_block(32, 64), conv_block(64, 128))
        self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Dropout(0.3),
                                  nn.Linear(128, num_classes))

    def forward(self, x):
        return self.head(self.features(x))

model = SmallCNN().to(device)
print(model(torch.randn(2, 3, 32, 32, device=device)).shape)   # torch.Size([2, 10])

mean, std = [0.4914, 0.4822, 0.4465], [0.2470, 0.2435, 0.2616]
train_tf = v2.Compose([v2.ToImage(), v2.RandomCrop(32, padding=4), v2.RandomHorizontalFlip(),
                       v2.ToDtype(torch.float32, scale=True), v2.Normalize(mean, std)])
test_tf = v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True), v2.Normalize(mean, std)])

if __name__ == "__main__":
    train_dl = DataLoader(datasets.CIFAR10("data", train=True, download=True, transform=train_tf),
                          batch_size=128, shuffle=True, num_workers=2)
    test_dl = DataLoader(datasets.CIFAR10("data", train=False, download=True, transform=test_tf),
                         batch_size=512)
    epochs = 20
    optimizer = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=5e-4)
    scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=3e-3,
                                                    total_steps=epochs * len(train_dl))
    loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1)
    for epoch in range(epochs):
        model.train()
        for xb, yb in train_dl:
            xb, yb = xb.to(device), yb.to(device)
            optimizer.zero_grad()
            loss = loss_fn(model(xb), yb)
            loss.backward()
            optimizer.step()
            scheduler.step()                   # OneCycle steps per batch
        model.eval()
        correct = 0
        with torch.no_grad():
            for xb, yb in test_dl:
                correct += (model(xb.to(device)).argmax(1) == yb.to(device)).sum().item()
        print(f"epoch {epoch + 1}: test acc {correct / 10_000:.3f}")

Shape Walk-Through (32×32 input)

Stage Output shape
Input (B, 3, 32, 32)
Block 1 (B, 32, 16, 16)
Block 2 (B, 64, 8, 8)
Block 3 (B, 128, 4, 4)
Global average pool + flatten (B, 128)
Linear (B, 10)

Common Pitfalls

  • Channels-last input (B, H, W, C); permute to (B, C, H, W).

  • A hard-coded nn.Linear input size that breaks when the image size changes (global pooling avoids this).

  • Forgetting model.eval(), which matters for BatchNorm and Dropout.

Interview Tips

  • Explain convolution, pooling and why CNNs share weights.

  • Walk through the output shapes layer by layer.

35

How do you save and load a trained PyTorch model?

The recommended way is to save the model’s state_dict (a dictionary of parameter and buffer tensors) with torch.save(model.state_dict(), path), then load it by creating the same model class and calling model.load_state_dict(torch.load(path, weights_only=True)), followed by model.eval() for inference. For resuming training, save a checkpoint dictionary that also holds the optimizer and scheduler states, the epoch and the best metric. Saving the whole model object with pickle is fragile (it ties the file to your code layout), and torch.load defaults to weights_only=True since PyTorch 2.6 for safety.

Read full answer

Applies toPyTorch 2.x

Code Example

import torch
import torch.nn as nn

torch.manual_seed(42)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 2))

    def forward(self, x):
        return self.layers(x)

model = Net()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10)
x, y = torch.randn(64, 10), torch.randint(0, 2, (64,))
for _ in range(5):
    optimizer.zero_grad()
    nn.functional.cross_entropy(model(x), y).backward()
    optimizer.step()
    scheduler.step()

# 1) Weights only, for inference
torch.save(model.state_dict(), "model_weights.pt")
restored = Net()
restored.load_state_dict(torch.load("model_weights.pt", weights_only=True, map_location="cpu"))
restored.eval()
with torch.no_grad():
    print(torch.allclose(restored(x), model(x)))      # True

# 2) Full checkpoint, for resuming training
torch.save({
    "epoch": 5,
    "model": model.state_dict(),
    "optimizer": optimizer.state_dict(),
    "scheduler": scheduler.state_dict(),
}, "checkpoint.pt")

ckpt = torch.load("checkpoint.pt", weights_only=True)
model2 = Net()
optimizer2 = torch.optim.AdamW(model2.parameters(), lr=1e-3)
scheduler2 = torch.optim.lr_scheduler.StepLR(optimizer2, step_size=10)
model2.load_state_dict(ckpt["model"])
optimizer2.load_state_dict(ckpt["optimizer"])
scheduler2.load_state_dict(ckpt["scheduler"])
start_epoch = ckpt["epoch"] + 1
model2.train()
print("resume from epoch", start_epoch)               # resume from epoch 6

Options

Method Use
state_dict + load_state_dict Standard, portable
Checkpoint dict Resume training exactly
torch.save(model) (whole object) Avoid: breaks when code moves
safetensors Safe, fast format used by Hugging Face
torch.export / ONNX Deployment without the Python class

Common Pitfalls

  • Loading GPU-saved weights on a CPU machine without map_location.

  • Keys prefixed with module. from DDP or _orig_mod. from torch.compile; save the unwrapped model.

  • Forgetting model.eval() after loading for inference.

  • Loading untrusted files with weights_only=False, which can execute arbitrary code.

Interview Tips

  • Recommend state_dict and explain why pickling the whole model is fragile.

  • Mention checkpoint contents for resuming and weights_only=True.

36

How do you handle imbalanced classes when training a classifier in PyTorch?

Handle class imbalance in PyTorch by rebalancing either the loss or the data: pass class weights to the loss (nn.CrossEntropyLoss(weight=...) or pos_weight in nn.BCEWithLogitsLoss), oversample rare classes with a WeightedRandomSampler, or use a focal loss that down-weights easy examples; then judge the model with imbalance-aware metrics (per-class recall, F1, PR-AUC) and tune the decision threshold on validation data. Data-level fixes such as collecting more minority examples or targeted augmentation often help the most.

Read full answer

Applies toPyTorch 2.x

Techniques

Technique PyTorch implementation Note
Class-weighted loss CrossEntropyLoss(weight=w) Weight ∝ 1 / class frequency
Positive weight (binary) BCEWithLogitsLoss(pos_weight=neg/pos) Single-logit binary tasks
Oversampling WeightedRandomSampler Balanced batches; risk of overfitting rare samples
Focal loss Custom: (1 − p_t)^γ · CE Focuses on hard examples
Augmentation of minority class Transforms applied more to rare samples Adds diversity
Threshold tuning Choose cut-off on validation PR curve No retraining
Metrics F1, recall per class, PR-AUC Accuracy misleads

Code Example

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset, WeightedRandomSampler

torch.manual_seed(42)
X = torch.randn(1000, 8)
y = torch.cat([torch.zeros(950), torch.ones(50)]).long()     # 95% / 5%

counts = torch.bincount(y)                                    # tensor([950, 50])
class_weights = counts.sum() / (len(counts) * counts.float())
print(class_weights.round(decimals=3))                        # tensor([ 0.5260, 10.0000])
weighted_ce = nn.CrossEntropyLoss(weight=class_weights)

pos_weight = counts[0] / counts[1]                            # 19.0 for a single-logit model
bce = nn.BCEWithLogitsLoss(pos_weight=pos_weight)

sample_weights = class_weights[y]                             # each sample weighted by its class
sampler = WeightedRandomSampler(sample_weights, num_samples=len(y), replacement=True,
                                generator=torch.Generator().manual_seed(42))
loader = DataLoader(TensorDataset(X, y), batch_size=100, sampler=sampler)
xb, yb = next(iter(loader))
print("minority share in a sampled batch:", yb.float().mean().item())   # about 0.5

def focal_loss(logits, targets, gamma=2.0, alpha=None):
    ce = F.cross_entropy(logits, targets, weight=alpha, reduction="none")
    p_t = torch.exp(-F.cross_entropy(logits, targets, reduction="none"))
    return ((1 - p_t) ** gamma * ce).mean()

logits = torch.randn(4, 2)
targets = torch.tensor([0, 1, 0, 1])
print(round(weighted_ce(logits, targets).item(), 4), round(focal_loss(logits, targets).item(), 4))
print(round(bce(logits[:, 1], targets.float()).item(), 4))

The weights use the formula n_samples / (n_classes · count): 1000 / (2 · 950) ≈ 0.526 and 1000 / (2 · 50) = 10.

Common Pitfalls

  • Reporting accuracy (95% by always predicting the majority class).

  • Combining a weighted loss and a balanced sampler, which over-corrects.

  • Oversampling before the train/validation split (duplicates leak into validation).

  • Probabilities from weighted training are no longer calibrated; recalibrate if you need them.

Interview Tips

  • Offer loss weighting, sampling and focal loss, plus the right metrics.

  • Mention threshold tuning as a cheap, effective step.

37

How do you use PyTorch for real-time inference and what are the concerns?

For real-time inference, export or optimize the trained model (model.eval(), torch.inference_mode(), torch.compile, quantization, or conversion through torch.export/ONNX to TensorRT or ONNX Runtime), load it once into a long-running service, and serve requests with tight latency budgets, often batching small requests together dynamically. The main concerns are latency (especially tail latency such as p99), throughput, warm-up and compilation time, hardware cost, memory, consistent preprocessing between training and serving, model accuracy after optimization, and monitoring for drift and failures.

Read full answer

Applies toPyTorch 2.x

Optimization Levers

Lever Effect Trade-off
model.eval() + torch.inference_mode() No autograd overhead None
torch.compile Kernel fusion, faster steady state Compile time, warm-up needed
Half precision (fp16/bf16) ~2× faster on GPUs, less memory Small accuracy risk
Quantization (int8) Faster on CPU, smaller model Accuracy check required
TensorRT / ONNX Runtime Highly optimized kernels Export complexity
Dynamic batching Higher GPU throughput Adds a few ms of waiting
Smaller models (distillation, pruning) Lower latency Retraining effort
CUDA graphs Lower launch overhead for small batches Static shapes

Code Example: Measuring Latency

import time

import torch
import torch.nn as nn

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = nn.Sequential(nn.Linear(256, 512), nn.ReLU(), nn.Linear(512, 10)).to(device).eval()
x = torch.randn(1, 256, device=device)

def measure(fn, runs=200, warmup=20):
    with torch.inference_mode():
        for _ in range(warmup):                  # warm-up: caches, compilation, cuDNN autotuning
            fn(x)
        times = []
        for _ in range(runs):
            start = time.perf_counter()
            fn(x)
            if device == "cuda":
                torch.cuda.synchronize()
            times.append((time.perf_counter() - start) * 1000)
    times = sorted(times)
    return round(times[len(times) // 2], 3), round(times[int(len(times) * 0.99) - 1], 3)

print("eager p50/p99 ms:", measure(model))
quantized = torch.ao.quantization.quantize_dynamic(model.cpu(), {nn.Linear}, dtype=torch.qint8)
if device == "cpu":
    print("int8 dynamic p50/p99 ms:", measure(quantized))

Concerns Checklist

  • Latency SLOs: track p50, p95 and p99, not only averages.

  • Cold starts: load and warm up before taking traffic.

  • Preprocessing parity: identical resizing, normalization and tokenization.

  • Concurrency: thread settings (torch.set_num_threads), multiple workers, GPU sharing.

  • Accuracy after optimization: compare outputs on a validation set.

  • Reliability: timeouts, fallbacks, health checks, versioned models.

  • Monitoring: latency, errors, input drift and prediction distributions.

Common Pitfalls

  • Benchmarking without warm-up or GPU synchronization.

  • Loading the model on every request.

  • Leaving dropout active by forgetting eval().

Interview Tips

  • Separate model-level optimizations from serving-level ones.

  • Talk about tail latency, warm-up and preprocessing parity.

38

When would you convert a PyTorch model to ONNX format?

You convert a PyTorch model to ONNX (Open Neural Network Exchange) when it must run outside the Python PyTorch runtime: for example, a C#, Java or C++ service using ONNX Runtime, an NVIDIA TensorRT pipeline for low-latency GPU inference, a browser via ONNX Runtime Web, edge or mobile hardware whose toolchain accepts ONNX, or a team that standardizes on one runtime for models from several frameworks. A typical scenario is a PyTorch-trained image classifier that a .NET backend must serve: export with torch.onnx.export(..., dynamo=True), declare dynamic batch size, validate the ONNX outputs against PyTorch on real inputs, and optionally quantize or optimize it in ONNX Runtime.

Read full answer

Applies toPyTorch 2.5+ (`dynamo=True` exporter), ONNX Runtime 1.17+

Scenario: Serving a Classifier from a .NET API

  1. Train in PyTorch; save the state_dict.

  2. Export to ONNX with a dynamic batch dimension.

  3. Check the model (onnx.checker) and compare outputs with PyTorch (np.allclose, tolerance ~1e-4).

  4. Load in the .NET service with Microsoft.ML.OnnxRuntime, CPU or GPU execution provider.

  5. Optionally quantize to int8 with ONNX Runtime tools and re-check accuracy.

Code Example

import numpy as np
import onnxruntime as ort
import torch
import torch.nn as nn

torch.manual_seed(42)
model = nn.Sequential(nn.Conv2d(3, 8, 3, padding=1), nn.ReLU(), nn.AdaptiveAvgPool2d(1),
                      nn.Flatten(), nn.Linear(8, 4)).eval()
example = torch.randn(2, 3, 64, 64)

batch = torch.export.Dim("batch", min=1, max=64)
onnx_program = torch.onnx.export(
    model, (example,), dynamo=True,
    input_names=["images"], output_names=["logits"],
    dynamic_shapes={"input": {0: batch}},       # keyed by the forward argument name ("input" for Sequential)
)
onnx_program.save("classifier.onnx")

session = ort.InferenceSession("classifier.onnx", providers=["CPUExecutionProvider"])
test = torch.randn(5, 3, 64, 64)                # a different batch size than the example
ort_out = session.run(None, {"images": test.numpy()})[0]
with torch.no_grad():
    torch_out = model(test).numpy()
print(ort_out.shape, np.allclose(ort_out, torch_out, atol=1e-4))   # (5, 4) True

When ONNX Is (and Isn’t) the Right Choice

Good fit Poor fit
Non-Python serving stacks Models with heavy custom Python logic
TensorRT / hardware accelerators Unsupported custom operators
One runtime for many frameworks Teams already serving with PyTorch / TorchServe
Browser or edge deployment Rapidly changing research code

Common Pitfalls

  • Forgetting dynamic axes, so the model only accepts the example batch size.

  • Not validating numerics after export.

  • Data-dependent control flow or custom ops that the exporter can’t capture.

  • Mismatched preprocessing between the training code and the consuming application.

Interview Tips

  • Give a concrete scenario (non-Python service or TensorRT).

  • Describe export, validation and runtime choice.

39

How would you deploy a PyTorch model as a REST API service?

A common approach is to wrap the model in a FastAPI application: load the model once at start-up (in eval mode, on the chosen device), define a Pydantic schema for the request, run preprocessing and inference under torch.inference_mode() in a POST /predict endpoint, and return JSON; then run it with Uvicorn behind a load balancer, package it in a Docker image, and add health checks, logging, metrics and versioning. For heavier traffic or GPUs, dedicated servers such as TorchServe, NVIDIA Triton or Ray Serve add dynamic batching, multi-model management and scaling.

Read full answer

Applies toPyTorch 2.x, FastAPI 0.110+, Pydantic 2.x

Code Example (app.py)

from contextlib import asynccontextmanager

import torch
import torch.nn as nn
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field

N_FEATURES, CLASSES = 4, ["setosa", "versicolor", "virginica"]
state = {}

def build_model():
    return nn.Sequential(nn.Linear(N_FEATURES, 16), nn.ReLU(), nn.Linear(16, len(CLASSES)))

@asynccontextmanager
async def lifespan(app: FastAPI):
    torch.manual_seed(42)
    model = build_model()
    # In production: model.load_state_dict(torch.load("model.pt", weights_only=True, map_location="cpu"))
    state["model"] = model.eval()
    yield
    state.clear()

app = FastAPI(title="Iris classifier", version="1.0.0", lifespan=lifespan)

class PredictRequest(BaseModel):
    features: list[list[float]] = Field(..., min_length=1, max_length=256)

class PredictResponse(BaseModel):
    labels: list[str]
    probabilities: list[list[float]]
    model_version: str

@app.get("/health")
def health():
    return {"status": "ok", "model_loaded": "model" in state}

@app.post("/predict", response_model=PredictResponse)
def predict(req: PredictRequest):
    if any(len(row) != N_FEATURES for row in req.features):
        raise HTTPException(status_code=422, detail=f"each row needs {N_FEATURES} features")
    x = torch.tensor(req.features, dtype=torch.float32)
    with torch.inference_mode():
        probs = torch.softmax(state["model"](x), dim=1)
    return PredictResponse(
        labels=[CLASSES[i] for i in probs.argmax(dim=1).tolist()],
        probabilities=probs.round(decimals=4).tolist(),
        model_version=app.version,
    )

# Run: uvicorn app:app --host 0.0.0.0 --port 8000 --workers 2
# Call: curl -X POST localhost:8000/predict -H "Content-Type: application/json" \
#            -d '{"features": [[5.1, 3.5, 1.4, 0.2]]}'

Production Checklist

Concern Approach
Packaging Docker image with pinned versions; model file or registry download
Scaling Multiple workers or replicas behind a load balancer; GPU servers for heavy models
Batching TorchServe / Triton dynamic batching for throughput
Validation Pydantic schemas, size limits
Observability Request logs, latency and error metrics, prediction distribution
Versioning Version in the response, canary or blue-green rollouts
Security Authentication, rate limits, HTTPS
Performance inference_mode, torch.set_num_threads, compile/quantize

Common Pitfalls

  • Loading the model inside the request handler.

  • Running blocking GPU inference in an async def endpoint, which stalls the event loop (use def or a thread pool).

  • Several Uvicorn workers each loading a large model onto one GPU and running out of memory.

Interview Tips

  • Walk through load-once, validate, infer, respond, then containerize and monitor.

  • Mention TorchServe/Triton for batching and scale.

40

How would you fine-tune a pre-trained model in PyTorch for a new task?

Fine-tuning starts from a pre-trained model, replaces its task head with one sized for the new task, and trains on the new data with a small learning rate, usually in stages: first train only the new head with the backbone frozen, then unfreeze some or all backbone layers and continue with lower, layer-wise learning rates, warm-up, weight decay and early stopping on a validation set. Matching the pre-trained preprocessing, watching for overfitting on small datasets, and, for large models, parameter-efficient methods like LoRA are the key practical points.

Read full answer

Applies toPyTorch 2.x, torchvision 0.17+ (Hugging Face models follow the same pattern)

Approach

  1. Pick a backbone close to your domain (ImageNet for photos, domain-specific models for medical or text tasks).

  2. Prepare data with the backbone’s preprocessing; split train/validation/test.

  3. Replace the head: new nn.Linear with the right number of outputs.

  4. Stage 1: freeze the backbone, train the head (higher LR, e.g. 1e-3).

  5. Stage 2: unfreeze upper layers (or all), LR 10–100× lower, warm-up + cosine decay.

  6. Regularize: augmentation, weight decay, label smoothing, early stopping.

  7. Evaluate on a held-out test set; compare with a frozen-backbone baseline.

Code Example

import torch
import torch.nn as nn
from torchvision.models import ResNet50_Weights, resnet50

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"
num_classes = 3

weights = ResNet50_Weights.DEFAULT
model = resnet50(weights=weights)
preprocess = weights.transforms()                        # use for both training and validation inputs
model.fc = nn.Sequential(nn.Dropout(0.2), nn.Linear(model.fc.in_features, num_classes))
model = model.to(device)

def set_trainable(module, flag):
    for p in module.parameters():
        p.requires_grad = flag

# Stage 1: head only
set_trainable(model, False)
set_trainable(model.fc, True)
stage1_opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3, weight_decay=0.01)

# Stage 2: unfreeze the last two stages with layer-wise learning rates
set_trainable(model.layer3, True)
set_trainable(model.layer4, True)
stage2_opt = torch.optim.AdamW([
    {"params": model.layer3.parameters(), "lr": 2e-5},
    {"params": model.layer4.parameters(), "lr": 1e-4},
    {"params": model.fc.parameters(), "lr": 5e-4},
], weight_decay=0.01)
stage2_sched = torch.optim.lr_scheduler.OneCycleLR(
    stage2_opt, max_lr=[2e-5, 1e-4, 5e-4], total_steps=1000, pct_start=0.1)

def train_step(xb, yb, optimizer, scheduler=None):
    model.train()
    optimizer.zero_grad()
    loss = nn.functional.cross_entropy(model(xb), yb, label_smoothing=0.1)
    loss.backward()
    optimizer.step()
    if scheduler:
        scheduler.step()
    return loss.item()

xb = torch.randn(4, 3, 224, 224, device=device)          # stands in for a preprocessed batch
yb = torch.randint(0, num_classes, (4,), device=device)
print(round(train_step(xb, yb, stage1_opt), 4), round(train_step(xb, yb, stage2_opt, stage2_sched), 4))
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"trainable parameters in stage 2: {trainable:,}")

Choices by Data Size

Data Strategy
Very small (hundreds) Freeze backbone; train head; strong augmentation
Medium (thousands) Unfreeze top layers, low LR
Large / different domain Fine-tune everything, warm-up
Huge model, limited GPU LoRA / adapters (PEFT), mixed precision, gradient checkpointing

Common Pitfalls

  • A learning rate that is too high, which destroys pre-trained features (“catastrophic forgetting”).

  • Mismatched preprocessing.

  • BatchNorm statistics drifting with tiny batches; keep BN in eval mode or freeze it.

  • Evaluating on the validation set used for early stopping and reporting it as test accuracy.

Interview Tips

  • Describe the staged freeze-then-unfreeze schedule with layer-wise learning rates.

  • Mention LoRA for large models and preprocessing parity.

41

What are Graph Neural Networks and how do you implement them in PyTorch?

Graph Neural Networks (GNNs) learn from graph-structured data (nodes connected by edges, such as social networks, molecules, road networks or citation graphs) by message passing: in each layer every node gathers (“aggregates”) transformed features from its neighbours and combines them with its own, so after k layers a node’s representation reflects its k-hop neighbourhood. In PyTorch you can write a graph convolution by hand with a normalized adjacency matrix, but most projects use PyTorch Geometric (PyG) or DGL, which provide layers such as GCNConv, GATConv and SAGEConv, sparse message passing and graph datasets.

Read full answer

Applies toPyTorch 2.x, PyTorch Geometric 2.5+

Core Idea (GCN Layer)

  • H′ = σ( D̂^(−½)  D̂^(−½) H W ), where  = A + I adds self-loops and D̂ is its degree matrix.

  • Each node averages its neighbours’ (and its own) features, weighted by degree, then applies a learned linear map.

Common Layers and Tasks

Layer Aggregation
GCN Degree-normalized mean of neighbours
GraphSAGE Sampled neighbours, mean/max/LSTM aggregator; scales to large graphs
GAT Attention weights over neighbours
GIN Sum aggregation + MLP; expressive for graph classification
Task Output
Node classification A label per node (e.g. paper topic)
Link prediction Score per node pair (recommendations)
Graph classification A label per graph (molecule toxicity), via pooling

Code Example: GCN from Scratch

import torch
import torch.nn as nn

torch.manual_seed(42)

edges = torch.tensor([[0, 1], [1, 2], [2, 0], [3, 4], [4, 5], [5, 3], [2, 3]])   # two triangles, one bridge
n = 6
A = torch.zeros(n, n)
A[edges[:, 0], edges[:, 1]] = 1
A = A + A.T                                               # undirected
A_hat = A + torch.eye(n)                                  # self-loops
d_inv_sqrt = A_hat.sum(dim=1).pow(-0.5)
A_norm = d_inv_sqrt[:, None] * A_hat * d_inv_sqrt[None, :]

class GCN(nn.Module):
    def __init__(self, in_dim, hidden, out_dim):
        super().__init__()
        self.w1, self.w2 = nn.Linear(in_dim, hidden), nn.Linear(hidden, out_dim)

    def forward(self, x, a_norm):
        h = torch.relu(a_norm @ self.w1(x))               # aggregate transformed neighbour features
        return a_norm @ self.w2(h)

X = torch.eye(n)                                          # one-hot node features
labels = torch.tensor([0, 0, 0, 1, 1, 1])
train_mask = torch.tensor([True, False, False, True, False, False])   # one labelled node per class

model = GCN(n, 16, 2)
optimizer = torch.optim.Adam(model.parameters(), lr=0.05, weight_decay=5e-4)
for epoch in range(100):
    optimizer.zero_grad()
    out = model(X, A_norm)
    loss = nn.functional.cross_entropy(out[train_mask], labels[train_mask])
    loss.backward()
    optimizer.step()
print(model(X, A_norm).argmax(dim=1).tolist())   # labels spread to unlabelled nodes via the graph

With PyTorch Geometric

import torch
import torch.nn as nn
from torch_geometric.datasets import Planetoid
from torch_geometric.nn import GCNConv

dataset = Planetoid(root="data/Cora", name="Cora")       # citation graph, 2,708 nodes, 7 classes
data = dataset[0]

class PygGCN(nn.Module):
    def __init__(self, in_dim, hidden, out_dim):
        super().__init__()
        self.conv1, self.conv2 = GCNConv(in_dim, hidden), GCNConv(hidden, out_dim)

    def forward(self, x, edge_index):
        x = torch.relu(self.conv1(x, edge_index))
        x = nn.functional.dropout(x, p=0.5, training=self.training)
        return self.conv2(x, edge_index)

torch.manual_seed(42)
model = PygGCN(dataset.num_features, 16, dataset.num_classes)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
for epoch in range(200):
    model.train()
    optimizer.zero_grad()
    out = model(data.x, data.edge_index)
    nn.functional.cross_entropy(out[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()
model.eval()
pred = model(data.x, data.edge_index).argmax(dim=1)
print("test accuracy:", (pred[data.test_mask] == data.y[data.test_mask]).float().mean().item())   # about 0.8

Common Pitfalls

  • Too many layers, which over-smooths node features until they look alike.

  • Dense adjacency matrices for large graphs (use sparse edge lists).

  • Leaking test nodes’ labels in transductive settings.

Interview Tips

  • Explain message passing and the GCN formula.

  • Name PyG/DGL and the three task types.

42

What is neural architecture search and how is it applied with PyTorch?

Neural architecture search (NAS) automates the design of network architectures by searching a defined space of options (layer types, depths, widths, connections) with a search strategy (reinforcement learning, evolution, Bayesian optimization, or gradient-based relaxation) and a performance estimate for each candidate. Research moved from very expensive early methods (thousands of GPU-days) to efficient ones: weight-sharing one-shot supernets (ENAS, DARTS, Once-for-All), zero-cost proxies that score untrained networks, and hardware-aware search that optimizes accuracy together with latency on a target device, as in the MobileNetV3 and EfficientNet families. In PyTorch, NAS is done with general tuning tools (Optuna, Ray Tune) over model-building code, with libraries such as Microsoft NNI or Meta’s Ax, or with a hand-written differentiable supernet.

Read full answer

Applies toPyTorch 2.x; Optuna 3.x/4.x for the example

Components of NAS

Component Options
Search space Cell-based (repeated blocks), macro (whole network), layer options
Search strategy RL (NASNet), evolution (AmoebaNet), Bayesian optimization, gradient-based (DARTS), random search
Performance estimation Full training (costly), low-fidelity training, weight sharing, zero-cost proxies
Objectives Accuracy, plus latency, FLOPs, memory, energy (hardware-aware)

Research Directions

  • One-shot / weight sharing: train one supernet; sample sub-networks without retraining (Once-for-All).

  • Differentiable NAS (DARTS): relax discrete choices into softmax-weighted mixtures and learn them by gradient descent; known to be unstable and to favour parameter-free operations, which follow-up work addresses.

  • Zero-cost proxies: score untrained networks from gradients or activations in seconds.

  • Hardware-aware NAS: latency predictors for phones and accelerators.

  • Benchmarks: NAS-Bench-101/201 make comparisons reproducible; random search is a strong baseline.

  • Transformers and LLMs: search over width, depth and attention heads, often combined with pruning.

Code Example: Simple NAS with Optuna

import optuna
import torch
import torch.nn as nn

torch.manual_seed(42)
X = torch.randn(1000, 20)
y = ((X[:, 0] * X[:, 1] + X[:, 2]) > 0).long()
X_train, y_train, X_val, y_val = X[:800], y[:800], X[800:], y[800:]

def build(trial):
    layers, width_in = [], 20
    for i in range(trial.suggest_int("n_layers", 1, 3)):
        width = trial.suggest_categorical(f"width_{i}", [16, 32, 64, 128])
        act = {"relu": nn.ReLU(), "gelu": nn.GELU(), "tanh": nn.Tanh()}[
            trial.suggest_categorical(f"act_{i}", ["relu", "gelu", "tanh"])]
        layers += [nn.Linear(width_in, width), act]
        width_in = width
    layers.append(nn.Linear(width_in, 2))
    return nn.Sequential(*layers)

def objective(trial):
    torch.manual_seed(42)
    model = build(trial)
    opt = torch.optim.Adam(model.parameters(), lr=trial.suggest_float("lr", 1e-4, 1e-2, log=True))
    for _ in range(100):                     # short training = low-fidelity estimate
        opt.zero_grad()
        nn.functional.cross_entropy(model(X_train), y_train).backward()
        opt.step()
    with torch.no_grad():
        acc = (model(X_val).argmax(1) == y_val).float().mean().item()
    params = sum(p.numel() for p in model.parameters())
    trial.set_user_attr("params", params)
    return acc

study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=42))
optuna.logging.set_verbosity(optuna.logging.WARNING)
study.optimize(objective, n_trials=25)
print("best accuracy:", round(study.best_value, 3), "params:", study.best_trial.user_attrs["params"])
print(study.best_params)

Common Pitfalls

  • Selecting architectures on the test set.

  • Comparing NAS results without a random-search baseline or equal compute budgets.

  • Ignoring deployment constraints (latency, memory) until after the search.

Interview Tips

  • Name the three components: search space, strategy, estimation.

  • Mention weight sharing, DARTS and hardware-aware NAS, and that random search is a strong baseline.

43

How do you implement GANs in PyTorch and what are their challenges?

A generative adversarial network (GAN) trains two networks against each other: a generator G that turns random noise into fake samples and a discriminator D that classifies samples as real or fake; in PyTorch you build both as nn.Modules with separate optimizers and alternate updates, training D with BCEWithLogitsLoss on real (label 1) and detached fake (label 0) batches, then training G to make D label its fakes as real. The challenges are unstable training, mode collapse (G produces little variety), vanishing generator gradients when D wins, sensitivity to hyperparameters and architecture, and hard evaluation (metrics like FID); remedies include DCGAN design rules, spectral normalization, Wasserstein loss with gradient penalty, and two-timescale learning rates.

Read full answer

Applies toPyTorch 2.x

Code Example: GAN on a 2-D Distribution

import torch
import torch.nn as nn

torch.manual_seed(42)
device = "cuda" if torch.cuda.is_available() else "cpu"
latent_dim = 8

def real_batch(n):
    """Mixture of 4 Gaussians on a circle: a quick test for mode collapse."""
    angles = torch.randint(0, 4, (n,)).float() * (torch.pi / 2)
    centers = torch.stack([torch.cos(angles), torch.sin(angles)], dim=1) * 2
    return (centers + 0.1 * torch.randn(n, 2)).to(device)

G = nn.Sequential(nn.Linear(latent_dim, 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 2)).to(device)
D = nn.Sequential(nn.utils.parametrizations.spectral_norm(nn.Linear(2, 64)), nn.LeakyReLU(0.2),
                  nn.utils.parametrizations.spectral_norm(nn.Linear(64, 64)), nn.LeakyReLU(0.2),
                  nn.Linear(64, 1)).to(device)                       # outputs a logit

opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
bce = nn.BCEWithLogitsLoss()
batch = 256

for step in range(3000):
    # Discriminator: real -> 1, fake -> 0
    real = real_batch(batch)
    fake = G(torch.randn(batch, latent_dim, device=device)).detach()   # don't update G here
    d_loss = bce(D(real), torch.ones(batch, 1, device=device)) + \
             bce(D(fake), torch.zeros(batch, 1, device=device))
    opt_d.zero_grad()
    d_loss.backward()
    opt_d.step()

    # Generator: make D call fakes real (non-saturating loss)
    fake = G(torch.randn(batch, latent_dim, device=device))
    g_loss = bce(D(fake), torch.ones(batch, 1, device=device))
    opt_g.zero_grad()
    g_loss.backward()
    opt_g.step()

    if step % 1000 == 0:
        print(f"step {step}: D loss {d_loss.item():.3f}, G loss {g_loss.item():.3f}")

with torch.no_grad():
    samples = G(torch.randn(2000, latent_dim, device=device)).cpu()
quadrants = ((samples[:, 0] > 0).long() * 2 + (samples[:, 1] > 0).long())
print("samples per quadrant:", torch.bincount(quadrants, minlength=4).tolist())   # balanced counts suggest no collapse

Challenges and Remedies

Challenge Remedy
Training instability Spectral norm, lower LR, Adam β₁ = 0.5, TTUR (different LRs for G and D)
Mode collapse Minibatch discrimination, WGAN-GP, unrolled GANs, diversity checks
Vanishing generator gradients Non-saturating loss (shown), Wasserstein loss
D overpowering G Label smoothing, noise on inputs, balance update counts
Evaluation FID, precision/recall for generative models, human review
Compute for high resolution Progressive growing, StyleGAN architectures

Where GANs Stand Today

  • Diffusion models now lead most image generation benchmarks, but GANs remain valuable for fast single-step generation, super-resolution, image-to-image translation and as adversarial losses in other models.

Common Pitfalls

  • Forgetting .detach() on fakes in the discriminator step, which updates G with the wrong objective.

  • Using BCELoss with a sigmoid output instead of the more stable BCEWithLogitsLoss with logits.

  • Judging progress from losses alone; inspect samples.

Interview Tips

  • Write the alternating update with .detach() and the non-saturating generator loss.

  • Name mode collapse and instability with concrete remedies.

44

What is model quantization in PyTorch and when is it useful?

Model quantization stores weights, and often activations, in low-precision integer formats such as int8 (or int4 for weights in LLMs) instead of 32-bit floats, cutting model size by about 4× for int8 and often speeding up inference on CPUs, mobile chips and accelerators with integer support, usually at a small accuracy cost. PyTorch offers dynamic quantization (weights quantized ahead of time, activations on the fly; easiest, good for linear- and LSTM-heavy models on CPU), static post-training quantization (activations calibrated on sample data), and quantization-aware training (fake quantization during training for the best accuracy); newer work uses the torchao library and the torch.export-based PT2E flow. It is useful when deploying to edge or mobile devices, serving under tight latency or cost budgets, or fitting large models into limited memory.

Read full answer

Applies toPyTorch 2.x (`torch.ao.quantization`, torchao)

Methods

Method What’s quantized Calibration data Accuracy Best for
Dynamic PTQ Weights ahead of time, activations at runtime None Good Linear/LSTM/transformer layers on CPU
Static PTQ Weights and activations A few hundred samples Good with care CNNs on CPU / mobile
Quantization-aware training (QAT) Simulated during training Training data Best When PTQ loses too much accuracy
Weight-only int8 / int4 (torchao, GPTQ, AWQ) Weights Optional Good LLM inference on GPUs
fp16 / bf16 Lower-precision floats None Near-lossless GPU inference

Code Example: Dynamic Quantization

import io
import time

import torch
import torch.nn as nn

torch.manual_seed(42)
model = nn.Sequential(nn.Linear(1024, 1024), nn.ReLU(), nn.Linear(1024, 1024), nn.ReLU(),
                      nn.Linear(1024, 10)).eval()
quantized = torch.ao.quantization.quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8)

def size_mb(m):
    buffer = io.BytesIO()
    torch.save(m.state_dict(), buffer)
    return buffer.getbuffer().nbytes / 1e6

def latency_ms(m, x, runs=50):
    with torch.inference_mode():
        for _ in range(5):
            m(x)
        start = time.perf_counter()
        for _ in range(runs):
            m(x)
    return (time.perf_counter() - start) / runs * 1000

x = torch.randn(32, 1024)
print(f"size: {size_mb(model):.1f} MB -> {size_mb(quantized):.1f} MB")    # roughly 4x smaller
print(f"latency: {latency_ms(model, x):.2f} ms -> {latency_ms(quantized, x):.2f} ms (CPU)")
with torch.inference_mode():
    diff = (model(x) - quantized(x)).abs().max().item()
print(f"max output difference: {diff:.4f}")                               # small but non-zero

When to Use It

  • Mobile and edge deployment (smaller downloads, battery and memory limits).

  • CPU inference services where latency or cost matters.

  • LLMs that otherwise don’t fit in GPU memory (weight-only int4/int8).

  • Not worth it when the model is already small and fast, or accuracy is critical and untested.

Common Pitfalls

  • Not measuring accuracy on a validation set after quantization.

  • Expecting dynamic quantization to speed up convolutions (it targets Linear and recurrent layers).

  • Calibrating static quantization on unrepresentative data.

  • Quantizing layers that are sensitive (often the first and last) without checking.

Interview Tips

  • Contrast dynamic, static and QAT.

  • Mention size/latency/accuracy trade-offs and weight-only quantization for LLMs.

45

What role does PyTorch play in reinforcement learning research?

PyTorch is the main framework for deep reinforcement learning research: its eager execution makes it easy to write and debug the unusual training loops RL needs (interleaving environment interaction, replay buffers, multiple networks and custom losses), and most reference implementations and libraries build on it, including Stable-Baselines3, CleanRL, TorchRL (PyTorch’s official RL library), Tianshou and RLlib’s PyTorch backend. Researchers use it to define policy and value networks, compute policy-gradient or TD losses with autograd, and scale training with vectorized environments and GPUs.

Read full answer

Applies toPyTorch 2.x, Gymnasium 1.x

Ecosystem

Library Focus
Stable-Baselines3 Reliable implementations (PPO, DQN, SAC, TD3)
CleanRL Single-file, readable research implementations
TorchRL Official PyTorch RL components: environments, buffers, losses
Tianshou Modular research framework
RLlib Distributed RL at scale
Hugging Face TRL RL from human feedback for language models

Example: REINFORCE on CartPole

import gymnasium as gym
import torch
import torch.nn as nn

torch.manual_seed(42)
env = gym.make("CartPole-v1")
policy = nn.Sequential(nn.Linear(4, 64), nn.Tanh(), nn.Linear(64, 2))   # action logits
optimizer = torch.optim.Adam(policy.parameters(), lr=1e-2)
gamma = 0.99

def run_episode(seed):
    obs, _ = env.reset(seed=seed)
    log_probs, rewards, done = [], [], False
    while not done:
        logits = policy(torch.as_tensor(obs, dtype=torch.float32))
        dist = torch.distributions.Categorical(logits=logits)
        action = dist.sample()
        obs, reward, terminated, truncated, _ = env.step(action.item())
        log_probs.append(dist.log_prob(action))
        rewards.append(float(reward))
        done = terminated or truncated
    return torch.stack(log_probs), rewards

returns_history = []
for episode in range(500):
    log_probs, rewards = run_episode(seed=episode)
    G, returns = 0.0, []
    for r in reversed(rewards):                     # discounted return for each step
        G = r + gamma * G
        returns.insert(0, G)
    returns = torch.tensor(returns)
    returns = (returns - returns.mean()) / (returns.std() + 1e-8)   # simple baseline / normalization
    loss = -(log_probs * returns).sum()             # policy-gradient loss, differentiated by autograd
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    returns_history.append(sum(rewards))
    if (episode + 1) % 100 == 0:
        recent = returns_history[-100:]
        print(f"episode {episode + 1}: mean return of last 100 = {sum(recent) / len(recent):.1f}")
# The mean return usually rises well above the ~20 of a random policy

Why PyTorch Fits RL Research

  • Autograd handles custom objectives (policy gradients, entropy bonuses, KL penalties).

  • torch.distributions gives sampling and log-probabilities for policies.

  • Dynamic graphs suit variable-length episodes.

  • GPU acceleration and torch.compile speed up large-batch algorithms like PPO.

Common Pitfalls

  • Forgetting to detach targets in value-based losses.

  • Evaluating with the stochastic training policy only.

  • Single-seed results (RL is high-variance).

Interview Tips

  • Name a few PyTorch-based RL libraries.

  • Show how a policy-gradient loss uses torch.distributions and autograd.

46

How do you create a PyTorch extension with custom C++/CUDA operations?

A PyTorch C++/CUDA extension implements an operation in C++ (and CUDA kernels for GPU) using the ATen tensor library, registers it with PyTorch, and builds it into a Python-loadable module, either ahead of time with setuptools and torch.utils.cpp_extension.CppExtension / CUDAExtension, or just-in-time with torch.utils.cpp_extension.load or load_inline. Current PyTorch recommends registering operators with TORCH_LIBRARY so they work with torch.compile, autograd (via torch.library.register_autograd or a custom autograd.Function) and fake tensors for tracing. You write one only when existing operations are too slow or don’t exist; often Triton kernels or torch.compile are an easier route to speed.

Read full answer

Applies toPyTorch 2.4+ (custom operator API), CUDA 12.x for GPU kernels

Steps

  1. Write the C++ function taking and returning torch::Tensor (at::Tensor).

  2. Write a CUDA kernel and a launcher for GPU tensors (optional).

  3. Register the operator with TORCH_LIBRARY(my_ops, m) and TORCH_LIBRARY_IMPL per device.

  4. Build: setup.py with CUDAExtension, or JIT load(...).

  5. Add autograd: register a backward formula in Python.

  6. Test: compare against a reference implementation and run torch.library.opcheck.

Example: Fused “scaled ReLU” (JIT build)

import torch
from torch.utils.cpp_extension import load_inline

cpp_source = r"""
#include <torch/extension.h>

torch::Tensor scaled_relu(torch::Tensor x, double scale) {
    TORCH_CHECK(x.is_floating_point(), "x must be a floating point tensor");
    return torch::clamp_min(x, 0) * scale;
}

TORCH_LIBRARY(demo_ops, m) {
    m.def("scaled_relu(Tensor x, float scale) -> Tensor");
}

TORCH_LIBRARY_IMPL(demo_ops, CPU, m) {
    m.impl("scaled_relu", &scaled_relu);
}
"""

load_inline(name="demo_ops_ext", cpp_sources=cpp_source, is_python_module=False, verbose=False)

def backward(ctx, grad_out):
    x, = ctx.saved_tensors
    return grad_out * (x > 0) * ctx.scale, None

def setup_context(ctx, inputs, output):
    x, scale = inputs
    ctx.save_for_backward(x)
    ctx.scale = scale

torch.library.register_autograd("demo_ops::scaled_relu", backward, setup_context=setup_context)

@torch.library.register_fake("demo_ops::scaled_relu")
def _(x, scale):
    return torch.empty_like(x)                      # shape/dtype rule for torch.compile tracing

x = torch.tensor([-1.0, 0.5, 2.0], requires_grad=True)
y = torch.ops.demo_ops.scaled_relu(x, 3.0)
y.sum().backward()
print(y.detach(), x.grad)                           # tensor([0.0000, 1.5000, 6.0000]) tensor([0., 3., 3.])

For a CUDA version, add a .cu file with a __global__ kernel, launch it from a C++ function for CUDA tensors, register it under TORCH_LIBRARY_IMPL(demo_ops, CUDA, m), and build with CUDAExtension in setup.py.

Alternatives Before Writing C++/CUDA

Option When
torch.compile Fusing standard ops automatically
Triton kernels Custom GPU kernels written in Python
torch.library.custom_op in Python Wrapping non-PyTorch code as an operator
Existing libraries (xFormers, FlashAttention) Common fused kernels

Common Pitfalls

  • No backward or fake-tensor registration, which breaks training or torch.compile.

  • Assuming contiguous inputs in CUDA kernels (call .contiguous() or handle strides).

  • ABI or CUDA version mismatches between the extension and the installed PyTorch.

Interview Tips

  • Describe write, register, build, add autograd, and test.

  • Mention Triton and torch.compile as simpler alternatives.

47

How should you answer a question about contributing to PyTorch or using community tools?

Answer with a specific, honest example: what you contributed or used, why, what you did technically, and the result; if you haven’t contributed to PyTorch itself, talk about community tools you have used seriously (PyTorch Lightning, Hugging Face Transformers, PyTorch Geometric, timm, torchmetrics, Weights & Biases) or contributions to smaller projects, bug reports and forum answers. Interviewers are checking whether you can work in a real ecosystem: read other people’s code, file a good issue, follow contribution rules, and choose tools for good reasons. Never invent contributions; maintainers’ names and pull requests are easy to check.

Read full answer

Applies tobehavioural / experience questions in ML engineering interviews

What Interviewers Look For

Signal How to show it
Depth with the ecosystem Name tools, versions and why you picked them
Open-source etiquette Issues with minimal reproductions, following contribution guides, tests, code review
Debugging ability Tracing a bug into library code
Judgement Knowing when to use a library and when to write your own
Impact What changed for your team or users

Structure (STAR)

  1. Situation: the project and the problem.

  2. Task: what you needed (a fix, a feature, a faster pipeline).

  3. Action: what you did: read the source, wrote a reproduction, opened an issue or PR, or adopted a tool.

  4. Result: merged PR, workaround, measured improvement, what you learned.

Example Answer Shapes

  • Contribution: “Our mixed-precision training crashed with a specific layer configuration. I reduced it to a 20-line reproduction, found the dtype issue in the library code, opened an issue with the script, and submitted a small PR with a test after a maintainer confirmed the approach. It was reviewed and merged, and we removed our local patch.”

  • Community tools: “We moved our training scripts to PyTorch Lightning to get multi-GPU training, checkpointing and logging without maintaining our own code. I wrote the migration plan, kept a parity test against the old loop’s metrics, and the switch cut our training script by about half.”

  • No open-source contributions yet: “I haven’t contributed to the PyTorch core repository. I use Hugging Face Transformers and PEFT daily, and I’ve reported two bugs with reproductions. Next, I’d like to start with documentation or a good-first-issue in a library I use.”

(Replace the details with your own real experience.)

Ways to Get Real Experience Quickly

  • Fix documentation or examples in a library you use.

  • Pick “good first issue” labels in PyTorch, torchvision, PyG or Lightning.

  • Answer questions on the PyTorch forums.

  • Publish a small, well-tested utility or reproduction of a paper.

Common Pitfalls

  • Overstating a typo fix as “core contributor”.

  • Listing tools without saying what problem they solved.

  • Badmouthing maintainers or projects.

Interview Tips

  • Prepare one contribution story and one tool-adoption story.

  • Be ready for follow-up questions about the code you touched.

48

How do you describe a project where PyTorch played a key role?

Pick one project you know end to end and walk through it in a clear order: the business or research problem and how success was measured, the data, why you chose PyTorch and which model, how you trained and evaluated it (with baselines and numbers), how it was deployed or used, and what you would do differently. The interviewer wants to see judgement and ownership, so emphasize decisions and trade-offs (why this architecture, how you handled imbalance or overfitting, how you served it) rather than listing libraries, and use only real results.

Read full answer

Applies toproject / experience questions in ML interviews

A Structure That Works

Part What to cover Time
Problem Who needed what; the metric that mattered 20–30 s
Data Size, labelling, quality issues, splits 30 s
Approach Baseline, model choice in PyTorch, key design decisions 1 min
Training and evaluation Loss, tricks (augmentation, schedulers, AMP), validation strategy, results vs baseline 1 min
Deployment Export (TorchScript/torch.export/ONNX), serving, latency, monitoring 30 s
Impact and lessons Effect on users or business; what you’d change 30 s

Example Outline (adapt to your own work)

  • Problem: detect defective parts on a production line from camera images; target high recall at a fixed false-alarm rate.

  • Data: several thousand labelled images, few defects; split by production batch to avoid leakage.

  • Approach: started from a pre-trained ResNet in torchvision; fine-tuned with class-weighted loss and strong augmentation; compared with a classical computer-vision baseline.

  • Training: AdamW, cosine schedule, mixed precision; early stopping on validation recall.

  • Deployment: exported to ONNX for the factory’s inference runtime; measured latency on the target hardware; added drift monitoring on image statistics.

  • Lessons: data labelling consistency mattered more than architecture changes.

Points That Impress

  • A baseline and a clear metric tied to the goal.

  • An honest failure and how you fixed it (data leakage, a bad split, overfitting).

  • Concrete PyTorch details: custom Dataset, DataLoader performance, AMP, DDP, export.

  • Your own role, separated from the team’s.

Common Pitfalls

  • Spending most of the time on architecture details nobody asked about.

  • No numbers, or numbers without a baseline.

  • Saying “we” for everything, so your contribution is unclear.

  • Inventing or inflating results.

Interview Tips

  • Prepare a 2-minute version and a 10-minute deep dive.

  • Expect follow-ups on why you made each choice.

49

How do you make PyTorch experiments reproducible?

To make PyTorch experiments reproducible, seed every random source (Python random, NumPy, torch.manual_seed, which also seeds CUDA), make data loading deterministic (seeded DataLoader generator and worker_init_fn), turn on deterministic algorithms (torch.use_deterministic_algorithms(True), torch.backends.cudnn.benchmark = False, plus the CUBLAS_WORKSPACE_CONFIG environment variable on CUDA), and record the environment: code commit, library versions, hardware, configuration and data version. Even then, bit-identical results are only guaranteed on the same hardware, software versions and settings, so report results over several seeds as well.

Read full answer

Applies toPyTorch 2.x

Sources of Non-Determinism

Source Control
Python / NumPy randomness random.seed, np.random.seed (or a seeded Generator)
PyTorch CPU/GPU randomness torch.manual_seed (seeds all devices)
cuDNN algorithm selection torch.backends.cudnn.benchmark = False
Non-deterministic CUDA kernels torch.use_deterministic_algorithms(True); CUBLAS_WORKSPACE_CONFIG=:4096:8
DataLoader shuffling and workers generator=torch.Generator().manual_seed(s), worker_init_fn
Different hardware or library versions Pin versions; record GPU and driver
Data changes Version datasets (DVC, hashes)

Code Example

import os
import random

import numpy as np
import torch
from torch.utils.data import DataLoader, TensorDataset

def set_seed(seed=42, deterministic=True):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)                       # seeds CPU and all CUDA devices
    if deterministic:
        os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"   # set before CUDA work starts
        torch.backends.cudnn.benchmark = False
        torch.use_deterministic_algorithms(True, warn_only=True)

def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    np.random.seed(worker_seed)
    random.seed(worker_seed)

def make_loader(dataset, seed=42):
    g = torch.Generator()
    g.manual_seed(seed)
    return DataLoader(dataset, batch_size=16, shuffle=True, num_workers=0,
                      worker_init_fn=seed_worker, generator=g)

def run(seed=42):
    set_seed(seed)
    ds = TensorDataset(torch.randn(128, 10), torch.randint(0, 2, (128,)))
    model = torch.nn.Linear(10, 2)
    opt = torch.optim.SGD(model.parameters(), lr=0.1)
    for xb, yb in make_loader(ds, seed):
        opt.zero_grad()
        torch.nn.functional.cross_entropy(model(xb), yb).backward()
        opt.step()
    return model.weight.detach().clone()

print(torch.equal(run(42), run(42)))              # True: identical weights
print(torch.equal(run(42), run(7)))               # False: different seed

print({"torch": torch.__version__, "cuda": torch.version.cuda, "numpy": np.__version__})   # log with results

Beyond Seeds

  • Track configs and metrics with tools such as Hydra, MLflow or Weights & Biases.

  • Save checkpoints with optimizer, scheduler and RNG states for exact resumption.

  • Pin dependencies (requirements.txt with versions, lock files, Docker images).

  • Report mean and standard deviation over several seeds.

Common Pitfalls

  • Seeding only PyTorch and not NumPy or Python’s random.

  • Forgetting DataLoader worker seeds.

  • Expecting identical results across GPU models or PyTorch versions.

  • Leaving cudnn.benchmark = True (fast but non-deterministic).

Interview Tips

  • List seeds, deterministic flags, data loading and environment tracking.

  • Note the speed cost of deterministic mode and the limits across hardware.

50

How does PyTorch Lightning simplify the standard PyTorch workflow?

PyTorch Lightning (the lightning package) removes the engineering boilerplate around plain PyTorch: you put the model, the training_step, validation_step and configure_optimizers in a LightningModule, and the Trainer runs the loop, moving data to devices, calling zero_grad / backward / step, switching train and eval modes, and adding logging, checkpointing, early stopping, mixed precision, gradient accumulation and multi-GPU or multi-node training through arguments instead of code. Your research code stays plain PyTorch inside the module, so it is easier to read, share and scale; the trade-off is a layer of abstraction that can make highly unusual training loops harder.

Read full answer

Applies toLightning 2.x (`import lightning as L`), PyTorch 2.x

What Lightning Handles

Concern Plain PyTorch Lightning
Training loop Hand-written Trainer.fit
Device placement .to(device) everywhere Automatic (accelerator="auto")
train/eval mode, no_grad Manual Automatic
Multi-GPU / multi-node DDP setup code devices=4, strategy="ddp"
Mixed precision autocast + GradScaler precision="bf16-mixed"
Checkpointing / early stopping Custom code Callbacks
Logging Manual self.log(...) with TensorBoard, W&B, MLflow loggers
Gradient clipping / accumulation Manual gradient_clip_val, accumulate_grad_batches

Code Example

import lightning as L
import torch
import torch.nn as nn
from lightning.pytorch.callbacks import EarlyStopping, ModelCheckpoint
from torch.utils.data import DataLoader, TensorDataset, random_split

L.seed_everything(42)

class Classifier(L.LightningModule):
    def __init__(self, in_dim=20, hidden=64, n_classes=3, lr=1e-3):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.Sequential(nn.Linear(in_dim, hidden), nn.ReLU(), nn.Linear(hidden, n_classes))
        self.loss_fn = nn.CrossEntropyLoss()

    def forward(self, x):
        return self.net(x)

    def training_step(self, batch, batch_idx):
        x, y = batch
        loss = self.loss_fn(self(x), y)
        self.log("train_loss", loss, prog_bar=True)
        return loss                                     # Lightning calls backward and step

    def validation_step(self, batch, batch_idx):
        x, y = batch
        logits = self(x)
        self.log("val_loss", self.loss_fn(logits, y), prog_bar=True)
        self.log("val_acc", (logits.argmax(1) == y).float().mean(), prog_bar=True)

    def configure_optimizers(self):
        opt = torch.optim.AdamW(self.parameters(), lr=self.hparams.lr)
        sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=20)
        return {"optimizer": opt, "lr_scheduler": sched}

X = torch.randn(2000, 20)
y = (X[:, 0] > 0).long() + (X[:, 1] > 0.5).long()
train_ds, val_ds = random_split(TensorDataset(X, y), [1600, 400], generator=torch.Generator().manual_seed(42))

trainer = L.Trainer(
    max_epochs=20, accelerator="auto", devices=1,
    precision="32-true",                                 # "bf16-mixed" on supported GPUs
    callbacks=[EarlyStopping(monitor="val_loss", patience=3),
               ModelCheckpoint(monitor="val_loss", save_top_k=1)],
    gradient_clip_val=1.0, log_every_n_steps=10, enable_progress_bar=False,
)
trainer.fit(Classifier(), DataLoader(train_ds, batch_size=64, shuffle=True), DataLoader(val_ds, batch_size=256))
print(trainer.callback_metrics)

When to Use It (or Not)

Use Lightning Stay with plain PyTorch
Standard supervised training at scale Very custom loops (some RL, meta-learning)
Teams that want shared conventions Small scripts where the abstraction adds little
Multi-GPU without DDP boilerplate When you need full control of every step

Lightning Fabric is a middle ground: it handles devices, precision and distribution while you keep your own loop.

Common Pitfalls

  • Calling .to(device) or .cuda() manually inside the module (let Lightning place tensors; use self.device when needed).

  • Calling loss.backward() or optimizer.step() yourself under automatic optimization.

  • Forgetting self.save_hyperparameters(), which makes checkpoints harder to reload.

Interview Tips

  • Name the three core methods and the Trainer arguments that replace boilerplate.

  • Mention Fabric for custom loops.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 50 PyTorch answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects