What is transfer learning and how does it differ from traditional machine learning?
Transfer learning reuses knowledge from a model trained on a large source task, such as ImageNet images or a huge text corpus, to solve a related target task with much less data. Traditional machine learning trains each model from scratch on data for one task only. Transfer learning works because early layers learn general features (edges, textures, word meanings) that are useful across tasks; its trade-off is the risk of negative transfer when the source and target are too different.
Read full answerHide full answer
Applies todeep learning concepts (examples in PyTorch 2.x with torchvision 0.19+, Python 3.12)
Key Differences
| Aspect | Traditional machine learning | Transfer learning |
|---|---|---|
| Starting point | Random initialization | Weights from a pretrained model |
| Data needed | Large labelled dataset per task | Small target dataset can be enough |
| Training time | Long | Short (often only a new head is trained) |
| Assumption | Train and test come from the same distribution and task | Source and target differ but share structure |
| Knowledge reuse | None | Features or weights reused across tasks |
Main Strategies
| Strategy | What is trained | When to use |
|---|---|---|
| Feature extraction | Only a new head; backbone frozen | Small data, similar domain |
| Fine-tuning | Some or all layers with a small learning rate | More data or a less similar domain |
| Gradual unfreezing | Head first, then deeper layers step by step | Avoid destroying pretrained features |
| Parameter-efficient fine-tuning | Small adapters or LoRA matrices | Large language or vision models |
Code Example
import torch
from torch import nn
from torchvision import models
# Downloads ImageNet weights on first use
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)
for p in model.parameters(): # freeze the pretrained backbone
p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, 5) # new head for 5 target classes (trainable)
optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)
x, y = torch.randn(8, 3, 224, 224), torch.randint(0, 5, (8,)) # stand-in batch
loss = nn.functional.cross_entropy(model(x), y)
loss.backward()
optimizer.step()
print("trainable parameters:", sum(p.numel() for p in model.parameters() if p.requires_grad)) # 2565
Practical Details
-
Use the preprocessing the pretrained model expects (input size, normalization), e.g.
models.ResNet18_Weights.DEFAULT.transforms(). -
Use a lower learning rate for pretrained layers than for the new head.
-
Check the license of the pretrained weights before commercial use.
Common Pitfalls
-
Negative transfer: source knowledge hurts when domains are very different.
-
Overfitting: unfreezing everything on a tiny dataset.
-
Catastrophic forgetting: a high learning rate erases useful pretrained features.