How does LLM fine-tuning differ from pretraining?
Fine-tuning continues training an already pretrained LLM on targeted data to improve particular behavior, domain terminology, or output structure. Pretraining builds broad language capabilities from a large corpus, usually through self-supervised next-token prediction.
Read full answerHide full answer
Applies toLLM fine-tuning concepts (provider-agnostic)
Core Difference
| Aspect | Pretraining | Fine-tuning |
|---|---|---|
| Starting point | Model initialization | Pretrained checkpoint |
| Data | Very large, broad corpora | Smaller, curated task or domain data |
| Main objective | Learn general language patterns | Specialize behavior, knowledge access, or preferences |
| Supervision | Usually self-supervised | Examples, demonstrations, corrections, or preference pairs |
| Compute | Expensive, distributed training | Usually cheaper, but still potentially GPU-intensive |
| Result | Broad base capabilities | More consistent task performance, style, or format |
Pretraining is analogous to learning general language and reasoning, while fine-tuning is analogous to learning a particular job’s conventions. Continued pretraining, which trains on domain text, is another option when new terminology or distributions are the main problem; it is different from ordinary task fine-tuning.
Main Fine-Tuning Approaches
-
Supervised fine-tuning (SFT): Train on prompt-completion examples, commonly masking prompt tokens from the loss so the model learns to generate only the assistant response.
-
Instruction tuning: SFT on diverse instructions and responses to improve general instruction following.
-
Parameter-efficient fine-tuning: LoRA trains low-rank adapters while freezing the base model; QLoRA additionally stores frozen base weights in quantized form, commonly 4-bit. Separate adapter modules are another PEFT option.
-
Preference optimization: DPO trains directly on preferred and rejected responses. Classic RLHF trains a reward model and then optimizes the policy against that reward, adding pipeline complexity.
-
Full-parameter tuning: Updates all model weights and generally requires more compute and memory than PEFT, but can provide greater capacity for substantial domain shifts.
Managed tuning is available from providers such as OpenAI and Google; Anthropic does not currently expose a generally available fine-tuning API comparable to those services. Meta’s Llama, Mistral, and Qwen are common open-weight bases for locally managed tuning.
Data and Training Workflow
-
Define the behavior and baseline: Identify measurable failures and compare the candidate against strong prompting and retrieval-augmented generation (RAG) baselines.
-
Prepare data: Use representative, high-quality examples with correct chat roles, diverse cases, deduplication, and separation of training, validation, and test data. Check for prompt leakage and near-duplicates.
-
Choose the objective: Use SFT for demonstrations, DPO for reliable pairwise preferences, and continued pretraining for domain-language adaptation. Do not assume that SFT alone can reliably teach frequently changing facts.
-
Train conservatively: Start with PEFT and a small learning rate, commonly around
1e-4to5e-4, while selecting sequence length and effective batch size from available GPU memory. Use validation loss, task metrics, checkpoint selection, and early stopping rather than training solely for a fixed number of epochs. -
Evaluate independently: Compare against the original checkpoint and prompted baseline using correctness, exact-match or rubric-based quality, refusal accuracy, bias, hallucination rate, and adversarial test sets.
Memory, Cost, and Serving Trade-Offs
For a hypothetical fully trainable 7B model in a mixed-precision layout:
-
BF16 weights:
7B × 2 bytes = 14 GB -
BF16 gradients:
7B × 2 bytes = 14 GB -
FP32 master weights:
7B × 4 bytes = 28 GB -
Two FP32 Adam moments:
2 × 7B × 4 bytes = 56 GB -
Subtotal:
14 + 14 + 28 + 56 = 112 GB, before activations, temporary buffers, and framework overhead
Actual frameworks may avoid some copies or shard parameters and optimizer states. With QLoRA, the frozen 4-bit weights occupy approximately 7B × 0.5 bytes = 3.5 GB, plus quantization metadata. For a Llama-7B-shaped model with rank-8 LoRA on q_proj and v_proj in 32 layers, the adapter count is:
2 projections × 8 × (4096 + 4096) × 32 = 4,194,304 parameters
If those parameters and gradients are FP32, they occupy about 16.8 MB each; two FP32 Adam moments occupy about 33.6 MB. Activations and workspace can still dominate memory, especially with long sequences.
Fine-tuning generally costs less than pretraining and may reduce prompt tokens, inconsistent formatting, and repeated instructions. It does not guarantee lower latency: longer generations or unmerged adapter serving can add overhead. Merge adapters for efficient conventional serving when appropriate, while retaining the ability to roll back. Prefer RAG for private or rapidly changing knowledge; use fine-tuning for repeatable, behavior-specific improvements.
Reliability and Best Practices
-
Do not fine-tune unnecessarily: Prompting is cheaper for simple instructions, and RAG is usually better for dynamic facts.
-
Protect data: Remove or transform PII, verify ownership and consent, restrict access, and review provider retention and training policies.
-
Treat fine-tuning as behavior shaping, not a security boundary: Prompt injection, retrieval poisoning, tool misuse, and unsafe outputs still require isolation, least privilege, input validation, and output controls.
-
Measure reliability: Test hallucination, refusal behavior, bias, prompt-injection resistance, and out-of-distribution cases, not only aggregate accuracy.
-
Monitor deployment: Track quality, safety failures, latency, cost, and data drift; retain evaluation gates and rollback procedures.