How do Large Language Models work?
A Large Language Model (LLM) is a neural network trained to predict text tokens, enabling it to generate responses, answer questions, summarize information, and perform other language-based tasks. Most modern decoder-only systems use a Transformer architecture and learn from large datasets through next-token prediction. Their main trade-off is that greater capability and context capacity usually require more compute, memory, latency, and data than smaller models.
Read full answerHide full answer
Applies tomachine learning concepts (all versions)
Core Mechanism
-
Tokenization: A tokenizer, often using a subword algorithm such as Byte Pair Encoding (BPE), converts text into token IDs. Common words may become single tokens, while unusual words can be split into smaller subword tokens.
-
Embeddings: Each token ID is mapped to a learned dense vector. Because attention alone does not inherently encode token order, positional information is also incorporated.
-
Transformer blocks: Each block combines causal self-attention, multi-head attention, nonlinear feed-forward layers, residual connections, and normalization. Multi-head attention lets different heads capture different relationships, while residual connections and normalization stabilize training.
-
Output prediction: The final hidden state is projected into probabilities for the next token:
P(w_t | w_1, ..., w_(t-1)) = softmax(W * h_t)
Here, P denotes conditional probability, w_t is the next token, w_1, ..., w_(t-1) is the preceding context, h_t is the final hidden representation, W is the output projection matrix, and softmax converts scores into probabilities.
Attention is commonly calculated as:
Attention(Q, K, V) = softmax((Q * K^T) / sqrt(d_k)) * V
Q, K, and V are the query, key, and value matrices. The product QK^T produces dot-product similarity scores, K^T is the transposed key matrix, d_k is the key dimension, and V contains the values that are aggregated.
Training and Adaptation
LLMs typically pass through three broad training stages:
-
Pretraining: The model learns next-token prediction on a large corpus.
-
Supervised instruction fine-tuning: The model is trained on instruction-response demonstrations to follow user requests.
-
Preference tuning: Human or AI-generated preferences are used to rank responses. RLHF combines reward modeling with reinforcement learning, while DPO directly optimizes a preference objective without the full RLHF pipeline.
| Method | How it works | Strengths | Weaknesses | Best fit and typical scale |
|---|---|---|---|---|
| Prompting | Supplies instructions in the context | No training required | Prompt-sensitive; consumes context | General tasks; inference-time cost only |
| Few-shot prompting | Provides examples of the desired pattern | No weight updates | Uses context and depends on example quality | Formatting and simple task guidance |
| RAG | Retrieves external documents and includes them in the prompt | Adds private or changing knowledge | Requires retrieval quality and adds latency | Knowledge-intensive systems with an external index |
| Full fine-tuning | Updates all model weights | Maximum training flexibility | High compute and memory cost; overfitting risk | Specialized domains with sufficient data |
| LoRA | Trains low-rank adapters while freezing the base model | Parameter-efficient training | Restricted adapter capacity | Domain or task adaptation on limited hardware |
| QLoRA | Trains adapters over a quantized base model | Very low training-memory use | Adds quantization complexity and possible quality trade-offs | Fine-tuning with limited GPU memory |
RAG supplements a model without retraining its generator. It can nevertheless introduce prompt-injection risks because retrieved content may contain malicious instructions.
Inference and Efficiency
During autoregressive generation, the model produces one token at a time and appends it to the context. The context window is the maximum sequence length the model can condition on, including both prompt and generated tokens.
-
Greedy decoding selects the highest-probability token; it is deterministic but can be repetitive.
-
Temperature sharpens or flattens the probability distribution. Lower values favor predictable output.
-
Top-k restricts selection to the
kmost probable tokens. -
Top-p restricts selection to the smallest token set whose combined probability reaches
p.
A KV cache stores previously computed attention keys and values, avoiding recomputation for earlier tokens. Compute still grows with the number of processed tokens: prompt processing becomes more expensive as prompt length grows, and each decoding step must account for the cached context.
Quantization represents weights, and sometimes activations or KV-cache values, in lower precision to reduce memory use. Distillation trains a smaller model to imitate a larger one, improving efficiency at some cost to capability.
Limitations
-
Hallucination: Models may generate fluent but factually incorrect information.
-
Knowledge cutoff: Learned information can become outdated unless updated or supplemented through RAG.
-
Bias: Training data and alignment processes can reproduce societal stereotypes.
-
Prompt injection: Malicious instructions in user or retrieved text may manipulate behavior.
-
Reasoning errors: Plausible-looking reasoning can still be logically flawed.
-
Evaluation difficulty: Benchmarks may not reflect real-world quality, and automatic metrics may disagree with human preferences.
Interview Tips
Distinguish pretraining from instruction tuning and preference tuning, and explain why RAG and fine-tuning solve different problems. Common mistakes are treating an LLM as an authoritative database, overlooking evaluation leakage, using inappropriate metrics for subjective tasks, and assuming fluent output is necessarily correct.