What is probability and how is it used in machine learning?
Probability is the mathematics of uncertainty: it assigns each event a number between 0 and 1 that follows three axioms (probabilities are non-negative, the whole sample space has probability 1, and the probabilities of mutually exclusive events add up), and it can be read as a long-run frequency or as a degree of belief. Machine learning uses it everywhere: data are modelled as samples from probability distributions, models are trained by maximizing likelihood (cross-entropy and mean squared error are negative log-likelihoods), classifiers output class probabilities, Bayesian methods combine priors with data, and evaluation, calibration, A/B testing and uncertainty estimates all rest on probability theory.
Read full answerHide full answer
Applies togeneral concepts; examples in Python 3.12 with NumPy 2.x and SciPy 1.13+
Where Probability Appears in ML
| Area | Example |
|---|---|
| Data modelling | Features and noise as random variables; Gaussian noise in regression |
| Training objectives | Maximum likelihood: log loss for classification, MSE for Gaussian regression |
| Model outputs | predict_proba, softmax probabilities |
| Bayesian methods | Priors, posteriors, Naive Bayes, Bayesian optimization |
| Generative models | VAEs, diffusion models, language models predicting next-token probabilities |
| Evaluation | Confidence intervals, hypothesis tests, calibration |
| Decision making | Expected cost, thresholds, reinforcement learning |
Code Example: Probability Behind a Classifier
import numpy as np
from scipy import stats
from sklearn.calibration import calibration_curve
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(42)
p_exact = 1 - stats.binom.cdf(5, 10, 0.5) # P(more than 5 heads in 10 fair flips)
p_sim = np.mean(rng.binomial(10, 0.5, 100_000) > 5)
print(f"exact {p_exact:.4f}, simulated {p_sim:.4f}") # exact 0.3770 (386/1024)
X, y = make_classification(n_samples=4000, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)
proba = LogisticRegression(max_iter=1000).fit(X_tr, y_tr).predict_proba(X_te)[:, 1]
print(f"log loss (negative log-likelihood per sample): {log_loss(y_te, proba):.3f}")
frac_pos, mean_pred = calibration_curve(y_te, proba, n_bins=5)
print(np.c_[mean_pred.round(2), frac_pos.round(2)]) # predicted vs observed rates: close if calibrated
P(more than 5 heads) = Σ from k = 6 to 10 of C(10, k)/2¹⁰ = (210 + 120 + 45 + 10 + 1)/1024 = 386/1024 ≈ 0.377.
Common Pitfalls
-
Treating model scores as probabilities without checking calibration.
-
Confusing P(A | B) with P(B | A) (the base-rate fallacy).
-
Assuming independence where it doesn’t hold.
Interview Tips
-
Give the axioms briefly, then several concrete ML uses.
-
Connect loss functions to likelihood.