What is the Naive Bayes algorithm and how does it work?
The Naive Bayes classifier is a probabilistic algorithm for classification tasks. It applies Bayes’ theorem with a strong “naive” assumption that all features are conditionally independent given the class label. This makes it extremely fast and effective for high-dimensional data like text, but the independence assumption often fails in practice, leading to poorly calibrated probabilities. The main trade-off is between computational efficiency and the accuracy of probability estimates.
Read full answerHide full answer
Applies toscikit-learn 1.5+ (Python 3.12)
How It Works
Bayes’ theorem states that the posterior probability of a class y given features x is:
P(y | x) = P(x | y) P(y) / P(x)
where P(y) is the prior, P(x | y) is the likelihood, and P(x) is the evidence. For classification, we want the class with the highest posterior. Since P(x) is constant for all classes, we can maximize P(y) P(x | y).
The “naive” assumption is that all features x_i are conditionally independent given y. This allows us to factor the likelihood as a product of per-feature probabilities:
P(x | y) = ∏_{i=1}^{n} P(x_i | y)
Thus the decision rule becomes: choose y that maximizes P(y) ∏ P(x_i | y). In practice, we compute in log space to avoid underflow:
log P(y | x) ∝ log P(y) + ∑ log P(x_i | y)
The class with the highest log-score is the prediction.
Variants
| Variant | Feature Type | Use Case | Strengths | Weaknesses |
|---|---|---|---|---|
| GaussianNB | Continuous (real numbers) | General continuous data, e.g. iris dataset | Simple, no need for scaling | Assumes Gaussian distribution per feature |
| MultinomialNB | Counts (non-negative integers) | Text classification with word counts | Fast, works well with high-dimensional sparse data | Assumes multinomial distribution; not ideal for continuous features |
| BernoulliNB | Binary features (0/1) | Text with binary word presence | Simple, fast | Loses information from counts |
| ComplementNB | Counts | Imbalanced text classification | Handles class imbalance better than MultinomialNB | Slightly more complex |
| CategoricalNB | Categorical (nominal) | Features with discrete categories | Handles categorical variables | Requires encoding |
Smoothing and Log Space
Laplace/additive smoothing adds a small constant α (alpha) to every count to avoid zero probabilities when a feature value never appears in a class. In scikit-learn, the alpha parameter defaults to 1.0 for MultinomialNB and BernoulliNB. GaussianNB uses a var_smoothing parameter (default 1e-9) to smooth variance estimates. Setting alpha=0 disables smoothing and can cause underflow if a test feature never seen in training. Computing in log space prevents numerical underflow when multiplying many small probabilities.
Code Example
import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB, GaussianNB
from sklearn.pipeline import Pipeline
# Example 1: Text classification with MultinomialNB
texts = [
"free money now",
"hello friend",
"win a prize",
"let's meet for lunch",
"earn cash today",
"nice to see you"
]
labels = [1, 0, 1, 0, 1, 0] # 1=spam, 0=ham
X_train_text, X_test_text, y_train_text, y_test_text = train_test_split(
texts, labels, test_size=0.33, random_state=42
)
text_clf = Pipeline([
('vect', CountVectorizer()),
('clf', MultinomialNB())
])
text_clf.fit(X_train_text, y_train_text)
print("Text example - predicted:", text_clf.predict(X_test_text))
print("Text example - probabilities:\n", text_clf.predict_proba(X_test_text))
# Example 2: GaussianNB on iris
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
gnb = GaussianNB()
gnb.fit(X_train, y_train)
print("Iris example - predicted:", gnb.predict(X_test))
print("Iris example - probabilities:\n", gnb.predict_proba(X_test))
Strengths and Weaknesses
-
Strengths: Fast training and prediction (O(n * d)), works well with little data, handles high-dimensional sparse data (text), simple to implement, robust to noise.
-
Weaknesses: Naive assumption often unrealistic, features are rarely independent; probabilities are poorly calibrated (not reliable); sensitive to the choice of smoothing parameter; cannot learn feature interactions; performance can be worse than more complex models when dependencies are strong.
Common Pitfalls
-
Data leakage: preprocessing (like vectorization) must be fit on training data only; using
Pipelineprevents this. -
Wrong variant: using GaussianNB for count data or MultinomialNB for continuous data without scaling.
-
Ignoring smoothing: setting
alpha=0can cause zero probabilities and underflow. -
Over-relying on probabilities: Naive Bayes probabilities are not well-calibrated; use
predictfor classification, not thresholding onpredict_probaunless calibrated (e.g., with Platt scaling). -
Not handling correlated features: the model double-counts correlated features, which can degrade performance.
-
Using accuracy on imbalanced data: use precision, recall, F1, or ROC-AUC instead.