What is machine learning classification?
Classification is a supervised machine-learning task that assigns discrete labels to observations, such as identifying whether an email is spam or whether a tumor is benign or malignant. It learns relationships between input features and possible classes, with a practical trade-off among predictive performance, interpretability, computational cost, and robustness.
Read full answerHide full answer
Applies tomachine learning concepts (all versions); scikit-learn 1.5+ for the named Python APIs
Core Idea
Many classifiers estimate the probability that an observation belongs to each possible class and select the most likely label:
ŷ = argmax_c p(c | x)
Here, x is an observation’s feature vector, c is a candidate class, p(c | x) is the estimated probability of that class, and ŷ is the predicted label. Not every classifier supports probability estimates; some produce only hard labels.
-
Binary classification has two classes, such as fraud versus legitimate transactions.
-
Multiclass classification has three or more mutually exclusive classes.
-
Multilabel classification allows an observation to have multiple labels simultaneously.
Common multiclass approaches include:
-
One-vs-rest: Train one binary classifier for each class.
-
One-vs-one: Train classifiers for class pairs and combine their results.
-
Softmax: Model all mutually exclusive classes jointly and normalize their scores into class probabilities.
Algorithm Comparison
| Algorithm | How it works | Training and prediction cost | Scaling required? | Interpretability | When to use |
|---|---|---|---|---|---|
| Logistic regression | Models class probabilities with a linear decision boundary | Binary training is roughly O(I*n*d) for I optimization passes; multiclass training is roughly O(I*n*c*d); prediction is O(d) for binary or O(c*d) for multiclass |
Yes | High | Linear, interpretable baseline models |
| k-NN | Predicts from nearby labeled observations | Stores training data; brute-force prediction is roughly O(k*n*d) |
Yes | Low | Small datasets with meaningful distance metrics |
| Naive Bayes | Applies conditional independence between features | Training roughly O(n*d); prediction roughly O(c*d) |
Not usually | Moderate | High-dimensional text and spam classification |
| Decision tree | Splits features using rules | Training roughly O(n*d*log(n)); prediction O(depth) |
No | High | Clear rules and mixed feature types |
| Random forest | Combines many randomized decision trees | Roughly O(T*n*d*log(n)) worst-case training; roughly O(T*depth) prediction |
No | Low to moderate | Robust nonlinear tabular models |
| Gradient boosting | Builds trees sequentially to reduce errors | Roughly O(T*n*d*depth) for T shallow trees |
No | Low to moderate | Strong tabular performance; XGBoost and LightGBM are common |
| SVM | Finds a maximum-margin boundary | Training can be expensive; prediction depends on the number of support vectors | Yes | Low to moderate | Small-to-medium clean datasets; kernel variants handle nonlinear data |
| Neural networks | Learns layered nonlinear transformations | Often the highest training cost; prediction depends on network size; GPUs help for large workloads | Yes | Low | Images, text, audio, and complex large-scale patterns |
Here, n is the number of samples, d is the number of features, c is the number of classes, k is the number of neighbors, T is the number of trees, I is the number of optimization passes, and depth is tree depth.
No algorithm is universally best. Linear methods are strong baselines, tree ensembles often perform well on structured data, and neural networks are particularly useful for high-dimensional data or complex feature interactions.
Outputs and Evaluation
predict() normally returns hard class labels, while predict_proba() returns class probabilities when supported. CalibratedClassifierCV can wrap a base classifier and learn a calibration mapping to improve probability reliability; calibration does not necessarily improve ranking metrics.
Useful evaluation measures include:
-
Confusion matrix: Counts correct and incorrect predictions for each class.
-
Precision: The proportion of predicted positives that are actually positive.
-
Recall: The proportion of actual positives that the model detects.
-
F1 score: The harmonic mean of precision and recall.
-
ROC-AUC: Measures ranking performance across decision thresholds.
-
PR-AUC: Focuses on the positive class and is often more informative for severe imbalance.
Accuracy can be misleading when one class dominates. A model that always predicts the majority class may achieve high accuracy while detecting almost no minority-class cases. A binary decision threshold should therefore reflect business costs, such as whether false positives or false negatives are more expensive.
Common Pitfalls
-
Data leakage: Fit scalers, encoders, imputers, feature selection, and resampling only on training folds. Use a
PipelineorColumnTransformer. -
Wrong metric: Avoid relying on accuracy for imbalanced data; consider precision, recall, F1, ROC-AUC, and PR-AUC.
-
Ignoring threshold costs: A default threshold of
0.5may not match the relative cost of false positives and false negatives. -
Untreated imbalance: Consider
class_weight="balanced", carefully validated resampling such as SMOTE applied within the training folds, or threshold tuning. -
Overfitting the validation set: Use cross-validation for model selection and reserve a separate test set for final evaluation.
-
Invalid preprocessing: Scale features for distance-based, margin-based, PCA, and many gradient-based models; scaling is generally unnecessary for tree models.
-
Missing values and reproducibility: Use appropriate imputation and set fixed random seeds where supported.