What is data preprocessing in machine learning?
Data preprocessing is the process of collecting, cleaning, transforming, integrating, reducing, and validating data before it is used to train or evaluate a machine-learning model. It addresses missing values, inconsistent formats, duplicates, outliers, incompatible feature types, and leakage. Its main trade-off is that aggressive transformations can remove useful information or overfit the training data, so every step must be justified and validated.
Read full answerHide full answer
Applies toPython 3.12 with pandas 2.2+ and scikit-learn 1.5+
Core Workflow
A typical preprocessing workflow includes:
-
Collection: Gather representative samples with documented provenance, correct labels, and appropriate sampling.
-
Cleaning: Correct invalid types, standardize formats, remove accidental duplicates, investigate missing values, and distinguish errors from legitimate extreme values.
-
Transformation: Encode categorical variables, scale numerical features when required, aggregate related records, and create derived variables.
-
Integration: Join datasets using validated keys while checking for unmatched records and unexpected row multiplication.
-
Reduction: Remove redundant features, apply feature selection or dimensionality reduction, and discard columns that cannot be justified.
-
Validation: Confirm schemas, ranges, uniqueness rules, class or target distributions, and consistency between training and serving environments.
Missing Values and Outliers
Missingness mechanisms help determine how it should be handled:
-
MCAR — Missing Completely at Random: Missingness is unrelated to observed or unobserved information. Complete-case deletion may be acceptable if the amount is small.
-
MAR — Missing at Random: Missingness depends on observed variables. Imputation can often model it appropriately.
-
MNAR — Missing Not at Random: Missingness depends on information not captured in the data. Domain knowledge, additional data, or sensitivity analysis may be required.
Available approaches include:
-
Dropping rows: Simple, but reduces sample size and can introduce bias.
-
SimpleImputer: Replaces missing values using the mean, median, or most frequent value. -
KNNImputer: Estimates values from similar records; useful for low-missingness data but can be expensive and sensitive to scaling. -
IterativeImputer: Predicts missing values iteratively and remains experimental; enabling it requiresfrom sklearn.experimental import enable_iterative_imputer. -
Missing-indicator features: Record whether a value was absent. These are especially useful when missingness itself is informative.
Outliers should first be checked against business rules and measurement errors. Valid extreme observations should not be removed automatically; transformations such as RobustScaler or winsorization may be more appropriate.
Encoding and Scaling
| Technique | Use | Important consideration |
|---|---|---|
OneHotEncoder |
Nominal categories without an inherent order | Unknown categories should use handle_unknown="ignore" |
OrdinalEncoder |
Ordered categories such as low, medium, high | The ordering must have a defensible business meaning |
TargetEncoder |
High-cardinality categories | Uses cross-fitting and must still be fitted only on training data |
StandardScaler |
Models sensitive to scale, such as k-NN, SVM, PCA, and gradient descent | Centers features around zero with unit variance |
MinMaxScaler |
Algorithms requiring a bounded feature range | Sensitive to outliers |
RobustScaler |
Data containing outliers | Uses median and interquartile range |
Tree-based models generally do not require scaling. Standardization and min-max scaling are approximately O(nd), while one-hot encoding can consume O(n × categories) memory and become impractical for high-cardinality features.
Integration, Reduction, and Validation
Use joins carefully by validating key uniqueness and expected row counts before and after merging. Many-to-many joins can silently multiply records, while missing or mismatched keys can discard useful observations.
Feature selection, PCA, aggregations, and sampling are also fitted transformations. Place them inside the same validation procedure as the model so they are learned separately on each training fold. Evaluate the final pipeline with metrics appropriate to the target: MAE, RMSE, and R² for regression; precision, recall, F1, ROC-AUC, and PR-AUC for imbalanced classification. Accuracy can be misleading when class frequencies are unequal.
Common Pitfalls
-
Data leakage: Fitting imputers, encoders, scalers, or feature selectors on the full dataset exposes validation information to training.
-
Incorrect split order: Split by time, group, or entity before preprocessing when observations are related.
-
Blind outlier deletion: Remove only values shown to be erroneous or invalid.
-
Arbitrary category ordering: Use ordinal encoding only when the category order has meaning.
-
Ignoring schema drift: Validate columns, types, ranges, and categories before batch or production processing.
-
Excessive complexity: Prefer interpretable transformations and document every preprocessing decision.
Interview follow-ups commonly ask which missing-value mechanism you assume, how you prevent leakage, why scaling is unnecessary for trees, and how you would validate the pipeline on unseen data.