Skip to content
Machine Learning & Data Science

Feature Engineering interview questions and answers

50 questions, 25 with code. About 5 h 6 min to read them all.

By TestLab · Updated

Plan for an interview
50 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

How does feature engineering affect machine learning model performance?

Feature engineering is the process of transforming raw data into input variables that expose useful signal to a learning algorithm. It affects model performance by making patterns easier to learn from attributes that would otherwise be noisy, poorly scaled, or misaligned with the model’s assumptions, and for tabular data it often improves performance more than switching algorithms. Its main trade-off is effort and risk: well-designed features can add strong signal, but poorly designed ones add complexity and can leak information from the target.

Read full answer

Applies tomachine learning concepts (all versions)

How It Affects Performance

  • Signal quality: Features that capture domain relationships (ratios, interactions, counts, recency) let the model separate classes or predict continuous targets with less noise.

  • Model fit: Appropriate representations reduce the burden on the model. For example, cyclic sin/cos encoding helps linear models treat December as close to January, while raw month numbers do not.

  • Generalization: Good features can improve validation performance, but overly specific or target-derived features can inflate training results and fail in production.

  • Algorithm sensitivity: Distance-based and gradient-based methods (k-NN, SVM, PCA, linear models with gradient descent) usually benefit from scaling; tree-based models are often insensitive to monotonic scaling but still benefit from informative derived features.

Techniques by Data Type

  • Numeric: StandardScaler or MinMaxScaler where required, log/Box-Cox/Yeo-Johnson transforms for skew, binning for non-linear thresholds, ratios, interactions, and polynomial features.

  • Categorical: one-hot encoding for low cardinality, ordinal encoding when order is meaningful, target encoding with cross-fitting to avoid leakage, frequency encoding, or hashing for very high cardinality.

  • Dates and times: extract parts (month, weekday, hour), use cyclic sin/cos encoding, or compute time since an event.

  • Text: TF-IDF vectors for sparse linear models or pretrained embeddings for semantic similarity.

  • Entity aggregations: counts, means, or rolling-window statistics per user or item, computed only from information available before prediction time.

Feature Selection and Reuse

Method How it works Strengths Weaknesses When to use
Filter (mutual information, correlation) Scores each feature against the target Fast, model-agnostic Ignores feature interactions Quick baseline, high-dimensional data
Wrapper (RFE) Trains a model repeatedly on subsets Captures model-specific value Expensive; cost grows with features and model training cost Small/medium feature sets
Embedded (L1, tree importances, SelectFromModel) Selection happens during training Balanced cost and quality Tied to the chosen model Common default for tabular models

Feature stores such as Feast help reuse the same feature definitions consistently between training and serving.

Common Pitfalls

  • Data leakage: computing target encodings, group aggregates, or scaling on the full dataset uses future or target information. Fit every data-dependent step on training folds only, use Pipeline/ColumnTransformer, and respect time order with TimeSeriesSplit.

  • Wrong metric: accuracy can mislead on imbalanced classes; prefer precision, recall, F1, ROC-AUC, or PR-AUC for classification and MAE, RMSE, or R^2 for regression.

  • Missing scaling or poor missing-value handling: scale when the algorithm requires it and impute inside the training pipeline.

  • Overfitting to validation choices: use cross-validation (for example, StratifiedKFold for classification) and a fixed random_state for reproducibility.

  • Interview follow-ups: “How do you prevent target leakage?”, “When would a tree model not need scaling?”, and “How would you engineer features for a time series?”

2

What feature types are commonly used in machine learning?

Feature engineering converts raw observations into model inputs that expose useful predictive signal. Common feature types include numeric, categorical, temporal, textual, aggregate, and interaction features, and their usefulness often matters more than the choice of algorithm for tabular data. The main trade-off is that richer features can improve performance but also increase complexity, computation, overfitting risk, and data-leakage risk.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Common Feature Types

Feature type Description Example
Raw numeric A directly measured continuous value Age, price, temperature
Binary A value representing two states is_active, has_churned
Count The number of events or items Purchases in the last 30 days
Categorical A label describing a group or category Product type, country
Ordinal A category with a meaningful order Education level, satisfaction rating
Temporal A date, timestamp, or duration derived from time Sign-up date, days since purchase
Aggregated A statistic calculated across related records Customer average order value
Interaction A feature combining two or more variables Price × discount
Text-derived A numerical representation of unstructured text TF-IDF values or embeddings
Target-derived A feature calculated using the target variable Cross-fitted target encoding

Techniques by Data Type

  • Numeric features:

  • Scaling: Standardization or min-max scaling is generally required for distance-based models such as k-NN and SVM, as well as PCA and gradient-descent-based models. Tree-based models usually do not require scaling.

  • Distribution transformations: log reduces right skew; Box-Cox seeks approximately normal, positive data; Yeo-Johnson can handle zero and negative values.

  • Binning: Converts continuous values into intervals. It can make relationships easier to model but may discard useful information.

  • Interactions and polynomials: Capture relationships such as combined effects that individual features cannot represent.

  • Categorical features:

  • One-hot encoding creates one binary column per category and works well for low-cardinality data.

  • Ordinal encoding represents ordered categories as integers, but should be used only when the order is meaningful.

  • Frequency encoding replaces a category with its occurrence count or proportion.

  • Target encoding replaces a category with an aggregate of the target. It must be computed using cross-fitting to avoid leakage.

  • Hashing maps categories to a fixed number of dimensions, providing a memory-bounded alternative for high-cardinality data.

  • Date and time features:

  • Extract year, month, day, weekday, hour, and holiday indicators.

  • Use cyclical sin and cos features for naturally repeating periods such as hours or months.

  • Calculate time since an event, such as days since the last purchase.

  • Respect chronology when splitting data; random cross-validation can leak future information into training.

  • Text features:

  • TF-IDF emphasizes terms that are frequent in one document but uncommon across the corpus.

  • Embeddings represent text as dense vectors that can capture semantic similarity.

  • Additional features may include document length, language, named entities, or domain-specific indicators.

  • Entity-level aggregates:

  • Counts, means, sums, standard deviations, recency, and trends can be calculated for each customer, account, device, or product.

  • Rolling windows are useful for behavior over time, such as purchases in the previous 7, 30, or 90 days.

Feature Selection

Feature engineering creates candidate features; feature selection chooses a useful subset.

  • Filter methods: Rank features independently of a model using correlation, mutual information, or statistical tests. They are fast but may miss interactions.

  • Wrapper methods: Search for a subset by repeatedly training and evaluating a model. Recursive Feature Elimination (RFE) is a common example, but it can be computationally expensive.

  • Embedded methods: Use model training to select features. L1 regularization can drive coefficients to zero, while tree-based models provide feature importances.

  • SelectFromModel: Retains features based on their importance or coefficient magnitude from a fitted estimator.

The selection procedure must be fitted inside each training fold during cross-validation so that information from validation data never influences the selected features.

Best Practices

  • Define every feature using only information that would genuinely be available at prediction time.

  • Fit scalers, encoders, imputers, target encoders, feature selectors, and aggregations on training data only. Use Pipeline and ColumnTransformer to enforce this separation.

  • For time-dependent data, use chronological splits or TimeSeriesSplit rather than random folds.

  • Use StratifiedKFold for imbalanced classification and compare models using suitable metrics such as precision, recall, F1, ROC-AUC, or PR-AUC.

  • Monitor missing values, rare categories, outliers, and distribution shifts between training and production.

  • Version feature definitions so training and serving compute identical values. Feature stores such as Feast help organize reusable, consistently defined features.

  • Validate whether added features genuinely improve out-of-sample performance rather than merely increasing training accuracy.

Common Pitfalls

  • Computing target encodings or aggregates on the full dataset before splitting causes data leakage.

  • Applying a scaler before splitting leaks validation-set statistics into training.

  • Using random splits for future-dependent data allows information from the future to influence the past.

  • Creating too many correlated or highly granular features increases memory use and overfitting risk.

  • Assuming feature importance proves causality is incorrect; importance only describes model behavior.

  • Selecting features or tuning hyperparameters repeatedly against the same validation set eventually overfits that set.

Interview Tips: Interviewers often ask how you prevent leakage, when scaling is necessary, how high-cardinality categories are handled, how temporal features are validated, and how you distinguish a useful feature from a spurious correlation.

3

What is the difference between feature selection and feature extraction?

Feature selection chooses a subset of the original input columns, while feature extraction creates a transformed representation whose features may not correspond directly to the original columns. Both are forms of feature engineering, which turns raw data into model inputs that expose useful signal and can matter more than algorithm choice for tabular data. The main trade-off is that selection usually preserves interpretability, whereas extraction can reveal hidden structure but may also reduce interpretability or increase dimensionality.

Read full answer

Applies tomachine learning concepts; scikit-learn 1.5+ (Python 3.12) for the named APIs

Core Difference

Aspect Feature selection Feature extraction
Output A subset of existing columns Newly constructed or transformed features
Dimensionality Usually stays the same or decreases May increase, remain similar, or decrease
Original meaning Retained features keep their original meaning Extracted features may be abstract or less interpretable
Methods Correlation, mutual information, RFE, L1 regularization, tree importances PCA, embeddings, TF-IDF, aggregation, learned representations
Typical cost Filters are inexpensive; wrappers repeatedly train models Ranges from simple transformations to expensive model or matrix computations
Best use Removing irrelevant, redundant, or noisy columns Encoding complex data or reducing dimensionality

For example, selecting age, income, and region from a customer table is feature selection. Creating income_per_month, account_age_days, or a one-hot representation of region is feature extraction because the model receives newly constructed inputs.

Feature Selection Methods

  • Filter methods: Rank features using correlation, mutual information, or statistical tests. They are fast and model-independent, but univariate methods may miss useful interactions.

  • Wrapper methods: Evaluate subsets by repeatedly training a model, as with recursive feature elimination (RFE). They can capture interactions but are computationally expensive and may overfit the selection process.

  • Embedded methods: Use a model to identify useful features, such as L1-regularized linear models or tree-based feature importances.

  • SelectFromModel: A scikit-learn utility that uses model coefficients, feature importances, or another model-specific criterion to retain features according to a threshold.

Feature selection should occur within each cross-validation training fold. Selecting features before splitting the data can leak validation or test information and produce overly optimistic results.

Extraction Techniques by Data Type

  • Numeric data: Scaling, log or Yeo-Johnson transformations, binning, ratios, interactions, and polynomial features can expose nonlinear or multiplicative relationships. Scaling is important for distance- or margin-based methods such as k-NN, SVM, PCA, and gradient-descent models, but is generally unnecessary for decision trees.

  • Categorical data: One-hot encoding represents categories independently. Ordinal encoding is appropriate only when categories have a genuine order; otherwise, it imposes a false order. Frequency encoding records category prevalence, while target encoding must be leakage-safe.

  • Dates and times: Extract parts such as hour, weekday, or month, encode periodic values with sine and cosine, and calculate elapsed time since relevant events.

  • Text: TF-IDF highlights important words, while pretrained embeddings represent semantic relationships and are useful for similarity or context-sensitive tasks.

  • Entity histories: Counts, means, trends, and rolling-window aggregates summarize behavior over time using only information available at the prediction timestamp.

  • Dimensionality reduction: PCA creates uncorrelated principal components, while autoencoders learn compact nonlinear representations. PCA is generally faster and easier to inspect; autoencoders need representative training data and more computation.

Combining Selection and Extraction

The techniques are complementary rather than mutually exclusive. A common workflow is to engineer domain features, extract a suitable representation, and then select or regularize the resulting inputs.

For mixed tabular data, a scikit-learn Pipeline can contain a ColumnTransformer with separate branches, such as StandardScaler for numeric columns and OneHotEncoder for categorical columns. Feature selection can follow that transformer. Fitting these steps separately within each training fold prevents preprocessing and selection information from leaking into validation folds.

Leakage and Operational Concerns

  • Target encoding: Compute category-to-target mappings from the current training fold only. Use cross-fitting when creating training features, and apply the training-fold mapping to validation and test data.

  • Historical aggregates: Respect the prediction timestamp. Randomly splitting time-series data can leak future information; use chronological splits or TimeSeriesSplit.

  • Preprocessing leakage: Fit scalers, imputers, encoders, transformations, and selectors only on each training fold.

  • Missing values: Impute within the pipeline and add missing-value indicators when absence itself may carry information.

  • Overfitting: Aggressive selection or extraction can discard minority-class signals or create unstable representations, especially with small datasets.

  • Imbalanced classes: Evaluate the downstream model with recall, F1, PR-AUC, or ROC-AUC rather than accuracy alone. Use stratified cross-validation such as StratifiedKFold for classification.

  • Reproducibility: Fix random seeds and record the feature list, preprocessing configuration, and model version.

A feature store such as Feast can help teams define, version, and reuse consistent features during training and serving. This can reduce training-serving skew, although temporal correctness and feature freshness still require validation.

Common Pitfalls and Interview Tips

  • Do not describe feature extraction as merely “feature selection with more columns”; extraction creates a new representation.

  • Do not select features using the full dataset before cross-validation.

  • Remember that one-hot encoding can substantially increase dimensionality for high-cardinality categories.

  • Match the transformation to the data: log requires positive values, while Yeo-Johnson can handle zero and negative values.

  • A strong interview example is distinguishing raw transactions from aggregate customer features, then selecting those aggregates with a leakage-safe pipeline.

4

What challenges arise during feature engineering?

Feature engineering transforms raw observations into model inputs that expose useful predictive signal. For tabular data, it often matters more than the choice of algorithm, but richer features also increase complexity, leakage and overfitting risks, and maintenance costs.

Read full answer

Applies tomachine learning concepts; pandas 2.2+ and scikit-learn 1.5+ where named

Common Challenges

  • Data leakage: A feature must contain only information available when the prediction is made. Target encoding is especially dangerous if it uses target values from the row being predicted, while aggregates can leak future observations into historical rows.

  • Temporal validity: Lagged and expanding features must respect event order. Centered windows, random train-validation splits, or aggregates containing later events can produce look-ahead bias.

  • Missing values: Missingness may be random, informative, or caused by collection problems. Impute values, add missingness indicators, or use models that natively handle missing data.

  • Outliers and skew: Heavy-tailed numeric variables can destabilize scaling and linear models. Clipping, log transformations, or robust scaling may help, but thresholds and fitted transformations must come only from training data.

  • High cardinality and unseen categories: One-hot encoding many categories produces sparse representations, while production may contain categories absent from training. Frequency encoding, hashing, cross-fitted target encoding, and documented fallbacks can help.

  • Overfitting and validation reuse: Trying many transformations against one validation set can indirectly overfit it. Prefer nested cross-validation and preserve a final test set.

  • Train-serving inconsistency: Offline and online pipelines must calculate the same values from the same definitions. Feature stores such as Feast can support reusable, versioned feature definitions.

  • Cost and latency: Entity aggregates and text embeddings can be expensive. Most tabular transformations run efficiently on CPUs, whereas embedding generation may benefit from GPUs.

Techniques by Data Type

  • Numeric: Scaling is important for k-NN, SVM, PCA, and gradient-based models but usually unnecessary for trees. A plain log transform requires nonnegative values, Box-Cox requires strictly positive values, and Yeo-Johnson supports zero and negative values. Binning, ratios, interactions, and polynomial features are other options.

  • Categorical: Use one-hot encoding for low-cardinality nominal variables, ordinal encoding only when order is meaningful, and frequency encoding for suitable high-cardinality variables. Compute encodings on training data only; use cross-fitting for target encoding.

  • Dates and times: Extract parts such as month, weekday, quarter, and holiday indicators. Encode cyclical values with sine and cosine so December and January remain adjacent, and calculate time-since-event features where appropriate.

  • Text: TF-IDF represents lexical importance, while pretrained embeddings represent semantic similarity. Embeddings generally require more computation but can share information across related words.

  • Entity aggregations: Counts, means, trends, and windowed statistics summarize customer or device behavior. Use only historical data or data from the current training fold.

Feature Selection

Approach How it works Strengths Weaknesses Typical scale
Filter: mutual information or correlation Ranks features by individual association Simple and inexpensive Misses interactions Correlation is roughly O(n × d); mutual-information cost depends on its estimator
Wrapper: RFE Repeatedly removes features while evaluating a model Captures interactions Expensive and model-dependent Roughly O(M × model-fit cost) for M candidate subsets
Embedded: L1 regularization or tree importance Learns feature importance during training Uses model behavior Importance is model-dependent Usually one or a few model fits
SelectFromModel Retains features using an estimator’s coefficients or importances Flexible and automatable Requires a suitable estimator and threshold One estimator fit, plus any threshold search

Feature selection that learns from data should be fitted inside the training pipeline rather than on the complete dataset.

Best Practices

  • Define each feature’s source, transformation time, availability, and production owner before implementing it.

  • Split data before fitting imputers, scalers, encoders, selectors, or learned aggregations; use Pipeline or ColumnTransformer to enforce this boundary.

  • Use StratifiedKFold for typical classification and TimeSeriesSplit for ordered data.

  • Choose metrics appropriate to the problem. Accuracy can be misleading with imbalanced classes, so also consider precision, recall, F1, ROC-AUC, and PR-AUC.

  • Keep seeds, data versions, feature definitions, and preprocessing parameters reproducible.

  • Monitor feature drift, missingness, schema changes, and offline-versus-online differences after deployment.

Interview Tips

  • Distinguish deterministic transformations from transformations that must be fitted on training data.

  • Explain how target encoding, aggregation, and temporal windows can leak information.

  • Know when scaling is necessary and when tree models do not require it.

  • Expect follow-ups about unseen categories, feature drift, interpretability, feature stores, and training-serving consistency.

5

What is feature normalization and standardization in machine learning?

Feature normalization and standardization are preprocessing techniques that rescale numeric features so they share a comparable range or distribution, helping models that are sensitive to feature magnitudes learn more effectively. They solve the problem of one feature dominating distance calculations or gradient updates because of its units or scale. The main trade-off is that scaling parameters (such as min, max, mean, and standard deviation) must be learned from training data only, adding pipeline complexity to avoid data leakage.

Read full answer

Applies toscikit-learn 1.5+ with pandas 2.2+ and NumPy 2.x (Python 3.12)

How Normalization and Standardization Work

  • Standardization (z-score scaling): z = (x - μ) / σ

  • x: the original feature value.

  • μ (mu): the mean of that feature computed on the training data.

  • σ (sigma): the standard deviation of that feature computed on the training data.

  • Result: the transformed feature has approximately mean 0 and standard deviation 1. It does not bound values to a fixed range.

  • Min-max normalization: x_scaled = (x - x_min) / (x_max - x_min)

  • x: the original feature value.

  • x_min: the minimum value of that feature in the training data.

  • x_max: the maximum value of that feature in the training data.

  • Result: values are rescaled to [0, 1] (or another range if configured). Outliers in new data can push values outside this range.

  • Other numeric transforms (often used alongside scaling): log, Box-Cox (positive data), or Yeo-Johnson (handles zeros/negatives) can reduce skew; binning can turn continuous values into categories; interactions and polynomial features can expose non-linear relationships.

When to Use Each Method

Method How it works Strengths Weaknesses When to use
Standardization (StandardScaler) Centers by mean, scales by standard deviation Works well with outliers compared to min-max; suits gradient-based and distance-based models Output is unbounded k-NN, SVM, PCA, linear/logistic regression with regularization, neural networks
Min-max normalization (MinMaxScaler) Maps training min/max to a fixed interval Preserves zero entries in sparse-friendly settings less directly, but gives bounded range Sensitive to outliers; new extremes exceed bounds Image pixels, features needing a known range, some neural network inputs
Robust scaling (RobustScaler) Uses median and IQR Resistant to outliers Less common default; distribution still not Gaussian Numeric features with heavy outliers

Tree-based models (decision trees, random forests, gradient boosting) are generally invariant to monotonic scaling, so normalization/standardization is usually unnecessary for them.

Common Pitfalls

  • Data leakage: Fitting a scaler on the full dataset before splitting lets test statistics influence training. Always fit on training folds only; Pipeline/ColumnTransformer enforce this in cross-validation.

  • Wrong order with time series: Use TimeSeriesSplit and compute scaling parameters from past data only.

  • Scaling target inappropriately: For regression, scale y only if the model or loss requires it, and inverse-transform predictions; keep classification labels unchanged.

  • Ignoring outliers with min-max: A single extreme value can compress most data into a narrow range; consider RobustScaler or clipping.

  • Assuming scaling helps trees: It rarely changes tree predictions and adds unnecessary steps.

Interview tips: Be ready to explain why distance-based models need scaling (e.g., k-NN prediction is O(n * d) per query and distances are dominated by large-scale features) and to describe how you would prevent leakage in a production feature pipeline or feature store (e.g., Feast) by reusing the same fitted transformation logic for training and serving.

6

Why does domain knowledge matter in feature engineering?

Domain knowledge helps identify which raw variables contain meaningful signal and how they should be transformed into model-ready features. It prevents invalid assumptions about units, relationships, business rules, and prediction-time availability while revealing useful ratios, interactions, and aggregations. Its main trade-off is the time required to develop and validate domain-specific features, along with the risk of overfitting to business assumptions that may later change.

Read full answer

Applies tomachine learning concepts

Why Domain Knowledge Matters

Feature engineering turns raw data into inputs that expose useful patterns to a model. For tabular problems, thoughtful features can improve results more than switching between similar algorithms because many models cannot discover domain relationships automatically.

Domain knowledge helps answer questions such as:

  • What does the measurement mean? Revenue recorded in thousands and revenue recorded in cents require consistent unit handling.

  • Is a relationship meaningful? A debt_to_income ratio may be more informative than debt and income separately.

  • Is the feature available at prediction time? A calculation using future transactions creates target leakage.

  • What population does the model serve? Age restrictions or regional differences may make some features inappropriate.

  • How should categories be represented? Treating an unordered category such as city as an integer imposes a false ordering.

Domain knowledge does not prove causality or guarantee generalization. It guides feature hypotheses that must still be validated out of sample.

Domain-Guided Features by Data Type

Data type Possible techniques Domain questions to consider
Numeric Scaling, log/Box-Cox/Yeo-Johnson transforms, binning, ratios, interactions, polynomial features Are values skewed or heavy-tailed? Is a ratio meaningful? Which interactions reflect a real relationship?
Categorical One-hot encoding, justified ordinal encoding, cross-fitted target encoding, frequency encoding, hashing Does the category have a natural order? Is it high-cardinality? Will the encoding remain stable over time?
Date and time Year/month/day parts, business-hour indicators, sin/cos cyclic encoding, time since an event Does seasonality matter? Should midnight and 23:00 be close rather than numerically distant?
Text TF-IDF, embeddings, domain vocabulary features Which terms or concepts indicate the target outcome? Could embedding similarity conflict with domain semantics?
Entity history Counts, sums, means, rates, and rolling-window aggregates What information was genuinely available at each prediction timestamp?

Scaling is needed for distance- and gradient-based methods such as k-NN, SVM, PCA, and regularized linear models. Tree-based models generally do not require it. Missing-value handling and class imbalance also depend partly on how the data was generated.

Feature Selection and Validation

Domain knowledge can reduce the feature space before or during model training:

  • Filter methods: Mutual information, correlation, or statistical tests identify weak or redundant variables.

  • Wrapper methods: RFE evaluates feature subsets using model performance.

  • Embedded methods: L1 regularization and tree importances identify useful variables during training.

  • SelectFromModel: Applies thresholds based on coefficients, importances, or other model-derived scores.

These methods can select unstable or leakage-prone features, so domain review remains important. Prefer features with a defensible business interpretation, stable distributions, and availability in production.

Production and Team Practices

Document each feature’s definition, owner, calculation time, units, and expected population. Reuse feature definitions through a feature store such as Feast so that offline training data and online serving inputs use consistent logic. Validate feature pipelines against held-out or time-ordered data, and monitor feature drift after deployment.

Common Pitfalls

  • Leakage: Do not calculate target encodings, historical aggregates, selection decisions, or preprocessing statistics using validation or test data.

  • Ignoring time order: Random splits can leak future patterns into training; use chronological splits or TimeSeriesSplit when observations depend on time.

  • Encoding every category: High-cardinality one-hot features increase dimensionality, memory use, and overfitting risk.

  • Using inappropriate ordinal encoding: Integer codes imply an order that may not exist.

  • Treating importance as causality: Model importance measures predictive contribution, not business causality.

  • Overfitting the validation set: Repeated experimentation eventually overfits that set, so reserve a final untouched test set.

Common follow-up questions include how to prevent leakage in target encoding, when to use log versus Yeo-Johnson transformation, how to validate temporal features, and how to distinguish predictive value from causal impact.

7

How does feature scaling affect gradient descent performance?

Feature scaling rescales numeric inputs so their gradients contribute more evenly during gradient descent, reducing the ill-conditioning caused by features with very different magnitudes. It usually improves convergence speed and makes the learning rate less sensitive to feature scale, but it adds preprocessing and can change how coefficients and regularization penalties are interpreted. Feature engineering more broadly turns raw data into model inputs that expose useful signal, and it often matters more than the choice of algorithm for tabular data.

Read full answer

Applies toscikit-learn 1.5+ and pandas 2.2+ (Python 3.12)

Why Scaling Changes Convergence

For linear regression, batch gradient descent commonly uses:

θ(t+1) = θ(t) - η × ∇J(θ(t))

J(θ) = (1 / (2n)) × Σᵢ(yᵢ - θᵀxᵢ)²

Here, θ is the vector of model parameters, η is the learning rate, n is the number of training examples, yᵢ is the observed target for example i, xᵢ is its feature vector, and ∇J is the gradient of the loss, which indicates how to change the parameters to reduce the loss.

If one feature ranges from 0 to 1,000,000 while another ranges from 0 to 10, the corresponding gradient components can have very different magnitudes. The optimizer may therefore take steps that are too aggressive for one feature and too conservative for another, causing oscillation or slow convergence.

A common standardization is:

zⱼ = (xⱼ - μⱼ) / σⱼ

For feature j, xⱼ is the raw value, μⱼ is its training-set mean, σⱼ is its training-set standard deviation, and zⱼ is the scaled value. Standardized features generally have zero mean and unit variance, which reduces scale imbalance and often improves the condition number of the optimization problem. Scaling cannot remove correlations between features, so it does not eliminate every source of slow convergence.

For an unregularized linear model, scaling followed by inverse transformation does not change the optimal predictions. It changes the optimization path and the appropriate learning-rate range. With L1 or L2 regularization, it changes the effective penalty applied to each original feature, so regularization strength may need retuning.

Algorithm Guidance

Approach How it works Scaling need Strength Weakness Typical scale
Gradient descent Repeatedly updates parameters using gradients Usually important Efficient and widely reusable Sensitive to feature scale and learning rate Batch update is approximately O(n × d)
Tree ensembles Split features using thresholds Usually unnecessary Handles nonlinear relationships and mixed feature types May miss smooth trends and extrapolate poorly Common implementations scale well to large tabular datasets
k-nearest neighbors Compares distances to nearby observations Essential Simple and effective locally Slow prediction and high memory usage Prediction is approximately O(n × d) per query
SVMs with RBF or polynomial kernels Learn a distance- or interaction-based boundary Usually beneficial Powerful for nonlinear boundaries Sensitive to feature scale and regularization Training cost varies substantially with kernel and data size

Tree-based models generally do not require scaling because monotonic linear transformations do not change the ordering of split thresholds. Neural networks and PCA also commonly benefit from scaling. Missing values must be handled separately: scaling does not perform imputation.

Feature Engineering by Data Type

  • Numeric: Use standardization, robust scaling, or min-max scaling as appropriate. Log, Box-Cox, or Yeo-Johnson transformations can reduce skew; binning can capture thresholds, while interactions and polynomial features can expose relationships between variables.

  • Categorical: Use one-hot encoding for nominal categories and ordinal encoding only when order is meaningful. Frequency encoding and hashing can handle high-cardinality features. Target encoding must be cross-fitted to avoid leakage.

  • Dates and times: Extract parts such as year, month, and weekday; encode cyclical values with sine and cosine; and calculate elapsed time since relevant events.

  • Text: TF-IDF emphasizes discriminative terms, while embeddings represent semantic relationships.

  • Entity histories: Compute counts, means, trends, and rolling-window aggregates for each entity, using only information available at prediction time.

  • Practical concerns: Check for outliers, class imbalance, missingness, and overfitting. Use reproducible seeds and identical feature transformations during training and serving; scaling has the same mathematical effect on CPU and GPU implementations.

Feature Selection and Leakage

Feature selection can complement feature engineering:

  • Filter methods: Mutual information and correlation can rank features without fitting a predictive model.

  • Wrapper methods: Recursive feature elimination evaluates feature subsets using a model.

  • Embedded methods: L1 regularization and tree importances select features during model training.

  • Model-based selection: SelectFromModel keeps features according to an estimator’s importance or coefficient criteria.

Selection must be fitted inside the training pipeline or within each cross-validation fold. Otherwise, the selector can inspect validation information and leak it into the model. For time-dependent data, use time-ordered splits such as TimeSeriesSplit; for classification, use StratifiedKFold when appropriate. Feature stores such as Feast help ensure that training and serving compute the same features consistently.

Common Pitfalls

  • Scaling before splitting: Fit the scaler only on training data; a pipeline ensures that each cross-validation fold learns its own preprocessing.

  • Using future information: Aggregates, target encodings, and historical counts must be available at the prediction timestamp.

  • Ignoring regularization changes: Compare scaled and unscaled models with appropriate, retuned regularization strengths.

  • Using the wrong metric: For regression, report RMSE for error magnitude and R² for explained variance. For imbalanced classification, precision, recall, F1, ROC-AUC, and PR-AUC are often more informative than accuracy.

  • Overfitting preprocessing choices: Define transformations from domain knowledge and validate them within training folds rather than repeatedly tuning them on the test set.

8

What is one-hot encoding and when should you use it?

One-hot encoding converts each category in a categorical feature into a separate binary column, marking presence with 1 and absence with 0 so machine learning algorithms can process non-numeric labels. It solves the problem of representing nominal categories without implying an artificial order, which ordinal integers would introduce. Its main trade-off is that high-cardinality features create many columns, increasing memory use and making distance-based or linear models harder to fit.

Read full answer

Applies tomachine learning concepts (examples in scikit-learn 1.5+ and pandas 2.2+, Python 3.12)

How It Works

  • Mapping: For a feature with k unique categories seen during fitting, the encoder creates k binary columns (or k - 1 when a reference category is dropped to avoid redundancy in linear models).

  • Example: A color column with values red, green, blue becomes color_red, color_green, and color_blue.

  • Fit/transform rule: Categories are learned from training data only. Unseen categories at prediction time are handled by producing all-zero indicators (when unknown categories are ignored) or by raising an error, depending on configuration.

When to Use It

  • Nominal categories: Use it when categories have no natural order, such as city, product type, or browser.

  • Low to moderate cardinality: It works well when the number of unique values is manageable; very high cardinality (thousands of categories) can produce wide, sparse feature matrices.

  • Model compatibility: Linear models, k-NN, SVM, and neural networks typically need explicit numeric encoding. Tree-based models can use one-hot features, but for high-cardinality data other encodings or native categorical handling may be more efficient.

  • Preprocessing discipline: Apply one-hot encoding inside a pipeline or on training folds only (for example, with ColumnTransformer and OneHotEncoder in scikit-learn) so preprocessing is fitted on training data and not on validation or test data.

Comparison with Other Categorical Encodings

Method How it works Strengths Weaknesses When to use
One-hot One binary column per category No false ordering; simple Many columns for high cardinality Nominal, low/moderate cardinality
Ordinal Integer per category Compact Implies order Truly ordered categories
Target encoding Replaces category with a target statistic Compact for high cardinality Leakage risk without cross-fitting High-cardinality supervised tasks
Frequency/hashing Count or hash bucket Scales to many categories Collisions or lost detail Very high cardinality, streaming

Common Pitfalls

  • Fitting on all data: Always fit the encoder inside a pipeline or on training folds only; fitting before splitting leaks category distribution information.

  • High cardinality blow-up: One-hot encoding IDs, ZIP codes, or free-text categories can create thousands of columns; prefer hashing or target encoding (with cross-fitting) instead.

  • Ignoring unknown categories: Production data may contain new categories; configure unknown-category handling and monitor for unseen values.

  • Misreading the dummy-variable trap: For regularized linear models, dropping a reference column is often unnecessary, but unregularized linear regression can become collinear without it.

  • Using ordinal encoding for nominal data: Assigning integers to unordered categories implies distances and order that do not exist.

  • Interview follow-ups: Expect questions on one-hot vs. target encoding, handling unseen categories, sparse vs. dense output, and why ordinal encoding is wrong for nominal data.

9

What is dimensionality reduction and how does it benefit machine learning?

Dimensionality reduction transforms many input variables into a lower-dimensional representation, either by creating new features or retaining a subset of the originals. It can remove redundancy, suppress noise, reduce computational cost, and make high-dimensional data easier to visualize; distance-based and regularized models may also generalize better. The central trade-off is information loss because discarded dimensions may contain task-relevant signal that an unsupervised method cannot recognize.

Read full answer

Applies tomachine-learning concepts (all versions)

What It Does

For n observations with d original features, dimensionality reduction produces a representation with k < d features. Transformation methods such as PCA create new combinations of features, while feature-selection methods retain original features that appear useful.

PCA finds orthogonal directions along which the data varies most. Standard implementations center the data automatically, but they do not automatically standardize it, so features measured on very different scales should usually be standardized first. Supervised methods such as linear discriminant analysis use the target while constructing the reduced representation.

Main Methods

Method How it works Strengths Weaknesses When to use Complexity or scale
PCA Projects data onto directions maximizing variance Fast, deterministic, useful for correlated numeric data Linear, potentially hard to interpret, sensitive to scaling Compression, denoising, visualization Dense full SVD is roughly O(n * d²) when n >= d
Linear discriminant analysis Finds directions maximizing class separation Uses label information and can improve classification Can overfit with small or imbalanced datasets Supervised classification Covariance estimation and eigendecomposition can be costly in d
Truncated SVD Computes a low-rank matrix factorization Effective for sparse text data and linear compression Components may be less intuitive TF-IDF matrices and large sparse datasets Cost depends on data density and retained rank
Autoencoders Learns a nonlinear encoder and decoder Captures complex nonlinear structure Needs substantial data and tuning; weak interpretability Images, audio, or complex representations Cost per epoch depends on network size and training duration
t-SNE or UMAP Builds a nonlinear neighborhood-based embedding Strong results for local visualization Geometry is unstable and usually unsuitable as default training preprocessing Exploratory visualization and clustering t-SNE is commonly O(n²); UMAP uses approximate nearest neighbors

Benefits and Limitations

  • Removes redundancy: PCA can replace strongly correlated variables with a few orthogonal components.

  • Reduces cost: Fewer inputs can shorten model training and prediction, particularly for dense linear models.

  • May improve generalization: Compression can reduce variance, although this must be verified through validation.

  • Supports visualization: Two-dimensional embeddings can reveal clusters, outliers, and manifold structure.

  • Can hide important signal: Low-variance features may still be highly predictive, and unsupervised reduction does not optimize for the task.

  • Reduces interpretability: New components often lack the direct business meaning of original variables.

  • Does not automatically fix imbalance or poor features: It cannot replace thoughtful feature engineering or appropriate evaluation.

Feature Engineering Context

Feature engineering turns raw data into model inputs that expose useful signal; for many tabular problems, it matters more than the choice of algorithm. Common choices include:

  • Numeric data: Scaling, log1p, Box-Cox or Yeo-Johnson transforms, binning, ratios, interactions, and polynomial features.

  • Categorical data: One-hot encoding for nominal values, ordinal encoding only when order has meaning, frequency or hashing encoding for high-cardinality values, and cross-fitted target encoding for supervised use.

  • Dates and times: Date parts, elapsed time since an event, and cyclic sine/cosine encodings so adjacent periods such as midnight and the end of day remain close.

  • Text data: TF-IDF vectors or pretrained embeddings.

  • Entity data: Counts, historical means, totals, and rolling-window aggregates.

Feature selection is related but distinct: filter methods rank features using correlation, mutual information, or statistical tests; wrapper methods evaluate model performance, as with recursive feature elimination; embedded methods use model-based importance, including L1 regularization and tree importances. SelectFromModel wraps an estimator’s importance scores into a selector.

Practical Guidance

  • Fit every learned transformation, including scalers, encoders, PCA, and aggregates, on training data only and apply it to validation or test data.

  • Use Pipeline and ColumnTransformer so cross-validation fits preprocessing within each training fold.

  • Select the number of components using validation performance and cumulative explained variance rather than choosing k solely from a training-set curve.

  • Compare the reduced model with a baseline that retains the original features.

  • Scale for PCA, k-NN, SVM, PCA, and gradient-descent methods; tree models generally do not require scaling.

  • Use suitable metrics: MAE, RMSE, and R² for regression; precision, recall, F1, ROC-AUC, and PR-AUC for imbalanced classification. Accuracy can mislead when classes are imbalanced.

  • A feature store such as Feast can help reuse consistent feature definitions between training and serving and reduce offline-online skew.

Common Pitfalls

  • Leakage: Target encodings and aggregates must be cross-fitted, and time-dependent features must use only information available before prediction.

  • Over-reduction: A smaller representation is not automatically better if it removes predictive signal.

  • Wrong purpose: t-SNE and UMAP are primarily visualization tools, not default preprocessing for prediction.

  • Ignoring scale: Unscaled PCA components may be dominated by high-magnitude features.

  • Validation overuse: Repeated tuning against the same validation set turns it into a training set; nested cross-validation gives more reliable estimates.

  • Interview follow-up: Be ready to explain how PCA differs from feature selection, why k is chosen, and how leakage is prevented.

10

How should categorical variables be handled in machine learning?

Categorical encoding converts nonnumeric labels into numerical features that models can process while preserving useful distinctions between categories. It solves the inability of most machine-learning algorithms to consume strings directly and often matters more than algorithm choice for tabular data. The main trade-off is retaining information without introducing excessive dimensionality, biased assumptions, or data leakage.

Read full answer

Applies toscikit-learn 1.5+ and pandas 2.2+ (Python 3.12)

Encoding Approaches

Choose an encoding based on the variable’s meaning, cardinality, and the model:

Data type Common techniques When to use and trade-offs
Nominal categories One-hot encoding, frequency encoding One-hot encoding is safe for linear models but creates one column per category. Frequency encoding is compact but may create misleading correlations.
Ordered categories Ordinal encoding Appropriate only when the order is meaningful, such as basic < plus < pro. The model may incorrectly assume equal spacing between levels.
High-cardinality categories Cross-fitted target encoding, hashing, embeddings Target encoding is compact but must be computed without target leakage. Hashing bounds dimensionality but can create collisions.
Dates and times Date parts, cyclic sin/cos encoding, time since an event Extract month, weekday, season, and elapsed time. Cyclical encoding avoids treating December and January as far apart.
Text TF-IDF, embeddings TF-IDF captures word frequency; embeddings represent semantic relationships.
Entity histories Counts, means, rolling-window aggregates Useful for customer, device, or account behavior, but only use information available at prediction time.

For nominal variables, one-hot encoding is usually the safest starting point. For high-cardinality features, use cross-fitted target encoding or hashing and validate carefully.

Code Example

The following example engineers date, ratio, and customer-level features, then handles nominal and ordinal categories with a leakage-safe Pipeline:

import numpy as np
import pandas as pd

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

rng = np.random.default_rng(42)
n = 60

df = pd.DataFrame({
    "customer_id": [f"C{i % 12:02d}" for i in range(n)],
    "city": rng.choice(["London", "Paris", "Berlin"], size=n),
    "plan": rng.choice(["basic", "plus", "pro"], size=n),
    "signup_date": pd.to_datetime("2024-01-01")
                    + pd.to_timedelta(rng.integers(0, 365, size=n), unit="D"),
    "monthly_spend": rng.normal(50, 15, size=n),
    "support_tickets": rng.integers(0, 8, size=n),
})

df["churn"] = (
    (df["monthly_spend"] < 55) | (df["support_tickets"] > 5)
).astype(int)

# Split before computing any learned statistics.
X_train, X_test, y_train, y_test = train_test_split(
    df.drop(columns="churn"),
    df["churn"],
    test_size=0.25,
    random_state=42,
    stratify=df["churn"],
)

def add_row_features(frame):
    result = frame.copy()
    month = result["signup_date"].dt.month
    result["signup_month"] = month
    result["signup_weekday"] = result["signup_date"].dt.weekday
    result["month_sin"] = np.sin(2 * np.pi * month / 12)
    result["month_cos"] = np.cos(2 * np.pi * month / 12)
    result["spend_per_ticket"] = (
        result["monthly_spend"] / (result["support_tickets"] + 1)
    )
    return result

X_train = add_row_features(X_train)
X_test = add_row_features(X_test)

# Customer aggregates are learned from training data only.
customer_mean = X_train.groupby("customer_id")["monthly_spend"].mean()
X_train["customer_avg_spend"] = X_train["customer_id"].map(customer_mean)
X_test["customer_avg_spend"] = X_test["customer_id"].map(customer_mean)

numeric_features = [
    "signup_month",
    "signup_weekday",
    "month_sin",
    "month_cos",
    "spend_per_ticket",
    "customer_avg_spend",
]

numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])

nominal_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])

ordinal_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OrdinalEncoder(
        categories=[["basic", "plus", "pro"]],
        handle_unknown="use_encoded_value",
        unknown_value=-1,
    )),
])

preprocessor = ColumnTransformer([
    ("numeric", numeric_pipeline, numeric_features),
    ("nominal", nominal_pipeline, ["city"]),
    ("ordinal", ordinal_pipeline, ["plan"]),
])

model = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", LogisticRegression(max_iter=1000)),
])

model.fit(X_train, y_train)
predictions = model.predict(X_test)

print("Test accuracy:", round(accuracy_score(y_test, predictions), 3))

Feature Selection

High-dimensional encodings can be simplified after preprocessing:

  • Filter methods: Use correlation or mutual information to remove weak or redundant features.

  • Wrapper methods: Use RFE to evaluate feature subsets using a chosen estimator.

  • Embedded methods: Use L1 regularization or tree-based importances to identify useful variables.

  • SelectFromModel: Retains features according to a model’s importance threshold.

  • Categorical groups: Consider related one-hot columns together so correlated dummies are not evaluated in isolation.

Best Practices

  • Treat nominal and ordinal variables differently; never impose numeric order on categories without a real ordering.

  • Handle unseen categories explicitly with handle_unknown="ignore" or another documented policy.

  • Impute missing values inside Pipeline so preprocessing is fitted only on training data.

  • Scale numeric inputs for k-NN, SVM, PCA, and gradient-based models; tree models generally do not require it.

  • For time-dependent data, use TimeSeriesSplit and calculate rolling aggregates only from earlier observations.

  • Store reusable definitions in a feature store such as Feast so training and serving compute identical features.

Common Pitfalls

  • Computing target encodings or aggregates on the full dataset leaks validation information.

  • Using target encoding without cross-fitting can make rare categories memorize labels.

  • Applying one-hot encoding before splitting is incorrect when it depends on discovered categories.

  • Using accuracy alone can mislead on imbalanced classification; also consider precision, recall, F1, ROC-AUC, and PR-AUC.

  • Treating high-cardinality identifiers as ordinary categories creates excessive sparsity.

  • Common interview follow-ups ask when to use target encoding versus one-hot encoding, how unknown categories are handled, and how leakage is prevented.

11

How do filter methods select features in machine learning?

Filter methods rank features independently of the final machine-learning model, usually by measuring each feature’s relationship with the target. They remove irrelevant or redundant variables before model training, which can improve speed, interpretability, and generalization. Their main trade-off is efficiency and model independence versus an inability to recognize interactions or context that require combinations of features.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How Filter Methods Work

Feature engineering turns raw data into inputs that expose useful signal to the model; for many tabular problems, it matters more than switching algorithms. A filter then evaluates candidate features and applies a selection rule:

  1. Compute an association score for each feature.

  2. Rank features from most to least useful.

  3. Keep the top k features or remove scores below a threshold.

  4. Fit the final model using the selected features.

Many common filters are univariate and score each feature separately. Others, such as mutual-information-based redundancy methods, can also account for relationships among features. Simple correlation or ANOVA scoring is often approximately O(n × d) for n rows and d features, although mutual-information estimation is costlier because it estimates relationships stochastically.

Common Filter Methods

  • Correlation filtering: Measures the linear relationship between a numeric feature and the target. It is fast and interpretable but misses nonlinear relationships, interactions, and redundancy among highly correlated features.

  • ANOVA F-test: Commonly used for classification through f_classif; f_regression provides the regression counterpart. The test measures whether feature means differ across target groups.

  • Mutual information: Available through mutual_info_classif and mutual_info_regression, it can detect nonlinear dependencies but may be unstable with small samples or poorly estimated discrete features.

  • Chi-squared test: Useful for suitable categorical features and targets, such as class-contingency tests.

  • Variance or threshold filters: Remove low-variance features or features whose score falls below a predefined boundary. Variance filters do not measure usefulness for the target.

Missing values must usually be handled before applying these filters. Imputation, scaling, encoding, and feature selection must be fitted on training data only.

Feature Engineering Context

Useful features depend on the data type:

  • Numeric: Scaling, log or Box-Cox/Yeo-Johnson transformations, binning, interactions, and polynomial features can expose different relationships.

  • Categorical: One-hot encoding is a common baseline. Ordinal encoding is appropriate only when categories have a real order; target encoding requires cross-fitting, while frequency encoding and hashing offer alternatives for high cardinality.

  • Dates and times: Useful parts include year, month, day, and day of week; cyclic sine and cosine encodings represent periodic values, and elapsed-time features measure time since an event.

  • Text: TF-IDF emphasizes discriminative terms, while embeddings represent semantic relationships.

  • Entity histories: Counts, rolling means, trends, and other window aggregations can summarize behavior per customer, device, or account.

Feature engineering and selection are complementary: good features expose signal, while filters remove those that contribute little individually.

When to Use Filters

Filters are especially useful for large feature spaces, quick baselines, constrained training or serving costs, and interpretable input contracts. They are less effective when predictive information exists mainly through interactions, when removing one feature disrupts a redundant group, or when categorical encodings require careful target-aware validation.

For production systems, a feature store such as Feast can help ensure that feature definitions are consistent between training and serving.

Comparison With Other Selection Approaches

Approach How it works Strengths Weaknesses Typical use and scale
Filter Scores features independently or using simple redundancy criteria Fast, simple, model-independent Misses interactions and some redundancy Initial reduction; often roughly O(n × d) per score
Wrapper Repeatedly evaluates model performance, as in recursive feature elimination (RFE) Captures feature combinations Expensive and model-dependent Small-to-medium feature sets
Embedded Learns importance during training through L1 regularization or tree importances Often more predictive than univariate filters Depends on model assumptions Models already supporting built-in selection
SelectFromModel Keeps coefficients or importances above a threshold Flexible and easy to integrate Still model-dependent Linear models and tree ensembles

Common Pitfalls

  • Data leakage: Compute imputations, encodings, target encodings, aggregate features, filter scores, and thresholds from training folds only.

  • Validation overfitting: Repeatedly changing selected features against the same validation set makes performance estimates optimistic.

  • Univariate blind spots: A weak feature may become valuable in an interaction; compare filtering with wrapper or embedded selection when appropriate.

  • Incorrect metrics: Accuracy can mislead with class imbalance; also consider precision, recall, F1, PR-AUC, and ROC-AUC.

  • Missing values and scaling: Handle missing values before filtering, and scale when required by the scoring method or downstream estimator.

  • Time-series data: Respect chronological order and compute rolling or historical aggregates only from information available at prediction time.

  • Common interview follow-ups: Why correlation misses nonlinear relationships, how mutual information is estimated, why filtering differs from regularization, and how feature-selection leakage is prevented during cross-validation.

12

What are wrapper methods for feature selection in machine learning?

Wrapper methods are feature selection techniques that evaluate subsets of input variables by training and scoring a predictive model on each candidate subset. They solve the problem of choosing features that maximize a model’s performance, rather than relying only on feature statistics. Their main trade-off is stronger, model-aware selection at the cost of much higher computation and a greater risk of overfitting to the chosen model and validation scheme.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12, scikit-learn 1.5+)

How Wrapper Methods Work

  • Subset search: A search strategy proposes which features to include or remove.

  • Model evaluation: A learning algorithm is trained on each subset and scored with a metric (for example, F1 for classification or RMSE for regression).

  • Selection rule: The subset with the best validation or cross-validation score is kept.

  • Common search strategies:

  • Forward selection: Start with no features and add the feature that improves the score most at each step.

  • Backward elimination: Start with all features and remove the least useful feature at each step.

  • Recursive elimination: Repeatedly fit a model, rank features (for example, by coefficients or importances), and drop the lowest-ranked features until the target count remains.

  • Exhaustive search: Evaluate every possible subset; practical only for very small feature sets.

Common Wrapper Methods

Method How it works Strengths Weaknesses When to use Typical complexity
Sequential Forward Selection (SequentialFeatureSelector, forward) Adds one feature at a time by model score Simple; can find useful small subsets Greedy; can miss interactions Moderate d, need a compact subset About O(d * k) model fits for k selected features (more with CV)
Sequential Backward Selection (SequentialFeatureSelector, backward) Removes one feature at a time Considers features in context of the full set Expensive; greedy Small to moderate d About O(d^2) model fits in the worst case (more with CV)
RFE / RFECV Recursively removes low-importance features using a fitted estimator Efficient for linear/tree models with importance scores; RFECV tunes subset size Depends on estimator’s ranking quality Models exposing coef_ or feature_importances_ Roughly O(m * d) fits for m elimination steps, times CV folds for RFECV
Exhaustive feature selection Scores all subsets of a given size or all sizes Finds the global best subset under the scoring setup Combinatorial explosion Very small d (often < 15) O(2^d) subsets total

Advantages and Limitations

  • Advantages: Capture interactions between features and the target model; usually select more task-relevant subsets than simple filters.

  • Limitations: Training many models is slow for large datasets or many features; results depend on the estimator, metric, and split strategy; repeated tuning can overfit the validation data.

  • Leakage control: Any preprocessing and the wrapper selection must be fitted on training folds only. For time-dependent data, respect time order (for example, TimeSeriesSplit) and avoid features computed from future information.

Common Pitfalls

  • Data leakage: Running scaling, imputation, or feature selection on the full dataset before cross-validation inflates scores. Use Pipeline/ColumnTransformer so preprocessing and selection run inside each training fold.

  • Wrong metric: Accuracy can mislead on imbalanced classes; prefer precision, recall, F1, ROC-AUC, or PR-AUC as appropriate.

  • Overfitting to validation: Repeatedly tuning the subset size on the same validation split can produce optimistic results; use nested CV for unbiased estimates.

  • Ignoring model assumptions: Wrappers are model-specific; features selected for logistic regression may not be best for trees.

  • Interview follow-ups: How do wrapper methods differ from filter and embedded methods? Why is RFECV often preferred over RFE? How would you prevent leakage when using target encoding with a wrapper?

13

How do embedded feature-selection methods work and improve models?

Embedded feature selection selects useful inputs while fitting the predictive model, using learned coefficients or feature importances rather than ranking variables separately as filter methods do. It can reduce noise and dimensionality, improve generalization and deployment efficiency, and simplify interpretation; the main trade-off is that the result is model-dependent and can be unstable. Feature engineering turns raw observations into inputs that expose signal, and embedded selection chooses among those inputs, so on tabular data thoughtful feature engineering often matters at least as much as the choice of model.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12); the concepts are model-family independent

How It Works

  • Regularization-based selection: Methods such as L1 and ElasticNet include a penalty in model fitting. L1 can drive some coefficients exactly to zero, while ElasticNet combines L1 sparsity with L2 stability.

  • Tree-based selection: An ensemble learns splits and derives impurity-based feature importances during the same fit. A threshold can then determine which features to retain.

  • SelectFromModel: In scikit-learn, this is a meta-estimator rather than an embedded algorithm itself. It fits a wrapped estimator exposing coef_ or feature_importances_ and keeps features according to a threshold.

  • Comparison with wrappers: Embedded methods normally avoid the repeated model fits required by wrapper methods such as recursive feature elimination, although SelectFromModel performs its own estimator fit and selection step.

Main Methods

Method How it works Strengths Weaknesses When to use and typical cost
L1 regularization Penalizes absolute coefficients and can set them exactly to zero Sparse, interpretable linear models Sensitive to scaling; may choose arbitrarily among correlated predictors High-dimensional linear problems; one penalized fit
ElasticNet Combines L1 sparsity with L2 stability More stable with correlated predictors Requires tuning alpha and l1_ratio Correlated numerical features; one penalized fit
Tree feature importance Scores features from their contribution to impurity-reducing splits Captures nonlinear relationships and interactions without scaling Importance can be split unpredictably among correlated predictors; impurity measures can favor high-cardinality features Nonlinear tabular models; one ensemble fit plus importance calculation
SelectFromModel Applies a threshold to an estimator’s coefficients or importances Consistent interface for supported estimators Selection depends on the wrapped model and threshold; not itself an embedded method Reusing model-derived scores; one estimator fit

How It Can Improve Models

  • Reduce overfitting: Removing irrelevant variables can reduce variance and improve validation performance.

  • Lower operational cost: Fewer inputs mean less computation during training and serving.

  • Improve interpretability: Sparse linear models and manageable tree ensembles expose the variables they rely on.

  • Preserve model-aware behavior: Scores reflect the selected estimator’s objective rather than a generic statistical association.

  • Avoid guaranteed gains: Selection can discard useful predictors, favor the wrong feature structure, or reduce performance; compare the selected and full pipelines on untouched validation or test data.

Practical Guidance

  • Put feature construction and selection inside a scikit-learn Pipeline, and evaluate the entire pipeline with cross-validation so every transformation is fitted only on training folds.

  • Compute target encodings and entity aggregates only from training data, using cross-fitting or out-of-fold estimates. For temporal data, preserve chronology and use TimeSeriesSplit.

  • Scale numeric features for L1 and ElasticNet; scaling is unnecessary for tree models.

  • Use PR-AUC, precision, recall, and F1 for imbalanced classification, and MAE, RMSE, or R² for regression. Accuracy can be misleading when the minority class is rare.

  • Engineer candidates appropriate to each data type: transformed or interacted numeric variables, one-hot or cross-fitted target encodings for categorical variables, date parts and cyclic encodings for time, TF-IDF or embeddings for text, and leakage-safe per-entity aggregates.

  • Use a feature store such as Feast when training and serving systems must apply consistent feature definitions.

Common Pitfalls

  • Leakage: Calculating encodings or aggregates before splitting leaks information into validation data.

  • Ignoring correlation: L1 may retain only one member of a correlated group, while tree models may distribute importance unpredictably.

  • Unstable thresholds: A threshold near a coefficient or importance boundary can change the selected set across samples.

  • Overfitting the validation set: Repeatedly tuning the estimator, threshold, and feature set against the same validation data turns it into a training resource.

  • Misreading importance: Predictive importance is not causal evidence, and an important feature may appear weak because another correlated feature received the credit.

Interviewers commonly ask how L1 creates sparse solutions, how tree importances are computed, why correlated predictors complicate selection, how thresholds should be validated, and how embedded selection differs from filter and wrapper methods.

14

How does feature correlation with the target influence feature selection?

Correlation with the target variable measures how strongly a feature moves together with the outcome, and it is often used as a simple signal of predictive usefulness during feature selection. It helps prioritize features that carry information about the target, which can reduce noise and model complexity. The main trade-off is that correlation captures only specific kinds of relationships and can be distorted by leakage, outliers, or redundant features.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

How Correlation Informs Selection

  • Filter ranking: In filter methods, features are scored by their relationship to the target and the highest-scoring ones are kept before model training. Correlation is a fast, model-agnostic baseline for this step.

  • Signal strength: A correlation magnitude close to 0 suggests little linear or monotonic association with the target, while a larger magnitude suggests the feature may help prediction.

  • Direction: The sign indicates whether higher feature values tend to align with higher or lower target values, which can aid interpretation but does not by itself prove causation.

  • Redundancy check: Features highly correlated with the target but also highly correlated with each other can add duplicate information; pairwise feature-feature correlation helps detect this.

Correlation Measures and Limits

Measure Relationship captured Typical use Limitation
Pearson correlation Linear association between numeric variables Continuous feature and continuous target Misses nonlinear patterns; sensitive to outliers
Spearman rank correlation Monotonic association based on ranks Skewed data or ordinal relationships Still misses non-monotonic patterns
Point-biserial / ANOVA-style checks Association with a binary or categorical target Mixed feature/target types Assumptions and encoding choices affect results
Mutual information General dependence, including nonlinear Broader filter-based selection Harder to interpret; estimates depend on sample size and binning/estimator settings

Key Considerations

  • Nonlinearity: A feature can be important even with near-zero Pearson correlation, for example when the relationship is U-shaped. Mutual information or model-based importance can complement correlation.

  • Multicollinearity: Selecting several features that are all correlated with the target and with each other can make linear model coefficients unstable. Regularization (L1/L2) or removing redundant features can help.

  • Scaling: Correlation itself does not require feature scaling, but downstream models that use the selected features (such as k-NN, SVM, PCA, or gradient descent) often do.

  • Time and memory: Computing pairwise correlations across n samples and d features is typically O(n * d^2) for a full feature-feature matrix, so very wide datasets may need sampling or cheaper filters.

Interview Tips

  • Typical mistakes: Computing correlations on the full dataset before splitting (leakage), using Pearson for clearly nonlinear relationships, treating correlation as causation, or dropping low-correlation features without checking interactions and nonlinear effects.

  • Leakage reminder: Any statistic used for selection, including target correlation or target encoding, must be estimated on training folds only; wrap selection inside an sklearn.pipeline.Pipeline or cross-validation loop.

  • Follow-up questions: “How would you handle nonlinear relationships?”, “Why can two features both correlate with the target but hurt a linear model together?”, and “How do you perform feature selection correctly inside cross-validation?”

15

What is the purpose of RFE in feature selection?

Recursive Feature Elimination (RFE) is a wrapper-based feature-selection method that progressively removes less important input features until a chosen number remains. It reduces dimensionality and may improve generalization and interpretability by removing noisy or redundant variables. Its main trade-offs are computational cost and the risk of discarding useful features when the underlying estimator produces unreliable importance scores.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

Why RFE Matters

Feature engineering transforms raw data into model inputs that expose useful signal. For tabular data, it often matters more than changing the learning algorithm. RFE complements feature engineering by removing features that appear redundant, noisy, or irrelevant to the model.

RFE can help:

  • Reduce dimensionality: Fewer columns can reduce training time, memory use, and prediction latency.

  • Improve generalization: Removing noisy features may reduce variance.

  • Improve interpretability: A smaller feature set is easier to explain and debug.

  • Prevent overfitting: This can be useful when the original dataset contains many weak predictors.

It does not guarantee better performance. If the estimator assigns importance incorrectly, RFE can remove genuinely predictive features.

How RFE Works

RFE repeatedly performs these steps:

  1. Fit an estimator on the current feature set.

  2. Obtain importance scores from the estimator’s coef_ or feature_importances_ attribute.

  3. Remove the least important feature, or step features at once.

  4. Repeat until n_features_to_select features remain.

  5. Fit the final estimator using the selected features.

The estimator must support feature-importance information. Scikit-learn uses the absolute magnitudes of linear-model coefficients and tree-based estimators commonly expose impurity-based importances.

RFE is a wrapper method because it uses a fitted model to rank features, not because it searches subsets by predictive performance. Other selection approaches include filter methods such as mutual information or correlation, embedded methods such as L1 regularization and tree importances, and model-based selection with SelectFromModel.

Feature Engineering Context

RFE should generally be applied after creating useful candidate features, but it cannot repair features that were computed incorrectly or contain future information.

Data type Common feature-engineering techniques
Numeric Scaling, log, Box-Cox or Yeo-Johnson transforms, binning, ratios, interactions, and polynomial features
Categorical One-hot encoding, ordinal encoding, frequency encoding, hashing, and cross-fitted target encoding
Dates and times Year, month, weekday, holiday flags, cyclic sin/cos encoding, and time since an event
Text TF-IDF, word or character n-grams, and pretrained embeddings
Entity histories Counts, means, trends, and windowed aggregations per customer, account, or device

For target encoding and entity aggregates, compute statistics using training data only. Cross-fitting target encoders when they are used within model selection. For time-dependent data, preserve chronological order and use TimeSeriesSplit. Feature stores such as Feast can help reuse consistently defined features between training and serving.

Evaluation and Trade-offs

Evaluate the complete pipeline on a held-out test set or with cross-validation. For classification, use StratifiedKFold; for time series, use TimeSeriesSplit. Choose metrics appropriate to the problem: precision, recall, F1, ROC-AUC, and PR-AUC for imbalanced classification, or MAE, RMSE, and R² for regression. Accuracy can be misleading when classes are imbalanced.

RFE’s computational cost grows roughly with the number of elimination rounds. If k features remain from an initial d, it performs approximately d - k + 1 estimator fits when step=1. Tune n_features_to_select and step inside cross-validation rather than repeatedly checking the test set.

Common Pitfalls

  • Feature leakage: Do not fit imputers, scalers, encoders, feature selection, or target statistics on the full dataset before splitting.

  • Incorrect importance assumptions: Coefficient magnitude and tree impurity importance have different limitations and may rank features poorly.

  • Removing useful nonlinear signals: A linear estimator may rank interactions poorly. Compare against tree models or use permutation importance when appropriate.

  • Ignoring data type requirements: Scale numeric features for distance-based and gradient-based models; scaling is generally unnecessary for trees, but missing-value handling remains important.

  • Overfitting the validation set: Tune the number of features within training folds, not by repeatedly inspecting the final test set.

  • Ignoring deployment constraints: Feature transformations and selection rules must be reproduced consistently at serving time.

  • Reproducibility: Set the wrapped estimator’s random_state and the cross-validation splitter’s random_state where supported.

16

How does Random Forest estimate feature importance?

Random Forest feature importance assigns each feature a score indicating how much it contributes to the forest’s predictions. It helps identify useful inputs and diagnose a model, but the scores are model-dependent and do not establish causality. The main trade-off is that impurity-based importance is fast but can favor high-cardinality features, whereas permutation importance avoids that particular bias but is slower and can be affected by correlated predictors.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How Random Forest Estimates Importance

A Random Forest averages many decision trees, each of which selects splits according to an impurity criterion:

  • Classification: commonly Gini impurity, entropy, or log loss.

  • Regression: commonly variance or mean squared error.

For mean decrease in impurity (MDI), a split’s contribution to feature (j) is:

contribution(s, j) = (N_node / N_train) * [impurity(parent) - (N_left / N_node) * impurity(left) - (N_right / N_node) * impurity(right)]

Here, (s) is a split, (j) is the feature used by that split, N_node is the number of training samples reaching the parent node, N_train is the total number of training samples, and the child impurities are weighted by their sample counts. Contributions are summed over all qualifying splits in all trees and then normalized. In scikit-learn, RandomForestClassifier.feature_importances_ and RandomForestRegressor.feature_importances_ expose these normalized MDI values, which sum to 1.

Permutation importance uses a different procedure: it measures the model’s validation score before and after randomly shuffling one input feature:

importance(j) = baseline_score - shuffled_feature_score

A large positive decrease means the feature was useful. Negative values are possible when random shuffling happens to improve performance.

Comparing Importance Approaches

Method What it measures Strengths Limitations
Impurity importance (feature_importances_) Total weighted reduction in impurity Fast and built into scikit-learn Biased toward features with many possible splits, including continuous or high-cardinality variables
Permutation importance Validation-performance decrease after shuffling a feature Estimator-agnostic and avoids split-count bias Slower; correlated predictors can dilute, redistribute, or obscure individual importance
Out-of-bag importance Performance change measured on OOB predictions Uses observations excluded from the tree making their predictions Noisier than importance measured on independent validation data; not exposed directly as feature_importances_

Feature Engineering Context

Feature engineering turns raw data into model inputs that expose useful signal; for many tabular problems, it matters more than switching between algorithms.

  • Numeric: scaling, log or Yeo-Johnson transformations, binning, ratios, interactions, and polynomial features.

  • Categorical: one-hot encoding, ordinal encoding when order is meaningful, cross-fitted target encoding, frequency encoding, or hashing.

  • Dates and times: calendar parts, cyclic sine/cosine features, and elapsed time since an event.

  • Text: TF-IDF vectors or embeddings.

  • Entity histories: counts, means, and rolling-window aggregates computed only from information available at prediction time.

Scaling is not required by Random Forest because its trees do not use distances, but transformations can still improve usefulness or model behavior.

Feature Selection and Leakage

  • Filter methods: mutual information or correlation can rank features without training a wrapper model.

  • Wrapper methods: recursive feature elimination (RFE) selects features by repeatedly evaluating model performance.

  • Embedded methods: L1 regularization and tree-based importances identify features during model fitting.

  • SelectFromModel: keeps features according to a fitted model’s importance or coefficient threshold.

All learned preprocessing—including imputation, scaling, encoding, target encoding, and entity aggregates—must be fitted on training folds only. ColumnTransformer and Pipeline help enforce this separation. Chronological data additionally requires preserving time order, and feature stores such as Feast can keep training and serving feature definitions consistent.

Common Pitfalls

  • Treating importance as causal evidence or as the direction of a feature’s effect.

  • Using impurity importance without noting its bias toward high-cardinality predictors.

  • Interpreting permutation importance incorrectly when predictors are correlated.

  • Computing aggregates or encodings from the full dataset before splitting it.

  • Reporting a single unstable importance estimate without repeated cross-validation or independent held-out data.

  • Using accuracy on an imbalanced problem without considering precision, recall, F1, ROC-AUC, or PR-AUC.

  • Leaving missing-value handling unspecified; any imputation must be fitted within the training pipeline.

Interview Tips

Interviewers commonly ask how importance changes across folds, whether it remains stable when correlated features are added, how it differs from SHAP values, and how missing values, class imbalance, and time-dependent leakage should be handled.

17

How does regularization perform feature selection implicitly in linear models?

Regularization adds a penalty to a model’s training objective to discourage unnecessary coefficients, reducing overfitting. In L1-regularized linear regression, called Lasso, that penalty can set some coefficients exactly to zero, performing implicit feature selection. The main trade-off is that excessive regularization may discard useful predictors or underfit the data.

Read full answer

Applies toscikit-learn 1.5+ and pandas 2.2+ (Python 3.12)

How It Works

The Lasso objective is:

minimize: RSS + alpha * sum(|beta_j|)

  • RSS is the residual sum of squares: the sum of squared differences between observed and predicted values.

  • alpha controls the strength of regularization.

  • beta_j is the fitted coefficient for feature j.

  • The absolute-value penalty is the L1 norm.

Because the L1 constraint has corners, the optimizer can set coefficients exactly to zero rather than merely shrinking them. A zero coefficient means that feature contributes nothing to the model’s predictions.

Method Penalty Exact-zero feature selection
Lasso L1: ‘sum( beta_j )‘ Yes; coefficients can become exactly zero
Ridge L2: sum(beta_j^2) No; it primarily shrinks coefficients
Elastic Net L1 and L2 Yes; it can retain groups of correlated predictors

Feature Selection

Lasso selects features as a side effect of optimization:

  • Standardize predictors so their measurement scales do not unfairly affect the penalty.

  • Select alpha with cross-validation rather than repeatedly inspecting the test set.

  • Interpret dropped features carefully: a correlated predictor may be redundant rather than useless.

  • Do not assume every useful feature will survive a large penalty.

Other feature-selection approaches include:

  • Filter methods: mutual information or correlation.

  • Wrapper methods: recursive feature elimination (RFE).

  • Embedded methods: tree-based importances or an L1 model.

  • Model-based selection: SelectFromModel.

Feature Engineering Context

Feature engineering converts raw data into model inputs that expose useful signal. For tabular data, it often matters more than choosing a more complex algorithm.

  • Numeric data: scaling, log, Box-Cox or Yeo-Johnson transformations, binning, ratios, interactions and polynomial features.

  • Categorical data: one-hot encoding for nominal categories, ordinal encoding for ordered categories, frequency encoding, hashing, and cross-fitted target encoding.

  • Dates and times: year, month, weekday, holiday flags, elapsed time since an event, and cyclic sin/cos encodings.

  • Text: TF-IDF, word or character n-grams, and pretrained embeddings.

  • Entity histories: counts, means and trends calculated over available history.

Features must be available at prediction time. Target encodings and aggregates must be fitted using training-fold information only, and temporal data must respect time order. A feature store such as Feast can help reuse consistent definitions between training and serving.

Code Example

import numpy as np
import pandas as pd

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import Lasso
from sklearn.metrics import mean_absolute_error, r2_score, root_mean_squared_error
from sklearn.model_selection import GridSearchCV, TimeSeriesSplit
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


class FeatureBuilder(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        X = X.copy()
        X["event_date"] = pd.to_datetime(X["event_date"])
        X["month"] = X["event_date"].dt.month
        X["weekday"] = X["event_date"].dt.dayofweek
        X["days_since_start"] = (
            X["event_date"] - pd.Timestamp("2024-01-01")
        ).dt.days
        X["amount_per_item"] = X["amount"] / X["items"]
        return X


class GroupStats(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        self.group_means_ = X.groupby("entity")["amount"].mean()
        self.default_mean_ = 0.0
        return self

    def fit_transform(self, X, y=None, **fit_params):
        self.fit(X, y)
        ordered = X.sort_values("event_date").copy()
        prior_sum = (
            ordered.groupby("entity")["amount"].cumsum()
            - ordered["amount"]
        )
        prior_count = ordered.groupby("entity").cumcount()
        prior_mean = prior_sum.div(prior_count.where(prior_count > 0)).fillna(0.0)

        result = X.copy()
        result["entity_amount_mean"] = prior_mean.reindex(result.index)
        return result

    def transform(self, X):
        result = X.copy()
        result["entity_amount_mean"] = (
            result["entity"].map(self.group_means_).fillna(self.default_mean_)
        )
        return result


rng = np.random.default_rng(42)
n = 180

data = pd.DataFrame({
    "amount": rng.normal(20, 5, n).round(2),
    "items": rng.integers(1, 6, n),
    "entity": rng.choice(["A", "B", "C", "D", "E", "F"], n),
    "category": rng.choice(["basic", "premium"], n),
    "event_date": pd.date_range("2024-01-01", periods=n),
})

data["target"] = (
    2 * data["amount"]
    + 5 * data["category"].eq("premium")
    + rng.normal(0, 2, n)
)

X = data.drop(columns="target")
y = data["target"]

# Preserve chronological order.
split = int(0.8 * len(data))
X_train, X_test = X.iloc[:split], X.iloc[split:]
y_train, y_test = y.iloc[:split], y.iloc[split:]

preprocessor = ColumnTransformer([
    ("numeric", StandardScaler(), [
        "amount", "items", "month", "weekday",
        "days_since_start", "amount_per_item", "entity_amount_mean"
    ]),
    ("category", OneHotEncoder(
        handle_unknown="ignore", sparse_output=False
    ), ["category"]),
], remainder="drop", sparse_threshold=0)

pipeline = Pipeline([
    ("features", FeatureBuilder()),
    ("group_stats", GroupStats()),
    ("preprocess", preprocessor),
    ("lasso", Lasso(max_iter=10_000)),
])

search = GridSearchCV(
    pipeline,
    {"lasso__alpha": np.logspace(-3, 2, 20)},
    scoring="neg_mean_absolute_error",
    cv=TimeSeriesSplit(n_splits=5),
)
search.fit(X_train, y_train)

model = search.best_estimator_
predictions = model.predict(X_test)

feature_names = model.named_steps["preprocess"].get_feature_names_out()
coefficients = model.named_steps["lasso"].coef_
selected = [
    name for name, coefficient in zip(feature_names, coefficients)
    if coefficient != 0.0
]

print("Selected features:", selected)
print("Best alpha:", search.best_params_["lasso__alpha"])
print("MAE:", mean_absolute_error(y_test, predictions))
print("RMSE:", root_mean_squared_error(y_test, predictions))
print("R²:", r2_score(y_test, predictions))

Common Pitfalls

  • Failing to standardize predictors can make the penalty depend on measurement units.

  • Checking the test set while choosing alpha causes test-set leakage.

  • Computing aggregates from future rows or from the current row can leak unavailable information.

  • A feature dropped by Lasso may be redundant with another predictor rather than intrinsically useless.

  • Temporal predictions require chronological validation, such as TimeSeriesSplit, rather than shuffled folds.

18

How does mutual information select relevant features?

Mutual information is a filter criterion that ranks features by how strongly each one is associated with the target, including nonlinear relationships. It solves feature-selection problems without repeatedly fitting a model, but it evaluates features individually and therefore cannot identify redundancy among correlated features. Feature engineering still matters because it determines whether the raw data exposes useful signal to the model.

Read full answer

Applies toscikit-learn 1.5+ and pandas 2.2+ (Python 3.12)

How It Works

For a feature X and target Y, mutual information is:

I(X; Y) = ∫ p(x, y) log[p(x, y) / (p(x)p(y))] dx dy

  • X is a candidate feature and Y is the prediction target.

  • p(x, y) is their joint probability distribution.

  • p(x) and p(y) are their marginal distributions.

  • The logarithm measures departure from independence, while dx dy denotes integration over continuous values.

  • For discrete variables, the integral is replaced by a sum over possible values.

A theoretical score of zero means independence; larger values indicate stronger dependence. mutual_info_classif estimates this for classification, while mutual_info_regression estimates it for regression. Continuous features are generally estimated using nearest-neighbor methods, making the results sensitive to sample size and estimator settings.

A selector such as SelectKBest(score_func=mutual_info_classif, k=...) computes a score for every transformed feature and retains the k highest-scoring columns. Tune k inside nested cross-validation rather than against the final test set.

Preparing Features

Feature engineering turns raw records into model inputs that expose useful signal; for tabular data, it often matters more than changing algorithms.

Data type Useful techniques
Numeric Scaling, log or Yeo-Johnson transforms, binning, ratios, polynomials, interactions
Categorical One-hot, ordinal, cross-fitted target, frequency, or hash encoding
Dates and times Parts, cyclic sin/cos, and time since an event
Text TF-IDF or pretrained embeddings
Entity history Counts, means, trends, and windowed aggregates

Every feature must use only information available at prediction time. Fit imputers, encoders, scalers, and selectors on training folds only.

Selection Strategies

Method Type Main idea Main limitation
Mutual information Filter General feature-target dependence Misses redundancy and pure interactions
Correlation Filter Linear relationship Misses nonlinear relationships
RFE Wrapper Removes features using model performance Expensive and model-dependent
L1 or tree importance Embedded Learns importance during fitting Depends on the model family
SelectFromModel Meta-estimator Uses fitted model importance Inherits the model’s biases

Code Example

import numpy as np
import pandas as pd
from functools import partial
from sklearn.compose import ColumnTransformer
from sklearn.feature_selection import SelectKBest, mutual_info_classif
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score, classification_report, roc_auc_score
)
from sklearn.model_selection import (
    StratifiedKFold, cross_validate, train_test_split
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
n = 200
base_date = pd.Timestamp("2023-01-01")
prediction_date = pd.Timestamp("2024-01-01")

signup_offsets = rng.integers(0, 180, n)
customers = pd.DataFrame({
    "customer_id": np.arange(n),
    "signup_date": base_date + pd.to_timedelta(signup_offsets, unit="D"),
    "plan": rng.choice(["basic", "premium"], n),
    "region": rng.choice(["north", "south", "west"], n),
    "age": rng.integers(18, 75, n),
    "monthly_usage": rng.normal(50, 15, n).clip(5, None),
    "support_tickets": rng.poisson(2, n),
})

# Generate historical transactions and aggregate them by customer.
counts = rng.integers(1, 6, n)
customer_index = np.repeat(np.arange(n), counts)
transactions = pd.DataFrame({
    "customer_id": customers["customer_id"].to_numpy()[customer_index],
    "amount": rng.gamma(2.0, 30.0, len(customer_index)),
    "transaction_date": base_date + pd.to_timedelta(
        signup_offsets[customer_index]
        + rng.integers(0, 180, len(customer_index)),
        unit="D",
    ),
})
transactions = transactions[
    transactions["transaction_date"] < prediction_date
]
aggregates = transactions.groupby("customer_id").agg(
    transaction_count=("amount", "size"),
    avg_transaction_amount=("amount", "mean"),
).reset_index()
customers = customers.merge(aggregates, on="customer_id")

# Engineer date, ratio, and aggregate features.
customers["signup_month"] = customers["signup_date"].dt.month
customers["account_age_days"] = (
    prediction_date - customers["signup_date"]
).dt.days
customers["usage_to_age"] = (
    customers["monthly_usage"] / customers["age"]
)

logit = (
    -2.0
    + 0.04 * customers["age"]
    + 0.7 * (customers["plan"] == "premium")
    - 0.02 * customers["monthly_usage"]
)
probability = 1 / (1 + np.exp(-logit))
customers["churn"] = rng.binomial(1, probability)

numeric = [
    "age", "monthly_usage", "support_tickets", "signup_month",
    "account_age_days", "usage_to_age", "transaction_count",
    "avg_transaction_amount",
]
categorical = ["plan", "region"]
X = customers[numeric + categorical]
y = customers["churn"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

preprocess = ColumnTransformer([
    ("numeric", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
    ]), numeric),
    ("categorical", OneHotEncoder(
        handle_unknown="ignore", sparse_output=False
    ), categorical),
])

model = Pipeline([
    ("preprocess", preprocess),
    ("select", SelectKBest(
        score_func=partial(mutual_info_classif, random_state=42),
        k=5,
    )),
    ("classifier", LogisticRegression(max_iter=1000)),
])

scoring = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc",
    "pr_auc": "average_precision",
}
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
results = cross_validate(model, X_train, y_train, cv=cv, scoring=scoring)
model.fit(X_train, y_train)

names = model.named_steps["preprocess"].get_feature_names_out()
selector = model.named_steps["select"]
print("Selected:", names[selector.get_support()].tolist())
for name, score in sorted(
    zip(names, selector.scores_), key=lambda pair: pair[1], reverse=True
)[:5]:
    print(f"{name}: MI score {score:.4f}")
for metric in scoring:
    print(f"CV {metric}: {results[f'test_{metric}'].mean():.3f}")

y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]
print(classification_report(y_test, y_pred))
print(f"Test ROC-AUC: {roc_auc_score(y_test, y_prob):.3f}")
print(f"Test PR-AUC: {average_precision_score(y_test, y_prob):.3f}")

Practical Guidance

  • Standardize for k-NN, SVM, PCA, and many gradient-descent models; trees generally do not require scaling.

  • Mutual information is invariant to one-to-one transformations in theory; ordinary affine rescaling does not change its information content.

  • mutual_info_classif and mutual_info_regression reject missing values, so impute within the pipeline.

  • For imbalanced churn targets, prefer precision, recall, F1, ROC-AUC, and PR-AUC because accuracy can mislead.

  • Set random_state for reproducibility. Point-in-time-correct stores such as Feast help keep training and serving features consistent.

  • Mutual-information filtering is primarily CPU-oriented; GPUs are more relevant when generating embeddings.

Common Pitfalls

  • Computing mutual information, encodings, or aggregates before splitting causes data leakage.

  • Aggregating transactions without enforcing a prediction-time cutoff leaks future behavior.

  • Selecting k against the test set biases the final result.

  • High marginal scores do not establish causality or reveal pure feature interactions.

  • Correlated features may all score highly even though only one is needed.

19

What is PCA and how does it support feature engineering?

Principal Component Analysis (PCA) is an unsupervised dimensionality-reduction method that replaces correlated numeric variables with orthogonal components ordered by the variance they capture. In feature engineering, PCA can compress redundant inputs, suppress noise, and provide components for later models or visualization. Its main trade-off is compact, decorrelated features versus possible loss of predictive information and reduced interpretability.

Read full answer

Applies toall versions

How PCA Works

For a centered feature matrix X with n samples and p features, PCA chooses a unit direction v that maximizes the variance of the component scores z = Xv. Here, X is the input data, v is a direction in feature space, and z contains one component score per sample.

Using singular value decomposition, X = U Σ Vᵀ, U contains left singular vectors, Σ contains singular values, and V contains right singular vectors. Columns of V define the principal-component directions; the squared singular values determine their variances. The explained variance ratio is each component’s variance divided by the total variance.

The first component captures the most variance, while each later component captures the next-largest variance subject to being orthogonal to the earlier ones. Orthogonality means uncorrelated linear combinations, not necessarily statistical independence.

Full dense SVD generally requires O(np min(n,p)) time. Randomized or truncated methods are often approximately O(npr + nr²), where r is the number of retained components.

How PCA Supports Feature Engineering

  • Compression: A small number of components can replace many correlated columns.

  • Decorrelation: Orthogonal components can stabilize models sensitive to multicollinearity.

  • Denoising: Discarding low-variance components may suppress noise, although it can also remove useful signal.

  • Visualization: The first two components can provide a two-dimensional view of high-dimensional data.

  • Downstream efficiency: Fewer inputs can reduce later training and inference costs.

Feature engineering more generally turns raw data into inputs that expose useful signal to a model, and it often matters more than algorithm choice for tabular data. PCA is useful only when preserving variance is an appropriate proxy for preserving information. Because it is unsupervised, a high-variance component may be unrelated to the target, while a low-variance component may be highly predictive.

Inputs and Alternatives

PCA requires a numeric matrix, so features should first be engineered for their data type:

  • Numeric: scaling, log/Box-Cox/Yeo-Johnson transformations, binning, interactions, and polynomial features.

  • Categorical: one-hot or ordinal encoding, frequency encoding, hashing, or target encoding computed with cross-fitting.

  • Dates and times: calendar parts, cyclic sine/cosine encodings, and time since a known event.

  • Text: TF-IDF vectors or embeddings; sparse text matrices generally suit truncated SVD better than dense PCA.

  • Entity histories: counts, means, and rolling-window aggregates computed using only information available at prediction time.

Unlike feature selection, PCA replaces original columns rather than retaining a subset.

Technique How it works Strength Weakness When to use Typical complexity or scale
PCA Rotates data into orthogonal components Handles correlated numeric features Components can be difficult to interpret Dense, correlated feature sets Full: O(np min(n,p)); randomized: O(npr + nr²)
Filter methods Rank features using correlation or mutual information Fast and model-independent May miss interactions Initial reduction O(np) for simple scores; estimator-dependent for mutual information
Wrapper methods Repeatedly evaluate models, as with RFE Accounts for model behavior Expensive and prone to overfitting Small-to-medium datasets Roughly O(p) model fits for p features (about p - k when eliminating one feature at a time)
Embedded methods Use L1 coefficients or tree importances during fitting Model-aware Depends on the model and regularization When model-specific selection is appropriate Model-fitting cost plus extraction
SelectFromModel Applies thresholds to fitted model coefficients or importances Flexible and pipeline-compatible Results depend on threshold choice Linear, tree, or ensemble models One model fit plus thresholding

Best Practices

  • Scale appropriately: PCA directions are affected by feature units. Use standardization or robust scaling when features have different scales; tree models themselves do not require scaling.

  • Prevent leakage: Fit imputers, scalers, target encodings, aggregates, and PCA on training folds only. Use StratifiedKFold for classification and TimeSeriesSplit for time-ordered data.

  • Select components responsibly: An explained-variance threshold is a reasonable unsupervised default, but component count should be validated within the training process rather than chosen from test results.

  • Evaluate the downstream task: Use metrics suited to the objective, such as precision, recall, F1, ROC-AUC, and PR-AUC for imbalanced classification, or MAE, RMSE, and R² for regression.

  • Manage feature consistency: A feature store such as Feast can centralize definitions and reduce training-serving skew.

Common Pitfalls

  • Fitting PCA before splitting or cross-validation leaks test-distribution information.

  • Applying PCA to unscaled numerical features lets large-magnitude columns dominate components.

  • Assuming the highest-variance component must be the most predictive.

  • Interpreting loadings as original features; components are linear combinations of the input features.

  • Densifying a very large sparse one-hot matrix instead of using a sparse-appropriate method.

  • Randomly splitting temporal data when deployment requires prediction from future observations.

Useful follow-up topics include how PCA differs from LDA, why scaling affects PCA, how explained variance is interpreted, and how component count is selected without test-set leakage.

20

How do autoencoders perform feature extraction?

Autoencoders are neural networks that learn to compress input data into a low-dimensional latent representation and then reconstruct the original input from it. They solve the problem of turning raw, high-dimensional data into compact features that expose useful signal without manual engineering. The main trade-off is that the learned features are often hard to interpret and can capture noise or fail to help a downstream task if the bottleneck and training objective are poorly chosen.

Read full answer

Applies toPyTorch 2.x (Python 3.12)

How Autoencoders Extract Features

An autoencoder has two parts: an encoder that maps an input x to a latent vector z = f(x), and a decoder that reconstructs x_hat = g(z). Training minimizes a reconstruction loss, commonly mean squared error:

MSE = (1/n) * sum((x_i - x_hat_i)^2)

  • n: number of samples in the batch or dataset.

  • x_i: the original input vector for sample i.

  • x_hat_i: the reconstructed vector produced by the decoder for sample i.

  • sum(...): adds the squared reconstruction error across samples (and, in practice, across features).

For feature extraction, you train the network on input features, discard the decoder, and use the encoder output z as the new representation for downstream models (for example, logistic regression or a classifier).

Types and Practical Guidance

  • Undercomplete autoencoder: the latent layer is smaller than the input, forcing compression.

  • Denoising autoencoder: trained to reconstruct clean input from a corrupted version, which encourages robust features.

  • Sparse/variational variants: add constraints (sparsity penalty or probabilistic latent space) to shape the representation.

Autoencoders are most useful for high-dimensional or unlabeled data (images, sensor readings, wide tabular data). Inputs should be scaled because training uses gradient descent. If the extracted features feed a supervised task, fit the autoencoder and any scaler on the training split only to avoid data leakage.

Code Example

import numpy as np
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Reproducible synthetic data
rng = np.random.default_rng(42)
torch.manual_seed(42)

X, y = make_classification(
    n_samples=800, n_features=20, n_informative=10, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

# Fit preprocessing on training data only (prevents leakage)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)

class Autoencoder(nn.Module):
    def __init__(self, in_dim: int, latent_dim: int) -> None:
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(in_dim, 12),
            nn.ReLU(),
            nn.Linear(12, latent_dim),  # latent features z
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 12),
            nn.ReLU(),
            nn.Linear(12, in_dim),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        z = self.encoder(x)
        return self.decoder(z)

model = Autoencoder(in_dim=X_train_s.shape[1], latent_dim=5)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()

train_loader = DataLoader(
    TensorDataset(torch.tensor(X_train_s, dtype=torch.float32)),
    batch_size=64,
    shuffle=True,
)

model.train()
for epoch in range(30):
    for (batch_x,) in train_loader:
        recon = model(batch_x)
        loss = loss_fn(recon, batch_x)  # reconstruction MSE
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

# Extract features: use encoder output only
model.eval()
with torch.no_grad():
    z_train = model.encoder(torch.tensor(X_train_s, dtype=torch.float32)).numpy()
    z_test = model.encoder(torch.tensor(X_test_s, dtype=torch.float32)).numpy()

print("original train shape:", X_train_s.shape)  # (600, 20)
print("latent train shape:", z_train.shape)      # (600, 5)
print("latent test shape:", z_test.shape)        # (200, 5)

Common Pitfalls

  • Data leakage: fitting the scaler or autoencoder on the full dataset before splitting lets test information influence features; always fit on training data (or training folds) only.

  • Identity mapping: an over-capacity network can memorize inputs; use a tight bottleneck, regularization, or a denoising objective.

  • Unhelpful features: low reconstruction error does not guarantee features that improve a supervised metric; validate with cross-validation on the downstream task.

  • Missing scaling: unscaled inputs make optimization unstable for gradient-based training.

  • Interview follow-ups: when to prefer PCA vs. autoencoders, how denoising/variational autoencoders differ, and how you would prevent leakage in a pipeline.

21

What is t-SNE and how does it visualize high-dimensional data?

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a nonlinear dimensionality-reduction technique that projects high-dimensional observations into two or three dimensions while attempting to preserve local neighborhoods. It helps reveal clusters, patterns, and anomalies in complex data, with the main trade-off being that local relationships are more reliable than global distances or cluster spacing.

Read full answer

Applies tot-SNE (all versions)

How It Works

t-SNE converts similarities between observations into conditional probabilities and arranges the projected points so that input-space neighbors remain close.

For observation x_i, similarity to another observation x_j is modeled with a Gaussian probability:

P(j|i) = exp(-||x_i - x_j||² / (2 * sigma_i²)) / sum over k != i of the same numerator

Here, x_i and x_j are original-space observations, ||x_i - x_j||² is their squared Euclidean distance, and sigma_i determines the width of the similarity kernel around observation i. The denominator normalizes the similarities so they sum to one.

The embedding y_i uses a Student’s t-distribution:

Q(j|i) = (1 + ||y_i - y_j||²)^-1 / sum over k != i of the same numerator

Here, y_i and y_j are low-dimensional embeddings. The Student-t kernel has heavy tails, so attraction between points that are far apart in the embedding decays more slowly than it would with a Gaussian kernel. This helps prevent all observations from collapsing into one crowded cluster and allows separated groups to spread apart.

The layout minimizes the Kullback–Leibler divergence:

KL(P || Q) = sum over i and j of P(j|i) * log(P(j|i) / Q(j|i))

P and Q are the original-space and embedding-space similarity distributions. The logarithm measures their divergence, so minimizing it encourages input-space neighbors to appear near one another in the embedding.

Parameters and Interpretation

  • Perplexity: Roughly controls how many neighbors influence each observation’s local structure.

  • Learning rate: Controls how aggressively the layout changes during optimization.

  • Early exaggeration: Temporarily strengthens attractive forces to help separate neighborhoods.

  • Initialization: Affects the starting arrangement and potentially the final result.

  • Random state: Makes stochastic optimization reproducible when fixed.

Unlike PCA, t-SNE is nonlinear and does not require components to be linear combinations of the original variables. Its axes have no direct meaning, and the main information lies in relative neighborhood structure rather than absolute coordinates.

What It Reveals

t-SNE can reveal local neighborhoods, apparent clusters, separated groups, and potential outliers in image, text, genomic, or sensor embeddings. It does not use labels during optimization, so visible groups are not automatically classes; they should be checked using domain knowledge, held-out data, or other statistical methods.

Adding or removing points can change the result because each observation influences the joint optimization. Consequently, t-SNE is primarily an exploratory, transductive visualization method rather than a standard inductive transformation for scoring unseen observations.

Comparison With Other Methods

Method How It Works Strengths Weaknesses When to Use Complexity or Scale
PCA Linear orthogonal projection preserving maximum variance Fast and deterministic; interpretable axes Captures mainly linear structure High-dimensional preprocessing or global variance analysis Efficient for moderate and large matrices
t-SNE Optimizes local probability similarities in 2D or 3D Strong neighborhood visualization Expensive, stochastic, weak global-distance preservation Discovering local clusters and outliers Exact similarities cost O(n²d), optimization O(n²) per iteration, and memory O(n²); Barnes–Hut is roughly O(n log n) per iteration
UMAP Optimizes a fuzzy topological representation Often fast and scalable; can preserve more global structure Parameter- and initialization-sensitive Large datasets or exploratory embeddings Uses approximate nearest-neighbor graphs and sparse optimization, scaling well beyond exact t-SNE

Feature Engineering Context

Feature engineering turns raw data into model inputs that expose useful signal; for many tabular problems, it matters more than choosing between similar algorithms. Techniques include:

  • Numeric: scaling, log, Box–Cox or Yeo–Johnson transforms, binning, ratios, interactions, and polynomial features.

  • Categorical: one-hot, ordinal, frequency, hashing, and cross-fitted target encoding.

  • Dates and times: date parts, cyclic sin/cos features, and time since an event.

  • Text: TF-IDF or embeddings.

  • Entity aggregations: counts, means, trends, and rolling-window statistics per customer, device, or account.

Feature selection includes filter methods such as mutual information and correlation, wrappers such as recursive feature elimination (RFE), and embedded methods using L1 regularization or tree importances. SelectFromModel retains features according to a fitted model’s importance or coefficient threshold.

Fit imputers, scalers, encoders, selectors, target encodings, and aggregate features on training folds only. Use cross-fitting for target encodings, and for time-dependent data construct aggregate or lagged features using only information available before the prediction time. Feature stores such as Feast help reuse consistent feature definitions and reduce training-serving skew.

Common Pitfalls

  • Interpreting distant cluster spacing or cluster sizes as quantitatively meaningful.

  • Assuming every visible cluster is statistically significant or represents a true class.

  • Treating t-SNE axes or absolute coordinates as having inherent meaning.

  • Using only one perplexity or random seed; compare several settings.

  • Applying t-SNE to nonnumeric, missing, or carelessly scaled features without a defensible preprocessing design.

  • Adding test observations and then claiming evidence of out-of-sample generalization.

  • Using t-SNE as the default production feature representation; PCA, UMAP, or a supervised transformation may be more suitable.

Interviewers commonly ask how perplexity affects the result, why cluster sizes and spacing are misleading, how t-SNE differs from PCA, whether it supports out-of-sample prediction, and how randomness and convergence should be handled.

22

How does Linear Discriminant Analysis extract features?

Linear Discriminant Analysis (LDA) is a supervised dimensionality reduction technique that projects data onto a lower-dimensional space by finding linear combinations of features that best separate the classes. It solves the problem of reducing feature count while preserving class-discriminative information, which is useful for visualization and as input to downstream classifiers. Its main trade-off is that it assumes classes are roughly Gaussian with similar covariance and can produce at most n_classes - 1 components, so it may miss non-linear structure.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

LDA maximizes the Fisher criterion: J(w) = (w^T S_B w) / (w^T S_W w)

  • w: projection vector (direction) that maps original features to a new component.

  • S_B: between-class scatter matrix, measuring how far class means are from the overall mean (class separation).

  • S_W: within-class scatter matrix, measuring how spread out samples are within each class (class compactness).

  • w^T: transpose of w; the quadratic forms w^T S_B w and w^T S_W w become scalars representing projected between-class and within-class variance.

For multiple components, LDA solves the generalized eigenvalue problem S_B w = λ S_W w and keeps the top eigenvectors. The number of extracted features is min(n_features, n_classes - 1). Because it uses class labels, the resulting features are chosen for discrimination, not just variance.

LDA vs PCA for Feature Extraction

Aspect LDA PCA
Learning type Supervised (uses labels) Unsupervised (ignores labels)
Objective Maximize class separation Maximize variance
Max components n_classes - 1 n_features
Strengths Features aligned with class boundaries; can improve classifier performance Works without labels; captures dominant structure
Weaknesses Assumes Gaussian, equal covariance; limited components; sensitive to outliers High-variance directions may not separate classes
When to use Labeled data and you want discriminative features for classification Unlabeled data, noise reduction, or preprocessing

Code Example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)  # 150 samples, 4 features, 3 classes

# Split before fitting anything that learns from data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Fit LDA on training data only to avoid leakage (LDA uses labels)
lda_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("lda", LinearDiscriminantAnalysis(n_components=2)),  # max = n_classes - 1
])
X_train_lda = lda_pipe.fit_transform(X_train, y_train)  # supervised: uses y_train
X_test_lda = lda_pipe.transform(X_test)

print("Original train shape:", X_train.shape)      # (105, 4)
print("LDA train shape:", X_train_lda.shape)       # (105, 2)
print("LDA test shape:", X_test_lda.shape)         # (45, 2)
print("Explained variance ratio (per component):",
      lda_pipe.named_steps["lda"].explained_variance_ratio_)  # deterministic for this split

Time complexity is dominated by computing scatter matrices and solving the eigenproblem, roughly O(n * d^2) to form scatters and O(d^3) for eigendecomposition (n samples, d features), so it scales poorly to very high-dimensional sparse data.

Common Pitfalls

  • Data leakage: fitting LDA on the full dataset before splitting lets label information from test samples influence the projection; always fit inside a training fold or Pipeline.

  • Ignoring assumptions: strong non-Gaussian class shapes or very different covariances can produce poor projections; consider Quadratic Discriminant Analysis or non-linear methods in that case.

  • Overestimating capacity: you cannot extract more than n_classes - 1 components, so LDA is not a general high-dimensional compressor.

  • Class imbalance: priors and scatter estimates can be skewed; inspect class distributions and consider stratified splits and appropriate downstream metrics.

  • Interview follow-ups: difference between LDA for feature extraction vs. classification, why the n_classes - 1 limit exists, LDA vs. PCA, and how you would prevent leakage in a cross-validation setup.

23

What is Independent Component Analysis (ICA) in machine learning?

Independent Component Analysis (ICA) is an unsupervised method that separates a set of mixed observed signals into additive components that are as statistically independent and non-Gaussian as possible. It solves the blind source separation problem, such as recovering individual voices from microphone recordings that each capture a mixture of them. Its main trade-off is that ICA can recover meaningful hidden sources when its assumptions hold, but it cannot determine the original scale, sign, or order of those sources and it is sensitive to noise and violations of independence.

Read full answer

Applies tomachine learning concepts (scikit-learn 1.5+ for implementation reference, Python 3.12)

How It Works

ICA models the observed data as a linear mixture of hidden sources: - X = A * S - X: the matrix of observed (mixed) signals, with one row per observation and one column per measured variable. - A: the unknown mixing matrix that describes how each source contributes to each observed variable. - S: the matrix of unknown source signals that ICA tries to recover, with one row per observation and one column per independent component.

The algorithm estimates an unmixing matrix W such that S_hat = W * X approximates the original sources. Because two independent random variables are, in general, less Gaussian than their mixture, common ICA algorithms such as FastICA find W by maximizing a measure of non-Gaussianity (for example, an approximation of negentropy) of the recovered components. Preprocessing typically includes centering (subtracting the mean) and whitening (decorrelating and scaling variables to unit variance), which simplifies the search to an orthogonal rotation.

Key Assumptions

  • Statistical independence: the source signals do not carry information about each other.

  • Non-Gaussian sources: at most one source may be Gaussian. ICA cannot separate multiple Gaussian sources because uncorrelated Gaussian variables are already independent.

  • Linear mixing: each observation is a weighted sum of the sources, without nonlinear interactions.

  • Enough observations: the number of observed mixtures should be at least the number of sources to estimate.

ICA vs. PCA

Aspect ICA PCA
Goal Find statistically independent components Find orthogonal directions of maximum variance
Criterion Independence and non-Gaussianity Variance explained (second-order statistics)
Component order No inherent ordering or scale Ordered by explained variance
Typical use Blind source separation, artifact removal, latent factor discovery Dimensionality reduction, noise reduction, decorrelation
Output ambiguity Scale, sign, and permutation of components are arbitrary Sign ambiguity only; scale tied to variance

Practical Considerations

  • Feature scaling and whitening: center and whiten inputs before ICA; libraries such as scikit-learn’s FastICA handle this internally by default.

  • Reproducibility: FastICA uses random initialization, so set a fixed random_state to get consistent components across runs.

  • Number of components: choosing too many components can fit noise; domain knowledge or stability checks help select n_components.

  • Complexity: FastICA is iterative; its cost is roughly O(n * m * k * i) for n samples, m observed features, k components, and i iterations, with memory dominated by storing X and the unmixing matrix.

  • Noise and nonlinearity: ICA assumes a noise-free linear model in its basic form; heavy noise or nonlinear mixing can degrade separation.

Interview Tips

  • Common pitfalls:

  • Confusing ICA with PCA: PCA decorrelates using variance, while ICA targets statistical independence and relies on higher-order statistics.

  • Forgetting the ambiguities: ICA cannot recover the true amplitude, sign, or order of sources without extra information.

  • Applying ICA to Gaussian or highly dependent sources, where separation is not identifiable.

  • Ignoring preprocessing: failing to center or whiten data can hurt convergence and results.

  • Follow-up questions interviewers often ask:

  • How does ICA differ from PCA, and when would you choose one over the other?

  • Why must ICA sources be non-Gaussian?

  • What is the cocktail party problem, and how does ICA address it?

  • How does FastICA use non-Gaussianity to estimate independent components?

24

How to handle missing values in feature engineering?

Missing value handling is the process of detecting and treating absent entries during feature engineering so a model receives complete, usable inputs. It solves the problem that most algorithms cannot operate on gaps and that naive fixes can distort distributions or leak information. The main trade-off is between preserving signal (richer imputation) and adding complexity or bias (simpler fills or dropping data).

Read full answer

Applies toscikit-learn 1.5+ and pandas 2.2+ (Python 3.12)

Common Strategies by Data Type

Strategy How it works Best for Watch out for
Drop rows/columns Remove records or features with missing values Very small missing fraction or uninformative columns Loses data; can bias results if missingness is not random
Constant / indicator fill Fill with a sentinel (e.g., 0, "missing") and optionally add a binary “was missing” flag Categorical data or when missingness itself is predictive Can create artificial categories or shift numeric scales
Statistical imputation Replace with mean, median, or most frequent value Quick baseline; median for skewed numeric features Underestimates variance; ignores relationships between features
Model-based imputation Predict missing values from other features (e.g., KNNImputer, iterative imputation) Correlated features where structure can inform the fill Slower; higher leakage risk if not fit inside training folds
Time-aware fills Forward/backward fill or interpolation along time Time series or ordered events Must respect temporal order; future values must not fill past gaps
  • Numeric: use median for skewed data, mean for roughly symmetric data, or transforms (e.g., Yeo-Johnson) after imputation when needed.

  • Categorical: treat missing as its own category or fill with the most frequent value before encoding.

  • Dates/times: derive parts (year, month) or “time since event” and impute the source timestamp consistently.

Preventing Data Leakage

  • Split first: separate training and test sets before any imputation that learns from data.

  • Fit on training folds only: compute fill values, target encodings, and aggregates inside a Pipeline or cross-validation loop so test data never influences them.

  • Respect prediction time: only use information that would be available when the model is served; for time series, use TimeSeriesSplit and avoid look-ahead fills.

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
n = 200
df = pd.DataFrame({
    "age": rng.integers(18, 70, size=n).astype(float),
    "income": rng.normal(50000, 15000, size=n),
    "city": rng.choice(["A", "B", "C", None], size=n),  # includes missing
    "target": rng.integers(0, 2, size=n),
})
# Introduce additional missing numeric values
df.loc[rng.choice(n, 25, replace=False), "age"] = np.nan
df.loc[rng.choice(n, 20, replace=False), "income"] = np.nan

X = df[["age", "income", "city"]]
y = df["target"]

# Split before fitting any transformer that learns from data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

numeric_features = ["age", "income"]
categorical_features = ["city"]

numeric_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="median")),  # fit on training data only
    ("scaler", StandardScaler()),
])

categorical_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore")),
])

preprocessor = ColumnTransformer(transformers=[
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features),
])

model = Pipeline(steps=[
    ("preprocessor", preprocessor),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
])

model.fit(X_train, y_train)
print("Train shape:", X_train.shape)  # (150, 3)
print("Test shape:", X_test.shape)    # (50, 3)
print("Test accuracy around 0.50 (varies slightly by version)")

Interview Tips

  • Common pitfalls: imputing on the full dataset before splitting (leakage), using mean on heavily skewed features, dropping rows when missingness is informative, and ignoring that missingness patterns can differ between training and production.

  • Follow-up questions: How would you test whether data is missing completely at random? When would you add a missingness indicator feature? How do you handle missing values in a time series without look-ahead bias?

25

How do feature engineering methods address imbalanced datasets?

Feature engineering turns raw data into model inputs that expose predictive signal; this can make minority-class behavior easier to learn without changing how many examples exist. It helps most when features capture behavior, exposure, or context that distinguishes minority cases, whereas class weighting and resampling alter the learning process rather than create features. The main trade-off is that aggressive handling of rare values can amplify noise or overfit, so transformations and imbalance-aware evaluation must avoid leakage.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+, and NumPy 2.x (Python 3.12)

How Feature Engineering Helps

Feature engineering does not directly change the class distribution. Instead, it can make the minority class more distinguishable by representing signals such as transaction frequency, time since an event, spending rate, unusual behavior, or interactions between variables. For tabular data, feature quality often matters more than switching algorithms because a strong feature can be informative for both majority and minority cases.

Rare categorical values and imbalanced target classes are different problems. Grouping rare category levels can stabilize encodings, but it does not increase the number of minority-class observations and could even remove a useful rare signal.

Techniques by Data Type

Data type Useful techniques Imbalance and leakage safeguards
Numeric Scaling, log or Yeo-Johnson transforms, robust scaling, binning, interactions, and polynomial features Fit transforms on training data only; check whether extreme minority values dominate a scaling parameter
Categorical One-hot encoding, ordinal encoding when order is real, frequency encoding, hashing, smoothed target encoding, and rare-category grouping Compute frequencies from training folds; use cross-fitted target encoding to prevent target leakage
Date and time Calendar parts, time since an event, recency, and cyclic sine/cosine encodings Calculate features relative to the prediction timestamp, not the dataset’s maximum date
Text TF-IDF, domain-specific indicators, pretrained embeddings, or sentiment and topic features Fit the vectorizer on training text only and avoid representations containing future or test-set information
Entity aggregates Counts, means, rates, trends, and rolling-window aggregates per customer or account Normalize by exposure and use only events available before prediction; calculate aggregates within training folds

Missing-value imputation and feature selection must also be fitted within each training fold. Filter methods include mutual information and correlation; wrapper methods include recursive feature elimination (RFE); embedded methods include L1 coefficients and tree-based importances. SelectFromModel is another option for retaining features according to a model-based threshold. Because class imbalance can make univariate relationships unstable, feature selection should use the same folds and imbalance-aware scoring as the final evaluation.

Balancing and Model Integration

Class weighting, such as scikit-learn’s class_weight="balanced", changes the relative influence of training errors. Oversampling duplicates minority observations, undersampling discards majority observations, and methods such as SMOTE generate synthetic minority points. These are resampling or optimization strategies, not feature engineering.

Perform any sampling within each training fold, never before the train-test split or across validation folds. Put imputation, scaling, encoding, and learned feature selection in a scikit-learn Pipeline or ColumnTransformer. Scale for distance-based, kernel, PCA, and gradient-based models; trees generally do not require scaling. Configure one-hot encoding to handle unseen categories safely, and tune decision thresholds on validation data when false positives and false negatives have different business costs.

Evaluation and Leakage Controls

Use StratifiedKFold for ordinary imbalanced classification so folds retain similar class proportions, with the number of folds no greater than the minority-class count. For historical or transactional data, respect time order with TimeSeriesSplit rather than random folds.

Evaluate minority-class performance with precision, recall, F1, ROC-AUC, and PR-AUC; PR-AUC is especially useful when positives are rare. Accuracy can mislead because predicting the majority class for every observation may produce a high score. Compare feature sets using identical folds, models, and metrics, and use fixed random seeds for reproducibility.

Compute target encodings, frequency tables, imputations, and aggregates using only information available at prediction time. A feature store such as Feast can help keep training and serving definitions consistent.

Common Pitfalls and Interview Tips

  • Data leakage: Full-dataset target encodings or aggregates can expose validation and test information.

  • Incorrect resampling: Duplicating minority rows before splitting can place copies in both training and test sets.

  • Unstable rare categories: Arbitrary grouping thresholds can remove meaningful signals or create unstable encodings.

  • Overfitting the validation set: Repeatedly selecting features and thresholds on one split produces optimistic results.

  • Misreading accuracy: A high score may merely reflect majority-class prevalence.

  • Ignoring deployment time: A feature is invalid if it uses information unavailable when the prediction is made.

Useful interview follow-ups include: Which features are available at prediction time? How would you prevent leakage from target encoding? Which metric represents the business cost best? And how would you verify that an apparent improvement persists across time or folds?

26

How should outliers be detected and handled in feature engineering?

Outlier detection and treatment identify observations or feature values that differ substantially from an expected pattern, then correct, transform, cap, or remove them. They expose more stable signal to the model, which often matters more than algorithm choice for tabular data. The main trade-off is robustness versus preserving legitimate rare cases.

Read full answer

Applies toPython 3.12 with pandas 2.2+, NumPy 2.x, and scikit-learn 1.5+

Detection Strategies

Choose a method according to the data type and whether outliers are univariate, multivariate, temporal, or model-relative.

Method How it works Strengths Weaknesses When to use Typical cost
IQR rule Flags values outside q1 - 1.5 * IQR or q3 + 1.5 * IQR Simple and interpretable Ignores relationships between features Skewed numeric features O(n × d × log n) for sorting-based percentiles
Robust z-score Uses the median and median absolute deviation Resistant to extreme values Requires a chosen threshold Robust numeric standardization Approximately O(n × d)
Isolation Forest Uses random partitions; anomalies are isolated in fewer splits Detects multivariate anomalies May flag valid rare cases High-dimensional numeric data Roughly O(n) expected training work and O(n) memory
Model residuals Flags large errors from a fitted baseline Learns expected structure Depends on baseline quality After establishing a baseline model Baseline training plus scoring

For the IQR rule, q1 and q3 are the 25th and 75th percentiles, and IQR = q3 - q1. Treat thresholds as investigation signals, not automatic proof that a record is invalid.

Treatment by Data Type

  • Numeric: Use scaling, log1p for positive right-skewed data, Box-Cox for strictly positive data, or Yeo-Johnson when zeros or negatives occur. Binning, interactions, and polynomial features are also common, but inspect derived features for amplified tails.

  • Categorical: Use one-hot encoding, meaningful ordinal encoding, cross-fitted target encoding, frequency encoding, or hashing. Group rare levels as "other" rather than deleting them automatically.

  • Dates and times: Extract date parts, elapsed time, and cyclical sin/cos features. Apply thresholds to meaningful derived values rather than raw timestamps.

  • Text: Review document length, token counts, and rare-token frequency; TF-IDF, embeddings, and truncation can limit unstable effects.

  • Entity aggregates: Compute counts, means, and rolling statistics with robust statistics, using only information available at prediction time.

  • Feature selection: Use filter methods such as mutual information or correlation, wrappers such as RFE, embedded methods such as L1 or tree importances, and SelectFromModel. Fit selection only on training data.

Remove records only after confirming a data-quality error. Otherwise, capping, transformation, robust models, or retention is usually safer.

Code Example

This example constructs date-part, ratio, and historical entity-aggregate features, learns clipping bounds on training data, and evaluates the complete pipeline consistently.

import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_predict,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


class QuantileClipper(BaseEstimator, TransformerMixin):
    """Clip numeric features using bounds learned during fit."""

    def __init__(self, lower=0.01, upper=0.99):
        self.lower = lower
        self.upper = upper

    def fit(self, X, y=None):
        values = np.asarray(X, dtype=float)
        if not np.isfinite(values).all():
            raise ValueError("Missing values must be imputed before clipping.")
        self.lower_bounds_ = np.quantile(values, self.lower, axis=0)
        self.upper_bounds_ = np.quantile(values, self.upper, axis=0)
        return self

    def transform(self, X):
        values = np.asarray(X, dtype=float)
        if values.shape[1] != len(self.lower_bounds_):
            raise ValueError("Unexpected number of features.")
        return np.clip(values, self.lower_bounds_, self.upper_bounds_)


rng = np.random.default_rng(42)
n = 240
latent = rng.normal(size=n)
account_id = rng.integers(0, 60, n)
income = rng.normal(50_000, 12_000, n) + 15_000 * latent
age = rng.normal(35, 8, n)

outlier_rows = rng.choice(n, size=8, replace=False)
income[outlier_rows] += (
    rng.choice([-1, 1], size=8)
    * rng.uniform(150_000, 300_000, size=8)
)

dates = pd.Series(
    pd.to_datetime("2022-01-01")
    + pd.to_timedelta(rng.integers(0, 730, n), unit="D")
)

# Simulated pre-existing history, independent of the target.
history = pd.DataFrame({
    "account_id": np.repeat(np.arange(60), 5),
    "prior_amount": rng.normal(100, 15, 300),
})
account_means = (
    history.groupby("account_id", as_index=False)["prior_amount"]
    .mean()
    .rename(columns={"prior_amount": "prior_amount_mean"})
)

data = pd.DataFrame({
    "account_id": account_id,
    "income": income,
    "age": age,
    "city": rng.choice(["London", "Paris", "Berlin"], n),
}).merge(account_means, on="account_id", validate="many_to_one")

data["income_age_ratio"] = data["income"] / (data["age"] + 1)
data["signup_year"] = dates.dt.year
data["signup_month"] = dates.dt.month
target = (latent + rng.normal(0, 0.4, n) > 0).astype(int)

indices = np.arange(n)
train_idx, test_idx = train_test_split(
    indices, test_size=0.25, stratify=target, random_state=42
)
X_train, y_train = data.iloc[train_idx], target[train_idx]
X_test, y_test = data.iloc[test_idx], target[test_idx]

numeric_columns = [
    "income", "age", "income_age_ratio",
    "signup_year", "signup_month", "prior_amount_mean",
]
numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("clipper", QuantileClipper(lower=0.01, upper=0.99)),
    ("scaler", StandardScaler()),
])
preprocessor = ColumnTransformer([
    ("numeric", numeric_pipeline, numeric_columns),
    ("categorical", OneHotEncoder(handle_unknown="ignore"), ["city"]),
])
model = Pipeline([
    ("features", preprocessor),
    ("classifier", LogisticRegression(max_iter=1000)),
])


def report(y_true, probabilities):
    predictions = (probabilities >= 0.5).astype(int)
    return {
        "precision": precision_score(y_true, predictions),
        "recall": recall_score(y_true, predictions),
        "f1": f1_score(y_true, predictions),
        "roc_auc": roc_auc_score(y_true, probabilities),
        "pr_auc": average_precision_score(y_true, probabilities),
    }


cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
oof_probabilities = cross_val_predict(
    model, X_train, y_train, cv=cv, method="predict_proba"
)[:, 1]
print("Cross-validation metrics:", report(y_train, oof_probabilities))

model.fit(X_train, y_train)
test_probabilities = model.predict_proba(X_test)[:, 1]
print("Test metrics:", report(y_test, test_probabilities))

clipper = model.named_steps["features"].named_transformers_[
    "numeric"
].named_steps["clipper"]
test_values = X_test[numeric_columns].to_numpy()
outlier_mask = (
    (test_values < clipper.lower_bounds_)
    | (test_values > clipper.upper_bounds_)
)
print("Test values outside training bounds:", int(outlier_mask.sum()))

Validation and Leakage

  • Split before learning clipping bounds, imputations, encodings, aggregates, or scalers.

  • Put learned transformations and feature selection inside a Pipeline so cross-validation fits them on each training fold only.

  • Cross-fit target encodings and preserve temporal order for time-dependent features.

  • Compare treated and untreated pipelines with cross-validation; report precision, recall, F1, ROC-AUC, and PR-AUC for imbalanced classification.

  • A feature store such as Feast can keep offline and online definitions consistent.

Common Pitfalls

  • Treating every statistical extreme as erroneous.

  • Removing valid rare observations without investigation.

  • Computing thresholds or aggregates before splitting.

  • Leaking test or future information into features.

  • Mixing training-set and out-of-fold metrics.

  • Reporting only accuracy for an imbalanced target.

27

How do more features affect overfitting risk in machine learning?

Feature engineering transforms raw data into model inputs that expose useful patterns to a learning algorithm. For tabular data, thoughtful feature engineering often matters more than choosing a more complex algorithm. Adding informative features can reduce bias and improve generalization, but irrelevant or redundant features can increase effective model complexity and raise variance and overfitting risk. Leaky features create a separate failure because they expose information unavailable at prediction time and can make validation results misleading.

Read full answer

Applies tomachine learning concepts across versions; scikit-learn 1.5+ (Python 3.12) for the named preprocessing and feature-selection APIs

The Core Trade-Off

A useful conceptual decomposition is:

prediction error ≈ bias² + variance + irreducible error

  • Bias²: systematic error caused by an overly simple model or insufficiently informative features.

  • Variance: sensitivity of the model to changes in the training sample.

  • Irreducible error: noise that no model can predict reliably.

Good features can lower bias by making the target easier to express. However, additional features give a sufficiently flexible model more opportunities to fit noise. The effective complexity often depends on the number of samples relative to the number of features and on the model’s regularization.

For example, adding age, income, occupation, and many interactions to a dataset with only 50 observations may create thousands of derived columns. A flexible model could obtain excellent training performance while performing poorly on unseen data.

Feature Choices by Data Type

Data type Useful techniques Important cautions
Numeric Scaling, log or Box-Cox/Yeo-Johnson transforms, binning, ratios, interactions, polynomial features Transformations should make relationships more learnable without creating unstable or redundant columns. Scaling is important for k-NN, SVM, PCA, and gradient-descent models, but usually unnecessary for trees.
Categorical One-hot encoding, ordinal encoding, cross-fitted target encoding, frequency encoding, hashing Target encoding and frequency statistics must be computed without using validation or test information. Rare categories and excessive cardinality can cause overfitting.
Dates and times Year, month, weekday, hour, cyclic sin/cos features, time since an event Future timestamps must never influence features used to predict earlier observations. Respect chronological splitting when calculating rolling aggregates.
Text TF-IDF, embeddings, n-grams Vocabulary and document statistics must be learned from the training data only. High-dimensional text features still require regularization or feature selection.
Entity histories Counts, means, trends, and rolling-window aggregates Aggregates must be restricted to information available at prediction time; otherwise temporal or target leakage occurs.

Reducing Overfitting While Adding Features

  • Use cross-validation: Compare candidate feature sets using KFold for ordinary data or StratifiedKFold for classification. Use TimeSeriesSplit for time-dependent observations.

  • Apply regularization: Ridge, Lasso, Elastic Net, or tree regularization can constrain model flexibility even when several features are retained.

  • Select features systematically:

  • Filter methods: Mutual information, correlation, or statistical tests rank features independently of a model.

  • Wrapper methods: RFE repeatedly removes features according to model performance.

  • Embedded methods: Lasso coefficients or tree-based importances identify useful features during training.

  • SelectFromModel: Uses a fitted model’s importance or coefficient values to select features.

  • Control dimensionality: Remove duplicates, limit highly correlated features, and avoid creating every possible interaction by default.

  • Use proper pipelines: Fit imputers, encoders, scalers, and feature selectors on training data only. Put column-specific preprocessing in a ColumnTransformer and that transformer in a Pipeline. Scikit-learn 1.5+ provides sklearn.preprocessing.TargetEncoder, which cross-fits during fit_transform and can be used as a ColumnTransformer step; fit it within each training fold.

  • Check learning curves: If performance improves on training data but not validation data as features are added, added complexity is probably overfitting.

Practical Decision Framework

Before adding a feature, ask:

  1. Does it represent information genuinely available at prediction time?

  2. Does it have a plausible relationship with the target?

  3. Does it improve cross-validated performance, not just training performance?

  4. Is its benefit stable across folds, subgroups, and time periods?

  5. Is the added storage and computation justified?

Feature stores such as Feast can help teams define, version, and reuse features consistently between training and serving. This reduces accidental mismatches, but a feature store does not automatically prevent leakage; feature definitions and update times must still be correct.

Interview Tips

Common mistakes include selecting features before splitting data, computing target encodings on the full dataset, using future information for historical rows, evaluating many feature combinations on the validation set, and relying on accuracy for imbalanced classification. For imbalanced problems, inspect precision, recall, F1, ROC-AUC, and PR-AUC; accuracy can remain high when the minority class is rarely predicted.

A useful follow-up answer is that feature selection and regularization are related but different: feature selection removes input variables, while regularization changes how strongly the model uses the remaining variables. Interviewers may also ask how you would distinguish leakage from ordinary overfitting, how you would tune a feature pipeline without repeatedly overfitting the validation fold, or why cross-fitted target encoding is safer than encoding each category using the entire training set.

28

How do you evaluate engineered feature effectiveness?

Feature effectiveness is the measurable, out-of-sample improvement a model gains from engineered features. Feature engineering turns raw data into inputs that expose useful signal, which often matters more than algorithm choice for tabular data. The main trade-off is better signal versus increased complexity, overfitting, and leakage risk.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12 with pandas 2.2+ and scikit-learn 1.5+)

Evaluation Strategy

Compare the same model and identical folds with and without each engineered feature group.

Method What it measures
Baseline versus engineered features Overall incremental value
Leave-one-group-out ablation Value of a feature or feature family
Permutation importance A model’s reliance on a feature
Fold-by-fold differences Stability of the improvement
Out-of-time evaluation Generalization under deployment conditions
  • Regression: use MAE, RMSE, and R².

  • Classification: use precision, recall, F1, ROC-AUC, and PR-AUC for imbalanced classes. Accuracy can mislead when class frequencies differ.

  • Compare performance on unseen data, not training performance.

  • Check missingness, outliers, cardinality, distribution shift, and availability at prediction time.

  • Prefer paired fold-level comparisons or confidence intervals when deciding whether an improvement is credible.

Engineering and Leakage Checks

  • Numeric: scaling, log or Yeo-Johnson transformations, binning, interactions, and polynomial features. Scaling matters for k-NN, SVM, PCA, and gradient-descent optimization, but usually not for trees.

  • Categorical: one-hot, ordinal, frequency, hashing, and cross-fitted target encoding.

  • Date and time: date parts, cyclic sin/cos encodings, and time since an event.

  • Text: TF-IDF or pretrained embeddings.

  • Entities: counts, means, and rolling-window aggregates computed from information available at prediction time.

Feature-selection methods include mutual-information and correlation filters, RFE, L1 regularization, tree importances, and SelectFromModel. Evaluate selection inside a leak-free pipeline or nested cross-validation.

Fit scalers, encoders, imputers, target encoders, and aggregates using training folds only. Target encoding should be cross-fitted, and time-dependent features must respect temporal order. Use fixed seeds, record library versions, and do not repeatedly tune against the same validation set. A feature store such as Feast can keep training and serving definitions consistent. Preprocessing usually benefits less from GPU acceleration than the downstream model does.

Code Example

This regression example compares raw inputs with date parts, a ratio, and an entity aggregate. Data are ordered chronologically, and every learned transformation remains inside the pipeline.

import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import (
    TimeSeriesSplit, cross_validate, train_test_split
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
n = 400

entity_number = rng.integers(0, 8, n)
entity_effect = np.array([-2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5])
dates = pd.to_datetime("2024-01-01") + pd.to_timedelta(
    rng.integers(0, 365, n), unit="D"
)

data = pd.DataFrame({
    "amount": rng.lognormal(mean=2.0, sigma=0.4, size=n),
    "latency": np.clip(
        rng.normal(15 + 2 * entity_effect[entity_number], 3, n), 1, None
    ),
    "category": rng.choice(["A", "B", "C"], size=n),
    "entity": [f"E{i:02d}" for i in entity_number],
    "signup_date": dates,
})

ratio = data["amount"] / (data["latency"] + 1)
day = (data["signup_date"] - pd.Timestamp("2024-01-01")).dt.days
y = (
    2 * entity_effect[entity_number]
    + 1.5 * np.sin(2 * np.pi * day / 365)
    + 1.2 * ratio
    + rng.normal(0, 0.5, n)
)

order = np.argsort(data["signup_date"].to_numpy())
data = data.iloc[order].reset_index(drop=True)
y = y[order]

class FeatureBuilder(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        return self

    def transform(self, X):
        out = X.copy()
        dt = pd.to_datetime(out["signup_date"])
        out["month"] = dt.dt.month.astype(float)
        out["day_of_week"] = dt.dt.dayofweek.astype(float)
        out["amount_ratio"] = out["amount"] / (out["latency"] + 1)
        return out


class EntityAggregateTransformer(BaseEstimator, TransformerMixin):
    def __init__(self, value_col, group_col, output_col):
        self.value_col = value_col
        self.group_col = group_col
        self.output_col = output_col

    def fit(self, X, y=None):
        self.group_mean_ = X.groupby(self.group_col)[self.value_col].mean()
        self.global_mean_ = float(X[self.value_col].mean())
        return self

    def transform(self, X):
        out = X.copy()
        means = out[self.group_col].map(self.group_mean_)
        out[self.output_col] = means.fillna(self.global_mean_)
        return out

    def fit_transform(self, X, y=None, **fit_params):
        self.fit(X)
        out = self.transform(X)
        sums = out.groupby(self.group_col)[self.value_col].transform("sum")
        counts = out.groupby(self.group_col)[self.value_col].transform("count")
        values = out[self.value_col]
        loo = (sums - values) / (counts - 1).replace(0, np.nan)
        out[self.output_col] = loo.fillna(self.global_mean_)
        return out


baseline = Pipeline([
    ("preprocess", ColumnTransformer([
        ("numeric", StandardScaler(), ["amount", "latency"]),
        ("categorical", OneHotEncoder(
            handle_unknown="ignore", sparse_output=False
        ), ["category"]),
    ])),
    ("model", Ridge(alpha=1.0)),
])

engineered = Pipeline([
    ("features", FeatureBuilder()),
    ("aggregate", EntityAggregateTransformer(
        "latency", "entity", "entity_latency_mean"
    )),
    ("preprocess", ColumnTransformer([
        ("numeric", StandardScaler(), [
            "amount", "latency", "month", "day_of_week",
            "amount_ratio", "entity_latency_mean"
        ]),
        ("categorical", OneHotEncoder(
            handle_unknown="ignore", sparse_output=False
        ), ["category"]),
    ])),
    ("model", Ridge(alpha=1.0)),
])

X_train, X_test, y_train, y_test = train_test_split(
    data, y, test_size=0.2, shuffle=False
)
scoring = {
    "mae": "neg_mean_absolute_error",
    "rmse": "neg_root_mean_squared_error",
    "r2": "r2",
}
cv = TimeSeriesSplit(n_splits=5)

for name, model in [("Baseline", baseline), ("Engineered", engineered)]:
    scores = cross_validate(model, X_train, y_train, cv=cv, scoring=scoring)
    model.fit(X_train, y_train)
    prediction = model.predict(X_test)
    print(
        name,
        "CV MAE:", -scores["test_mae"].mean(),
        "CV RMSE:", -scores["test_rmse"].mean(),
        "CV R2:", scores["test_r2"].mean(),
        "Test MAE:", mean_absolute_error(y_test, prediction),
        "Test RMSE:", mean_squared_error(y_test, prediction) ** 0.5,
        "Test R2:", r2_score(y_test, prediction),
    )

Common Pitfalls

  • Fitting transformations before splitting or computing future information.

  • Comparing feature sets with different models, folds, or tuning budgets.

  • Treating correlated feature importance as causal importance.

  • Optimizing repeatedly against one validation split.

  • Ignoring missing values, class imbalance, temporal drift, serving-time availability, and computational cost.

29

How does feature engineering mitigate the curse of dimensionality?

The curse of dimensionality describes how data becomes sparse and distances become less informative as the number of features grows, degrading methods such as k-NN and making models prone to overfitting. Feature engineering transforms raw data into model-ready inputs that expose predictive signal and, when carefully designed, produce compact representations with less noise and redundancy. Its main trade-off is improved signal versus greater engineering complexity, maintenance effort, and leakage risk; for tabular data, it often matters as much as or more than the choice of algorithm.

Read full answer

Applies tomachine learning concepts (all versions)

How Feature Engineering Helps

For two observations, Euclidean distance is:

distance(x, y) = sqrt(sum from i=1 to d of (x_i - y_i)^2)

Here, x and y are observations, d is the number of features, i is the feature index, and x_i and y_i are their values for feature i. As d increases with a fixed sample size:

  • Distance concentration: nearest and farthest observations become less distinguishable.

  • Sparsity: observations occupy an increasingly small portion of the available space.

  • Sample requirements: uniformly covering the space can require exponentially more samples.

  • Variance: models may fit noise in irrelevant columns.

  • Computational cost: distance calculations and dense model inputs grow with the feature count; for example, a k-NN distance query costs approximately O(n * d).

Feature engineering can mitigate these problems by:

  • Distilling information: Replacing correlated measurements with domain-driven ratios, indices, or aggregate features.

  • Reducing noise: Using logarithms, transformations, smoothing, or robust scaling to make genuine patterns easier to model.

  • Creating compact representations: Using embeddings, hashing, carefully chosen aggregates, or feature selection instead of retaining many raw columns.

  • Exposing useful structure: Encoding seasonality, interactions, ratios, and time since an event so models do not need to infer them from many weak inputs.

  • Improving distance geometry: Scaling can prevent one large-range feature from dominating a distance calculation, although scaling alone does not reduce dimensionality.

Feature engineering does not automatically solve the curse of dimensionality. Creating many one-hot columns, polynomial terms, or interactions can instead increase sparsity, computation, and overfitting.

Feature Engineering by Data Type

Data type Useful techniques Important consideration
Numeric Scaling, log/Yeo-Johnson transforms, binning, interactions, polynomial features Scaling is normally required for k-NN, SVM, PCA, and gradient descent; trees generally do not require it.
Categorical One-hot encoding, ordinal encoding, cross-fitted target encoding, frequency encoding, hashing One-hot encoding can increase dimensionality, so it is primarily suitable for low-cardinality features.
Dates and times Year, month, weekday, season, time since an event, cyclic sin/cos encoding Features must not reveal information unavailable at prediction time.
Text TF-IDF, embeddings, token or phrase indicators TF-IDF can be high-dimensional; embeddings often provide a more compact representation.
Entity history Counts, means, trends, recency, rolling-window aggregates Compute aggregates from the appropriate historical window, not from future observations.

Feature Selection and Leakage Control

Feature selection removes features that add little signal and complements the dimensionality reduction achieved through feature engineering:

  • Filter methods: Mutual information or correlation; inexpensive but may miss interactions.

  • Wrapper methods: RFE evaluates feature subsets but can be expensive.

  • Embedded methods: L1 regularization or tree-based importances identify importance while fitting a model.

  • SelectFromModel: Retains features according to a fitted model’s coefficient or importance threshold.

Place selection inside cross-validation so each fold selects features using only its training partition. Use StratifiedKFold for classification and TimeSeriesSplit for temporal data. Target encodings must be cross-fitted, and entity aggregates must respect information available at prediction time. Feature stores such as Feast help reuse consistent feature definitions and reduce training-serving skew.

Common Pitfalls

  • Data leakage: Computing target encodings, aggregates, selections, or imputation statistics before splitting produces unrealistically strong validation results.

  • Ignoring imbalance: Accuracy can look high when the minority class is rare; inspect precision, recall, F1, ROC-AUC, and PR-AUC.

  • Creating too many features: One-hot encoding or indiscriminate interactions may worsen sparsity and overfitting.

  • Using inappropriate preprocessing: Scaling is generally unnecessary for trees but important for distance- and gradient-based models.

  • Ignoring missing values and outliers: Transformations may create NaN or infinite values; validate and impute them within the training pipeline.

  • Overfitting the validation set: Use cross-validation rather than repeatedly tuning against one holdout.

  • Ignoring temporal order: Random splits can leak future patterns into training data.

30

How do you engineer features for time-series data?

Time-series feature engineering transforms timestamped observations into predictors that expose temporal signal to a model. It uses lags, rolling statistics, calendar and cyclic components, ratios, and entity aggregates; the main trade-off is useful context versus leakage and overfitting. For tabular data, thoughtful features often matter more than switching to a more complex algorithm.

Read full answer

Applies topandas 2.2+, scikit-learn 1.5+, NumPy 2.x (Python 3.12)

Core Approach

Every feature must contain information available at the prediction origin:

  • Lagged values: previous observations such as sales_lag_1 or sales_lag_7.

  • Rolling statistics: shifted rolling means, standard deviations, sums, or counts.

  • Calendar and cyclic features: hour, weekday, month, holidays, and sine/cosine encodings.

  • Derived values: ratios, differences, time since an event, and interactions.

  • Entity features: per-customer, store, device, or region history.

Sort by time before computing features, and calculate lags and rolling statistics independently within each entity.

Techniques by Data Type

Data type Useful techniques Main safeguard
Numeric Scaling, log, Box-Cox, Yeo-Johnson, binning, ratios, interactions, polynomial features Match the transform to the distribution and scale when the model is sensitive to feature magnitude
Categorical One-hot, ordinal, frequency, hashing, cross-fitted target encoding Fit encoders on training data only
Date and time Parts, holidays, elapsed time, sine/cosine encoding Handle time zones, daylight-saving changes, and forecast horizon
Text TF-IDF, embeddings, token counts, sentiment Fit text vectorizers only on training data
Entity aggregates Counts, means, sums, trends, rolling windows Exclude unavailable future observations

Leakage-Safe Validation

Split chronologically and keep preprocessing and models inside a scikit-learn Pipeline, ensuring that imputers, scalers, encoders, and feature selectors learn only from each training fold. TimeSeriesSplit is preferable to shuffled cross-validation for temporal data.

For multi-step forecasting, future targets are unavailable; use recursive forecasting, direct horizon models, or only features known at the forecast origin. The example below is a rolling one-step backtest: when a date is scored, earlier dates’ actual sales are assumed to have been observed by then. It is not a fixed-origin multi-step backtest.

Feature Selection and Reuse

  • Filter methods: correlation or mutual information.

  • Wrapper methods: recursive feature elimination (RFE).

  • Embedded methods: L1 regularization, tree importances, or SelectFromModel.

Place feature selection inside the pipeline so validation data cannot influence it. A feature store such as Feast can version and reuse feature definitions consistently between training and serving.

Code Example

import numpy as np
import pandas as pd

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestRegressor
from sklearn.impute import SimpleImputer
from sklearn.metrics import mean_absolute_error, r2_score, root_mean_squared_error
from sklearn.model_selection import TimeSeriesSplit, cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
n_days = 180
dates = pd.date_range("2023-01-01", periods=n_days, freq="D")
time_index = np.arange(n_days, dtype=float)

# Synthetic daily data for two regions.
df = pd.DataFrame({
    "date": dates,
    "region": np.tile(["North", "South"], n_days // 2),
    "price": rng.normal(20, 3, n_days),
    "ad_spend": rng.uniform(1, 10, n_days),
    "promo": rng.integers(0, 2, n_days),
})
df["sales"] = (
    100
    + 30 * (df["region"] == "South")
    + 20 * (df["promo"] == 1)
    + 15 * np.sin(2 * np.pi * df["date"].dt.dayofweek / 7)
    + 0.05 * time_index
    + rng.normal(0, 5, n_days)
)

def add_features(data):
    # Sort each entity chronologically before calculating temporal features.
    result = data.sort_values(["date", "region"]).copy()
    result["day_of_week"] = result["date"].dt.dayofweek
    result["month"] = result["date"].dt.month
    result["sin_day"] = np.sin(
        2 * np.pi * result["date"].dt.dayofyear / 365.25
    )
    result["cos_day"] = np.cos(
        2 * np.pi * result["date"].dt.dayofyear / 365.25
    )

    grouped = result.groupby("region", sort=False)["sales"]
    result["lag_1"] = grouped.shift(1)
    result["lag_7"] = grouped.shift(7)

    # Shift first so the current target never enters its own rolling feature.
    result["rolling_7_mean"] = grouped.transform(
        lambda values: values.shift(1).rolling(7, min_periods=3).mean()
    )
    result["price_ratio"] = result["price"] / result["ad_spend"]
    return result

features = add_features(df)
X = features.drop(columns=["date", "sales"])
y = features["sales"]

# Chronological holdout; preprocessing is fitted only on X_train.
split = int(len(features) * 0.8)
X_train, X_test = X.iloc[:split], X.iloc[split:]
y_train, y_test = y.iloc[:split], y.iloc[split:]

numeric_features = [
    "lag_1", "lag_7", "rolling_7_mean", "day_of_week", "month",
    "sin_day", "cos_day", "price", "ad_spend", "promo", "price_ratio"
]
numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])
categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore")),
])
preprocessor = ColumnTransformer([
    ("numeric", numeric_pipeline, numeric_features),
    ("categorical", categorical_pipeline, ["region"]),
])
model = Pipeline([
    ("preprocessor", preprocessor),
    ("regressor", RandomForestRegressor(
        n_estimators=200, random_state=42, n_jobs=-1
    )),
])

model.fit(X_train, y_train)
predictions = model.predict(X_test)
print("Test MAE:", round(mean_absolute_error(y_test, predictions), 2))
print("Test RMSE:", round(root_mean_squared_error(y_test, predictions), 2))
print("Test R^2:", round(r2_score(y_test, predictions), 2))

# Expanding-window, rolling one-step temporal cross-validation.
cv = TimeSeriesSplit(n_splits=3)
scores = cross_validate(
    model, X_train, y_train, cv=cv,
    scoring={
        "mae": "neg_mean_absolute_error",
        "rmse": "neg_root_mean_squared_error",
        "r2": "r2",
    },
)
print("CV MAE:", round(-scores["test_mae"].mean(), 2))
print("CV RMSE:", round(-scores["test_rmse"].mean(), 2))
print("CV R^2:", round(scores["test_r2"].mean(), 2))

Common Pitfalls

  • Leakage: never include current or future targets in rolling or aggregate features.

  • Random splitting: it breaks temporal order and usually inflates performance.

  • Wrong metrics: use MAE, RMSE, and R² for regression; accuracy can mislead for imbalanced classification.

  • Missingness: missing recent observations may itself carry signal, so do not blindly discard it.

  • Overfitting: correlated temporal features can memorize a training period; test multiple time windows.

  • Reproducibility: track seeds, preprocessing, feature definitions, and data versions.

31

How do you create interaction features using domain knowledge?

Interaction features are new variables formed by combining two or more existing inputs so a model can capture relationships that are hard to learn from the original columns alone. Domain knowledge guides which combinations are meaningful (for example, rates, ratios, or products tied to real-world behavior), which helps expose signal in tabular data without blindly adding noise. The main trade-off is that useful interactions can improve performance, but extra or poorly chosen features increase dimensionality, overfitting risk, and the chance of data leakage if they use information unavailable at prediction time.

Read full answer

Applies toscikit-learn 1.5+ and pandas 2.2+ (Python 3.12)

How Domain Knowledge Guides Interactions

  • Start from the mechanism: Ask how variables act together in the real system (cost per unit, load per area, speed from distance and time) rather than generating all pairs.

  • Choose the right operation: Use ratios for efficiency or density (sales / visits), products for joint effects (age * dosage), differences for gaps (due_date - event_date), and thresholds for regime changes.

  • Respect units and constraints: Ensure denominators are nonzero, units are compatible, and the interaction is defined for all valid rows.

  • Validate, do not assume: Compare models with and without the feature using cross-validation and a metric that fits the problem (for imbalanced classification use precision, recall, F1, ROC-AUC, or PR-AUC; for regression use MAE, RMSE, or R^2, since R^2 alone can hide error scale).

Examples by Data Type and Domain

Data type Domain-informed interaction Why it helps
Numeric area_sqft / rooms, price / quantity Captures density or unit cost better than raw totals
Categorical + numeric Mean spend by segment, or segment-specific rate Encodes group behavior; compute aggregates on training folds only
Date/time days_since_event, cyclic sin/cos of hour or month Preserves recency and periodicity
Counts over windows Orders per customer in past 30 days Summarizes entity history; respect time order

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n = 300
df = pd.DataFrame({
    "area_sqft": rng.uniform(500, 3000, n),
    "rooms": rng.integers(1, 7, n),
    "age_years": rng.uniform(0, 80, n),
    "dist_km": rng.uniform(0.5, 30, n),
})

# Domain-informed interactions: all inputs are available at prediction time.
df["area_per_room"] = df["area_sqft"] / df["rooms"]  # housing density
df["age_x_dist"] = df["age_years"] * df["dist_km"]   # joint location/age effect

y = (2 * df["area_sqft"] - 500 * df["age_years"] - 300 * df["dist_km"]
     + rng.normal(0, 1000, n))

X_train, X_test, y_train, y_test = train_test_split(
    df, y, test_size=0.2, random_state=42
)

num_cols = df.columns.tolist()
preprocess = ColumnTransformer([("num", StandardScaler(), num_cols)])
model = Pipeline([("prep", preprocess), ("reg", Ridge(random_state=42))])
model.fit(X_train, y_train)

print(f"Test R^2: {model.score(X_test, y_test):.3f}")
# R^2 is typically high for this synthetic linear signal (varies slightly by version)
print("Feature count:", len(num_cols))  # 6

Validation and Leakage Considerations

  • No future information: Row-wise interactions from raw inputs are safe; target encodings and per-entity aggregates must be computed on training folds only (use cross-fitting) and, for time series, with TimeSeriesSplit.

  • Keep preprocessing in a Pipeline: Fit scalers and encoders on training data only; trees do not require scaling, but linear models, k-NN, SVM, PCA, and gradient descent do.

  • Check stability: A domain interaction should improve cross-validated performance consistently, not just on one split.

Common Pitfalls

  • Target leakage: Creating features from the target or post-outcome data (for example, price / target_price) inflates scores and fails in production.

  • Uncontrolled feature explosion: Generating all polynomial pairs (PolynomialFeatures) without selection can overfit; prefer domain-selected interactions or embedded selection such as L1 or tree importances.

  • Ignoring data issues: Division by zero, missing values, and rare categories can break interactions; impute and guard denominators.

  • Interview follow-ups: How do you detect leakage? When would you use target encoding versus one-hot? How do you validate an interaction added to a time-series model?

32

What is polynomial feature expansion in machine learning?

Polynomial feature expansion creates new numeric features by raising existing features to powers and multiplying them together, so a linear model can learn curved relationships in the original inputs. It solves the problem of underfitting when the true pattern is nonlinear but you want to keep an interpretable linear estimator. Its main trade-off is that the number of generated features grows quickly with the original feature count and the chosen degree, which increases computation and the risk of overfitting.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How It Works

  • For input features x1, ..., xn and degree d, polynomial expansion generates every term of the form x1^a1 * x2^a2 * ... * xn^an, where a1, ..., an are non-negative integers and a1 + ... + an <= d.

  • x1, ..., xn: the original numeric features. d: the maximum total degree. a1, ..., an: the exponent assigned to each feature in a generated term.

  • Example: with x1, x2 and d = 2, the expansion produces 1, x1, x2, x1^2, x1*x2, x2^2. The constant 1 is a bias term and can be omitted if the model already adds an intercept.

  • The output feature count grows combinatorially: roughly C(n + d, d) terms for n features and degree d (including the bias). With interaction_only=True, only products of distinct features are kept (no powers like x1^2).

When to Use and Trade-offs

Aspect Details
How it helps Lets linear and logistic regression capture interactions and curvature without switching to a nonlinear model.
Strengths Simple, interpretable with linear models, useful for small tabular datasets with smooth nonlinear effects.
Weaknesses Feature count and memory grow fast; high-degree terms amplify noise and outliers; coefficients become harder to interpret.
When to use Few numeric features, suspected interactions or mild nonlinearity, and you can regularize (Ridge, Lasso, Elastic Net).
When to avoid Many features, very high degree, sparse/high-dimensional data, or when tree-based models already capture interactions well.

Common Pitfalls

  • Fitting before splitting: applying PolynomialFeatures or StandardScaler to the full dataset leaks test information. Keep preprocessing inside a Pipeline fitted on training folds only.

  • Skipping scaling: polynomial terms can have very different magnitudes (for example x vs x^3). Scale features when using gradient descent, SVM, or regularized linear models; scaling is not required for tree models.

  • Choosing too high a degree: degree 3 or 4 can explode feature count and overfit. Start at degree 2, use cross-validation, and add regularization.

  • Forgetting regularization: expanded features are often correlated. Ridge or Lasso usually performs better than plain linear regression.

  • Interview follow-ups: How does polynomial expansion differ from kernel methods (e.g., polynomial kernel SVM)? How do you control overfitting? When would you prefer trees or splines instead?

33

Which feature engineering techniques are specific to text data?

Text feature engineering converts raw documents into numeric representations that expose lexical, structural, and semantic signal to a model. It solves the problem that most machine-learning estimators cannot consume strings directly, while balancing interpretability and simplicity against richer but more computationally expensive representations.

Read full answer

Applies tomachine-learning feature engineering (all versions)

Text-Specific Techniques

  • Cleaning and normalization: Lowercase text, remove irrelevant punctuation, normalize accents, and handle HTML or Unicode. Preserve case, punctuation, and negation when they carry meaning.

  • Tokenization: Split text into words, subwords, or characters. Subwords are useful for rare words, spelling variations, and languages with large vocabularies.

  • Stop-word removal, stemming, and lemmatization: Reduce vocabulary size by removing common words or reducing words to roots, but avoid aggressive removal when exact terms, negation, or domain vocabulary matter.

  • Bag-of-words: Represent documents with word counts, binary occurrence indicators, or character counts. These features are fast but ignore word order.

  • TF-IDF: Weight terms by their frequency within a document and rarity across the corpus. A smoothed form is TF-IDF(t, d) = TF(t, d) * log((1 + N) / (1 + DF(t)) + 1), where t is a term, d is a document, N is the number of training documents, and DF(t) is the number of training documents containing that term.

  • N-grams: Word n-grams capture local phrases, while character n-grams capture spelling, morphology, and word fragments. They can improve robustness but increase dimensionality.

  • Embeddings: Map words, sentences, or documents to dense vectors representing semantic relationships. They capture meaning beyond exact overlap but are less interpretable and require an embedding model.

  • Hashing: Map tokens into a fixed number of dimensions without retaining a vocabulary. It supports large-scale or streaming processing, although hash collisions can discard information.

  • Text-derived metadata: Add document length, language, source channel, named entities, topic, sentiment, or readability when these are available at prediction time.

  • Historical text aggregates: For text linked to users or documents, compute prior counts, recent frequency, average sentiment, or topic distributions using only earlier records.

Choosing a Representation

Technique How it works Strengths Weaknesses Typical scale or complexity
TF-IDF Weights terms by document frequency and corpus rarity Fast, sparse, interpretable Misses synonyms and broad context Construction is generally linear in processed tokens; sparse storage scales with vocabulary size
N-grams Adds adjacent words or character fragments Captures phrases, spelling, and morphology Larger feature space Word n-grams add sequence combinations; character n-grams grow with document length
Embeddings Encodes text as dense semantic vectors Handles similarity and context Less interpretable; additional dependencies Dense storage scales with vector width; inference depends on the embedding model
Hashing Hashes tokens into fixed dimensions Memory efficient and parallelizable Possible collisions; no natural vocabulary Processing is generally linear in tokens with fixed-dimensional output

Techniques That Are Not Text-Specific

  • Numeric data: Scaling, logarithmic, Box-Cox, or Yeo-Johnson transformations, binning, interactions, and polynomial features also apply to numerical variables.

  • Categorical data: One-hot, ordinal, frequency, and cross-fitted target encoding apply to categorical variables generally. Target encodings must be learned within training folds to avoid target leakage.

  • Dates and times: Extracting date parts, cyclic sine/cosine encodings, and time since an event are date-specific rather than text-specific.

  • Aggregations: Counts, means, rates, and rolling-window aggregates are generic features that may use text-derived values but are not limited to text.

Feature Selection

Text matrices can contain many sparse columns, so feature selection is often useful:

  • Filter methods: Use chi-square tests, mutual information, document frequency, or correlation to remove uninformative columns.

  • Wrapper methods: Use recursive feature elimination, or RFE, to evaluate feature subsets with a chosen estimator. These methods can be expensive.

  • Embedded methods: Use L1-regularized linear-model coefficients or tree-based importances to identify useful features.

  • SelectFromModel: Retains features according to coefficients or importances learned by a supervised model. Place it inside a pipeline so it is fitted only on training folds.

Best Practices and Serving

  • Split the data before fitting TfidfVectorizer, encoders, selectors, or other learned transformations. For classification cross-validation, use StratifiedKFold and keep the complete preprocessing pipeline within each fold.

  • Fit vocabulary, IDF values, normalization rules, and feature selection only on training data; use transform for validation, test, and production data.

  • Use the same tokenizer, vocabulary, IDF values, normalizer, and model version during training and inference.

  • Ensure every feature is available at prediction time. For time-dependent text, calculate historical aggregates strictly from earlier events.

  • Handle missing or null documents explicitly rather than allowing them to fail silently during vectorization.

  • Set random seeds where applicable and retain preprocessing artifacts so the pipeline is reproducible and auditable.

  • For imbalanced classification, use precision, recall, F1, ROC-AUC, or PR-AUC as appropriate. Accuracy can be misleading when a minority class is rare because a high-scoring majority classifier can achieve deceptively high accuracy.

  • Embedding inference may benefit from GPU acceleration for large batches, whereas TF-IDF and linear models are usually efficient on CPUs.

  • Feature stores such as Feast can centralize reusable definitions and reduce training-serving inconsistency.

  • Feature engineering often matters more than algorithm choice for tabular data. For text, compare meaningful representations with the same estimator before adopting a more expensive architecture.

Common Pitfalls

  • Data leakage: Fitting a vectorizer, target encoder, or feature selector before splitting or outside cross-validation folds.

  • Removing meaningful tokens: Aggressive stop-word removal can discard negation such as “not good.”

  • Ignoring context: Unigram TF-IDF treats “bank” in “river bank” and “bank account” as the same term.

  • Excessive dimensionality: Large vocabularies increase memory use and overfitting risk.

  • Unstable preprocessing: Different tokenization or normalization between training and serving creates distribution shift.

  • Poor evaluation: A small test set can make text-classification scores unstable; report appropriate metrics and use cross-validation.

Common follow-up questions include how TF-IDF differs from embeddings, how vocabulary leakage is prevented, when stemming helps multilingual text, and how training-serving consistency is maintained.

34

How does deep learning automate feature engineering?

Deep learning automates feature engineering by learning useful transformations and latent representations directly from data instead of requiring every derived feature to be specified manually. It can discover interactions and nonlinear structure, especially in text, images, and sequences, although hand-engineered features remain important for many tabular problems and can matter more than the choice of algorithm. The main trade-off is reduced manual effort in exchange for greater data, compute, tuning, and interpretability requirements.

Read full answer

Applies todeep learning concepts (all versions), with scikit-learn 1.5+ feature-selection concepts

How It Works

A neural network optimizes a task-specific loss while its layers progressively transform the input:

  1. Input preparation: Clean data, represent categorical values, handle missing values, and split training and evaluation data.

  2. Representation learning: Hidden layers combine inputs and learn nonlinear interactions.

  3. Feature extraction: Intermediate activations become a learned feature representation.

  4. Prediction or reconstruction: The network predicts the target or reconstructs its input.

  5. Deployment: Training, preprocessing, and serving must apply the same transformations.

An autoencoder commonly minimizes:

reconstruction_loss = (1/n) * sum((x_i - x_hat_i)^2)

Here, n is the number of examples, x_i is an original feature vector, and x_hat_i is its reconstruction. The loss is the mean squared difference between the original and reconstructed values.

Techniques by Data Type

Data type Conventional feature engineering What deep learning can automate
Numeric Scaling, log/Box-Cox/Yeo-Johnson transforms, binning, interactions, and polynomial features Learned nonlinear transformations, interactions, and latent factors
Categorical One-hot or ordinal encoding, cross-fitted target encoding, frequency encoding, and hashing Entity embeddings and learned category relationships
Date and time Date parts, cyclic sine/cosine encoding, and time since an event Temporal convolutions, attention, and learned lag or seasonality patterns
Text TF-IDF, tokenization, and n-grams Contextual embeddings and representations learned by language models
Entity aggregates Counts, means, and rolling-window statistics Neural sequence models and attention over historical records

Deep learning does not eliminate preprocessing. Numerical scaling can improve neural-network optimization; categorical identifiers need a defined encoding or embedding lookup; and missing values require an explicit policy. It also does not automatically discover useful aggregate features without suitable historical sequences and entity identifiers.

Feature Selection and Representations

  • Filter methods: Rank features with measures such as mutual information or correlation.

  • Wrapper methods: Use model performance to search for useful subsets, for example with recursive feature elimination.

  • Embedded methods: Use regularization or tree importances learned during model training, such as L1 penalties or tree-based importances.

  • Model-based selection: Methods such as SelectFromModel retain features according to a fitted estimator’s importance or coefficient criteria.

  • Neural selection: Neural networks can learn feature weights, gates, or masks, although the selected representation may be unstable and difficult to interpret.

Entity embeddings compact high-cardinality categories and can reveal similarity between customers, products, or locations. Autoencoders learn nonlinear compressed representations without labels, but they may discard rare yet predictive information. Supervised networks optimize directly for the target, while attention and convolution identify local or global patterns in sequences, text, images, and sensor data.

Practical Guidance

  • Split before learning: Fit scalers, encoders, target encodings, autoencoders, and feature selectors on training data only. Cross-fit target encodings and compute aggregates from training folds only.

  • Respect time order: Use TimeSeriesSplit or chronological windows for forecasting; random folds can expose future information.

  • Compare fairly: Use a strong linear or tree-based baseline and cross-validation such as StratifiedKFold for classification.

  • Choose suitable metrics: Accuracy can mislead with imbalanced classes; inspect precision, recall, F1, ROC-AUC, or PR-AUC as appropriate.

  • Control overfitting: Use validation-based early stopping, regularization, and careful tuning. Reproduce runs with fixed seeds.

  • Plan for serving: Store preprocessing metadata and learned artifacts. Feature stores such as Feast can help reuse feature definitions consistently between training and serving.

  • Use hardware appropriately: GPUs accelerate large neural models, while small tabular datasets may train faster on CPUs with conventional models.

Common Pitfalls

  • Data leakage: Computing target encodings, aggregates, scalers, or representations before splitting invalidates evaluation results.

  • Unsupervised information loss: An autoencoder may compress away rare but predictive signals.

  • Opaque features: Learned representations are harder to explain, monitor, and correct manually.

  • Missing categorical coverage: Unseen categories may have poorly trained embeddings or inconsistent serving behavior.

  • Wrong metric: A high accuracy can conceal poor minority-class performance.

  • Unjustified complexity: Deep learning is not automatically superior to simpler feature pipelines for small or structured datasets.

35

How can genetic algorithms be used for feature engineering?

Genetic algorithms (GAs) can be used for feature engineering by treating feature subsets or feature transformations as a population of candidates that evolves through selection, crossover, and mutation to maximize a model-based fitness score. They help search large, combinatorial feature spaces where exhaustive search is impractical. Their main trade-off is search flexibility versus cost: GAs can discover useful combinations but are computationally expensive, stochastic, and prone to overfitting if fitness is evaluated with data leakage.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

How Genetic Feature Selection Works

  • Chromosome encoding: represent a candidate as a binary mask [1, 0, 1, ...] where 1 keeps a feature and 0 drops it. For feature construction (genetic programming/symbolic regression), a chromosome can instead encode an expression tree of operators and inputs.

  • Fitness function: score each candidate with cross-validated model performance (for example, F1 for imbalanced classification, PR-AUC for rare positives, or RMSE for regression). Fitness must be computed on training folds only, with any scalers or encoders fitted inside each fold (for example, via an sklearn.pipeline.Pipeline) to prevent leakage.

  • Selection: prefer higher-scoring candidates (tournament or roulette selection) so useful feature combinations survive to the next generation.

  • Crossover and mutation: combine parts of two parent masks (for example, single-point crossover) to explore new subsets, and randomly flip bits at a small mutation rate to maintain diversity and avoid local optima.

  • Stopping and validation: run for a fixed number of generations or until fitness stops improving, select the best mask, and evaluate the final model on a held-out test set that the GA never saw.

Comparison with Other Feature Selection Methods

Method How it works Strengths Weaknesses When to use
GA (wrapper) Evolves feature subsets using model fitness Captures feature interactions; flexible search Slow (population * generations * CV folds fits); stochastic; leakage risk Medium feature sets where interactions matter
Filter (mutual information, correlation) Scores features independently of the model Fast, simple Misses interactions; model-agnostic Quick baseline, very high dimensions
Embedded (L1, tree importances) Selection happens during model training Efficient; tied to model Depends on model assumptions Tree or linear pipelines
RFE (wrapper) Recursively removes weakest features Systematic Expensive on many features Smaller feature sets

Best Practices

  • Prevent leakage: split data before any selection. Compute target encodings, group aggregates, and GA fitness on training folds only, and respect time order with TimeSeriesSplit for temporal data.

  • Control cost: start with filter or embedded methods, then apply a GA when interactions are likely to matter. Limit population size, generations, and the feature pool.

  • Reproducibility: fix random seeds (for example, numpy.random.default_rng(seed)); GA results can still vary across platforms and library versions.

Common Pitfalls and Interview Follow-ups

  • Overfitting to CV: a GA can exploit noise in the validation folds; mitigate with nested cross-validation or a final untouched test set.

  • Ignoring evaluation metrics: accuracy can mislead on imbalanced classes; choose F1, ROC-AUC, or PR-AUC to match the problem.

  • Typical follow-ups: “How would you evolve new features, not just select them?” (genetic programming/symbolic regression), “How do you avoid leakage in target encoding?” (compute encodings fold-wise on training data), and “When would a GA be worse than embedded selection?” (very high dimensions or tight compute budgets).

36

How do you normalize a feature vector in Python?

Feature normalization rescales numeric inputs so their magnitudes are comparable; for one vector, L2 normalization makes its Euclidean length 1, while standardization and min-max scaling operate on each feature across samples. It prevents large-scale features from dominating distance-based or regularized models, at the cost of learned scaling parameters that must be estimated without validation or test leakage. More broadly, feature engineering turns raw data into model inputs that expose signal and often matters more than algorithm choice for tabular data.

Read full answer

Applies toNumPy 2.x, pandas 2.2+, and scikit-learn 1.5+ (Python 3.12)

Normalization Methods

Method How it works When to use Main concern
L2 normalization Divides a vector by its Euclidean norm Text similarity, cosine similarity, per-sample normalization Does not center features; a zero vector cannot be normalized to unit length
Min-max scaling Scales each feature using training minima and maxima Bounded feature ranges Sensitive to outliers; unseen values can fall outside [0, 1]
Standardization Subtracts the training mean and divides by the training standard deviation k-NN, SVM, PCA, regularized linear models, neural networks Not bounded and can still be affected by extreme values
Robust scaling Uses the median and interquartile range Features with substantial outliers Less familiar to some model implementations and requires fitting preprocessing state

Key formulas:

  • L2: x' = x / ||x||_2, where ||x||_2 = sqrt(sum(x_i^2)); x is the input vector, x_i is one element, and i indexes the elements.

  • Min-max: x' = (x - x_min) / (x_max - x_min), where x_min and x_max are the training-set minimum and maximum for that feature.

  • Standardization: x' = (x - mu) / sigma, where mu is the training-set mean and sigma is the training-set standard deviation.

Feature Engineering by Data Type

Data type Useful techniques
Numeric Scaling, log transforms for positive skew, Box-Cox or Yeo-Johnson transforms, binning, interactions, and polynomial features
Categorical One-hot encoding, ordinal encoding when order is real, cross-fitted target encoding, frequency encoding, and hashing for very high cardinality
Date and time Year, month, weekday, holiday flags, elapsed time since an event, lags, and cyclic sine/cosine encodings
Text TF-IDF or pretrained embeddings, usually followed by model-appropriate scaling
Entity aggregates Counts, means, rates, and rolling-window aggregates grouped by customer, account, device, or another entity

Feature Selection and Consistency

  • Filter methods: Mutual information or correlation can rank features without training a complex model.

  • Wrapper methods: RFE evaluates feature subsets with a chosen estimator.

  • Embedded methods: L1 regularization and tree-based importances expose useful features during model fitting.

  • Model-based selection: SelectFromModel keeps features according to an estimator’s importance or coefficient threshold.

  • Feature stores: A system such as Feast can reuse feature definitions and values consistently between training and serving.

Code Example

import numpy as np
import pandas as pd

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import (
    StratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


def l2_normalize(vector: np.ndarray) -> np.ndarray:
    """Return a vector with unit L2 norm."""
    array = np.asarray(vector, dtype=np.float64)
    norm = np.linalg.norm(array)
    if norm == 0.0:
        raise ValueError("A zero vector cannot be normalized to unit length.")
    return array / norm


print(l2_normalize(np.array([3.0, 4.0])))  # [0.6 0.8]

rng = np.random.default_rng(42)
n_rows = 120

df = pd.DataFrame(
    {
        "event_date": pd.date_range("2024-01-01", periods=n_rows, freq="D"),
        "account_id": rng.integers(0, 20, size=n_rows),
        "amount": rng.lognormal(mean=8.0, sigma=0.6, size=n_rows),
        "category": rng.choice(["A", "B", "C"], size=n_rows),
    }
)

# Deterministic date parts and features based only on earlier events.
df["day_of_week"] = df["event_date"].dt.dayofweek
df["month"] = df["event_date"].dt.month
df["prior_amount"] = df.groupby("account_id")["amount"].shift(1)
df["amount_ratio"] = df["amount"] / df["prior_amount"].replace(0, np.nan)
df["account_prior_events"] = df.groupby("account_id").cumcount()

# A fixed domain threshold creates synthetic classification labels.
y = (df["amount"] > 3000).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    df,
    y,
    test_size=0.25,
    random_state=42,
    stratify=y,
)

numeric_features = [
    "amount",
    "amount_ratio",
    "day_of_week",
    "month",
    "account_prior_events",
]
categorical_features = ["category"]

numeric_pipeline = Pipeline(
    [
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler()),
    ]
)

preprocessor = ColumnTransformer(
    [
        ("num", numeric_pipeline, numeric_features),
        (
            "cat",
            OneHotEncoder(handle_unknown="ignore", sparse_output=True),
            categorical_features,
        ),
    ]
)

pipeline = Pipeline(
    [
        ("preprocessor", preprocessor),
        ("classifier", LogisticRegression(max_iter=1000)),
    ]
)

# The pipeline prevents preprocessing leakage inside every CV fold.
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_scores = cross_val_score(
    pipeline, X_train, y_train, cv=cv, scoring="accuracy"
)
print(f"Mean CV accuracy: {cv_scores.mean():.3f}")

pipeline.fit(X_train, y_train)
test_accuracy = accuracy_score(y_test, pipeline.predict(X_test))
print(f"Test accuracy: {test_accuracy:.3f}")

Common Pitfalls

  • Data leakage: Split before fitting scalers, encoders, imputers, or feature selectors. Compute target encodings and learned aggregates only from each training fold, and respect chronological order for time-dependent features.

  • Wrong transformation: Trees generally do not require scaling, while k-NN, SVM, PCA, and gradient-based optimization usually benefit from it.

  • Outliers and missing values: Check distributions before choosing min-max scaling; use imputation inside the pipeline rather than filling the full dataset beforehand.

  • Model evaluation: Accuracy can mislead with imbalanced classes; consider precision, recall, F1, ROC-AUC, and PR-AUC as appropriate. Avoid repeatedly tuning against the test set.

  • Production consistency: Apply the same fitted transformations at serving time and record random seeds for reproducibility.

Interviewers often ask: “Why split before scaling?”, “When is L2 normalization preferable to standardization?”, and “How do you prevent leakage when creating aggregates?”

37

How do I one-hot encode categorical variables in Python?

One-hot encoding converts each category into a binary feature so models can use nominal data without treating category labels as ordered numbers. It prevents false ordering relationships, but can consume substantial memory for high-cardinality features. In scikit-learn, a fitted OneHotEncoder inside a Pipeline keeps training and prediction preprocessing consistent.

Read full answer

Applies toNumPy 2.x, pandas 2.2+, and scikit-learn 1.5+ (Python 3.12)

How One-Hot Encoding Works

For a country column containing US, CA, and GB, one-hot encoding creates one binary column per category:

country country_US country_CA country_GB
US 1 0 0
CA 0 1 0
GB 0 0 1

For a known category, exactly one encoded column is active. OneHotEncoder(handle_unknown="ignore") represents categories unseen during fitting with an all-zero row, avoiding prediction-time errors.

Feature Engineering Context

Feature engineering turns raw data into model inputs that expose useful signal; for tabular data, it often matters more than choosing between similar algorithms.

Data type Common techniques Important consideration
Numeric Scaling, log, Box-Cox or Yeo-Johnson transforms, binning, ratios, interactions and polynomial features Scaling helps k-NN, SVM, PCA and gradient-based optimization but is usually unnecessary for trees
Categorical One-hot, ordinal, frequency, hashing and cross-fitted target encoding Target encoding can leak labels when fitted on all data
Dates and times Date parts, elapsed time since an event, cyclic sine/cosine features Preserve time order for temporal prediction
Text TF-IDF, token counts and embeddings High-dimensional representations need regularization or selection
Entity histories Counts, means and rolling-window aggregates Use only information available at prediction time

Feature selection can use filter methods such as mutual information or correlation, wrapper methods such as recursive feature elimination (RFE), embedded methods such as L1 regularization or tree importance, and model-based selection with SelectFromModel.

Code Example

This example creates date parts, a ratio and customer aggregates from the training partition before applying numeric and categorical preprocessing in a pipeline.

import numpy as np
import pandas as pd

from sklearn.compose import ColumnTransformer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


def engineer_features(
    frame: pd.DataFrame,
    customer_means: pd.Series,
    fallback_mean: float,
) -> pd.DataFrame:
    """Create model-ready features using precomputed training aggregates."""
    features = frame.copy()
    signup_date = pd.to_datetime(features["signup_date"])

    features["signup_month"] = signup_date.dt.month
    features["signup_dayofweek"] = signup_date.dt.dayofweek
    features["spend_per_visit"] = (
        features["orders"] / features["visits"].clip(lower=1)
    )
    features["mean_visits_per_customer"] = (
        features["customer_id"]
        .map(customer_means)
        .fillna(fallback_mean)
    )

    return features.drop(columns=["customer_id", "signup_date"])


rng = np.random.default_rng(42)
size = 60

data = pd.DataFrame({
    "customer_id": np.repeat(np.arange(20), 3),
    "country": rng.choice(["US", "CA", "GB"], size=size),
    "plan": rng.choice(["basic", "plus", "premium"], size=size),
    "signup_date": pd.date_range("2024-01-01", periods=size, freq="D"),
    "visits": rng.integers(1, 20, size=size),
    "orders": rng.integers(0, 8, size=size),
})

# Split before learning category values or calculating aggregates.
train_data, test_data = train_test_split(
    data,
    test_size=0.2,
    random_state=42,
)

customer_means = train_data.groupby("customer_id")["visits"].mean()
fallback_mean = float(customer_means.mean())

train_features = engineer_features(
    train_data, customer_means, fallback_mean
)
test_features = engineer_features(
    test_data, customer_means, fallback_mean
)

numeric_columns = [
    "visits",
    "orders",
    "signup_month",
    "signup_dayofweek",
    "spend_per_visit",
    "mean_visits_per_customer",
]
categorical_columns = ["country", "plan"]

preprocessor = ColumnTransformer(
    transformers=[
        ("numeric", StandardScaler(), numeric_columns),
        (
            "categorical",
            OneHotEncoder(
                handle_unknown="ignore",
                sparse_output=False,
            ),
            categorical_columns,
        ),
    ],
    remainder="drop",
)

pipeline = Pipeline([("preprocessor", preprocessor)])
pipeline.fit(train_features)

encoded_train = pipeline.transform(train_features)
encoded_test = pipeline.transform(test_features)

encoder = pipeline.named_steps["preprocessor"].named_transformers_[
    "categorical"
]

print("Encoded training shape:", encoded_train.shape)
print("Encoded test shape:", encoded_test.shape)
print("Categorical features:", encoder.get_feature_names_out())

Best Practices

  • Fit on training data only: Put encoders, scalers, imputers and selectors inside a Pipeline or ColumnTransformer.

  • Use suitable validation: Use StratifiedKFold for classification and TimeSeriesSplit for temporal data, fitting preprocessing within each fold.

  • Handle missing values explicitly: In scikit-learn 1.5, OneHotEncoder treats missing values as a category. If missingness is not meaningful, impute inside the pipeline and optionally add a missing-value indicator.

  • Control dimensionality: Prefer sparse output for high-cardinality data; consider frequency encoding, hashing or cross-fitted target encoding for identifiers or very large categorical domains.

  • Prevent target leakage: Compute target encodings and entity aggregates within training folds only, and use only historical information available at prediction time.

  • Evaluate imbalance appropriately: For imbalanced targets, examine precision, recall, F1, ROC-AUC and PR-AUC; accuracy alone can hide poor minority-class performance.

  • Keep serving consistent: Reuse identical feature definitions during training and serving. A feature store such as Feast can help organize consistently computed features.

Common Pitfalls

  • Data leakage: Fitting an encoder or calculating aggregates on the complete dataset exposes validation or test information.

  • Inconsistent encoders: Fitting separate fit_transform and transform calls on unrelated objects can produce incompatible columns.

  • Unseen categories: Without handle_unknown="ignore", prediction fails when a new category appears.

  • Excessive dimensionality: One-hot encoding is unsuitable for identifiers, free text or very high-cardinality variables without additional techniques.

  • False ordering: Assigning arbitrary integer codes to nominal categories implies an order that does not exist.

  • Invalid temporal validation: Random splits can leak future information; preserve chronological order for temporal tasks.

Interviewers commonly ask how unseen categories are handled, how target leakage is prevented, how high-cardinality features are reduced, and when ordinal or target encoding is preferable.

38

How do I implement PCA from scratch in Python 3.12?

Principal component analysis (PCA) is a feature-engineering transformation that replaces correlated variables with orthogonal components ordered by explained variance. It can reduce dimensionality, expose structure, and improve conditioning for later models. The main trade-off is reduced interpretability and the risk that discarded low-variance directions still contain predictive information.

Read full answer

Applies toPython 3.12 with NumPy 2.x, pandas 2.2+, and scikit-learn 1.5+

How It Works

For m observations and p features, PCA first centers each column by subtracting its training-set mean. It then decomposes the centered matrix using singular value decomposition:

X_centered = U × S × Vt

  • U contains the left singular vectors.

  • S is the diagonal matrix of singular values.

  • Vt contains the transposed right singular vectors, whose rows define the principal directions.

The sample variance associated with singular value s[j] is s[j]² / (m - 1). Retaining the first k components produces:

Z = X_centered @ Vt[:k].T

The explained-variance ratio for component i is λ[i] / sum(λ), where λ contains all component variances. The cumulative ratio sums these ratios and indicates how much total variance the retained components preserve.

Feature-Engineering Context

Feature engineering turns raw data into model inputs that expose useful signal. For tabular data, thoughtful features often matter more than the choice of learning algorithm.

  • Numeric data: Scaling balances units; log or Yeo-Johnson transforms reduce skew; binning captures thresholds; interactions and polynomial features represent nonlinear structure.

  • Categorical data: One-hot encoding fits low-cardinality fields, ordinal encoding represents ordered levels, cross-fitted target encoding handles supervised categories, and frequency encoding or hashing provides alternatives.

  • Dates and times: Useful features include date parts, cyclic sin/cos encodings, and elapsed time since an event.

  • Text: TF-IDF emphasizes word importance, while embeddings represent semantic relationships.

  • Entity aggregates: Counts, means, and rolling-window statistics summarize behavior per customer, account, or device.

Feature selection includes correlation or mutual-information filters, recursive feature elimination, L1 or tree-importance methods, and SelectFromModel. A feature store such as Feast can keep training and serving definitions consistent.

Code Example

import numpy as np
import pandas as pd

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


class ScratchPCA(BaseEstimator, TransformerMixin):
    """Dense PCA implemented with NumPy's SVD."""

    def __init__(self, n_components=2):
        self.n_components = n_components

    def fit(self, X, y=None):
        X = np.asarray(X, dtype=np.float64)
        if X.ndim != 2:
            raise ValueError("X must be a two-dimensional matrix.")
        if not np.isfinite(X).all():
            raise ValueError("X must contain only finite values.")

        n_samples, n_features = X.shape
        if n_samples < 2 or n_features == 0:
            raise ValueError("PCA requires at least two samples and one feature.")
        if (
            not isinstance(self.n_components, (int, np.integer))
            or self.n_components < 1
        ):
            raise ValueError("n_components must be a positive integer.")

        k = min(self.n_components, n_samples, n_features)
        self.n_features_in_ = n_features
        self.mean_ = X.mean(axis=0)
        X_centered = X - self.mean_

        _, singular_values, Vt = np.linalg.svd(
            X_centered, full_matrices=False
        )

        variances = singular_values**2 / (n_samples - 1)
        total_variance = variances.sum()
        self.components_ = Vt[:k]
        self.explained_variance_ = variances[:k]
        self.explained_variance_ratio_ = (
            variances[:k] / total_variance
            if total_variance > 0
            else np.zeros(k)
        )
        return self

    def transform(self, X):
        if not hasattr(self, "components_"):
            raise RuntimeError("ScratchPCA must be fitted before transform.")

        X = np.asarray(X, dtype=np.float64)
        if X.ndim != 2 or X.shape[1] != self.n_features_in_:
            raise ValueError("X has an incompatible feature shape.")
        if not np.isfinite(X).all():
            raise ValueError("X must contain only finite values.")

        return (X - self.mean_) @ self.components_.T


rng = np.random.default_rng(42)
n_samples = 12
df = pd.DataFrame(
    {
        "date": pd.date_range("2024-01-01", periods=n_samples, freq="2D"),
        "category": rng.choice(["A", "B", "C"], n_samples),
        "amount": rng.integers(10, 50, n_samples),
        "visits": rng.integers(1, 9, n_samples),
        "entity_id": [f"e{i % 4 + 1}" for i in range(n_samples)],
    }
)

# Deterministic row features do not learn from the data.
df["day_of_week"] = df["date"].dt.dayofweek
df["month"] = df["date"].dt.month
df["amount_per_visit"] = df["amount"] / df["visits"]

# Respect chronological order.
split_at = 8
train_df = df.iloc[:split_at].copy()
test_df = df.iloc[split_at:].copy()

# Learn the entity aggregate from training rows only.
entity_means = train_df.groupby("entity_id")["visits"].mean()
train_df["entity_visits_mean"] = train_df["entity_id"].map(entity_means)
test_df["entity_visits_mean"] = test_df["entity_id"].map(entity_means)

numeric_features = [
    "amount",
    "visits",
    "amount_per_visit",
    "entity_visits_mean",
    "day_of_week",
    "month",
]
categorical_features = ["category"]

preprocessor = ColumnTransformer(
    transformers=[
        (
            "numeric",
            Pipeline(
                [
                    ("impute", SimpleImputer(strategy="median")),
                    ("scale", StandardScaler()),
                ]
            ),
            numeric_features,
        ),
        (
            "categorical",
            OneHotEncoder(handle_unknown="ignore", sparse_output=False),
            categorical_features,
        ),
    ]
)

pipeline = Pipeline(
    [
        ("preprocess", preprocessor),
        ("pca", ScratchPCA(n_components=2)),
    ]
)

X_train_pca = pipeline.fit_transform(train_df)
X_test_pca = pipeline.transform(test_df)
pca = pipeline.named_steps["pca"]

print("Projected training shape:", X_train_pca.shape)  # (8, 2)
print("Projected test shape:", X_test_pca.shape)        # (4, 2)
print("Explained variance ratio:", pca.explained_variance_ratio_)
print("Components:\n", pca.components_)

Practical Guidance

  • Scale when units differ: PCA maximizes variance, so a large-scale numeric feature can otherwise dominate the components.

  • Choose k using validation: Examine cumulative explained variance, reconstruction error, and downstream performance. For temporal data, use TimeSeriesSplit and fit the entire pipeline inside each training fold.

  • Prevent leakage: Fit imputers, scalers, encoders, aggregates, and PCA on training data only. PCA is unsupervised, but fitting it on test observations still leaks test-distribution information.

  • Handle practical issues: Impute missing values within the pipeline, inspect outliers, and monitor downstream overfitting. Fixing random seeds supports reproducibility.

  • Complexity: Dense SVD costs approximately O(m × p × min(m, p)); randomized or incremental alternatives can help with very large datasets.

Common Pitfalls

  • Fitting preprocessing or PCA before splitting the data.

  • Assuming PCA standardizes features; it centers them but does not scale them.

  • Choosing k from test performance or interpreting component signs as meaningful.

  • Applying dense PCA directly to very sparse text matrices without considering truncation or another representation.

  • Treating low-variance components as automatically useless.

Interview follow-ups commonly concern scaling, covariance eigendecomposition versus SVD, component selection, leakage, and PCA versus feature selection.

39

How do I select top features with SelectKBest in Python?

SelectKBest is a scikit-learn filter that scores each feature independently against the target and retains the top k. It is fast and easy to interpret, but it cannot directly detect interactions among features.

Read full answer

Applies toscikit-learn 1.5+, pandas 2.2+, and NumPy 2.x (Python 3.12)

How SelectKBest Works

SelectKBest receives a scoring function through score_func. The function assigns one score to each feature, and the selector keeps the features with the highest scores.

Common choices include:

  • f_classif: ANOVA F-test for numeric classification features.

  • f_regression: F-test for numeric regression targets.

  • chi2: Chi-squared test requiring nonnegative features, commonly used with counts.

  • mutual_info_classif: Measures general dependence, including nonlinear relationships.

Because each feature is evaluated separately, SelectKBest is a filter method applied before the final predictive model.

Feature Engineering by Data Type

Feature engineering turns raw data into model inputs that expose useful signal; for tabular data, it often matters more than switching algorithms.

  • Numeric: Scaling, logarithms, Box-Cox or Yeo-Johnson transformations, binning, ratios, polynomial features, and interactions.

  • Categorical: One-hot encoding for nominal variables, ordinal encoding only for genuinely ordered categories, frequency encoding, hashing, and cross-fitted target encoding.

  • Dates and times: Year, month, weekday, day-of-year, elapsed time since an event, and cyclic sin/cos encodings.

  • Text: TF-IDF for lexical representations or pretrained embeddings for semantic representations.

  • Entity aggregations: Counts, means, rates, and rolling-window statistics calculated for customers, products, or other entities.

Feature Selection Methods

Method How it works Strengths Weaknesses Typical use
SelectKBest Ranks each feature independently Fast and simple Misses interactions Initial filtering
Mutual information Measures feature-target dependence Handles nonlinear relationships Noisy on small samples General filtering
Correlation Measures linear association between features Simple and interpretable Measures only linear relationships Removing redundant numeric features
RFE Removes features using model performance Accounts for combinations Expensive and model-dependent Small feature sets
L1 regularization Shrinks some coefficients to zero Embedded in linear models Sensitive to scaling and tuning Linear models
Tree importance Uses tree-based importance values Handles nonlinearities Can favor high-cardinality features Tree ensembles
SelectFromModel Retains features based on learned coefficients or importances Uses model behavior Requires model assumptions Post-training selection

Runtime depends on score_func. The F-test and chi-squared scores are approximately O(n * d), while mutual-information estimation can be substantially more expensive.

Code Example

This example creates balanced synthetic data, engineers date, ratio, and segment-aggregate features, and places the aggregate calculation inside a pipeline. Consequently, every cross-validation fold learns its aggregate only from that fold’s training data.

import numpy as np
import pandas as pd

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import (
    StratifiedKFold,
    cross_validate,
    train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
n = 500

segments = rng.choice(["A", "B", "C"], size=n)
plans = rng.choice(["basic", "premium"], size=n)
amount = rng.lognormal(mean=3.8, sigma=0.45, size=n)
previous_amount = rng.lognormal(mean=3.5, sigma=0.40, size=n)

signup_date = pd.Timestamp("2022-01-01") + pd.to_timedelta(
    rng.integers(0, 730, size=n), unit="D"
)
last_active = pd.Timestamp("2024-01-01") - pd.to_timedelta(
    rng.integers(0, 365, size=n), unit="D"
)

df = pd.DataFrame({
    "segment": segments,
    "plan": plans,
    "amount": amount,
    "previous_amount": previous_amount,
    "signup_date": signup_date,
    "last_active": last_active,
})

# Construct a balanced target containing signal from numeric and categorical data.
score = (
    df["amount"]
    + 0.5 * df["previous_amount"]
    + np.where(df["segment"].eq("A"), 5.0, -5.0)
    + rng.normal(0.0, 5.0, size=n)
)
df["target"] = (score > np.median(score)).astype(int)

# Deterministic row-level features do not learn from the data.
df["signup_month"] = df["signup_date"].dt.month.astype(float)
df["last_active_dow"] = df["last_active"].dt.dayofweek.astype(float)
df["days_since_signup"] = (
    pd.Timestamp("2024-01-01") - df["signup_date"]
).dt.days.astype(float)
df["amount_ratio"] = df["amount"] / (df["previous_amount"] + 1.0)

train, test = train_test_split(
    df,
    test_size=0.25,
    random_state=42,
    stratify=df["target"],
)


class SegmentAmountMean(BaseEstimator, TransformerMixin):
    """Learn segment means from the current pipeline training fold."""

    def fit(self, X, y=None):
        self.group_means_ = X.groupby("segment")["amount"].mean()
        self.fallback_mean_ = float(X["amount"].mean())
        return self

    def transform(self, X):
        values = (
            X["segment"]
            .map(self.group_means_)
            .fillna(self.fallback_mean_)
        )
        return pd.DataFrame(
            {"segment_amount_mean": values.to_numpy()},
            index=X.index,
        )

    def get_feature_names_out(self, input_features=None):
        return np.asarray(["segment_amount_mean"], dtype=object)


numeric_features = [
    "amount",
    "previous_amount",
    "signup_month",
    "last_active_dow",
    "days_since_signup",
    "amount_ratio",
]
categorical_features = ["segment", "plan"]

preprocessor = ColumnTransformer(
    transformers=[
        ("numeric", StandardScaler(), numeric_features),
        (
            "segment_mean",
            SegmentAmountMean(),
            ["segment", "amount"],
        ),
        (
            "categorical",
            OneHotEncoder(handle_unknown="ignore", sparse_output=False),
            categorical_features,
        ),
    ]
)

pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("selector", SelectKBest(score_func=f_classif, k=3)),
    (
        "classifier",
        LogisticRegression(max_iter=1000, class_weight="balanced"),
    ),
])

X_train = train[numeric_features + categorical_features]
y_train = train["target"]

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_results = cross_validate(
    pipeline,
    X_train,
    y_train,
    cv=cv,
    scoring=["f1", "roc_auc", "average_precision"],
)

pipeline.fit(X_train, y_train)
feature_names = pipeline.named_steps["preprocessor"].get_feature_names_out()
selector = pipeline.named_steps["selector"]
selected_mask = selector.get_support(indices=True)

print("Selected features:", feature_names[selected_mask].tolist())
print("ANOVA F-scores:", np.round(selector.scores_[selected_mask], 3).tolist())
print("CV F1:", round(cv_results["test_f1"].mean(), 3))
print("CV ROC-AUC:", round(cv_results["test_roc_auc"].mean(), 3))
print("CV average precision:", round(cv_results["test_average_precision"].mean(), 3))

Best Practices and Leakage Prevention

  • Keep the selector inside a Pipeline so preprocessing and feature selection are fitted on training data only.

  • Compute target encodings and aggregates within each training fold; for temporal data, use TimeSeriesSplit and preserve chronological order.

  • Verify that every feature is available at prediction time and excludes future observations.

  • Use StratifiedKFold for imbalanced classification. Evaluate F1, ROC-AUC, and PR-AUC because accuracy can be misleading.

  • Scale for k-NN, SVM, PCA, and regularized linear models; scaling is usually unnecessary for trees.

  • Handle missing values explicitly and use a feature store such as Feast when training and serving must share feature definitions.

Common Pitfalls

  • Fitting preprocessing before splitting causes data leakage.

  • Applying chi2 to negative or standardized values causes errors.

  • Treating an arbitrarily ordered category as ordinal creates a false ordering.

  • Assuming SelectKBest evaluates feature interactions is incorrect.

  • Repeatedly tuning k against the test set turns it into a validation set.

  • One-hot and transformed names may differ from source columns; inspect get_feature_names_out() to map them correctly.

40

How do you implement missing-data mean, median, or mode imputation in Python?

Missing-value imputation replaces missing entries with a representative value such as the mean, median, or mode. It prevents incomplete records from breaking model training, but the replacement may distort distributions or conceal uncertainty. Feature engineering turns raw data into model inputs that expose signal to the model, and effective imputation often matters as much as algorithm choice for tabular data.

Read full answer

Applies toPython 3.12 with pandas 2.2+, NumPy 2.x, and scikit-learn 1.5+

Imputation Strategies

Strategy Appropriate use Main limitation
Mean Approximately symmetric numeric features Sensitive to outliers and skew
Median Skewed numeric features or values with outliers Does not preserve multimodality
Mode Categorical features and occasionally discrete numeric features Unstable for high-cardinality features; ties need a consistent policy

In scikit-learn, mean and median are named mean and median, while mode is represented by most_frequent. Mean and median are generally inappropriate for nominal categorical columns, so those columns should use mode-based imputation.

Feature Engineering Context

Imputation is one part of converting raw observations into model-ready features:

  • Numeric data: scaling, log or Box-Cox transforms, Yeo-Johnson transforms, binning, interactions, and polynomial features.

  • Categorical data: one-hot encoding for low-cardinality features, ordinal encoding for ordered values, cross-fitted target encoding for high-cardinality features, and frequency or hash encoding.

  • Date and time data: year, month, weekday, cyclic sin/cos encoding, and time since an event.

  • Text data: TF-IDF for sparse representations or pretrained embeddings for dense semantics.

  • Entity and history data: counts, means, and rolling-window aggregates computed using only information available at prediction time.

  • Feature selection: filter methods such as mutual information or correlation, wrapper methods such as recursive feature elimination (RFE), embedded methods such as L1 or tree importances, and SelectFromModel. Selection must be fitted within cross-validation to avoid leakage.

Target encodings and historical aggregates must be calculated within training folds or using past data only. A feature store such as Feast can help ensure that the same feature definitions are used during training and serving.

Code Example

The following function builds a production-style preprocessing pipeline. The imputer and scaler are fitted on training data only, while the same fitted transformations are applied to test data.

from typing import Literal

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


def build_preprocessor(
    imputation: Literal["mean", "median", "mode"],
    numeric_columns: list[str],
    categorical_columns: list[str],
) -> ColumnTransformer:
    """Build a train-fitted preprocessing pipeline for missing values."""
    if imputation not in {"mean", "median", "mode"}:
        raise ValueError("imputation must be 'mean', 'median', or 'mode'")

    numeric_strategy = (
        imputation if imputation in {"mean", "median"} else "most_frequent"
    )

    numeric_pipeline = Pipeline([
        ("imputer", SimpleImputer(strategy=numeric_strategy)),
        ("scaler", StandardScaler()),
    ])

    categorical_pipeline = Pipeline([
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("onehot", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
    ])

    return ColumnTransformer([
        ("numeric", numeric_pipeline, numeric_columns),
        ("categorical", categorical_pipeline, categorical_columns),
    ])


rng = np.random.default_rng(42)

# Small synthetic dataset
data = pd.DataFrame({
    "age": [20, np.nan, 35, 42, np.nan, 28, 50, 33, 39, 45],
    "income": [50000, 60000, np.nan, 80000, 90000,
               55000, np.nan, 75000, 95000, 65000],
    "city": ["London", "Paris", "London", None, "Paris",
             "London", "Berlin", "Paris", "London", "Berlin"],
    "orders": rng.integers(1, 6, size=10),
    "signup_date": pd.to_datetime([
        "2024-01-05", "2024-02-11", "2024-03-20", "2024-04-08",
        "2024-05-16", "2024-06-01", "2024-07-19", "2024-08-02",
        "2024-09-12", "2024-10-21",
    ]),
})

# Split before fitting learned preprocessing
train = data.iloc[:6].copy()
test = data.iloc[6:].copy()

# Row-level engineered features
for frame in (train, test):
    frame["signup_month"] = frame["signup_date"].dt.month
    frame["income_to_age"] = frame["income"] / frame["age"]

# Fit the city aggregation on training data and apply it to both splits
city_order_means = train.groupby("city", dropna=True)["orders"].mean()
global_order_mean = train["orders"].mean()

train["city_order_mean"] = train["city"].map(city_order_means)
test["city_order_mean"] = test["city"].map(city_order_means)
train["city_order_mean"] = train["city_order_mean"].fillna(global_order_mean)
test["city_order_mean"] = test["city_order_mean"].fillna(global_order_mean)

numeric_columns = [
    "age", "income", "signup_month", "income_to_age", "city_order_mean"
]
categorical_columns = ["city"]

for strategy in ("mean", "median", "mode"):
    preprocessor = build_preprocessor(
        strategy, numeric_columns, categorical_columns
    )
    train_ready = preprocessor.fit_transform(train)
    test_ready = preprocessor.transform(test)
    print(strategy, train_ready.shape, test_ready.shape)

# Output:
# mean (6, 7) (4, 7)
# median (6, 7) (4, 7)
# mode (6, 7) (4, 7)

Best Practices

  • Prevent leakage: Fit imputers, encoders, and scalers only on training folds or training data. Reusing statistics calculated over the entire dataset exposes validation information to the model.

  • Use cross-validation correctly: Put preprocessing inside each fold’s pipeline. Use StratifiedKFold for classification and TimeSeriesSplit for temporal data.

  • Preserve missingness signals: If missingness is informative, use SimpleImputer(add_indicator=True) or create explicit missing indicators.

  • Scale when necessary: Scaling is important for k-NN, SVM, PCA, and gradient-descent models, but usually unnecessary for tree-based models.

  • Match evaluation to the problem: For imbalanced classification, inspect precision, recall, F1, ROC-AUC, and PR-AUC rather than accuracy alone. For regression, use MAE, RMSE, and R².

  • Keep preprocessing reproducible: Fix random seeds, record imputation strategies, and define consistent transformations for training and serving.

Common Pitfalls

  • Computing means or modes on the full dataset before splitting causes data leakage.

  • Using the mean for heavily skewed features or the mode for sparse high-cardinality features can produce weak representations.

  • Forgetting that an aggregate feature may contain future information can invalidate an otherwise sound model.

  • Encoding categorical values without handle_unknown="ignore" can fail when serving data contains unseen categories.

  • Repeatedly tuning against one validation split can cause overfitting to the validation set.

  • Imputation alone does not solve class imbalance, missing-not-at-random data, or genuinely erroneous records.

41

How to create new feature columns in SQL?

Feature engineering in SQL transforms raw table data into derived columns that expose useful signal to a machine learning model, such as ratios, date parts, or group aggregates. It solves the problem of preparing consistent, reusable inputs close to where the data lives, which can matter more than algorithm choice for tabular data. Its main trade-off is that complex logic is harder to maintain in SQL, and leakage is easy to introduce if features use information not available at prediction time.

Read full answer

Applies toANSI SQL (examples compatible with PostgreSQL 16)

Common SQL Approaches

  • SELECT expressions: Compute features on the fly without changing storage. Best for exploration and for building training datasets.

  • CREATE VIEW: Saves the feature logic as a reusable query. The underlying data stays current, but expensive expressions run on every read.

  • CREATE TABLE ... AS SELECT or INSERT ... SELECT: Materializes features for faster training and serving. You must refresh the table when source data changes.

  • Generated/computed columns: Persist row-level formulas managed by the database (syntax varies by database). Good for deterministic row-level transforms, not for cross-row aggregates.

Feature Engineering Techniques in SQL

Data type Example features SQL pattern
Numeric Ratios, bins, interactions amount / NULLIF(units, 0), CASE WHEN ... buckets, price * quantity
Categorical Prior frequency/count encoding, simple bucketing COUNT(*) OVER (PARTITION BY category ORDER BY event_date ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING), CASE grouping rare values
Dates/times Parts, day-of-week, time since event EXTRACT(YEAR FROM order_date), order_date - signup_date
Aggregations per entity Counts and means over history AVG(amount) OVER (PARTITION BY customer_id ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING)

Use cyclic sin/cos encoding for periodic time parts (for example, hour of day) in downstream code or with math functions when supported. For text, SQL can extract length or token counts, but TF-IDF or embeddings are usually computed outside SQL.

Code Example

-- Create a reusable feature view from existing columns.
-- Uses a synthetic-style table name: orders(customer_id, order_date, signup_date,
-- amount, units, category, label).

CREATE OR REPLACE VIEW order_features AS
SELECT
    customer_id,
    order_date,

    -- Date features: parts and time since signup
    EXTRACT(YEAR FROM order_date) AS order_year,
    EXTRACT(MONTH FROM order_date) AS order_month,
    EXTRACT(DOW FROM order_date) AS order_dow,
    (order_date - signup_date) AS days_since_signup,

    -- Numeric features: ratio and interaction
    amount / NULLIF(units, 0) AS amount_per_unit,
    amount * units AS total_value,

    -- Leakage-safe historical aggregate: prior category count only (excludes current row)
    COUNT(*) OVER (
        PARTITION BY category
        ORDER BY order_date
        ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
    ) AS category_prior_count,

    -- Leakage-safe historical aggregate: past orders only (excludes current row)
    AVG(amount) OVER (
        PARTITION BY customer_id
        ORDER BY order_date
        ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
    ) AS avg_prior_amount,

    label
FROM orders;

-- Materialize features when you need stable training data.
CREATE TABLE order_features_train AS
SELECT *
FROM order_features
WHERE order_date < DATE '2026-01-01';

Common Pitfalls and Interview Tips

  • Data leakage: Never use future or target information. Compute aggregates with windows that exclude the current row, and derive target or frequency encodings from training or prior data only.

  • Time order: For time series, split by date (as shown) rather than random sampling so features and labels reflect prediction time.

  • NULL handling: Division and historical windows can produce NULL (for example, the first row in a partition has no prior rows); use NULLIF and document the imputation strategy.

  • Training-serving skew: Reuse the same SQL logic (often via a view or feature store such as Feast) so production features match training features.

  • Interview follow-ups: Expect questions on preventing leakage in SQL windows, when to materialize versus use a view, and how you would move the same feature logic into a pandas/scikit-learn Pipeline or ColumnTransformer for cross-validated preprocessing.

42

How do you implement feature hashing in Python?

Feature hashing maps each categorical value or text token to a coordinate in a fixed-width numeric vector using a stable hash. It avoids a learned vocabulary and is useful for high-cardinality categories and sparse text, at the cost of collisions and reduced interpretability. For tabular data, thoughtful feature engineering often matters more than a marginal change of algorithm.

Read full answer

Applies toPython 3.12 with pandas 2.2+, NumPy 2.x, and scikit-learn 1.5+

How It Works

First, namespace each value so that fields such as city and device do not deliberately share names. In scikit-learn’s dictionary mode, a category such as city=Paris can be represented by a feature name with value 1.

For a feature name t, scikit-learn uses the MurmurHash3 calculation:

h = murmurhash3_32(t, seed=0)

index = abs(h) mod d

Here, t is the namespaced feature name, h is its signed hash, and d is the output dimension. The absolute hash is reduced modulo d to obtain the coordinate. With alternate_sign=True, the coefficient receives a sign derived from the hash, reducing collision-induced bias.

  • No vocabulary: Training and serving do not need to fit a category list.

  • Accumulation: Token counts can be added when multiple occurrences map to one coordinate.

  • Collisions: Different features can occupy the same coordinate; increasing n_features reduces this risk.

  • Cost: Processing a row is approximately O(k) for k nonzero features, with O(k) sparse storage.

Feature Engineering Choices

  • Numeric: Scale when required by the model; consider log, Box-Cox, or Yeo-Johnson transforms for skew. Ratios, interactions, binning, and polynomial features can expose relationships.

  • Categorical: Use one-hot encoding for low cardinality, ordinal encoding only for genuine order, and frequency or cross-fitted target encoding for higher cardinality. Hashing avoids a stored vocabulary.

  • Dates and times: Extract parts such as month, hour, and weekday; encode cyclic values with sine and cosine; calculate time since a relevant event.

  • Text: TF-IDF weights token frequencies by inverse document frequency, while embeddings represent semantic relationships.

  • Entity aggregates: Counts, means, and rolling windows summarize each customer, device, or account using only information available at prediction time.

  • Feature selection: Filter methods include mutual information and correlation, wrappers include RFE, and embedded methods include L1 regularization or tree importances. SelectFromModel applies a threshold to model-derived importance or coefficients.

Leakage and Evaluation

Fit preprocessing inside a Pipeline and ColumnTransformer so it learns only from each training fold. Cross-fit target encodings and compute aggregates within training folds; for event data, respect chronological order and never use future events.

For imbalanced classification, use precision, recall, F1, ROC-AUC, and PR-AUC because accuracy can be misleading. Scale for k-NN, SVM, PCA, and gradient-descent models, but generally not for trees. Handle missing values explicitly, control overfitting, and fix random seeds.

Code Example

import numpy as np
import pandas as pd

from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction import FeatureHasher
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import TimeSeriesSplit, cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
n = 120

df = pd.DataFrame({
    "user_id": rng.integers(0, 25, size=n),
    "city": rng.choice(["London", "Paris", "Berlin", "Madrid"], size=n),
    "device": rng.choice(["mobile", "desktop", "tablet"], size=n),
    "channel": rng.choice(["web", "store", "partner"], size=n),
    "amount": rng.lognormal(mean=2.2, sigma=0.5, size=n),
})

df["timestamp"] = pd.date_range("2025-01-01", periods=n, freq="h")
df["hour"] = df["timestamp"].dt.hour
df["day_of_week"] = df["timestamp"].dt.dayofweek
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)
df["amount_ratio"] = df["amount"] / 100.0

# This count uses only earlier events for the same user.
df["prior_user_events"] = df.groupby("user_id").cumcount()

# Independent of every predictor column above.
df["target"] = (df["hour"] % 3 == 0).astype(int)

hashed_features = ["user_id", "city", "device"]
numeric_features = [
    "hour",
    "day_of_week",
    "hour_sin",
    "hour_cos",
    "amount_ratio",
    "prior_user_events",
]
X = df[hashed_features + ["channel"] + numeric_features]
y = df["target"].to_numpy()
split = 90


def rows_to_named_counts(frame: pd.DataFrame) -> list[dict[str, int]]:
    """Represent each row as namespaced categorical features."""
    filled = frame.fillna("__missing__").astype(str)
    return [
        {
            f"{column}={value}": 1
            for column, value in zip(filled.columns, row)
        }
        for row in filled.itertuples(index=False, name=None)
    ]


hash_pipeline = Pipeline([
    ("prepare", FunctionTransformer(rows_to_named_counts, validate=False)),
    (
        "hasher",
        FeatureHasher(
            n_features=64,
            input_type="dict",
            alternate_sign=True,
        ),
    ),
])

preprocessor = ColumnTransformer([
    ("hashed", hash_pipeline, hashed_features),
    ("one_hot", OneHotEncoder(handle_unknown="ignore"), ["channel"]),
    (
        "numeric",
        Pipeline([
            ("impute", SimpleImputer(strategy="median")),
            ("scale", StandardScaler()),
        ]),
        numeric_features,
    ),
])

model = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", LogisticRegression(max_iter=1000)),
])

cv = TimeSeriesSplit(n_splits=4)
scores = cross_validate(
    model,
    X.iloc[:split],
    y[:split],
    cv=cv,
    scoring={
        "roc_auc": "roc_auc",
        "pr_auc": "average_precision",
        "f1": "f1",
    },
)

for metric, values in scores.items():
    print(f"CV {metric}: {values.mean():.3f}")

model.fit(X.iloc[:split], y[:split])
predictions = model.predict(X.iloc[split:])
probabilities = model.predict_proba(X.iloc[split:])[:, 1]

print("Transformed training shape:",
      model.named_steps["preprocessor"].transform(X.iloc[:split]).shape)
print("Precision:", precision_score(y[split:], predictions, zero_division=0))
print("Recall:", recall_score(y[split:], predictions, zero_division=0))
print("F1:", f1_score(y[split:], predictions, zero_division=0))
print("ROC-AUC:", roc_auc_score(y[split:], probabilities))
print("PR-AUC:", average_precision_score(y[split:], probabilities))

Common Pitfalls

  • Collision-heavy dimensions: Validate the dimension empirically rather than assuming more coordinates are always better.

  • Data leakage: Do not derive features from validation labels or future events.

  • Unstable hashes: Avoid Python’s process-randomized hash(); use a stable implementation such as FeatureHasher.

  • Train-serving mismatch: Reuse the same dimension, namespaces, hash implementation, and signing convention. A feature store such as Feast can keep definitions consistent.

  • Hashing is not anonymization: Hashed identifiers may still be brute-forced.

  • Holdout misuse: Repeated tuning against the holdout effectively turns it into training data.

43

How to detect and remove correlated features in Python?

Highly correlated features are input variables that carry nearly the same information, often measured by a high absolute Pearson correlation between them. Removing one feature from each highly correlated pair reduces redundancy, which can simplify a model, speed up training, and lessen multicollinearity in linear models. The main trade-off is that dropping a feature may discard small but useful signal, and the choice of which feature to keep is somewhat arbitrary.

Read full answer

Applies topandas 2.2+, NumPy 2.x, scikit-learn 1.5+ (Python 3.12)

How It Works

  • Correlation matrix: Compute pairwise correlations (usually Pearson for numeric features) with DataFrame.corr(). Values range from -1 to 1; abs(corr) near 1 indicates strong redundancy.

  • Thresholding: Choose a cutoff such as 0.8, 0.9, or 0.95. Higher thresholds are safer but remove fewer features.

  • Upper-triangle scan: Check only the upper triangle of the matrix so each pair is evaluated once, and collect one column from every pair above the threshold to drop.

  • Apply consistently: Drop the same columns from training, validation, and test data. Compute the correlation matrix on the training set only so the selection does not peek at test data.

Feature Selection Context

Approach How it works Strengths Weaknesses When to use
Correlation filter Removes one of two highly correlated inputs Fast, simple, model-agnostic Ignores the target; pairwise only Quick redundancy cleanup for numeric data
Mutual information filter Ranks features by dependence on the target Captures nonlinear target relationships Univariate; can miss interactions Pre-filter before model training
Embedded (L1 / trees) Model assigns low weight or importance Uses target and feature interactions Model-dependent When a fitted model can guide selection

Code Example

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(42)
n = 300
base = rng.normal(size=n)

# Small synthetic dataset with an intentionally redundant feature pair (f1, f2).
df = pd.DataFrame({
    "f1": base + rng.normal(scale=0.01, size=n),
    "f2": base + rng.normal(scale=0.01, size=n),  # highly correlated with f1
    "f3": rng.normal(size=n),
    "f4": rng.normal(size=n) * 0.5 + base * 0.1,
})
y = (df["f3"] + rng.normal(scale=0.5, size=n) > 0).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    df, y, test_size=0.25, random_state=42, stratify=y
)

# Compute absolute Pearson correlation on training data only.
corr = X_train.corr(numeric_only=True).abs()
upper = corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool))

threshold = 0.9
to_drop = [col for col in upper.columns if any(upper[col] > threshold)]

X_train_reduced = X_train.drop(columns=to_drop)
X_test_reduced = X_test.drop(columns=to_drop)  # apply the same columns to test

print("Dropped features:", to_drop)                 # ['f2']
print("Train shape before:", X_train.shape)         # (225, 4)
print("Train shape after:", X_train_reduced.shape)  # (225, 3)
print("Test shape after:", X_test_reduced.shape)    # (75, 3)

Common Pitfalls

  • Leaking test information: Do not decide which columns to drop using the full dataset. Compute correlations on training folds only; for cross-validation, fit the selection inside a custom transformer or fold loop.

  • Ignoring the target: Two features can be correlated with each other yet both useful for prediction. Validate with cross-validation (for example, StratifiedKFold for classification) before and after removal.

  • Wrong correlation for the data type: Pearson assumes a roughly linear numeric relationship. For ordinal or non-linear monotonic relationships, consider Spearman; for categorical features, correlation filtering is not directly appropriate.

  • Overfitting to the validation set: Repeatedly tuning the threshold on one validation split can inflate performance; use nested evaluation when selection choices are tuned.

  • Interview follow-ups: Expect questions on multicollinearity versus predictive redundancy, why tree models are less sensitive to correlated inputs than linear regression, how you would handle correlation in a Pipeline, and when to prefer PCA or regularization (L2/L1) instead of dropping features.

44

How do I generate polynomial and interaction features with PolynomialFeatures?

PolynomialFeatures creates powers and cross-products from numeric columns, making nonlinear relationships available to a model. It expands the representation of the data, but the additional dimensionality can increase computation, memory use, noise, and overfitting risk.

Read full answer

Applies toscikit-learn 1.5+ (Python 3.12)

How PolynomialFeatures Works

For d numeric inputs, degree=2 with include_bias=False produces the d original columns and d(d + 1) / 2 powers and pairwise interactions.

  • degree: Maximum polynomial degree.

  • interaction_only=True: Retains originals and cross-products but excludes squared and higher powers.

  • include_bias=False: Removes the constant column of ones.

  • get_feature_names_out(input_features=...): Supplies names matching the numeric input columns.

PolynomialFeatures requires numeric data, so categorical columns must be encoded separately.

Feature Engineering Context

Feature engineering turns raw data into model inputs that expose signal; for tabular data, it often matters more than a small algorithm change.

  • Numeric: Scaling, log/Box-Cox/Yeo-Johnson transforms, binning, ratios, polynomials, and interactions.

  • Categorical: One-hot, ordinal, frequency, hashing, and cross-fitted target encoding.

  • Dates: Parts, cyclic sin/cos features, and elapsed time since events.

  • Text: TF-IDF vectors or pretrained embeddings.

  • Entities: Counts and means per entity, including rolling windows where time ordering permits.

Code Example

This example constructs date, ratio, and training-fold-fitted entity-mean features, then places one-hot encoding, polynomial expansion, and scaling inside a pipeline.

import numpy as np
import pandas as pd

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import (
    OneHotEncoder,
    PolynomialFeatures,
    StandardScaler,
    TargetEncoder,
)
from sklearn.utils.validation import check_is_fitted


class GroupMeanTransformer(BaseEstimator, TransformerMixin):
    """Learn per-group means during fit and apply them during transform."""

    def __init__(self, group_column: str, value_column: str):
        self.group_column = group_column
        self.value_column = value_column

    def fit(self, X, y=None):
        self.global_mean_ = float(X[self.value_column].mean())
        self.group_means_ = X.groupby(self.group_column)[self.value_column].mean()
        return self

    def transform(self, X):
        check_is_fitted(self, ["global_mean_", "group_means_"])
        values = X[self.group_column].map(self.group_means_)
        return values.fillna(self.global_mean_).to_numpy().reshape(-1, 1)

    def get_feature_names_out(self, input_features=None):
        return np.asarray([f"{self.group_column}_{self.value_column}_mean"])


rng = np.random.default_rng(42)
n_samples = 240
entity_ids = rng.integers(0, 12, n_samples)
areas = rng.normal(1800, 350, n_samples)
rooms = rng.integers(1, 6, n_samples)
incomes = rng.normal(60000, 12000, n_samples)
listed_at = pd.to_datetime("2023-01-01") + pd.to_timedelta(
    rng.integers(0, 365, n_samples), unit="D"
)

frame = pd.DataFrame({
    "entity_id": entity_ids,
    "district": rng.choice(["North", "South", "West"], n_samples),
    "area": areas,
    "rooms": rooms,
    "income": incomes,
    "listed_at": listed_at,
})

# Deterministic date and ratio features do not learn from the target.
frame["listing_year"] = frame["listed_at"].dt.year
frame["listing_month"] = frame["listed_at"].dt.month
frame["income_per_area"] = frame["income"] / frame["area"]

entity_effect = rng.normal(0, 0.35, 12)
score = (
    1.2 * (frame["income"] / 60000)
    + 0.8 * (frame["area"] / 1800)
    + 0.2 * (frame["rooms"] - 3)
    + entity_effect[frame["entity_id"]]
    + rng.normal(0, 0.3, n_samples)
)
target = (score > score.median()).astype(int)

numeric_columns = [
    "area", "rooms", "income", "income_per_area",
    "listing_year", "listing_month",
]
features = numeric_columns + ["district", "entity_id"]
X, y = frame[features], target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("polynomial", PolynomialFeatures(
        degree=2,
        include_bias=False,
        interaction_only=False,
    )),
    ("scaler", StandardScaler()),
])

preprocessor = ColumnTransformer([
    ("numeric", numeric_pipeline, numeric_columns),
    ("categorical", Pipeline([
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("one_hot", OneHotEncoder(handle_unknown="ignore")),
    ]), ["district"]),
    ("group_mean", GroupMeanTransformer("entity_id", "area"),
     ["entity_id", "area"]),
    ("target", TargetEncoder(cv=5, shuffle=True, random_state=42),
     ["entity_id"]),
])

model = Pipeline([
    ("features", preprocessor),
    ("classifier", LogisticRegression(max_iter=1000)),
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cv_accuracy = cross_val_score(
    model, X_train, y_train, cv=cv, scoring="accuracy"
)

model.fit(X_train, y_train)
probabilities = model.predict_proba(X_test)[:, 1]
predictions = probabilities >= 0.5

print(f"Mean CV accuracy: {cv_accuracy.mean():.3f}")
print(f"Test accuracy: {accuracy_score(y_test, predictions):.3f}")
print(f"Test precision: {precision_score(y_test, predictions):.3f}")
print(f"Test recall: {recall_score(y_test, predictions):.3f}")
print(f"Test F1: {f1_score(y_test, predictions):.3f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, probabilities):.3f}")
print(f"Test PR-AUC: {average_precision_score(y_test, probabilities):.3f}")
print(
    "Transformed feature count:",
    len(model.named_steps["features"].get_feature_names_out()),
)

Feature Selection

Feature selectors should also be fitted inside cross-validation:

  • Filter: Correlation or mutual information.

  • Wrapper: Recursive feature elimination (RFE).

  • Embedded: L1 coefficients or tree feature importances.

  • Model-based: SelectFromModel.

Best Practices

  • Fit imputers, scalers, encoders, group aggregates, target encodings, and selectors only on training folds.

  • Scale for k-NN, SVM, PCA, gradient descent, and usually regularized linear models; trees generally do not require scaling.

  • Use domain knowledge to select meaningful interactions rather than every possible term.

  • For time-dependent data, use TimeSeriesSplit and calculate aggregates only from information available before prediction time.

  • Feature stores such as Feast can keep training and serving feature definitions consistent.

Common Pitfalls

  • Data leakage: Never fit target encoding or aggregates on validation or test data.

  • Raw categories: Do not pass strings or arbitrary category codes to PolynomialFeatures.

  • Dimensionality explosion: Higher degrees grow combinatorially and can increase overfitting.

  • Wrong metric: Accuracy can mislead with imbalanced classes; inspect precision, recall, F1, ROC-AUC, and PR-AUC.

  • Follow-up: Why does interaction_only=True retain original features, and should scaling occur before or after polynomial expansion?

45

How do you implement normalization and standardization in Python with NumPy?

Feature scaling transforms numeric features onto comparable scales so that no single feature dominates distance calculations or gradient updates. Normalization (min-max scaling) and standardization (z-score scaling) both solve this problem for algorithms such as k-NN, SVM, PCA, and gradient descent, but they trade off differently: normalization preserves a fixed range yet is sensitive to outliers, while standardization is more robust to outliers but does not bound values.

Read full answer

Applies toNumPy 2.x (Python 3.12)

Normalization vs Standardization

Method Formula Output range Strengths Weaknesses Typical use
Min-max normalization x_norm = (x - x_min) / (x_max - x_min) Usually [0, 1] Preserves relative spacing; bounded range Outliers compress most values into a small interval Neural network inputs, image pixels, algorithms needing bounded features
Z-score standardization z = (x - mu) / sigma Unbounded (centered near 0) Less affected by outliers than min-max; works well with many linear models Does not produce a fixed range Linear/logistic regression, SVM, PCA, k-means, gradient-based optimization
  • Symbols: x is the original feature value; x_min and x_max are the minimum and maximum of that feature; mu is the feature mean; sigma is the feature standard deviation.

  • For a constant feature (x_max - x_min == 0 or sigma == 0), replace the denominator with 1.0 to avoid division by zero.

How It Works

  • Compute scaling statistics per feature (column-wise), not across the whole array.

  • Apply the same statistics to transform new data; in production, store the fitted x_min/x_max or mu/sigma.

  • Trees (decision trees, random forests, gradient-boosted trees) usually do not need feature scaling because splits depend on order, not magnitude.

Code Example

import numpy as np

rng = np.random.default_rng(42)
# Synthetic data: 6 samples, 3 features with different scales
X = np.column_stack([
    rng.uniform(0, 1, size=6),                         # feature 1: small range
    rng.normal(100, 15, size=6),                       # feature 2: centered near 100
    rng.integers(1000, 5000, size=6).astype(np.float64),  # feature 3: thousands
])

def min_max_normalize(X):
    x_min = X.min(axis=0)
    x_max = X.max(axis=0)
    denom = x_max - x_min
    denom = np.where(denom == 0, 1.0, denom)  # protect constant features
    X_norm = (X - x_min) / denom
    return X_norm, x_min, x_max

def standardize(X):
    mu = X.mean(axis=0)
    sigma = X.std(axis=0, ddof=0)  # population std; sklearn StandardScaler uses ddof=0
    sigma = np.where(sigma == 0, 1.0, sigma)
    Z = (X - mu) / sigma
    return Z, mu, sigma

X_norm, x_min, x_max = min_max_normalize(X)
Z, mu, sigma = standardize(X)

print("Original shape:", X.shape)                 # (6, 3)
print("Normalized min:", X_norm.min(axis=0))      # [0. 0. 0.]
print("Normalized max:", X_norm.max(axis=0))      # [1. 1. 1.]
print("Standardized mean:", np.round(Z.mean(axis=0), 10))  # [0. 0. 0.]
print("Standardized std:", np.round(Z.std(axis=0), 10))    # [1. 1. 1.]

Common Pitfalls

  • Data leakage: compute x_min/x_max or mu/sigma on the training split only, then apply them to validation/test data. In scikit-learn, use StandardScaler or MinMaxScaler inside a Pipeline/ColumnTransformer so each cross-validation fold is fit correctly.

  • Scaling before splitting: fitting on the full dataset leaks test information and inflates metrics.

  • Outlier sensitivity: min-max scaling can collapse most values when one extreme outlier exists; consider standardization or RobustScaler.

  • Unnecessary scaling: applying scaling to tree-based models adds no benefit and extra complexity.

  • Interview follow-ups: “When would you choose normalization over standardization?”, “How do you prevent leakage during cross-validation?”, and “How would you handle a feature with zero variance?”

46

How do you perform feature engineering for recommendation systems?

Feature engineering turns raw interactions, metadata, and context into model-ready inputs that expose preference signal for candidate generation and ranking. For tabular recommendation data, it can matter more than a marginal algorithm change, but it adds pipeline complexity, serving latency, and leakage risk.

Read full answer

Applies toPython 3.12, pandas 2.2+, NumPy 2.x, and scikit-learn 1.5+

Recommended Workflow

  1. Define the prediction time, target, and candidate set; separate candidate generation from ranking.

  2. Create inference-available user, item, session, context, and collaborative features.

  3. Fit learned transformations and as-of aggregates only on training history; validate chronologically.

  4. Version feature definitions and monitor drift, latency, and training-serving skew.

Feature Toolkit

Data type Techniques Main caution
Numeric Scaling, log/Box-Cox/Yeo-Johnson, binning, ratios, interactions, polynomial features Impute and fit transforms within the training pipeline.
Categorical One-hot, ordinal, cross-fitted target, frequency, hashing Target encoding must use training folds only.
Dates and times Parts, time since events, rolling windows, cyclic sin/cos Never include future values.
Text TF-IDF, n-grams, pretrained or learned embeddings Balance dimensionality against semantic quality.
Entities and behavior Counts, means, recency, trends, purchase intervals Compute aggregates as of the prediction timestamp.
Collaborative User/item factors, similarity, co-occurrence Content features remain important for cold start.

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score, f1_score, precision_score,
    recall_score, roc_auc_score,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)

users = pd.DataFrame({
    "user_id": np.arange(20),
    "signup_date": pd.to_datetime("2022-01-01")
        + pd.to_timedelta(rng.integers(0, 700, 20), unit="D"),
})
items = pd.DataFrame({
    "item_id": np.arange(30),
    "category": rng.choice(["electronics", "books", "home"], 30),
    "price": rng.uniform(5, 300, 30).round(2),
})
items["price_vs_category_mean"] = (
    items["price"]
    / items.groupby("category")["price"].transform("mean")
)

n = 500
interactions = pd.DataFrame({
    "user_id": rng.integers(0, 20, n),
    "item_id": rng.integers(0, 30, n),
    "timestamp": pd.to_datetime("2024-01-01")
        + pd.to_timedelta(
            rng.choice(120 * 86_400, size=n, replace=False), unit="s"
        ),
    "rating": rng.integers(1, 6, n),
})
interactions["click"] = (
    rng.random(n) < interactions["rating"] / 10
).astype(int)

events = interactions.merge(users, on="user_id").merge(items, on="item_id")
events = events.sort_values("timestamp").reset_index(drop=True)
events["hour"] = events["timestamp"].dt.hour
events["dow"] = events["timestamp"].dt.dayofweek
events["days_since_signup"] = (
    events["timestamp"] - events["signup_date"]
).dt.days

split = int(0.8 * len(events))
train = events.iloc[:split].copy()
test = events.iloc[split:].copy()

# Point-in-time training aggregates use only earlier interactions.
train["user_prior_count"] = train.groupby("user_id").cumcount()
train["user_prior_mean_rating"] = (
    train.groupby("user_id")["rating"]
    .transform(lambda s: s.shift().expanding().mean())
    .fillna(0.0)
)
train["item_prior_count"] = train.groupby("item_id").cumcount()

# Test aggregates use the complete earlier training history.
user_stats = train.groupby("user_id")["rating"].agg(["size", "mean"])
test["user_prior_count"] = test["user_id"].map(user_stats["size"]).fillna(0)
test["user_prior_mean_rating"] = (
    test["user_id"].map(user_stats["mean"]).fillna(0.0)
)
test["item_prior_count"] = (
    test["item_id"].map(train.groupby("item_id").size()).fillna(0)
)

categorical = ["category"]
numeric = [
    "price", "price_vs_category_mean", "hour", "dow",
    "days_since_signup", "user_prior_count",
    "user_prior_mean_rating", "item_prior_count",
]
preprocess = ColumnTransformer([
    ("categorical", OneHotEncoder(handle_unknown="ignore"), categorical),
    ("numeric", StandardScaler(), numeric),
])
model = Pipeline([
    ("preprocess", preprocess),
    ("classifier", LogisticRegression(max_iter=1000, random_state=42)),
])
model.fit(train[categorical + numeric], train["click"])

probabilities = model.predict_proba(test[categorical + numeric])[:, 1]
predictions = (probabilities >= 0.5).astype(int)
scores = {
    "ROC-AUC": roc_auc_score(test["click"], probabilities),
    "PR-AUC": average_precision_score(test["click"], probabilities),
    "Precision": precision_score(test["click"], predictions, zero_division=1),
    "Recall": recall_score(test["click"], predictions, zero_division=1),
    "F1": f1_score(test["click"], predictions, zero_division=1),
}
for name, value in scores.items():
    print(f"{name}: {value:.3f}")

Evaluation and Feature Selection

Use TimeSeriesSplit or rolling-origin validation for temporal events. Click prediction requires precision, recall, F1, ROC-AUC, and PR-AUC; PR-AUC is especially important for imbalanced clicks because accuracy can hide minority failures. Ranking evaluation should include Recall@K, MAP, or NDCG@K.

Method How it works Strength Weakness Use and cost
Filter Correlation or mutual information Independent and fast Misses interactions Screening; estimator-dependent cost
Wrapper Repeated model evaluation, such as RFE Model-specific Expensive and can overfit Small datasets; M model fits
Embedded L1 coefficients or tree importances Uses model information Can be unstable Models already supporting these signals
SelectFromModel Threshold on model scores Pipeline-friendly Threshold-dependent Refinement; roughly one model fit

Run selection inside each validation fold. A feature store such as Feast supports versioned definitions and consistent offline/online computation. GPUs can accelerate embedding generation, whereas most tabular transformations remain CPU-bound.

Common Pitfalls

  • Using post-click information, future aggregates, or globally fitted target encodings.

  • Treating randomly sampled unexposed items as guaranteed negatives.

  • Imputing values before splitting, or overlooking missingness indicators.

  • Creating excessive correlated features that increase overfitting and latency.

  • Ignoring class imbalance, reproducibility, version changes, or training-serving skew.

Interviewers commonly ask about prediction timestamps, cold start, negative sampling, ranking metrics, leakage prevention, and feature-store consistency.

47

How do you engineer features for predictive maintenance?

Feature engineering turns raw telemetry, maintenance history, and asset metadata into model inputs that expose degradation signals. For predictive maintenance, useful features often include recent sensor trends, rolling baselines, time since service, and labels such as “failure within the next seven days.” The main trade-off is richer predictive signal versus leakage, dimensionality, and the operational cost of computing features consistently.

Read full answer

Applies tomachine learning concepts with Python 3.12, pandas 2.2+, NumPy 2.x, and scikit-learn 1.5+

Predictive-Maintenance Workflow

  1. Define the prediction time, failure horizon, and intervention policy.

  2. Explore telemetry for missingness, outliers, saturation, and sensor drift.

  3. Create features available at scoring time, then validate them chronologically.

  4. Compare classification, survival-time, or remaining-useful-life models.

  5. Monitor feature distributions and retrain when the fleet or sensors change.

Feature engineering frequently matters more than algorithm choice for tabular maintenance data, but a complex model cannot compensate for leaked or unavailable features.

Feature Families

Data type Useful transformations Important considerations
Numeric telemetry Scaling, log/Yeo-Johnson transforms, binning, ratios, polynomial terms, deltas Scale for linear models, SVMs, PCA, and neural networks; trees usually do not require it
Categorical metadata One-hot, ordinal, frequency encoding, hashing, cross-fitted target encoding Target encoding must be learned inside each training fold
Dates and time Month, weekday, holiday flags, elapsed time, sin/cos cyclic encoding, time since maintenance Decide whether the scheduled maintenance date was known at prediction time
Text TF-IDF, work-order descriptions, normalized terms, embeddings Remove boilerplate and prevent future work orders from entering earlier rows
Entity histories Counts, means, quantiles, slopes, rolling windows, time since last failure Use only historical observations; do not aggregate the current or future row

Leakage-Safe Validation

Use a chronological holdout for final evaluation and TimeSeriesSplit for rolling-origin cross-validation. Fit scalers, imputers, encoders, feature selectors, target encoders, and learned aggregates only on each training fold.

For example, an asset’s historical mean can be learned from a training fold and mapped to validation rows. In production, rolling features should contain only readings available before the scoring timestamp. A failure label may legitimately use future information, but the model’s input features may not.

Code Example

import numpy as np
import pandas as pd

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, classification_report, roc_auc_score
from sklearn.model_selection import TimeSeriesSplit, cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
start = pd.Timestamp("2024-01-01")
rows = []

# Small fleet with synthetic degradation and scheduled failures.
for asset_index, failure_day in enumerate([4, 8, 12, 16, 20, 24, 28, 32]):
    equipment_id = f"EQ-{asset_index:02d}"
    maintenance_day = int(rng.integers(0, max(1, failure_day // 2)))
    model_type = "A" if asset_index % 2 == 0 else "B"
    site = f"SITE-{asset_index % 2}"

    for day in range(failure_day + 1):
        wear = day / failure_day
        rows.append({
            "equipment_id": equipment_id,
            "model_type": model_type,
            "site": site,
            "date": start + pd.Timedelta(days=day),
            "maintenance_date": start + pd.Timedelta(days=maintenance_day),
            "temperature": 45 + 12 * wear + rng.normal(0, 0.5),
            "vibration": 0.5 + 1.5 * wear + rng.normal(0, 0.05),
            "pressure": 100 - 15 * wear + rng.normal(0, 1.0),
            "target": int(failure_day - day <= 7),
        })

raw = pd.DataFrame(rows).sort_values(["date", "equipment_id"])


def build_features(frame):
    out = frame.copy()
    month = out["date"].dt.month
    out["month"] = month
    out["dayofweek"] = out["date"].dt.dayofweek
    out["month_sin"] = np.sin(2 * np.pi * month / 12)
    out["month_cos"] = np.cos(2 * np.pi * month / 12)
    out["days_since_maintenance"] = (
        out["date"] - out["maintenance_date"]
    ).dt.days.clip(lower=0)
    out["vibration_temperature_ratio"] = (
        out["vibration"] / out["temperature"].clip(lower=1)
    )
    out["temperature_change_1"] = out.groupby(
        "equipment_id", sort=False
    )["temperature"].diff()
    out["vibration_change_1"] = out.groupby(
        "equipment_id", sort=False
    )["vibration"].diff()
    return out


class AssetAggregateTransformer(BaseEstimator, TransformerMixin):
    """Learn asset baselines from the fit data only."""

    def __init__(self, group_col="equipment_id",
                 value_cols=("temperature", "vibration")):
        self.group_col = group_col
        self.value_cols = value_cols

    def fit(self, X, y=None):
        self.group_means_ = X.groupby(self.group_col)[
            list(self.value_cols)
        ].mean()
        self.global_means_ = X[list(self.value_cols)].mean()
        return self

    def transform(self, X):
        out = X.copy()
        for column in self.value_cols:
            mapping = self.group_means_[column]
            out[f"{column}_asset_train_mean"] = (
                out[self.group_col].map(mapping)
                .fillna(self.global_means_[column])
            )
        return out


data = build_features(raw)
split_date = start + pd.Timedelta(days=18)
train = data[data["date"] < split_date].copy()
test = data[data["date"] >= split_date].copy()

numeric = [
    "temperature", "vibration", "pressure", "month",
    "month_sin", "month_cos", "days_since_maintenance",
    "vibration_temperature_ratio", "temperature_change_1",
    "vibration_change_1", "temperature_asset_train_mean",
    "vibration_asset_train_mean",
]
categorical = ["equipment_id", "model_type", "site", "dayofweek"]

numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
])
categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore", sparse_output=True)),
])
preprocessor = ColumnTransformer([
    ("numeric", numeric_pipeline, numeric),
    ("categorical", categorical_pipeline, categorical),
])

pipeline = Pipeline([
    ("asset_aggregates", AssetAggregateTransformer()),
    ("preprocessor", preprocessor),
    ("model", LogisticRegression(max_iter=1000, class_weight="balanced")),
])

X_train, y_train = train.drop(columns="target"), train["target"]
X_test, y_test = test.drop(columns="target"), test["target"]

cv = TimeSeriesSplit(n_splits=3)
cv_scores = cross_validate(
    pipeline, X_train, y_train, cv=cv,
    scoring=["roc_auc", "average_precision"],
)
print("CV ROC-AUC:", cv_scores["test_roc_auc"])
print("CV PR-AUC:", cv_scores["test_average_precision"])

pipeline.fit(X_train, y_train)
predictions = pipeline.predict(X_test)
print(classification_report(y_test, predictions, zero_division=0))
probabilities = pipeline.predict_proba(X_test)[:, 1]
print("Test ROC-AUC:", roc_auc_score(y_test, probabilities))
print("Test PR-AUC:", average_precision_score(y_test, probabilities))

Selection and Operations

  • Filter methods: mutual information or correlation can remove uninformative variables quickly.

  • Wrapper methods: RFE evaluates subsets but can be expensive.

  • Embedded methods: L1 coefficients, tree importances, and SelectFromModel estimate usefulness during training.

  • Feature stores: Feast and similar systems help reuse feature definitions between training and serving.

  • Deployment: Most tabular feature pipelines run efficiently on CPUs; GPUs become relevant mainly for high-volume sequence models or deep embeddings.

Common Pitfalls

  • Future leakage: Using the final reading in a failure window or computing a “future” rolling statistic invalidates evaluation.

  • Wrong metrics: Accuracy can mislead with rare failures; emphasize precision, recall, F1, ROC-AUC, and PR-AUC.

  • Poor splitting: Random train/test splits mix future and past conditions from the same equipment.

  • Overfitting: Thousands of generated features can overwhelm a small maintenance dataset; test stability and business cost of false alarms.

  • Distribution shift: Sensor replacements and changing operating conditions require drift monitoring.

  • Interview follow-up: Explain why each feature is available at prediction time, how you choose the failure horizon, and how you prevent overlapping windows from leaking information across folds.

48

How do you engineer features for customer churn prediction?

Feature engineering turns raw customer data into model inputs that expose predictive signals for churn. For tabular data, thoughtful features often matter more than the choice of algorithm; the main trade-off is stronger signal versus greater risk of overfitting, leakage, and production complexity.

Read full answer

Applies tomachine learning concepts and Python 3.12 with pandas 2.2+ and scikit-learn 1.5+

Churn Prediction Workflow

  1. Define the prediction point and label: Specify the customer snapshot and the churn horizon, such as cancellation within 30 days.

  2. Create snapshot-safe features: Include tenure, recency, spending, support activity, transactions, contract details, and demographics known at that time.

  3. Transform variables: Scale numeric columns, encode categoricals, extract date components, and create ratios or interactions.

  4. Validate without leakage: Fit learned transformations inside a cross-validation pipeline.

  5. Deploy consistently: Reuse the same feature definitions and point-in-time joins during serving.

Techniques by Data Type

Data type Useful techniques Main consideration
Numeric Scaling; log, Box-Cox, or Yeo-Johnson transforms; binning; ratios; interactions; polynomial features Avoid excessive granularity and unnecessary transformations
Categorical One-hot; ordinal encoding for genuinely ordered categories; cross-fitted target encoding; frequency encoding; hashing for high-cardinality data Target encoding can leak, and hashing loses interpretability and may cause collisions
Date/time Parts such as month and weekday; tenure and event recency; cyclic sin/cos encoding Calculate every value relative to the prediction timestamp
Text TF-IDF, token counts, embeddings Monitor vocabulary drift, dimensionality, and latency
Entity history Counts, means, trends, and rolling-window aggregates Use only events available before prediction

Code Example

This example builds date, ratio, and customer-aggregate features before training a leakage-safe logistic-regression pipeline. Every transaction predates the single prediction cutoff, and no aggregate uses the churn label.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score, f1_score, precision_score,
    recall_score, roc_auc_score
)
from sklearn.model_selection import (
    StratifiedKFold, cross_validate, train_test_split
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)
cutoff = pd.Timestamp("2025-01-01")
n = 400

customers = pd.DataFrame({
    "signup_date": cutoff - pd.to_timedelta(
        rng.integers(30, 730, n), unit="D"
    ),
    "last_login": cutoff - pd.to_timedelta(
        rng.integers(0, 90, n), unit="D"
    ),
    "monthly_spend": rng.gamma(2.0, 35.0, n),
    "support_tickets": rng.poisson(1.5, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "contract_type": rng.choice(["monthly", "annual"], n)
})

event_counts = rng.integers(1, 5, n)
events = pd.DataFrame({
    "customer_id": np.repeat(np.arange(n), event_counts),
    "event_date": cutoff - pd.to_timedelta(
        rng.integers(1, 180, event_counts.sum()), unit="D"
    ),
    "amount": rng.gamma(2.0, 20.0, event_counts.sum())
})
aggregates = events.groupby("customer_id").agg(
    transaction_count=("amount", "size"),
    avg_transaction_amount=("amount", "mean")
)
# customers is indexed by customer id (0..n-1), matching the aggregates index.
customers = customers.join(aggregates)

customers["tenure_months"] = (
    cutoff - customers["signup_date"]
).dt.days / 30.44
customers["last_login_days"] = (
    cutoff - customers["last_login"]
).dt.days
customers["spend_per_tenure_month"] = (
    customers["monthly_spend"] / customers["tenure_months"]
)

month = customers["signup_date"].dt.month
customers["signup_month_sin"] = np.sin(2 * np.pi * month / 12)
customers["signup_month_cos"] = np.cos(2 * np.pi * month / 12)

score = (
    -1.8
    + 0.12 * customers["support_tickets"]
    - 0.01 * customers["last_login_days"]
    + 0.6 * (customers["contract_type"] == "monthly")
    + 0.0002 * customers["monthly_spend"]
)
probability = 1.0 / (1.0 + np.exp(-score.to_numpy()))
customers["churn"] = rng.binomial(1, probability)

numeric = [
    "tenure_months", "last_login_days", "monthly_spend",
    "support_tickets", "spend_per_tenure_month",
    "signup_month_sin", "signup_month_cos",
    "transaction_count", "avg_transaction_amount"
]
categorical = ["region", "contract_type"]
X = customers[numeric + categorical]
y = customers["churn"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

preprocessor = ColumnTransformer([
    ("numeric", StandardScaler(), numeric),
    ("categorical", OneHotEncoder(
        handle_unknown="ignore", sparse_output=False
    ), categorical)
])
pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", LogisticRegression(max_iter=1000))
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_validate(
    pipeline, X_train, y_train, cv=cv,
    scoring=["roc_auc", "average_precision"]
)
for name, values in scores.items():
    print(f"CV {name}: {values.mean():.3f}")

pipeline.fit(X_train, y_train)
predictions = pipeline.predict(X_test)
probabilities = pipeline.predict_proba(X_test)[:, 1]

print(f"Precision: {precision_score(y_test, predictions):.3f}")
print(f"Recall: {recall_score(y_test, predictions):.3f}")
print(f"F1: {f1_score(y_test, predictions):.3f}")
print(f"ROC-AUC: {roc_auc_score(y_test, probabilities):.3f}")
print(f"PR-AUC: {average_precision_score(y_test, probabilities):.3f}")

Validation and Feature Selection

Use StratifiedKFold for ordinary classification and TimeSeriesSplit when snapshots are ordered in time. Keep scalers, imputers, encoders, and feature selection inside the pipeline so validation data cannot influence fitting. Cross-fitted target encoding must also compute each training fold’s encodings from the other folds.

Churn is often imbalanced, so accuracy can mislead. Evaluate precision, recall, F1, ROC-AUC, and PR-AUC, then choose the metric according to the cost of missed churners versus unnecessary retention actions.

Feature-selection methods include:

  • Filter: mutual information or correlation.

  • Wrapper: recursive feature elimination (RFE).

  • Embedded: L1 regularization or tree-based importance.

  • Model-based: SelectFromModel with an appropriate estimator.

Apply selection within each training fold or evaluation pipeline.

Production

A feature store such as Feast can provide versioned offline and online definitions, point-in-time historical retrieval, and consistent training-serving values. Monitor feature drift, missingness, and differences between offline and online values.

Common Pitfalls

  • Target leakage: Do not use cancellation status, future spending, or post-snapshot activity.

  • Aggregate leakage: Respect each prediction timestamp and never calculate aggregates from future events.

  • Wrong validation: Do not randomly split time-dependent snapshots or repeatedly tune against the test set.

  • Poor metric choice: Accuracy may hide poor performance on the minority churn class.

  • Unnecessary complexity: Too many bins, interactions, or high-dimensional text features can increase variance.

  • Incorrect scaling: Scale for k-NN, SVM, PCA, and gradient-descent models; scaling is usually unnecessary for trees.

  • Poor reproducibility: Fix random seeds and version feature code and data.

  • Training-serving mismatch: Reuse identical feature definitions and transformations in both environments.

49

How do you engineer features for real-time web traffic anomaly detection?

Feature engineering converts raw web-traffic events into point-in-time signals that expose unusual behavior to an anomaly detector. It solves the problem of sparse, noisy, and imbalanced traffic data; meaningful features often matter more than changing algorithms. The main trade-off is detection quality versus streaming latency, feature complexity, and sensitivity to traffic drift.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12 with NumPy 2.x, pandas 2.2+, and scikit-learn 1.5+)

Approach

Build features from causal event-time windows rather than individual requests alone. Useful signals include request rates, error ratios, latency, payload sizes, endpoint deviations, session behavior, and changes from each endpoint’s historical baseline.

Every feature must be available when an event is scored. In production, update streaming aggregates incrementally, apply identical transformations during training and serving, and monitor feature freshness and distribution drift.

Feature Toolkit

Data type Useful techniques Examples
Numeric Scaling, ratios, deviations, robust transformations Requests/minute, error ratio, latency versus baseline, log bytes
Categorical One-hot, frequency, cross-fitted target encoding, hashing Endpoint, HTTP method, status category, country
Time Date parts, cyclic encoding, time since event Hour, weekend, seconds since previous request
Text TF-IDF, hashing, embeddings Rare URL path, user-agent, suspicious payload token
Entity history Counts, means, standard deviations over windows Requests per IP, endpoint errors, session length
Traffic context Multi-window changes and interactions Current requests divided by prior-hour requests

Evaluate feature selection within the training process: filters include mutual information and correlation, wrappers include RFE, and embedded methods include L1 coefficients and tree importances. SelectFromModel is useful when model-derived importance is preferable to raw correlations.

Leakage and Online Design

  • Compute rolling aggregates causally; use shift(1) when a feature compares an observation only with prior history.

  • Fit scalers, imputers, encoders, and target encoders on training data only.

  • Cross-fit target encoding within training folds and never use validation or test labels.

  • Respect event time, late-arriving events, and watermarks; distinguish event time from processing time.

  • Store reusable definitions in a system such as Feast so training and serving use the same features.

  • Monitor cardinality, missingness, drift, and feature freshness.

Code Example

This labeled synthetic backtest creates causal endpoint-level features, selects a threshold with time-ordered out-of-fold predictions, and evaluates a final temporal holdout.

import numpy as np
import pandas as pd
from sklearn.base import clone
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import (
    accuracy_score, average_precision_score, f1_score,
    precision_recall_curve, precision_score, recall_score,
    roc_auc_score,
)
from sklearn.model_selection import TimeSeriesSplit
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

rng = np.random.default_rng(42)

times = pd.date_range("2025-01-01", periods=720, freq="h")
endpoints = ["/login", "/search", "/checkout", "/api"]
df = pd.DataFrame(
    [(ts, endpoint) for ts in times for endpoint in endpoints],
    columns=["timestamp", "endpoint"],
)
df["method"] = rng.choice(["GET", "POST"], len(df))

# Exogenous ground truth for the labeled synthetic backtest.
attack = (rng.random(len(df)) < 0.03).astype(int)
attack[[0, 4, 8]] = 1  # Ensure the earliest CV training fold has positives.
y = attack.copy()

df["requests"] = rng.poisson(18, len(df)) + attack * rng.poisson(55, len(df))
df["bytes"] = df["requests"] * rng.lognormal(7, 0.5, len(df))
df["status_5xx"] = rng.binomial(
    df["requests"], np.where(attack == 1, 0.25, 0.02)
)
df["latency_ms"] = rng.lognormal(3, 0.4, len(df)) + attack * 900

# Previous 24 hours only; the current observation is excluded.
group = df.groupby("endpoint", sort=False)
df["requests_24h"] = group["requests"].transform(
    lambda s: s.shift(1).rolling(24, min_periods=1).sum()
)
df["baseline_requests"] = group["requests"].transform(
    lambda s: s.shift(1).rolling(24, min_periods=1).mean()
)
df["baseline_latency"] = group["latency_ms"].transform(
    lambda s: s.shift(1).rolling(24, min_periods=6).mean()
)

safe_requests = df["requests"].clip(lower=1)
df["error_rate"] = df["status_5xx"] / safe_requests
df["bytes_per_request"] = df["bytes"] / safe_requests
df["requests_vs_baseline"] = safe_requests / df["baseline_requests"]
df["latency_vs_baseline"] = df["latency_ms"] / df["baseline_latency"]

hour = df["timestamp"].dt.hour
df["hour_sin"] = np.sin(2 * np.pi * hour / 24)
df["hour_cos"] = np.cos(2 * np.pi * hour / 24)
df["is_weekend"] = df["timestamp"].dt.dayofweek >= 5

numeric = [
    "requests", "requests_24h", "requests_vs_baseline", "error_rate",
    "bytes_per_request", "latency_vs_baseline", "hour_sin", "hour_cos",
    "is_weekend",
]
categorical = ["endpoint", "method"]

preprocessor = ColumnTransformer([
    ("numeric", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
    ]), numeric),
    ("categorical", OneHotEncoder(handle_unknown="ignore"), categorical),
])

def build_model():
    return Pipeline([
        ("features", clone(preprocessor)),
        ("classifier", RandomForestClassifier(
            n_estimators=300, class_weight="balanced",
            random_state=42, n_jobs=-1,
        )),
    ])

unique_times = pd.DatetimeIndex(sorted(df["timestamp"].unique()))
split_at = int(len(unique_times) * 0.8)
train_times, test_times = unique_times[:split_at], unique_times[split_at:]

oof = np.zeros(len(df))
covered = np.zeros(len(df), dtype=bool)
tscv = TimeSeriesSplit(n_splits=4)
for train_idx, valid_idx in tscv.split(train_times):
    train_mask = df["timestamp"].isin(train_times[train_idx])
    valid_mask = df["timestamp"].isin(train_times[valid_idx])
    model = build_model()
    model.fit(df.loc[train_mask], y[train_mask])
    oof[valid_mask] = model.predict_proba(df.loc[valid_mask])[:, 1]
    covered |= valid_mask

precision, recall, thresholds = precision_recall_curve(y[covered], oof[covered])
f1 = np.divide(
    2 * precision[:-1] * recall[:-1],
    precision[:-1] + recall[:-1],
    out=np.zeros_like(precision[:-1]),
    where=(precision[:-1] + recall[:-1]) > 0,
)
threshold = thresholds[np.argmax(f1)]

model = build_model()
train_mask = df["timestamp"].isin(train_times)
test_mask = df["timestamp"].isin(test_times)
model.fit(df.loc[train_mask], y[train_mask])
scores = model.predict_proba(df.loc[test_mask])[:, 1]
predictions = (scores >= threshold).astype(int)
y_test = y[test_mask]

print({
    "accuracy": accuracy_score(y_test, predictions),
    "precision": precision_score(y_test, predictions, zero_division=0),
    "recall": recall_score(y_test, predictions, zero_division=0),
    "f1": f1_score(y_test, predictions, zero_division=0),
    "roc_auc": roc_auc_score(y_test, scores),
    "pr_auc": average_precision_score(y_test, scores),
})

Validation and Operations

Evaluate imbalanced anomalies with precision, recall, F1, ROC-AUC, and PR-AUC; accuracy can be misleading when attacks are rare. Use time-ordered validation because random splits can expose future traffic to earlier predictions. Scaling is unnecessary for trees but required for distance-based, kernel, PCA, and many gradient-descent models; fixed seeds and recorded model versions support reproducibility.

Common Pitfalls

  • Leakage: calculating baselines, encodings, or aggregates with future information.

  • Wrong metric: optimizing accuracy for a highly imbalanced event stream.

  • Cold start: new endpoints or IPs lack historical baselines; use global or hierarchical fallbacks.

  • Overfitting: repeatedly tuning features or thresholds against the test set.

  • Concept drift: attack patterns and normal traffic change; monitor recall and feature distributions.

  • Oversized features: unbounded cardinality increases memory use and latency; cap, hash, or expire values.

50

How do you engineer features for sentiment prediction in NLP?

Feature engineering for sentiment prediction transforms raw text into numeric inputs that expose polarity cues (words, phrases, and context) to a model. It solves the problem that classifiers cannot consume text directly, and for many tabular-style NLP baselines it can matter as much as model choice. The main trade-off is between simple, interpretable sparse features and richer representations that capture meaning but require more data, compute, and care to avoid leakage.

Read full answer

Applies toscikit-learn 1.5+ and pandas 2.2+ (Python 3.12)

Feature Engineering Strategy

  • Text preprocessing: Normalize case, handle punctuation/emojis, and tokenize. Keep negation cues (not, n't) because they flip sentiment; aggressive stop-word removal can drop them.

  • Bag-of-words features: Use TfidfVectorizer with unigrams and bigrams to capture phrases like "not good". Tune min_df, max_df, and ngram_range to control sparsity.

  • Numeric/metadata features: Add review length, punctuation counts, exclamation/question marks, or lexicon scores (e.g., counts of positive/negative terms) as numeric columns.

  • Embeddings: Replace or augment TF-IDF with pretrained sentence/document embeddings for semantic similarity; average token vectors only as a simple baseline. Fit any downstream reducer on training data only.

  • Feature selection: Apply filter methods (mutual information), embedded methods (SelectFromModel with L1 logistic regression or tree importances), or RFE to reduce noise and overfitting.

  • Feature reuse in production: Use a feature store (e.g., Feast) to define and serve the same features consistently between training and inference.

Approach How it works Strengths Weaknesses When to use
TF-IDF + linear model Sparse term weights Fast, interpretable, strong baseline Misses word order/semantics beyond n-grams Small/medium labeled data, need transparency
Pretrained embeddings + classifier Dense semantic vectors Captures meaning and paraphrases Needs embedding model; less interpretable Limited labels or richer language variation
Handcrafted lexicon/metadata Counts and ratios Cheap, domain-specific signal Brittle, incomplete coverage As features combined with TF-IDF/embeddings

Code Example

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)

data = pd.DataFrame({
    "text": [
        "I love this product, it works great",
        "Absolutely wonderful experience and quality",
        "Best purchase I have made",
        "Terrible product, broke after one day",
        "I hate this, complete waste of money",
        "Very disappointing and poor quality",
        "It is okay, nothing special",
        "Average performance, does the job",
        "Not bad but not great either",
        "Fantastic value and excellent service",
        "Awful customer support and defective item",
        "Pretty good overall, would recommend",
    ],
    "sentiment": [1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 1],  # 1=positive, 0=negative
})
# Simple numeric metadata feature available at prediction time.
data["exclaim_count"] = data["text"].str.count("!").astype(float)

X = data[["text", "exclaim_count"]]
y = data["sentiment"]

# Split before fitting any transformer to prevent leakage.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

preprocess = ColumnTransformer([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2), min_df=1), "text"),
    ("num", StandardScaler(), ["exclaim_count"]),
])

pipeline = Pipeline([
    ("prep", preprocess),
    ("clf", LogisticRegression(max_iter=1000, random_state=42)),
])

cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
f1_scores = cross_val_score(pipeline, X_train, y_train, cv=cv, scoring="f1")
print(f"CV F1 mean={f1_scores.mean():.3f} std={f1_scores.std():.3f}")
# F1 varies; dataset is intentionally tiny for illustration.

pipeline.fit(X_train, y_train)
print(classification_report(y_test, pipeline.predict(X_test), zero_division=0))

Common Pitfalls

  • Data leakage: Fitting TfidfVectorizer, scalers, or target/aggregate features on the full dataset leaks label or test information. Keep all learned preprocessing inside a Pipeline/ColumnTransformer and fit on training folds only.

  • Wrong metric: Accuracy can mislead on imbalanced sentiment classes; prefer precision, recall, F1, ROC-AUC, or PR-AUC and inspect confusion matrices.

  • Ignoring negation and context: Plain unigrams may miss "not good"; include bigrams or richer representations.

  • Overfitting sparse n-grams: High max_features/rare n-grams can memorize training text; regularize and validate with cross-validation.

  • Interview follow-ups: How would you handle class imbalance, domain shift, sarcasm, or multilingual text? When would you choose TF-IDF over embeddings or a pretrained transformer?

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 50 Feature Engineering answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects