Skip to content
Machine Learning & Data Science

Computer Vision interview questions and answers

54 questions, 11 with code. About 5 h 22 min to read them all.

By TestLab · Updated

Plan for an interview
54 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is computer vision and how does it compare with human vision?

Computer vision is the field of enabling computers to interpret images and video, identifying objects, patterns, spatial relationships, and semantic meaning. It resembles human vision because both transform light into structured information, but computer systems typically depend on learned models and explicit training data, while human vision combines biological processing, context, experience, and rapid adaptation. The main trade-off is that machines can analyze images consistently and at scale, whereas humans generally generalize better to unfamiliar situations and use broader common-sense reasoning.

Read full answer

Applies tomachine learning and computer vision concepts (all versions)

How Computer Vision Works

A typical computer-vision system converts visual input into numerical representations and then analyzes those representations for a specific task:

  1. Acquire an image or video: Data may come from cameras, medical scanners, satellites, or stored files.

  2. Preprocess the input: Common operations include resizing, color conversion, normalization, cropping, noise removal, and augmentation.

  3. Extract features: Traditional methods use hand-designed features, while modern systems usually learn representations directly from pixels.

  4. Train or run a model: The model learns patterns during training and later performs inference on new data.

  5. Produce task-specific outputs: Examples include a class label, bounding boxes, pixel masks, object trajectories, or recognized text.

Pretrained models are often used with transfer learning, especially when the available dataset is small. For models pretrained on natural images, preprocessing normally follows the model’s expected normalization, such as its documented channel mean and standard deviation.

Core Tasks and Modern Approaches

Task Typical output Common approaches
Image classification One or more labels for the whole image ResNet, EfficientNet, Vision Transformers
Object detection Bounding boxes, labels, and confidence scores Faster R-CNN, YOLO family, DETR
Semantic segmentation One class label per pixel U-Net and other encoder-decoder architectures
Instance segmentation Pixel masks for individual objects Mask R-CNN, DETR-based models
Keypoint or pose estimation Locations of joints, facial landmarks, or other keypoints CNNs, Vision Transformers, pose-specific networks
Tracking Object identities and trajectories across frames Detection plus matching, optical flow, transformer trackers
OCR Text, words, characters, and sometimes layout Detection and recognition pipelines
Image generation New or modified images Diffusion models, GANs

Other important representations include CLIP embeddings, which place images and text in a shared vector space for tasks such as zero-shot classification and image-text retrieval.

Relationship to Human Vision

Both human and machine vision rely on related principles: detecting edges, contrasts, textures, shapes, motion, and spatial relationships. However, their implementations and limitations differ substantially:

Human vision Computer vision
Uses the retina, optic nerves, and visual cortex Uses cameras, sensors, algorithms, and learned models
Learns continuously from everyday experience Usually learns from explicitly prepared training data
Combines vision with memory, reasoning, and common sense Often performs a narrowly defined task
Handles occlusion, changing viewpoints, and unusual lighting well in many cases May struggle with distribution shifts and unfamiliar contexts
Automatically focuses, adapts, and uses depth cues Focus, exposure, depth, and alignment usually require explicit handling
Has substantial energy and computational cost Can process large volumes consistently and often in parallel

Computer vision therefore does not reproduce human sight in full. It provides specialized approximations of particular abilities.

Evaluation

Metrics must match the output of the task:

  • Classification: Accuracy, precision, recall, F1, top-k accuracy, ROC-AUC, and PR-AUC.

  • Object detection: Intersection over Union (IoU) and mean Average Precision (mAP).

  • Semantic and instance segmentation: Mean Intersection over Union (mIoU) and Dice score.

  • Keypoint estimation: Normalized keypoint error or percentage of correctly located points.

  • Tracking: Identity consistency, trajectory quality, and fragmentation or identity-switch rates.

Accuracy can be misleading for imbalanced datasets: a model predicting the majority class every time may achieve high accuracy while failing to detect rare but important cases.

Practical Considerations

  • Data quality: Label errors, inconsistent annotations, poor lighting, and underrepresented groups can reduce real-world performance.

  • Augmentation: Flips, crops, color jitter, and other transformations improve robustness when they represent plausible deployment conditions.

  • Overfitting: Validation data should not be repeatedly used for manual tuning, and final evaluation should use untouched test data.

  • Deployment: Latency, memory, model size, quantization, and hardware determine whether a model is suitable for cloud, mobile, or edge devices. ONNX export can improve interoperability with supported inference runtimes.

  • GPU versus CPU: GPUs accelerate large neural-network training and inference, while CPUs may be sufficient for compact models or low-throughput edge workloads.

  • Privacy and bias: Face recognition, medical imaging, and surveillance systems require careful attention to consent, demographic performance, security, and legal obligations.

Common Pitfalls

  • Confusing computer vision with image processing: preprocessing transforms pixels, whereas vision models infer higher-level meaning.

  • Evaluating on data similar to the training set and then claiming real-world generalization.

  • Ignoring class imbalance, localization quality, or per-class performance.

  • Using inappropriate augmentation, such as horizontal flips when text or anatomical direction makes them invalid.

  • Forgetting that OpenCV loads images in BGR order, while most image models expect RGB.

  • Reporting accuracy alone for safety-critical or imbalanced tasks.

Common interview follow-ups include: How do convolutional networks learn hierarchical features? Why use transfer learning? How would you detect data leakage? Which metrics matter for a medical-imaging system? How would you choose a deployment-friendly model?

2

What are the key components of a computer vision system?

A computer vision system converts raw pixels into useful outputs such as labels, bounding boxes, masks, text, object identities, or generated images. Its components must cover data, modeling, evaluation, and operations while balancing predictive quality, data requirements, latency, compute, and complexity.

Read full answer

Applies tomachine learning and computer vision concepts (Python 3.12 ecosystem)

Core Architecture

A typical production pipeline contains these layers:

  • Input acquisition: Images come from cameras, videos, scanners, satellites, or stored files. Acquisition must account for lighting, motion blur, calibration, frame rate, and privacy.

  • Preprocessing: Images are resized, cropped, color-corrected, normalized, or converted into tensors. OpenCV loads images in BGR, while most deep-learning models expect RGB.

  • Representation learning: A backbone converts images into feature maps. Common backbones include ResNet, EfficientNet, and Vision Transformer (ViT) models. CLIP provides joint image-text embeddings for retrieval and zero-shot classification.

  • Task-specific head: Backbone features are converted into the required output. Classification predicts class scores, while detection predicts object locations and labels.

  • Training and optimization: A task-appropriate loss is minimized using an optimizer and backpropagation. Training data update the model, validation data guide model selection, and checkpoints preserve the selected weights.

  • Inference engine: The trained weights and the same preprocessing, prediction, and postprocessing path are deployed together.

  • MLOps and monitoring: Versioned data, reproducible runs, model registries, deployment pipelines, latency monitoring, and drift detection support reliable operation.

Tasks, Models, and Metrics

Task Typical output Representative approaches Evaluation metrics
Image classification One or more class labels ResNet, EfficientNet, ViT Accuracy, top-k accuracy, precision, recall, F1, ROC-AUC, PR-AUC
Object detection Boxes, labels, confidence scores Faster R-CNN, YOLO family, DETR IoU-based mAP, precision, recall
Semantic segmentation Class label per pixel U-Net, encoder-decoder networks mIoU, Dice
Instance segmentation Separate mask per object Mask R-CNN Mask mAP, mask IoU, Dice
Promptable segmentation Mask for a prompted object or region SAM Mask IoU, Dice
Keypoint or pose estimation Joint or landmark coordinates CNN and ViT pose models PCK, MPJPE
Tracking Identities and trajectories over time SORT, DeepSORT, ByteTrack HOTA, IDF1, MOTA
OCR Text and confidence values Detection plus text recognition Character error rate, word error rate
Image generation Generated images or edited regions Diffusion models and GANs Human evaluation and complementary quantitative measures

IoU measures overlap between predicted and ground-truth regions, while Dice emphasizes pixel-level agreement. Accuracy can mislead for imbalanced classification because a majority-class-only model may score highly while failing on rare but important cases.

Data and Training Pipeline

Training data must represent deployment conditions and have accurate annotations. Split it into training, validation, and test sets. Data-derived preprocessing parameters, such as normalization statistics, must be learned from the training split only; a scikit-learn Pipeline or ColumnTransformer helps prevent data leakage in classical pipelines.

Useful augmentation includes crops, horizontal flips, rotations, color jitter, and blur. Choose it according to the domain: flips may be invalid for text or medical laterality. Transfer learning is the default for small datasets because it reuses pretrained representations and needs fewer labels. Fine-tuning often uses a lower learning rate and may freeze early layers. Address class imbalance with suitable sampling, loss weighting, or augmentation.

Use cross-validation when more robust estimates are needed: StratifiedKFold for classification and TimeSeriesSplit for time-ordered data. Keep the final test set untouched during training and model selection.

Evaluation and Reliability

Metrics must match the operational objective. Detection commonly matches predictions and annotations using IoU thresholds, while segmentation evaluates region overlap. Precision reflects false positives; recall reflects missed objects.

Test performance across relevant lighting, viewpoints, object sizes, environments, and demographic groups. Inspect false positives, false negatives, localization errors, and subgroup failures rather than relying only on an aggregate score. Record seeds, dataset versions, model weights, hyperparameters, software versions, and augmentation settings for reproducibility.

Deployment and Operations

  • Latency: Measure end-to-end response time, including decoding, preprocessing, inference, and postprocessing.

  • Model size and memory: Smaller backbones or reduced input resolution can support edge and mobile devices.

  • Hardware: CPUs can handle small models, while GPUs or specialized accelerators improve throughput for larger CNNs and transformers.

  • Optimization: Quantization, pruning, optimized runtimes, and ONNX export can reduce resource use, but outputs and accuracy must be revalidated.

  • Reliability: Handle missing frames, corrupted images, low-confidence results, and out-of-distribution inputs.

  • Safety and privacy: Face recognition, medical imaging, and surveillance require access controls, retention limits, consent, fairness testing, and bias monitoring.

Interview Tips

  • Common mistake: Fitting preprocessing before splitting the data and causing leakage.

  • Common mistake: Reporting accuracy for an imbalanced problem without precision, recall, F1, or PR-AUC.

  • Common mistake: Comparing models with different input resolutions, augmentation policies, or pretrained weights.

  • Common mistake: Measuring GPU inference time while ignoring end-to-end latency.

  • Useful follow-up questions: How are annotations produced? How will rare cases and demographic groups be evaluated? What happens when confidence is low? Which hardware and latency target define success? How will model updates be monitored and rolled back?

3

What is image segmentation in computer vision?

Image segmentation is the task of partitioning an image into regions by assigning a meaningful label to each pixel or group of pixels. It solves the problem of understanding where objects, boundaries, or regions are at pixel-level detail, which is essential for medical imaging, autonomous driving, and scene understanding. Its main trade-off is that pixel-accurate predictions require dense annotations and more computation than image classification or bounding-box detection.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Types of Image Segmentation

Type Output Typical use Example approaches
Semantic segmentation One class label per pixel; all pixels of the same class share that label Road/sky/building parsing, organ delineation U-Net, DeepLab-style encoder–decoder CNNs, SegFormer (transformer-based)
Instance segmentation A class label and a separate mask per object instance Counting and separating individual people, cells, or products Mask R-CNN, YOLACT-style models
Panoptic segmentation Combines semantic labels for “stuff” (sky, road) with instance masks for “things” (cars, people) Full scene parsing Mask2Former-style unified models
Promptable segmentation A mask generated from a user prompt (point, box, or text) Interactive annotation and general-purpose masking SAM (Segment Anything Model)

How It Works

  • Encoder–decoder networks: A backbone (e.g., ResNet, EfficientNet, or a Vision Transformer/ViT) extracts features at multiple scales, and a decoder upsamples them to produce a per-pixel class map. Skip connections (as in U-Net) preserve spatial detail lost during downsampling.

  • Transfer learning: For small datasets, start from a backbone pretrained on a large dataset and fine-tune the segmentation head; freeze early layers if data is very limited.

  • Preprocessing and augmentation: Resize or crop to a fixed input size, normalize using the pretrained model’s mean and standard deviation, and apply flips, random crops, and color jitter. Keep geometric transforms consistent between the image and its mask.

Evaluation Metrics

  • mIoU (mean Intersection over Union): For each class, IoU = intersection / union of predicted and ground-truth pixels; mIoU averages IoU across classes. It penalizes both false positives and false negatives.

  • Dice coefficient (F1 for pixels): Dice = 2 * intersection / (predicted + ground truth). Common in medical imaging, especially with small foreground regions.

  • Pixel accuracy: The fraction of correctly labeled pixels; it can be misleading on imbalanced scenes where a dominant class (e.g., background) inflates the score.

Practical Considerations

  • Class imbalance: Large background regions dominate loss and metrics; use class-weighted loss, focal loss, or Dice loss and report mIoU/Dice rather than pixel accuracy.

  • Complexity: Dense prediction scales with image resolution. For an input of H × W pixels, producing a full-resolution mask is roughly proportional to H × W times the network cost; high-resolution inference increases memory and latency.

  • Deployment: For real-time or edge use, reduce input size, use efficient backbones, and apply quantization or ONNX export. Validate latency and memory on the target device.

  • Bias and privacy: In face or person segmentation, evaluate performance across demographic groups and handle identifiable imagery according to privacy requirements.

Common Pitfalls

  • Mismatched masks: Applying augmentations to the image but not the mask, or using interpolation that corrupts label IDs (use nearest-neighbor for masks).

  • Wrong normalization: Forgetting the pretrained backbone’s mean/std, or feeding BGR images (OpenCV default) when the model expects RGB.

  • Metric mismatch: Reporting accuracy on highly imbalanced data instead of mIoU or Dice.

  • Leakage: Tiling or augmenting before the train/validation split, causing overlapping pixels across splits.

Interview follow-ups: How does semantic segmentation differ from instance segmentation? When would you choose Dice loss over cross-entropy? How does U-Net’s skip connection help? How would you segment a small dataset with limited labels?

4

How does image processing differ from computer vision?

Image processing transforms pixels through operations such as resizing, filtering, sharpening, and edge detection. Computer vision interprets visual content to identify objects, understand scenes, or make predictions, often using classical algorithms, machine learning, or deep neural networks. The main trade-off is that image processing is usually explicit and deterministic, whereas computer vision requires learned or engineered representations that can generalize but may demand more data and computation.

Read full answer

Applies toimage processing and computer vision concepts (all versions)

Core Difference

Aspect Image Processing Computer Vision
Primary goal Modify or extract low-level pixel information Derive meaningful, task-specific information from images or video
Typical inputs Pixel arrays, masks, and histograms Images, video frames, sensor data, 3D scans or point clouds, and images with labels or prompts
Common methods Resizing, cropping, convolution, thresholding, blur, and Canny edges CNNs, vision transformers, object detectors, segmentation models, OCR, and generative models
Typical outputs Enhanced images, filtered images, edges, masks, and measurements Labels, bounding boxes, pixel masks, keypoints, text, embeddings, and generated images
Strengths Predictable, interpretable, and often effective with limited data Learns complex patterns and handles varied real-world conditions
Weaknesses Limited semantic understanding Greater data, compute, training, and deployment requirements
When to use Preprocessing, image cleanup, measurement, and deterministic transformations Classification, detection, segmentation, recognition, tracking, and scene understanding
Scale Often approximately O(H × W) per fixed-size full-image operation, with O(H × W) output storage Depends heavily on the model, input resolution, architecture, and hardware; often has higher compute and memory costs

The fields overlap considerably. Image processing is commonly a preprocessing stage in a computer vision pipeline, while traditional computer vision systems can use substantial image processing without neural networks.

Main Computer Vision Tasks

Task Typical output Common approach Evaluation
Image classification One or more class labels ResNet, EfficientNet, ViT Accuracy, top-k accuracy
Object detection Bounding boxes, class labels, and confidence scores Faster R-CNN, YOLO family, DETR IoU, mAP
Semantic segmentation One class label per pixel U-Net, DeepLab, SegFormer mIoU, Dice
Instance segmentation A separate pixel mask for each object Mask R-CNN Mask IoU, AP
Promptable segmentation A pixel mask selected using points, boxes, or other prompts SAM Mask IoU and prompt-based evaluation
Keypoint or pose estimation Coordinates of body joints or other keypoints CNN and transformer pose models PCK, OKS
Tracking Object positions and persistent identities Kalman filters, Deep SORT, transformer trackers IDF1, HOTA
OCR Detected and recognized text Detection and text-recognition models Text accuracy, word error rate
Image-text matching Similarity scores, retrieved images, or aligned embeddings CLIP Recall@K or task-specific alignment metrics
Image generation A new or transformed image GANs and diffusion models Human evaluation and perceptual metrics

How They Work Together

A typical vision system follows a pipeline:

  1. Preprocess: Decode the image, convert color spaces, resize it, and normalize values.

  2. Prepare the input: Apply the pretrained model’s required channel order, mean, and standard deviation.

  3. Extract or learn features: Apply classical filters or feed the image into a trained neural network.

  4. Produce task output: Generate labels, boxes, masks, keypoints, text, or embeddings.

  5. Evaluate and deploy: Measure task-specific performance, then optimize the model for the target environment.

OpenCV reads images in BGR order, whereas most PyTorch and torchvision models expect RGB. Pretrained models also require their documented normalization values. For small datasets, transfer learning is usually a strong starting point; training augmentation should use only training data.

Engineering and Deployment Concerns

  • Latency: Favor smaller backbones, lower input resolution, optimized inference, or hardware-specific acceleration when real-time response is required.

  • Model size: Neural networks may require substantially more storage and memory than classical processing pipelines.

  • Quantization: Reducing numerical precision can lower memory use and accelerate compatible inference, but should be validated for accuracy.

  • ONNX export: torch.onnx.export can provide an interoperable deployment format, although operators and dynamic shapes require testing with the selected runtime.

  • Hardware: CPUs are often sufficient for classical processing or small models; GPUs are useful for training and demanding neural-network inference.

  • Robustness: Changes in lighting, viewpoint, blur, weather, camera quality, and background can reduce performance.

  • Privacy and bias: Face analysis, medical imaging, and surveillance require careful handling of sensitive data, demographic performance differences, and access controls.

Common Pitfalls

  • Treating an edge-detected image as equivalent to object recognition; edges are features, not semantic labels.

  • Using accuracy for a heavily imbalanced classification problem without also examining precision, recall, F1, ROC-AUC, or PR-AUC.

  • Comparing detection or segmentation models using only classification accuracy.

  • Resizing or normalizing inconsistently with the pretrained model’s requirements.

  • Applying augmentation to validation or test data.

  • Reporting a strong validation score without checking performance on the actual deployment hardware.

  • Ignoring dataset shift, subgroup performance, privacy, and failure cases.

5

What is edge detection and how does it identify image boundaries?

Edge detection identifies abrupt changes in pixel intensity that often correspond to object boundaries, texture transitions, or other structural features. It converts an image into an edge map that emphasizes likely contours. The main trade-off is sensitivity: detecting weaker boundaries also increases sensitivity to noise and false positives.

Read full answer

Applies toedge-detection concepts (all versions) and OpenCV 4.x for Canny and image-processing conventions

How It Works

Edge detection usually examines changes in brightness between neighboring pixels. For a grayscale image I(x, y), the horizontal and vertical intensity gradients are approximated with kernels such as Sobel:

  • Horizontal gradient (Gx): Estimates changes from left to right.

  • Vertical gradient (Gy): Estimates changes from top to bottom.

  • Gradient magnitude (G): Calculated as sqrt(Gx² + Gy²). Large values indicate strong intensity changes.

  • Gradient direction (theta): Calculated as atan2(Gy, Gx). It identifies the approximate orientation of an edge.

A gradient magnitude alone can produce thick, duplicated edges. Most algorithms therefore apply non-maximum suppression, which keeps only the strongest response across each edge direction.

The exact behavior depends on the algorithm:

  • Thresholding: Converts gradient magnitude into a binary edge map. It is simple and fast but sensitive to noise and threshold selection.

  • Sobel and Scharr: Compute smoothed first derivatives. Scharr often produces stronger gradient responses than Sobel.

  • Laplacian: Uses a second-derivative kernel and responds strongly to rapid intensity changes, including some texture details.

  • Canny: A multistage detector that performs Gaussian smoothing, gradient calculation, non-maximum suppression, and hysteresis thresholding. It is generally the standard classical choice for clean contours.

  • Learned detectors: CNNs or transformer-based models can infer boundaries from learned visual patterns. They may handle complex scenes better but require suitable training data and more computation.

Common Methods

Method How it works Strengths Weaknesses Typical use
Sobel Calculates horizontal and vertical gradients Simple and fast Thick or duplicate edges without refinement Quick gradient analysis
Laplacian Detects second-order intensity changes Sharp response to fine details Highly noise-sensitive Detail and texture analysis
Canny Uses smoothing, gradients, suppression, and hysteresis Clean, connected edges Requires threshold tuning General-purpose contour detection
Learned models Predict boundaries from training examples Handles complex appearance and context Training and inference cost Document, medical, or industrial imagery

Canny uses two thresholds during hysteresis: pixels above the high threshold become definite edges, while pixels above the low threshold remain edges only when connected to a definite edge. This is more reliable than using one threshold because it avoids isolated responses while preserving weaker continuous contours.

Practical Guidance

  • Preprocess appropriately: Convert color images to grayscale when color is irrelevant, but preserve color channels when chromatic boundaries matter.

  • Reduce noise first: Blur slightly before gradient detection if sensor noise creates many isolated responses.

  • Tune thresholds on representative samples: A fixed threshold that works for one image may miss weak edges or detect texture in another.

  • Remember color-order conventions: OpenCV uses BGR channel order, whereas most PyTorch and TensorFlow model pipelines expect RGB.

  • Consider model preprocessing: For pretrained neural networks, use the model’s documented normalization values and resize method rather than Canny preprocessing unless the model was trained on edge maps.

  • Evaluate against ground truth when possible: Use precision, recall, F1, or tolerant matching that accounts for small annotation differences. Pixel accuracy alone can be misleading because edge pixels are rare.

  • Edge maps are not object masks: A contour may represent texture, shadows, or internal boundaries rather than a complete object boundary.

Common Pitfalls

  • Using one threshold everywhere: Image-specific or automatically selected thresholds are often more reliable.

  • Skipping noise reduction: Raw gradients amplify sensor noise and compression artifacts.

  • Confusing edges with objects: Edge detection finds intensity transitions, not semantic object instances.

  • Ignoring resizing behavior: Downscaling can erase weak edges, while aggressive resizing and interpolation can soften or introduce artifacts.

  • Using inappropriate evaluation metrics: Compare predicted edges with tolerance or matching rather than treating every unmatched edge pixel as equally costly.

  • Applying grayscale conversion unnecessarily: Color transitions may disappear when luminance alone is retained.

Common interview follow-ups include how Canny differs from Sobel, why hysteresis thresholding is useful, how edge detection supports segmentation, and how learned edge detectors compare with classical image-processing methods.

6

What role do convolutional neural networks play in computer vision?

Convolutional neural networks (CNNs) learn reusable visual features by applying learned filters to local neighborhoods of an image. They underpin tasks including classification, detection, segmentation, OCR, tracking, and image generation while reducing parameters through weight sharing. Their main trade-off is an efficient, data-friendly spatial inductive bias versus a less natural fit for global, long-range relationships than vision transformers.

Read full answer

Applies tomachine learning concepts (all versions)

How CNNs Work

A convolution slides a learned kernel across an image and computes a weighted sum at each spatial location:

output[i, j, k] = bias[k] + sum(c, u, v) input[i+u, j+v, c] * weight[k, c, u, v]

  • output[i, j, k] is the response for output channel k at spatial position (i, j).

  • input[i+u, j+v, c] is an input-channel value at a kernel-relative position.

  • weight[k, c, u, v] is a learned filter coefficient.

  • bias[k] is the learned offset for output channel k.

  • c, u, and v enumerate input channels and the kernel’s vertical and horizontal offsets.

Multiple filters detect features such as edges, textures, shapes, and object parts. Deeper layers combine these into higher-level representations.

Key properties include:

  • Local connectivity: Each neuron examines only a small receptive field.

  • Weight sharing: The same filter is reused across spatial positions.

  • Translation equivariance: Shifting an input generally shifts its feature map.

  • Partial translation invariance: Pooling can make predictions less sensitive to small shifts.

  • Hierarchical features: Early layers detect simple patterns, while deeper layers represent complex objects.

For an H × W output, a K × K convolution with C_in input and C_out output channels costs approximately O(H × W × K² × C_in × C_out), where H and W are the output dimensions, K is the kernel size, and the C terms are channel counts. Activation memory and data movement often dominate practical performance.

Computer Vision Tasks

Task Typical output Representative approaches Common metrics
Image classification One label and class probabilities ResNet, EfficientNet, ViT Accuracy, top-k accuracy
Object detection Bounding boxes, labels, confidence scores Faster R-CNN, YOLO family, DETR IoU, mAP
Semantic segmentation One class label per pixel U-Net, fully convolutional networks mIoU, Dice
Instance segmentation Separate pixel mask for each object Mask R-CNN Mask IoU, AP
Keypoint or pose estimation Joint or landmark coordinates CNN heatmap or coordinate regressors PCK, OKS
Tracking Object identities and trajectories CNN features with SORT, DeepSORT, or learned trackers IDF1, HOTA
OCR Recognized text, often with character boxes CNN extractors with CTC or attention decoders CER, WER
Image generation Newly synthesized images U-Net backbones used by diffusion models Human evaluation, FID

CNNs are often used as the backbone in larger systems. Faster R-CNN combines a CNN backbone with region-proposal and classification components, while Mask R-CNN adds per-instance masks.

Training and Evaluation

A typical training pipeline includes:

  1. Split data into training, validation, and test sets before learning preprocessing statistics.

  2. Resize images to the model’s required dimensions.

  3. Normalize using the pretrained model’s documented mean and standard deviation.

  4. Apply training-only augmentation such as crops, horizontal flips, and mild color jitter.

  5. Train with an objective appropriate to the task, such as cross-entropy for classification.

  6. Select hyperparameters using validation data and evaluate once on the test set.

Transfer learning is the default for small image datasets. Initialize a pretrained backbone, replace its task-specific head, and either freeze the backbone initially or fine-tune it with a smaller learning rate.

Accuracy can mislead for imbalanced classification, so also consider precision, recall, F1, ROC-AUC, and PR-AUC. Detection uses IoU and mAP, while segmentation commonly uses mIoU and Dice. Prefer StratifiedKFold for classification cross-validation and time-aware splits for temporally collected data.

CNNs, Transformers, and Deployment

Approach How it works Strengths Weaknesses Typical use Complexity or scale
CNN Local convolutions build hierarchical features Efficient, data-efficient, strong spatial bias Long-range interactions require depth Small-to-medium datasets, real-time and edge vision Convolution cost is O(H × W × K² × C_in × C_out); activation memory can dominate
Vision transformer Self-attention operates over image patches Global interactions and effective large-scale pretraining Often needs more data or stronger pretraining Large datasets and foundation models Standard self-attention is O(N² × D) per layer for N patches and model width D
Hybrid model Combines CNN features with attention Balances local efficiency and global modeling More tuning and deployment complexity Advanced detection and segmentation Adds convolution and attention costs, often increasing latency and memory

Deployment decisions include:

  • Latency: Measure preprocessing, inference, and postprocessing end to end.

  • Model size: Smaller backbones and mixed precision can improve throughput and memory usage.

  • Quantization: FP16 or INT8 can reduce memory usage but requires accuracy validation.

  • ONNX export: Supports execution through compatible non-PyTorch runtimes and accelerators.

  • Hardware: GPUs benefit large training workloads; edge systems may use CPUs, NPUs, or inference accelerators.

  • Robustness: Test domain shifts, lighting changes, camera differences, adversarial inputs, and demographic performance.

  • Privacy and bias: Face-related systems require consent, secure processing, bias testing, and restrictions on identity inference.

Common Pitfalls

  • Data leakage: Fit learned preprocessing only on training folds.

  • Validation overfitting: Repeatedly tuning against one validation set can produce optimistic results.

  • Incorrect normalization: OpenCV loads images as BGR, while most torchvision models expect RGB and specific mean/std values.

  • Invalid augmentation: Flips can invalidate medical images or text with directional meaning.

  • Wrong metric: Accuracy can conceal poor minority-class performance.

  • Missing or corrupted images: Handle them explicitly instead of producing invalid tensors.

  • Train-deployment mismatch: Reproduce the training preprocessing and image resolution during deployment.

Likely follow-up questions include: Why are CNNs translation-equivariant rather than fully invariant? When would you prefer a ViT? Which metric suits imbalanced classification? How can deployment latency and memory be reduced?

7

Why is depth perception important in computer vision applications?

Depth perception lets a vision system infer distance and 3D structure from images or sensor data. It solves spatial problems in navigation, robotics, medical imaging, augmented reality, and autonomous driving. The main trade-off is that sensors and calibration improve accuracy but add cost, power, and complexity, while inexpensive monocular inference is often scale-ambiguous.

Read full answer

Applies tomachine learning and computer vision concepts (Python 3.12)

Why It Matters

Depth supplies geometric information that flat images do not make explicit:

  • Spatial reasoning: Separates foreground from background and reveals relative positions.

  • Navigation and robotics: Supports obstacle assessment, path planning, grasping, and collision avoidance.

  • Medical imaging: Enables measurement of anatomy and changes over time.

  • Augmented reality: Places virtual objects at plausible positions and scales.

  • Autonomous driving: Combines scene geometry with object detection to assess obstacles.

Depth is not necessarily metric. A system may correctly order objects by distance without knowing their absolute distances.

How Depth Is Estimated

  • Monocular cues: Shadows, perspective, texture gradients, object size, occlusion, and motion parallax infer depth from one view or video. Absolute scale remains ambiguous from a single image.

  • Stereo vision: Two calibrated cameras infer depth from the disparity between corresponding pixels.

  • Structured light: A camera measures distortion of a projected pattern to reconstruct surface geometry.

  • Time-of-flight: A sensor measures the return time of emitted light.

  • LiDAR or radar: These provide range measurements; LiDAR generally offers higher spatial resolution but is more expensive.

For rectified stereo, a basic relationship is:

Z = (f_px × B_px) / disparity_px

Here, Z is distance along the optical axis, f_px is focal length in pixels, B_px is camera separation expressed in pixels, and disparity_px is the horizontal pixel difference between matching points. This requires accurate calibration and correspondences. Occlusion, reflective surfaces, low texture, and poor lighting can make stereo unreliable.

Role Across Vision Tasks

Task Output Contribution of depth
Image classification Image-level label Separates objects from background and captures spatial context
Object detection Bounding boxes and labels Supports distance-aware obstacle assessment
Semantic segmentation Label per pixel Distinguishes surfaces and object regions
Instance segmentation Per-instance pixel masks Captures individual object shape and extent
Keypoint and pose estimation 2D or 3D landmarks Supports metrically meaningful pose estimates
Tracking Identity across frames Associates observations using motion and 3D location
OCR Text and character locations Corrects perspective and supports text orientation
Image generation Synthesized image or scene Encourages coherent geometry, scale, and object placement

Models and Evaluation

ResNet and EfficientNet CNN backbones and ViT backbones are common foundations for depth and scene-understanding models. Faster R-CNN, YOLO, and DETR can consume depth or support auxiliary depth objectives. U-Net and Mask R-CNN support segmentation, SAM provides promptable segmentation, and CLIP-style image-text embeddings can complement depth-aware representations.

Common depth-estimation metrics include:

  • AbsRel: Average relative depth error; lower is better.

  • RMSE: Penalizes large absolute errors; lower is better.

  • SqRel: Emphasizes squared relative errors; lower is better.

  • Threshold accuracy: Percentage of pixels below relative-error thresholds such as 0.05, 0.10, or 0.25.

  • Scale alignment: Used when monocular predictions have unknown global scale.

Downstream evaluation uses accuracy or top-k accuracy for classification, IoU and mAP for detection, and mIoU or Dice for segmentation. For imbalanced classification, precision, recall, F1, ROC-AUC, and PR-AUC may be more informative because accuracy can hide poor minority-class performance.

Practical Guidance

  • Preprocessing and augmentation: Normalize with the pretrained model’s training-dataset mean and standard deviation. Flips, crops, and color jitter should preserve geometric validity.

  • Transfer learning: Start with pretrained depth or scene-understanding models for small datasets.

  • Validation: Fit preprocessing only on training folds and avoid repeatedly tuning against the validation set.

  • Domain shift: Test across lighting, weather, camera, and scene changes.

  • Deployment: Measure latency and model size, use lighter backbones or quantization for edge devices, and consider ONNX export. Revalidate accuracy after conversion.

  • Risk: Depth and face-related systems can expose sensitive spatial information and perform unevenly across demographic groups.

Common Pitfalls

  • Treating scale-ambiguous monocular output as a metric measurement.

  • Ignoring occlusions, reflective surfaces, and calibration errors.

  • Selecting RMSE without considering whether large errors matter more than frequent small errors.

  • Applying invalid flips to tasks with meaningful left-right or handedness.

  • Leaking preprocessing information across validation folds.

Interviewers may ask about disparity and calibration, scale ambiguity, occlusion, suitable metrics, sensor trade-offs, and applications in autonomous driving or robotic grasping.

8

What challenges affect object recognition under lighting and orientation changes?

Robust object recognition requires a model to identify objects despite changes in illumination, viewpoint, scale, and rotation. These variations alter visible color, contrast, texture, and shape, making the same object look substantially different to the model. The main trade-off is that greater robustness usually requires broader training data or stronger augmentation, increasing model complexity and computational cost while potentially reducing accuracy on clean, consistent images.

Read full answer

Applies tomachine learning concepts (Python 3.12)

Why Lighting and Orientation Are Challenging

  • Lighting variation: Shadows, glare, low light, backlighting, fluorescent lighting, and color-temperature changes can hide edges or make identical colors appear different.

  • Local versus global illumination: One part of an image may be overexposed while another is poorly exposed. A single global brightness adjustment cannot model this variation.

  • Texture loss: Low light or blur removes fine textures that may distinguish one object category from another.

  • In-plane rotation: Rotating an object changes its apparent horizontal and vertical extents; with a fixed image canvas, the rotation can also move content outside the frame.

  • Viewpoint change: Perspective and 3D rotation reveal different surfaces, occlude visible features, and can change an object’s apparent proportions.

  • Scale and deformation: Objects appear at different sizes and may bend, deform, or become partially hidden.

  • Background interference: Lighting can create background edges or patterns that resemble object features.

Robustness therefore requires more than memorizing category examples. The model must learn which features are stable across conditions rather than relying on brightness, color, position, or a particular pose.

Common Computer Vision Tasks

Task Output Typical approaches
Image classification One or more category labels ResNet, EfficientNet, ViT; CLIP for image-text embeddings and zero-shot classification
Object detection Bounding boxes, labels, and confidence scores Faster R-CNN, YOLO family, DETR
Semantic segmentation A category label for each pixel U-Net, DeepLab, SegFormer; SAM for promptable segmentation
Instance segmentation A separate mask for each object Mask R-CNN
Keypoint or pose estimation Coordinates of joints or landmarks CNN and ViT pose models
Tracking Consistent object identities across frames Detection plus association
OCR Text and sometimes character-level geometry CNN/Transformer OCR systems
Image generation New images conditioned on text or other inputs Diffusion models

The same lighting and orientation challenges affect these tasks differently. Classification focuses on the category label, detection must also localize changing shapes, and segmentation must assign every pixel correctly.

How Models Improve Robustness

  • Transfer learning: Start with a pretrained ResNet, EfficientNet, or ViT backbone. This is the default for small datasets because pretrained features generally cover more appearances than training from scratch.

  • Photometric augmentation: Apply brightness, contrast, color-jitter, hue, saturation, shadow, and sometimes glare transformations.

  • Geometric augmentation: Use flips, rotations, affine transforms, perspective transforms, scaling, and random crops.

  • Pretrained-model normalization: Use the model’s documented mean and standard deviation rather than calculating normalization from the entire dataset.

  • Robust architectures: CNNs provide useful translation and local-pattern sensitivity, while attention-based vision transformers can model relationships between distant image regions.

  • Condition-aware modeling: For scenes with extreme illumination, models may estimate exposure or lighting conditions before classification, or operate on multiple image scales.

Augmentation must preserve the label. For example, horizontal flips are usually valid for animals but can invalidate left-right-sensitive classes such as directional traffic signs. Lighting augmentation should also reflect realistic physics; arbitrary color transformations may create images that never occur in deployment.

Evaluation and Deployment

Evaluate separate test sets for normal light, dim light, glare, shadow, and different orientations. Aggregate accuracy alone can hide poor performance under particular conditions.

  • Classification: Use accuracy and top-k accuracy; include per-condition results and the confusion matrix.

  • Detection: Use IoU-based matching and mAP, while checking missed and duplicate detections.

  • Segmentation: Use mIoU and Dice, with separate results for small or poorly lit objects.

  • Imbalanced classes: Accuracy can mislead, so include precision, recall, F1, ROC-AUC, or PR-AUC when appropriate.

Deployment adds constraints such as inference latency, model size, GPU versus CPU execution, quantization, and ONNX export. Edge devices require careful benchmarking. Face-related systems also require privacy controls, consent, demographic testing, and bias monitoring.

Common Pitfalls

  • Training and testing on nearly identical lighting makes robustness look better than it is.

  • Using augmentation without realistic physical constraints can create unrealistic training images.

  • Forgetting to use the pretrained model’s required normalization can sharply reduce accuracy.

  • Reporting only aggregate accuracy hides failures under specific conditions.

  • Using random image splits when near-duplicate video frames appear in both sets causes data leakage.

  • Testing orientation robustness without separating in-plane rotation from viewpoint change hides which problem remains unsolved.

  • Assuming a larger model automatically solves environmental variation; additional relevant data often matters more than parameter count.

9

What are common steps in computer vision image preprocessing?

Image preprocessing converts raw image pixels into clean, consistently formatted inputs for a computer vision model. Common steps include decoding, resizing, cropping, color conversion, normalization, and training-time augmentation. The main trade-off is preserving task-relevant detail while reducing noise and computation without introducing unrealistic transformations or data leakage.

Read full answer

Applies toOpenCV 4.x, NumPy 2.x, and PyTorch 2.x (Python 3.12)

Typical Pipeline

A preprocessing pipeline usually follows these stages:

  1. Read and validate the image: Decode the file, handle corrupt or unreadable inputs, and correct orientation using EXIF metadata when applicable.

  2. Resize or crop: Match the model’s expected spatial dimensions while retaining important objects.

  3. Convert color space: OpenCV loads images as BGR, while most pretrained PyTorch models expect RGB.

  4. Clean the image: Apply denoising, sharpening, contrast adjustment, or edge detection only when appropriate for the task.

  5. Normalize pixels: Scale values and standardize them using the pretrained model’s mean and standard deviation.

  6. Augment training data: Apply random crops, flips, rotations, color jitter, and other realistic transformations during training only.

  7. Batch and convert: Stack images into tensors and move them to the appropriate device.

Not every pipeline uses every step.

Common Preprocessing Steps

Step Purpose Important consideration
Decoding and validation Produce a usable pixel array Reject corrupt images and handle transparency separately
Orientation correction Align images consistently Apply EXIF rotation before resizing
Resizing Match model input dimensions Avoid unnecessary distortion or loss of small objects
Cropping Remove irrelevant background Ensure important content is not accidentally removed
Color conversion Produce the expected channel order OpenCV uses BGR; most models expect RGB
Normalization Make pixel distributions consistent Use the exact mean and standard deviation of the pretrained model
Augmentation Improve robustness to unseen images Use only for training unless the operation is inference-safe
Denoising or filtering Reduce sensor noise and artifacts Excessive smoothing can erase useful details
Histogram or contrast adjustment Handle lighting variation Can amplify noise in already low-contrast images

Task-Specific Preprocessing

Computer vision task Model output Preprocessing emphasis
Image classification Class label Resize, normalize, and use mild augmentation
Object detection Bounding boxes and labels Preserve aspect ratio and avoid cropping away objects
Semantic segmentation One label per pixel Resize labels without interpolation; transform images and masks identically
Instance segmentation Per-object masks and classes Apply geometric transforms jointly to images and masks
Keypoint or pose estimation Keypoint coordinates Preserve body geometry and apply coordinated transformations
Tracking Object identities across frames Keep frames temporally consistent and avoid arbitrary resizing
OCR Text and character positions Preserve text sharpness; grayscale or contrast correction may help
Image generation A generated image Training preprocessing depends heavily on the model and target resolution

For semantic and instance segmentation, interpolation must be handled carefully. A bilinear resize is appropriate for images, but nearest-neighbor resizing is generally required for label masks because interpolated class IDs do not represent valid categories.

Architecture also determines the input contract: ResNet, EfficientNet, and ViT commonly use fixed-size tensors; Faster R-CNN, the YOLO family, DETR, U-Net, Mask R-CNN, SAM, and CLIP have task-specific resize, normalization, box, or mask rules. Follow the selected checkpoint’s documented preprocessing pipeline.

Training and Evaluation

  • Split before fitting preprocessing: Compute normalization statistics or learn preprocessing parameters only from the training set. Using the full dataset can leak validation information.

  • Use the model’s expected statistics: For ImageNet-based models, common values are mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225], ordered for RGB channels.

  • Prefer transfer learning for small datasets: Start with a pretrained ResNet, EfficientNet, or ViT checkpoint, retain its expected normalization, and fine-tune it rather than training from scratch.

  • Augment only training data: Random horizontal flips can help natural-image tasks but may be invalid for text, medical orientation, satellite imagery, or other direction-sensitive domains.

  • Keep evaluation deterministic: Validation and test images should use fixed resize, crop, and normalization operations.

  • Preserve deployment parity: The preprocessing used during training should match what the deployed system can reproduce.

Code Example

import cv2
import numpy as np

rng = np.random.default_rng(42)

# Synthetic BGR image, as returned by OpenCV.
image = rng.integers(0, 256, size=(240, 320, 3), dtype=np.uint8)
cv2.rectangle(image, (80, 50), (230, 180), (255, 255, 255), thickness=3)

# Resize to a model-style input size.
resized = cv2.resize(image, (224, 224), interpolation=cv2.INTER_AREA)

# Convert OpenCV's BGR ordering to RGB.
rgb = cv2.cvtColor(resized, cv2.COLOR_BGR2RGB)

# Normalize using ImageNet pretrained-model statistics.
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
normalized = (rgb.astype(np.float32) / 255.0 - mean) / std

# Optional classic processing for edge-based tasks.
gray = cv2.cvtColor(resized, cv2.COLOR_BGR2GRAY)
blurred = cv2.GaussianBlur(gray, (5, 5), sigmaX=0)
edges = cv2.Canny(blurred, threshold1=50, threshold2=150)

print("Original shape:", image.shape)
print("Resized shape:", resized.shape)
print("Normalized shape:", normalized.shape)
print("Edge-map shape:", edges.shape)

Common Pitfalls

  • Ignoring color order: Feeding BGR data to a model expecting RGB silently degrades performance.

  • Using the wrong normalization: Pixel scaling alone is not equivalent to standardization with the model’s mean and standard deviation.

  • Augmenting evaluation data: Random transformations can make evaluation inconsistent.

  • Leaking preprocessing statistics: Fit learned transformations such as normalization or imputation on training data only.

  • Resizing away small objects: Detection and segmentation pipelines need padding, tiling, or higher resolution when objects occupy few pixels.

  • Overusing filters: Blurring, sharpening, and thresholding can remove texture or create artifacts.

  • Ignoring deployment constraints: End-to-end latency, memory use, and model size can dominate edge inference; evaluate quantization and ONNX export where appropriate. Face-related applications also require privacy and bias safeguards.

10

How does image resizing affect computer vision model performance?

Image resizing changes the spatial dimensions of an input image to match a model’s expected input size, which is required because most CNNs and vision transformers process fixed-size tensors and batches must have consistent shapes. It solves the problem of handling variable-resolution inputs, but its main trade-off is between preserving visual detail and controlling computational cost, since aggressive downscaling discards information while upscaling can introduce artifacts.

Read full answer

Applies tocomputer vision concepts (examples in Python 3.12 with OpenCV 4.x, NumPy 2.x, and PyTorch 2.x/torchvision)

How Resizing Impacts Performance

  • Information loss: Downscaling removes fine-grained features (small objects, thin edges, text), which can reduce accuracy for tasks like object detection (evaluated with IoU and mAP) and segmentation (evaluated with mIoU and Dice). Upscaling does not recover lost detail and may blur edges.

  • Geometric distortion: Stretching an image to a square without preserving aspect ratio changes object shapes, which can confuse models trained on natural proportions and shift boxes or keypoints.

  • Computational cost: Input resolution strongly affects latency and memory. For convolutional layers, compute and activation memory scale roughly with image area (H * W); larger inputs slow inference and increase GPU memory use.

  • Batch consistency: Training and inference pipelines resize or pad images so a batch forms a single tensor. Inconsistent preprocessing between training and deployment is a common source of accuracy drops.

Interpolation and Aspect Ratio Strategies

Strategy How It Works Strengths Weaknesses When to Use
Direct resize (stretch) Scales width/height independently Simple, fixed output size Distorts aspect ratio When distortion is acceptable or data is already normalized
Resize + pad (letterbox) Scales to fit, pads remainder Preserves aspect ratio Adds padding pixels the model must ignore or learn Object detection (common in YOLO-style pipelines)
Center/random crop after resize Resizes shorter side, crops Preserves aspect ratio, common augmentation May cut off important content Image classification, training augmentation
  • Interpolation: Area-based interpolation is preferred for downscaling; bilinear or bicubic interpolation are common for upscaling. For label masks in segmentation, use nearest-neighbor interpolation to avoid producing invalid class IDs.

  • Preprocessing alignment: Match the pretrained model’s expected input size, color order (OpenCV loads BGR while most models expect RGB), and normalization statistics (for example, ImageNet mean/std) between training and inference.

Common Pitfalls

  • Train/serve mismatch: Using different resize methods, interpolation, or normalization at inference than during training can noticeably hurt accuracy.

  • Ignoring aspect ratio: Stretching detection or pose images can shift object geometry and reduce mAP.

  • Wrong color order: Mixing OpenCV (BGR) with PyTorch/torchvision pipelines that expect RGB can silently degrade performance.

  • Resolution too small for the task: Small inputs may be fine for classification but miss small objects in detection or thin structures in segmentation.

  • Interview follow-ups: Why use letterboxing for detectors? How does input resolution affect FLOPs, latency, and GPU memory? Why is nearest-neighbor interpolation used for segmentation masks? How do you handle variable-sized images with vision transformers or feature pyramids?

11

What techniques reduce noise in images?

Image denoising is the process of reducing random pixel variations (noise) in an image to improve visual quality and make downstream tasks like classification, detection, segmentation, or OCR more reliable. It addresses corrupted measurements caused by sensor limitations, low light, transmission errors, or compression. Its main trade-off is that stronger noise removal often smooths away edges and fine details, so the technique must match the noise type and the application’s tolerance for detail loss.

Read full answer

Applies toimage denoising concepts (classical filter names referenced from OpenCV 4.x, Python 3.12)

Common Noise Types

  • Gaussian noise: Random intensity variations that follow a normal distribution, common in low-light sensor capture and electronic noise.

  • Salt-and-pepper (impulse) noise: Random pixels pushed to minimum or maximum intensity, often caused by transmission errors, faulty sensors, or bit errors.

  • Speckle noise: Multiplicative, grain-like noise typical in coherent imaging such as ultrasound, SAR, and laser imaging.

  • Poisson (shot) noise: Signal-dependent noise arising from the discrete nature of photon capture; it becomes more visible in dim regions.

Denoising Techniques

Technique How it works Strengths Weaknesses When to use Typical complexity/scale
Mean/box filter Replaces each pixel with the average of its local neighborhood Very fast, simple Blurs edges and textures Quick baseline smoothing of mild noise O(H*W) with separable/integral-image implementations
Gaussian blur Convolves with a Gaussian kernel, weighting nearer pixels more strongly Fast; effective for Gaussian noise Blurs edges and fine detail Mild Gaussian noise or preprocessing before other operations O(H*W*k) per channel (separable kernels reduce cost)
Median filter Replaces each pixel with the neighborhood median Robust to outliers; preserves edges better than mean blur Less effective on pure Gaussian noise; slower for large windows Salt-and-pepper noise Depends on window size; higher than simple linear filters
Bilateral filter Averages neighbors by both spatial distance and intensity similarity Reduces noise while preserving edges Slower; sensitive to spatial and range parameters Edge-preserving smoothing for photos Higher than Gaussian blur due to per-pixel range weighting
Non-local means (NLM) Averages pixels weighted by similarity of surrounding patches across a search region Strong denoising with good detail preservation Computationally expensive in time and memory High-quality offline denoising of photos or medical images Cost grows with image size, patch size, and search window
Wavelet/transform-domain denoising Transforms the image, suppresses noise-dominated coefficients, then reconstructs Can separate noise and structure across scales Can introduce ringing or artifacts if thresholds are poorly chosen Images with multi-scale structure or specific noise models Transform-dependent; often efficient for practical image sizes
Deep learning denoisers (CNN/transformer-based) Learn a mapping from noisy to clean images from training data Strong quality on noise distributions seen during training Require training data and more compute, often GPU; may hallucinate details Production pipelines with known noise characteristics or pretrained models Inference cost depends on model size; usually higher than classical filters

Practical Guidance

  • Match the filter to the noise: Median filtering is usually better for impulse noise, while Gaussian or NLM-style methods are more suitable for Gaussian-like noise.

  • Preserve task-relevant detail: For edge detection, OCR, or small-object detection, avoid aggressive smoothing that removes boundaries or thin structures.

  • Use the right color pipeline: OpenCV loads images as BGR, while many vision models expect RGB; apply the same preprocessing assumptions consistently.

  • Pretrained-model preprocessing: When denoising before a pretrained CNN/ViT, keep normalization consistent with the model’s expected mean and standard deviation.

  • Validate beyond appearance: Visual inspection helps, but paired-image metrics such as PSNR or SSIM and downstream task performance are better checks when available.

Common Pitfalls

  • Over-smoothing: Large kernels or aggressive parameters can erase edges, textures, and small objects.

  • Mismatched assumptions: Applying a filter designed for additive noise to impulse or speckle noise can leave artifacts.

  • Color-space mistakes: Denoising in the wrong channel order or after inappropriate conversions can degrade results.

  • Ignoring deployment constraints: NLM and learned denoisers may be too slow or memory-heavy for real-time or edge devices; consider latency, model size, quantization, or ONNX export.

  • Interview follow-ups: How do you choose between median and bilateral filtering? Why can denoising hurt detection or segmentation? How would you denoise video while preserving temporal consistency? When is a learned denoiser preferable to a classical filter?

12

How does image augmentation improve vision model performance?

Image augmentation creates varied training examples by applying transformations that should preserve labels, such as flips, crops, color jitter, and controlled geometric distortions. It regularizes a vision model, reduces memorization of individual images, and improves robustness to plausible variations in deployment data. The main trade-off is that overly aggressive or semantically invalid transformations can remove important visual evidence, change the correct label, and increase training cost without improving validation performance.

Read full answer

Applies tocomputer vision augmentation concepts (all current versions)

How It Works

Augmentation broadens the effective training distribution without collecting and labeling new images.

  • Regularization: Random transformations make it harder to memorize specific training examples.

  • Invariance: Appropriate transformations encourage features to remain stable under expected changes. For example, horizontal flipping can teach mirror invariance, but it is not a general substitute for viewpoint invariance.

  • Robustness: Blur, noise, and color jitter can reduce sensitivity to imperfect capture conditions.

  • Data efficiency: Transfer learning with a pretrained ResNet, EfficientNet, or ViT is often effective on small datasets when combined with suitable augmentation.

  • Annotation consistency: For detection, segmentation, pose estimation, and tracking, images must be transformed together with bounding boxes, masks, keypoints, or identity labels.

  • Label preservation: A transformation is useful only when it preserves the information required to determine the target. A crop that removes the object or a flip that reverses text semantics is harmful.

Augmentation does not guarantee better performance. Its effect should be established through controlled validation experiments against a no-augmentation baseline.

Tasks, Outputs, and Metrics

Task Model output Suitable augmentations Typical evaluation
Image classification Class label Crops, flips, color jitter Accuracy, top-k accuracy
Object detection Boxes, labels, confidence Flips, crops, scaling, color changes IoU, mAP
Semantic segmentation Label per pixel Flips, crops, color changes mIoU, Dice
Instance segmentation Per-instance masks Geometric transforms with synchronized masks Mask mAP, IoU
Keypoint/pose estimation Joint locations Flips with joint swapping, crops, scale changes PCK, MPJPE
Tracking Object identities over time Appearance and motion-consistent changes HOTA, IDF1, ID switches
OCR Recognized text Carefully controlled distortion and noise CER, WER
Image generation Generated image Domain-specific transforms that preserve training-pair semantics FID, precision and recall

Common approaches include Faster R-CNN, YOLO, and DETR for detection; U-Net and Mask R-CNN for semantic and instance segmentation; promptable models such as SAM for promptable segmentation; and CLIP for image-text representations.

Practical Guidance

  1. Start from a strong baseline: Compare the same architecture and training schedule with and without augmentation.

  2. Match the deployment domain: Use transformations that reflect expected changes in lighting, scale, viewpoint, blur, sensor noise, or image compression.

  3. Use mild transformations first: Tune probability and strength on validation data rather than applying an arbitrarily large collection.

  4. Transform annotations jointly: Geometric operations require synchronized updates to boxes, masks, keypoints, and temporal identity information.

  5. Keep evaluation deterministic: Use fixed validation and test preprocessing. Test-time augmentation is valid when multiple views are aggregated consistently and justified by measured gains, but it increases inference latency.

  6. Normalize appropriately: When using a pretrained model, resize to its expected input size and use that model’s documented mean and standard deviation.

  7. Handle color conventions correctly: OpenCV loads images in BGR order; convert to RGB before passing them to most torchvision models.

  8. Watch class imbalance: Accuracy can mislead because it may conceal poor minority-class performance. Also consider macro-F1, PR-AUC, or other task-appropriate metrics.

  9. Control reproducibility: Seed Python, NumPy, and PyTorch, and record the augmentation configuration and random states.

  10. Optimize the data path: Multiple DataLoader workers or tensor transformations on the accelerator can help when augmentation becomes a CPU bottleneck.

Augmentation complements rather than replaces balanced sampling, class weighting, high-quality data collection, and careful regularization. For edge deployment, evaluate latency, model size, quantization, and ONNX export together with the complete preprocessing pipeline. For face-related applications, augmentation does not remove privacy risks or demographic bias, and those issues require separate assessment.

Common Pitfalls

  • Semantic corruption: Removing the target with a crop, rotating medical images, or flipping text can change the correct answer.

  • Unrealistic transformations: Extreme distortions may not reflect real deployment conditions and can create shortcuts.

  • Inconsistent evaluation: Randomly evaluating a single augmented view makes results unstable; use fixed evaluation or a clearly defined aggregation procedure.

  • Annotation mismatch: Transforming an image without updating its boxes, masks, or keypoints corrupts supervision.

  • Overfitting the validation set: Repeatedly tuning augmentation strength on one validation split can overfit that split; use cross-validation or a separate final test set where appropriate.

  • Unjustified test-time augmentation: It increases latency and complexity, so retain it only when measured performance improves consistently.

13

What are color spaces and why do they matter in image processing?

A color space is a mathematical way to represent the colors in an image using multiple components or channels. Different spaces make different operations easier—for example, RGB supports display, HSV separates brightness from color, and CIELAB approximates perceptual color differences. The main trade-off is that computationally convenient spaces are often less perceptually uniform, while perceptually meaningful spaces usually require conversion.

Read full answer

Applies togeneral image-processing concepts (version-independent); OpenCV 4.x conventions (Python 3.12)

How Color Spaces Work

A color space defines:

  • Channels: The numeric components stored for each pixel.

  • Value range: What each channel represents and its permitted range.

  • Reference model: How the channel values map to visible colors.

  • Conversion rules: Formulas for transforming colors between spaces.

For example, a pixel in 8-bit RGB might be (B=20, G=120, R=200), while its HSV representation separates the hue, saturation, and brightness. Conversion is generally inexpensive, but repeated conversions between similar spaces can add processing overhead.

Common Color Spaces

Color space Main channels Strengths Limitations Typical use
RGB Red, green, blue Natural representation of display pixels; simple arithmetic Perceptually uneven; brightness and color are mixed Display, rendering, neural-network inputs
BGR Blue, green, red OpenCV’s native array channel order Same perceptual limitations as RGB OpenCV image-processing pipelines
Grayscale Luminance One channel; reduces computation and memory Discards chromatic information Edges, thresholds, document images
HSV Hue, saturation, value Separates color from brightness; intuitive for user-selected colors Not perceptually uniform; hue becomes unstable near low saturation Segmentation, color detection, graphics
CIELAB Lightness, a, b Designed to approximate perceptual color differences; useful for color matching More computational cost; uncommon in basic pipelines Color constancy, quality inspection, clustering
YUV / YCbCr Luminance plus chrominance Separates brightness from color; supports video and broadcast formats Requires careful channel interpretation Video encoding and compression

RGB is an additive model: increasing channel values generally makes a color brighter. HSV represents color through hue—the color family—saturation—color purity—and value—brightness. CIELAB instead separates lightness from two opponent color dimensions.

Why Color Spaces Matter

Choosing the right representation can determine whether an image-processing task succeeds:

  • Thresholding: A fixed threshold on the red channel may be unstable under warm lighting, while a threshold on HSV hue or saturation can better isolate a colored object.

  • Segmentation: Separating illumination from chromatic information often makes object boundaries easier to detect.

  • Feature extraction: Grayscale images are efficient for shape and texture analysis, while color channels provide additional information when color distinguishes objects.

  • Color constancy and matching: CIELAB can make numerical comparisons more closely resemble human perception, although its uniformity is only approximate.

  • Machine learning: Neural networks learn from raw channel values, so channel order and normalization must be handled correctly. For small datasets, transfer learning with pretrained weights is generally the default.

  • Image augmentation: Operations such as brightness, saturation, and hue changes are easier to control in HSV, though they may produce less natural results than carefully designed RGB operations.

A typical pipeline may load an image as BGR with OpenCV, convert it to HSV for color segmentation, and convert the resulting mask back to BGR when combining it with the original image.

Conversion and Preprocessing

Conversion functions accept an image array and a conversion code. The output remains an image array, but its channels have different meanings. For example, OpenCV uses cv2.cvtColor(image_bgr, cv2.COLOR_BGR2HSV) to convert BGR data to HSV and cv2.cvtColor(image_hsv, cv2.COLOR_HSV2BGR) to convert it back.

For pretrained PyTorch models, input usually follows the model’s expected RGB order and normalization. The expected mean and standard deviation must come from that specific model’s documentation; they are not interchangeable across all architectures. Channel swaps or inconsistent normalization can reduce model accuracy even when the image geometry remains unchanged.

Common Pitfalls

  • Confusing RGB and BGR: OpenCV uses BGR, while most PyTorch image pipelines use RGB. Swapping them can make colors look incorrect or reduce model accuracy.

  • Assuming HSV is perceptually uniform: Equal numerical hue differences do not necessarily look equally different to humans.

  • Applying color thresholds without lighting control: Shadows and highlights can alter saturation and value, producing incomplete masks.

  • Discarding color unnecessarily: Grayscale removes a useful signal when objects are distinguished primarily by color.

  • Using the wrong conversion code: The code must describe both the source and destination spaces, such as COLOR_BGR2HSV, not merely the destination.

  • Ignoring bit depth: Converting between 8-bit and floating-point representations can change clipping behavior and numerical precision.

Interview Tips

Common follow-up questions include when to use HSV instead of RGB, why color segmentation can fail under changing illumination, how to preserve color while extracting edges, and how to avoid channel-order errors in OpenCV and deep-learning pipelines. A strong answer explains both the mathematical representation and the practical imaging conditions that affect the chosen color space.

14

What are feature descriptors, and why are they important for visual recognition?

A feature descriptor is a numerical representation of an image or an image region that captures visually meaningful patterns such as edges, corners, textures, shapes, or semantic objects. Descriptors support tasks ranging from image retrieval to object detection and visual tracking, with the main trade-off being computational cost and storage versus robustness and discriminative power.

Read full answer

Applies tocomputer vision concepts (Python 3.12)

What a Feature Descriptor Represents

A descriptor usually converts an image or local region into a vector that can be compared with other vectors.

  • Feature detector: Locates interesting locations or regions in an image, such as corners or edges.

  • Feature descriptor: Describes the visual content at each detected location.

  • Feature matcher: Compares descriptors to find correspondences between images.

For example, SIFT represents local image structure using gradient magnitude and orientation. ORB uses binary features, making it faster and more compact but generally less robust to large appearance changes than learned descriptors.

Descriptors are designed to preserve useful properties while ignoring irrelevant variation:

  • Scale invariance: Remains useful when an object appears larger or smaller.

  • Rotation invariance: Handles image rotation and viewpoint changes.

  • Illumination robustness: Tolerates brightness and contrast differences.

  • Locality: Describes a region without depending heavily on the rest of the image.

Why Descriptors Matter

Descriptors convert pixels into representations suitable for comparison, learning, and geometric reasoning.

Vision task Main output How descriptors contribute
Image classification Class label A model can classify learned or handcrafted feature representations.
Object detection Bounding boxes and labels Local descriptors support region proposal, classification, and box matching.
Semantic segmentation Label for every pixel Learned feature maps encode visual evidence at each location.
Instance segmentation Per-instance masks and labels Features help distinguish separate objects of the same class.
Keypoint and pose estimation Landmark coordinates Correspondences between keypoint descriptors establish object or body geometry.
Tracking Object identities across frames Appearance descriptors associate detections with existing tracks.
OCR Text and character sequences Features represent strokes and character structures.
Image generation Generated image or conditioned output Latent features guide generation, similarity search, and controllable conditioning.

Other applications include image retrieval, visual place recognition, camera pose estimation, panorama stitching, and identifying duplicate images.

Classical and Learned Descriptors

Approach How it works Strengths Weaknesses Typical use
Handcrafted descriptors Compute statistics such as gradients, intensity histograms, or binary comparisons Explainable and often data-efficient Limited adaptation to unusual domains; many require careful preprocessing Small datasets, controlled environments, geometric matching
CNN feature embeddings Use intermediate convolutional layers to represent image regions Strong image-classification features and straightforward transfer learning Less explicit about invariance; representation depends on the training domain Classification, retrieval, detection pipelines
Vision transformer embeddings Represent image regions using self-attention and learned tokens Captures global context and supports modern transfer learning Usually requires more data and compute than small classical methods Large datasets, semantic image-text representation
CLIP embeddings Align images and text in a shared vector space Enables zero-shot classification and image-text search Fine-grained geometry may be weaker than task-specific local descriptors Semantic retrieval, zero-shot classification
Learned keypoint descriptors Train both where to sample points and how to describe them Strong matching robustness and adaptability More complex training and deployment Advanced matching, visual localization, tracking

Deep models do not remove the concept of a descriptor: their hidden activations or output embeddings are learned feature descriptors.

Practical Guidance

  • Choose the representation for the task: Global embeddings are effective for classification and retrieval, while local descriptors are valuable when precise spatial correspondences matter.

  • Use transfer learning for small datasets: Fine-tune a pretrained CNN or vision transformer rather than training a complete feature extractor from scratch.

  • Preserve the expected color format: OpenCV loads images in BGR, whereas most pretrained models expect RGB.

  • Match descriptors carefully: Use ratio testing or reciprocal matching, followed by RANSAC for geometric verification when handling transformations and outliers.

  • Control augmentation realistically: Flips, crops, color jitter, blur, and noise should resemble plausible deployment conditions rather than distorting task-defining shapes.

  • Account for scale: Pairwise descriptor matching can require (O(nm)) comparisons for (n) and (m) features; indexing, hashing, or approximate nearest-neighbor search can reduce latency.

  • Plan for deployment: Compare latency, memory use, model size, and hardware compatibility. Quantization and formats such as ONNX can improve edge deployment, but may reduce matching accuracy.

Common Pitfalls

  • Confusing detection with description: A detector finds keypoints; a descriptor represents them.

  • Assuming invariance is unlimited: No descriptor is equally reliable under every scale, rotation, occlusion, blur, or illumination change.

  • Using a global descriptor for precise localization: It may identify similar images but cannot replace local correspondences.

  • Ignoring outliers: Nearest-neighbor matches can be incorrect; geometric verification is often essential.

  • Leaking data during training: Fit feature-learning or preprocessing components only on the training split when evaluating on held-out data.

  • Ignoring the domain shift: Descriptors pretrained on generic images may perform poorly on medical, industrial, satellite, or low-light imagery.

15

How does the Scale-Invariant Feature Transform (SIFT) algorithm work?

The Scale-Invariant Feature Transform (SIFT) is a classic computer vision algorithm that detects distinctive keypoints in an image and describes each with a 128-dimensional vector so the same physical point can be matched across images. It solves the problem of finding reliable correspondences despite changes in scale, rotation, and moderate illumination or viewpoint, but its main trade-off is higher computational cost and memory use compared with simpler or binary descriptors.

Read full answer

Applies tocomputer vision concepts (classic algorithm, all versions)

How It Works

SIFT runs in four stages: 1. Scale-space extrema detection: Build a Gaussian scale space L(x, y, σ) = G(x, y, σ) * I(x, y), where I is the image, G is a Gaussian kernel with standard deviation σ, and * is convolution. Compute Difference-of-Gaussians: D(x, y, σ) = L(x, y, kσ) - L(x, y, σ), where k is a constant scale factor between adjacent blurs. Candidate keypoints are local extrema of D compared with 26 neighbors in space and scale. 2. Keypoint localization: Refine candidates with subpixel interpolation and reject unstable points (low contrast or strong edge response using the Hessian ratio) to improve repeatability. 3. Orientation assignment: Compute gradient magnitude m(x, y) = sqrt((L(x+1,y)-L(x-1,y))^2 + (L(x,y+1)-L(x,y-1))^2) and orientation θ(x, y) = atan2(L(x,y+1)-L(x,y-1), L(x+1,y)-L(x-1,y)) around each keypoint. Build an orientation histogram and assign the dominant direction (and extra keypoints for strong secondary peaks) to achieve rotation invariance. 4. Descriptor generation: Take a 16x16 window around the keypoint, split into 4x4 cells, and compute an 8-bin orientation histogram per cell, yielding a 128-value descriptor. Normalize it to reduce sensitivity to illumination changes.

Key Properties and Evaluation

  • Outputs: keypoint locations (x, y), scale σ, orientation θ, and a 128-float descriptor per keypoint.

  • Invariance: robust to uniform scaling, rotation, and small affine/illumination changes; less robust to large viewpoint or non-rigid deformation.

  • Matching: descriptors are compared with Euclidean distance; Lowe’s ratio test (best/second-best < ~0.8) filters ambiguous matches.

  • Complexity: detection/description is roughly O(n log n) to O(n * s) in image pixels n and number of scales s; matching m keypoints by brute force is O(m^2 * d) with d=128.

Aspect SIFT ORB (for contrast)
Descriptor 128-float, gradient-based 256-bit, binary
Invariance Strong scale/rotation Rotation, limited scale
Speed & memory Slower, larger Faster, compact
Typical use High-quality matching, stitching Real-time or resource-limited

Common Pitfalls

  • Expecting invariance to everything: SIFT degrades under heavy blur, large out-of-plane rotation, or non-rigid motion; consider learned local features (e.g., SuperPoint) or task-specific deep models in those cases.

  • Matching mistakes: using raw nearest-neighbor matches without Lowe’s ratio test or geometric verification (RANSAC) produces many false correspondences.

  • Interview tips: be ready to explain DoG vs. Laplacian-of-Gaussian, why 128 dimensions (4x4 cells x 8 bins), how the ratio test reduces ambiguity, and when to prefer binary descriptors (ORB/BRIEF) for latency or edge deployment.

16

What is the Histogram of Oriented Gradients (HOG) descriptor?

The Histogram of Oriented Gradients (HOG) descriptor represents local image structure by accumulating edge-gradient magnitudes into spatially organized histograms and normalizing them block by block. It converts a fixed-size image or detection window into a feature vector for classifiers such as support vector machines, especially in classic pedestrian and object-detection systems. Its main trade-off is a compact, interpretable, CPU-friendly descriptor versus less expressive hand-crafted features that generally underperform modern CNNs and vision transformers on large-scale recognition tasks.

Read full answer

Applies toHOG descriptor (all versions)

How It Works

  1. Convert to grayscale: HOG usually operates on one channel, reducing color variation and computation. Color-HOG variants also exist.

  2. Compute image gradients:

  3. Magnitude: m(p) = sqrt(gx(p)^2 + gy(p)^2)

  4. Unsigned orientation: θ(p) = atan2(gy(p), gx(p)) mod π

  5. Here, p is a pixel, gx and gy are its horizontal and vertical intensity derivatives, m is gradient or edge strength, and θ is orientation in radians. The modulo operation restricts an unsigned orientation to the interval from zero to pi. Some implementations also support signed gradients, which retain direction information.

  6. Build orientation histograms:

  7. Each pixel contributes its gradient magnitude to an orientation bin within a spatial cell.

  8. A more general cell histogram is H(c, b) = sum(m(p) * w(p, b)) over pixels p in cell c.

  9. Here, c identifies a cell, b identifies an orientation bin, m(p) is the pixel’s gradient magnitude, and w(p, b) is its assignment weight. The weight is normally one for its selected bin or may be split between adjacent bins through interpolation.

  10. Group cells into blocks: A typical block contains a 2 × 2 arrangement of cells. Each block histogram is normalized to reduce sensitivity to contrast and intensity scale.

  11. Concatenate block histograms: The normalized vectors from all blocks form the final descriptor.

Parameters and Descriptor Shape

Parameter Typical value Purpose
Cell size 8 × 8 pixels Spatial resolution of each histogram
Block size 16 × 16 pixels Number of cells combined for normalization
Block stride 8 × 8 pixels Movement between blocks; overlapping blocks are common
Orientation bins 9 Quantization of gradient directions
Normalization L2-Hys L2 normalization with clipping and subsequent renormalization

With a 64 × 64 window and the parameters above, there are 7 × 7 blocks. Each block contains four cells, and each cell contains nine bins:

descriptor length = 7 × 7 × 4 × 9 = 1,764

This length is fixed for that window size and parameterization. Changing the image size, cell size, block size, stride, or bin count changes the descriptor length.

What HOG Can Be Used For

HOG is a feature extractor, not a complete recognition system:

  • Image classification: HOG features can be passed to an SVM, logistic regression, or another classifier that predicts a label.

  • Object detection: A sliding-window detector can classify candidate windows and associate scores with boxes. Detection generates candidate boxes; non-maximum suppression then removes duplicate detections.

  • Semantic segmentation: HOG does not natively produce a label for every pixel. Dense variants can provide local features, but modern segmentation systems normally use learned encoders such as U-Net or Mask R-CNN.

  • Instance segmentation: HOG alone does not produce instance masks and labels; those require a downstream segmentation model.

  • Keypoint and pose estimation: HOG is not a native keypoint or pose estimator, although local gradients may contribute to hand-crafted features.

  • Tracking: HOG can describe candidate regions, but motion and identity tracking require a tracking algorithm.

  • OCR: Edge gradients can be useful preprocessing, but HOG does not directly recognize text.

  • Image generation: HOG is not a generative model.

For HOG-based classification, accuracy or top-k accuracy may be appropriate, while precision, recall, F1, and PR-AUC are preferable for imbalanced classes. Object detection is commonly evaluated with IoU and mAP, while segmentation systems are commonly evaluated with mIoU and Dice.

Strengths, Limitations, and Alternatives

  • Strengths:

  • Captures local edge shapes and their spatial orientation.

  • Produces an interpretable, fixed-size feature vector for a fixed window.

  • Can perform well on structured targets such as pedestrians, particularly with a tuned SVM.

  • Is efficient on CPUs and can be suitable for edge or resource-constrained systems.

  • Limitations:

  • Discards much of the color and semantic information available to modern models.

  • Fixed cell and orientation resolution can lose fine detail.

  • Multiscale sliding-window search can be expensive.

  • Performance generally trails learned CNN and vision-transformer features on large, diverse datasets.

For a small image-classification dataset, transfer learning with a pretrained CNN or vision transformer is usually the modern default. Suitable augmentations include flips, crops, and color jitter, using the pretrained model’s expected mean and standard deviation for normalization. End-to-end deployment should consider latency, memory, model size, and hardware: HOG with a compact SVM is attractive on CPUs and edge devices, whereas deep models may require more memory or GPU acceleration and may offer quantization or ONNX-export options.

Common Pitfalls

  • Confusing features with predictions: HOG produces a descriptor; a separate classifier produces labels or detection scores.

  • Using inconsistent preprocessing: Apply the same grayscale conversion, resizing, and intensity-scaling policy during training and inference. Separate mean normalization is not a standard HOG requirement.

  • Ignoring scale changes: Resize targets to the detector’s expected window size or search across multiple scales.

  • Using an expensive search strategy: Very small sliding-window steps improve localization but sharply increase computation.

  • Leaking information: Fit learned preprocessing or dimensionality reduction on training folds only, then apply the fitted transformation to validation and test data.

  • Using the wrong metric: Accuracy can hide poor minority-class performance, and detection should not be evaluated with classification accuracy alone.

  • Ignoring deployment risks: In face-related systems, assess privacy, consent, demographic bias, and disparate error rates in addition to model accuracy.

17

How do image feature extraction methods compare?

Image feature extraction converts pixels into numerical representations that retain information useful for classification, retrieval, detection, and segmentation. Handcrafted descriptors are fast and interpretable but encode less semantics; learned representations usually capture richer task-relevant information but require more data and computation. The main trade-off is efficiency and transparency versus representational power.

Read full answer

Applies tomachine learning and computer vision concepts (Python 3.12)

Common Feature Extraction Methods

Method How it works Strengths Weaknesses Typical use
Global descriptors Summarize color histograms, gradients, shape statistics, or pooled pixels Fast, interpretable, effective with limited data Loses spatial detail; sensitive to lighting and viewpoint Coarse classification and retrieval
Keypoint descriptors Detect corners or distinctive regions and describe them locally with SIFT or ORB Precise geometric matching Limited semantics; depends on local image structure Stitching, localization, matching
CNN backbones ResNet and EfficientNet learn hierarchical convolutional features Strong semantics; effective transfer learning Higher compute and storage cost; may encode bias Classification, detection, segmentation
Vision Transformers ViT converts image patches into tokens and applies self-attention Captures global relationships; flexible architecture Often needs large-scale pretraining; patch resolution limits fine detail Classification and dense prediction
CLIP embeddings Align image and text representations in a shared space Supports zero-shot classification and cross-modal retrieval Weak for precise localization; sensitive to prompts and domain shift Image-text search and zero-shot recognition
Self-supervised features Learn from unlabeled images using reconstruction, masking, or contrastive objectives Exploits large unlabeled datasets Training and evaluation require more care Domain-specific pretraining and retrieval

SIFT forms floating-point descriptors from gradient magnitudes and orientations and tolerates scale and rotation changes. ORB uses faster binary descriptors and is often preferred for real-time matching. SURF is less common than SIFT and ORB, with modern alternatives usually preferred in new systems.

Key Differences

  • Granularity: Keypoint descriptors are sparse; CNN and ViT feature maps cover many locations; global pooling discards precise location.

  • Semantic level: Handcrafted features represent color, gradients, and texture. Learned features can encode objects, parts, and abstract concepts.

  • Data requirements: Handcrafted methods remain useful with small datasets. Deep CNNs and ViTs generally benefit from large-scale pretraining.

  • Robustness: SIFT and ORB handle moderate scale, rotation, and viewpoint changes better than raw pixels. Learned robustness depends on its training distribution and augmentation.

  • Interpretability: Histograms and edge features are inspectable, whereas learned embeddings have limited human-readable meaning.

  • Cost: ORB generally has lower latency than SIFT. CNN cost generally grows with image pixels, channels, and depth. ViT global self-attention has approximately O(N²D) interaction cost, where N is the number of tokens and D is the embedding width.

Selecting Features by Task

  • Image classification: A trained classifier head over pooled CNN, ViT, or CLIP features produces a class label. Evaluate with accuracy or top-k accuracy.

  • Object detection: Faster R-CNN, the YOLO family, and DETR use learned features to predict bounding boxes, labels, and confidence scores. Evaluate with IoU and mAP.

  • Semantic segmentation: U-Net-style models assign a class to every pixel. Evaluate with mIoU and Dice. SAM supports promptable mask extraction rather than inherently predicting semantic classes.

  • Instance segmentation: Mask R-CNN and segmentation variants of the YOLO family predict a class, box, and separate mask per object. Evaluate with mask IoU and mAP.

  • Keypoint and pose estimation: Models predict landmark or joint coordinates. Evaluate with PCK, normalized error, or object-keypoint similarity.

  • Tracking: Appearance features associate detections with existing tracks. Evaluate identity consistency with metrics such as IDF1.

  • OCR: Features support text detection and character or word recognition. Evaluate detection separately from recognition accuracy.

  • Image generation: Generative models such as GANs and diffusion models produce images, sometimes conditioned on extracted features. Evaluate realism, diversity, and task-specific consistency.

Practical Guidance

  • Transfer learning: For small datasets, start with pretrained ResNet, EfficientNet, ViT, or CLIP representations. Freeze the encoder and train a classifier head, or fine-tune only selected layers.

  • Preprocessing and augmentation: Use the checkpoint’s expected mean and standard deviation. OpenCV loads images as BGR, while most PyTorch and torchvision models expect RGB. Flips, crops, resized crops, and suitable color jitter can improve robustness, but must be semantically valid.

  • Preprocessing fit: Fit imputation, scaling, and learned transformations such as PCA on training folds only. Handle missing values explicitly. Scale for distance-, gradient-, or PCA-based methods when required; trees generally do not need scaling.

  • Evaluation: Use stratified cross-validation for classification and inspect precision, recall, F1, PR-AUC, and ROC-AUC for imbalanced data, because accuracy can conceal minority-class failures.

  • Deployment: Compare latency, memory, model size, and CPU versus GPU requirements. Quantization and ONNX export can reduce deployment cost, but require accuracy validation. Consider edge-device support explicitly.

  • Privacy and reproducibility: Face representations may reveal or infer sensitive attributes, so minimize retention, enforce access controls, and test demographic performance. Record checkpoints, preprocessing parameters, seeds, splits, and library versions.

Common Pitfalls

  • Fitting preprocessing on the full dataset before cross-validation introduces data leakage.

  • Assuming deep features are always superior ignores latency, domain shift, and labeled-data requirements.

  • Mismatched normalization or RGB/BGR ordering can silently degrade pretrained features.

  • Comparing methods with different splits, augmentation, or evaluation protocols makes the comparison unreliable.

  • Repeatedly tuning against one validation set causes overfitting to that split; reserve a final test set or use nested cross-validation.

Interviewers commonly ask about scale and rotation invariance, local versus global representations, transfer learning, feature-map use in detectors, and how computational cost changes with image resolution and model depth.

18

How does Haar cascade object detection work in OpenCV?

Haar cascade detection uses a trained cascade of simple rectangular-feature classifiers to locate objects such as faces in an image. It provides fast, small-model detection on CPUs by rejecting most candidate windows early, at the cost of lower accuracy and robustness than modern deep-learning detectors under difficult lighting, poses, or occlusion. In OpenCV, a Haar detector produces bounding boxes; the object label is implicit in the particular cascade that was loaded.

Read full answer

Applies toOpenCV 4.10–4.12 (Python 3.12)

How It Works

  • Haar-like features: Each feature compares weighted sums of pixel intensities in adjacent rectangular regions. A common two-rectangle feature can measure whether one region is darker than a neighboring region.

  • Integral images: OpenCV precomputes an integral representation of the image so the sums for rectangular features can be obtained quickly without repeatedly summing individual pixels.

  • AdaBoost training: Positive and negative training examples are used to select informative weak classifiers, normally simple thresholded decision stumps. AdaBoost assigns weights to those classifiers so their combined vote becomes stronger.

  • Cascade stages: The trained weak classifiers are organized into stages ordered from inexpensive checks to more detailed ones. A candidate window is discarded immediately when it fails a stage, avoiding the cost of evaluating every feature.

  • Image scanning and output: detectMultiScale scans candidate windows over an image pyramid and returns bounding boxes as (x, y, width, height) coordinates. The XML file fixes the feature template relative to each detection window, while scaleFactor controls how many image scales are searched.

  • Neighbor filtering: minNeighbors groups nearby detections and rejects isolated or redundant boxes. A higher value generally leaves only detections with stronger local support.

Using and Training

  • Preprocessing: Convert the BGR image to grayscale, then optionally apply histogram equalization or contrast normalization. Haar cascades do not use the RGB mean and standard-deviation normalization required by most pretrained neural models.

  • Cascade selection: Choose an XML classifier trained for the intended target. Official OpenCV distributions commonly bundle frontal- and profile-face, eye, and body-related classifiers, but exact files vary by package; a pedestrian cascade is not guaranteed to be included.

  • Detection controls: scaleFactor values closer to 1 search more scales and can improve recall at higher cost. minSize and maxSize constrain candidate dimensions, while minNeighbors controls grouping and suppression.

  • Custom objects: cv2.CascadeClassifier loads and applies compatible cascade XML files, but current Python distributions do not provide a complete end-to-end training workflow. A custom detector therefore requires a compatible training toolchain and validation of the exported XML.

  • Data preparation: Keep training, validation, and test data separate, include varied examples and genuine negatives, and avoid tuning thresholds on the test set. For deep-learning alternatives, use the pretrained model’s mean and standard deviation, augment with flips, crops, and color jitter, and use transfer learning as the default for small datasets.

Performance and Evaluation

Haar cascades are attractive when CPU latency and model size matter. Their main limitations are sensitivity to illumination, viewpoint, scale, occlusion, and non-frontal faces, as well as weaker performance on cluttered scenes than modern detectors. Candidate-window count, image scales, and early-stage rejection determine inference cost.

Detection evaluation requires bounding-box ground truth. Use intersection over union (IoU) and mean average precision (mAP); precision and recall are also useful for imbalanced cases. Ordinary classification accuracy can mislead because it does not measure localization quality or account for duplicate and missed detections.

For deployment, compare latency, memory, model size, and hardware support. Deep alternatives may also require evaluating ONNX export, quantization, and edge-device compatibility. Face-related systems additionally require privacy safeguards, consent and retention policies, and testing for demographic bias.

Computer-Vision Context

Haar cascades address object detection, whose output is boxes and labels. Other principal tasks have different outputs: classification produces an image label, semantic segmentation assigns a label to every pixel, instance segmentation produces a mask for each object, keypoint or pose estimation predicts keypoint coordinates, tracking assigns identities across frames, OCR extracts text, and image generation produces an image.

Common current approaches include ResNet and EfficientNet or ViT for classification; Faster R-CNN, the YOLO family, and DETR for detection; U-Net for semantic segmentation; Mask R-CNN for instance segmentation; SAM for promptable segmentation; and CLIP for image-text embeddings. Haar cascades remain useful as simple legacy or specialized detectors, but deep detectors generally offer better accuracy and robustness.

Common Pitfalls

  • Wrong color channel: OpenCV images are normally BGR, while most deep models expect RGB.

  • Confusing a detector with a classifier: detectMultiScale returns boxes, not confidence values or class scores.

  • Excessive search: A scaleFactor too close to 1 can make inference unnecessarily slow.

  • Over-filtering: A high minNeighbors value can suppress genuine detections, especially in crowded scenes.

  • Wrong metric: Accuracy alone does not evaluate detection localization.

  • Unrealistic expectations: Haar cascades are vulnerable to poor lighting, occlusion, profile faces, and unusual poses.

  • Oversight in face use: Model performance, privacy impact, and demographic bias must all be evaluated before deployment.

19

How do CNNs differ architecturally from fully connected networks?

A convolutional neural network (CNN) differs from a fully connected network by using local filters and shared weights over spatial data, whereas a fully connected layer connects every flattened input to every output unit. CNNs exploit locality and parameter sharing for tasks such as image classification, with the main trade-off being stronger spatial assumptions in exchange for less architectural flexibility.

Read full answer

Applies toCNN and fully connected neural network architectures (all versions)

Core Architecture

A traditional fully connected neural network treats each input as a flat vector. Every neuron in one layer connects to every neuron in the next layer:

  • An image of shape 224 × 224 × 3 becomes 150,528 values before entering dense layers.

  • A dense layer with 1,000 neurons requires 150,528,000 learned weights from that image alone.

  • The network does not inherently know that nearby pixels form edges or that the same object may appear in different positions.

A CNN preserves spatial organization while extracting features:

  1. Convolution layer: Applies small filters across local regions.

  2. Activation function: Introduces nonlinearity, commonly ReLU.

  3. Normalization layer: Stabilizes training, often using batch normalization.

  4. Pooling or strided convolution: Usually reduces spatial resolution; output channel depth is determined by the number of filters.

  5. Task head: Converts the extracted representation into predictions.

CNNs often learn a feature hierarchy: early layers detect edges and color contrasts, middle layers combine them into textures and object parts, and deeper layers represent objects or abstract visual concepts. A dense head is commonly used only after the CNN has extracted useful spatial features.

Weight Sharing and Translation Equivariance

A key CNN property is weight sharing: the same convolutional filter is applied at every spatial location.

Property CNN Fully connected network
Input representation Often preserves spatial structure Usually expects a flat vector
Connectivity Local receptive fields Every input connects to every output
Weight sharing One filter is reused across positions Separate weight for each input-output pair
Spatial assumptions Exploits local relationships No built-in spatial structure
Translation behavior Approximately translation-equivariant No built-in translation behavior
Parameter efficiency Usually high Usually low
Typical use Images, video, audio, grids Flattened features, tabular data, generic vector inputs

If a filter detects an edge, it can activate wherever that edge occurs. CNNs are not automatically fully translation-invariant: padding, stride, boundary effects, pooling, and the final prediction head can affect the result.

Scale and Complexity

For a dense layer connecting d inputs to h units:

  • Parameters: d × h + h

  • Multiply-accumulate operations per example: approximately d × h

For a convolution with C input channels, F × F filters, and K filters:

  • Parameters: F × F × C × K + K

  • Work per output position: F × F × C × K

  • Total work: approximately H_out × W_out × F × F × C × K, where H_out and W_out are the output height and width.

CNNs avoid learning a separate detector for every image position, but they can still require substantial memory for activations, especially at high resolution or with many channels.

Practical Design Choices

  • Classification and detection: CNN backbones such as ResNet and EfficientNet support label or probability prediction. Faster R-CNN, YOLO, and DETR additionally produce bounding boxes, labels, and confidence scores for detected objects.

  • Segmentation: U-Net and other CNN encoder-decoder systems assign a semantic class to each pixel. Mask R-CNN produces an instance mask and class for each object. SAM supports promptable masks but generally requires another model or procedure to assign semantic categories.

  • Other vision tasks: Keypoint and pose estimation predict joint or landmark coordinates. Tracking associates detections across frames and assigns persistent object identities. OCR predicts text and often character or word locations. Image-generation systems such as GANs and diffusion models synthesize new images; CNNs may provide generator, decoder, or U-Net components.

  • Transfer learning and preprocessing: For small datasets, pretrained weights are usually preferable. Pretrained models require their documented normalization, and augmentation such as flips, crops, and color jitter should match the deployment distribution.

  • Architecture alternatives: Vision transformers use self-attention instead of convolutional locality, while hybrid models combine both.

  • Deployment: Profile CPU and GPU latency, memory use, and model size for the target workload. Quantization and ONNX export can improve CPU or edge deployment, but converted-model accuracy should be validated. Face-related systems also require explicit privacy, bias, and fairness assessments.

Interview Tips

Common mistakes include claiming that CNNs understand objects exactly like humans, asserting that convolution guarantees translation invariance, or comparing CNNs with dense networks without specifying the input dimensions. Flattening a CNN’s feature maps into a large dense layer can erase spatial structure and create many parameters.

Typical follow-up questions include: How does padding affect output shape? How do stride and dilation change the receptive field? Why does flattening reduce parameter efficiency? When is a dense layer still appropriate? and how should architecture choices change under a strict latency or memory budget?

20

What is the purpose of pooling layers in CNNs?

Pooling layers downsample a CNN’s feature maps by summarizing local regions, reducing spatial dimensions and computation. They help the network tolerate small shifts in an object’s position, but this translation-invariance trade-off can discard fine spatial details needed for localization or segmentation.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

How Pooling Works

A pooling layer slides a window over each feature map, such as a 2 × 2 region, and replaces each region with one value. The number of channels remains unchanged, while the height and width are usually reduced.

For example, max pooling can be represented as:

y[i, j, c] = max(x[m, n, c]) for all (m, n) in the pooling region

Here:

  • x is the input feature map.

  • y is the pooled feature map.

  • i and j identify the output position.

  • c identifies the feature channel.

  • The maximum is calculated independently for each channel.

Unlike a convolution, standard pooling has no learned weights or biases.

Why Pooling Is Useful

  • Spatial downsampling: A 2 × 2 pooling layer with stride 2 halves the height and width, producing a feature map one-quarter the size.

  • Computational efficiency: Later convolutional layers process smaller tensors, reducing memory use and operation count.

  • Larger effective receptive fields: As pooling reduces spatial dimensions, later layers can combine information from a wider portion of the original image.

  • Limited translation invariance: If an important feature moves slightly within the pooling window, max pooling may produce the same output. This helps classification but is not true or unlimited invariance.

  • Hierarchical features: Early layers detect local edges and textures, while later layers combine them into parts and objects.

Common Pooling Choices

Method Operation Characteristics Typical use
Max pooling Keeps the largest value in each region Preserves strong local activations and is widely used in CNNs Classification and object detection
Average pooling Computes the mean in each region Smooths feature maps and retains more aggregate information Classification, feature reduction
Global average pooling Averages each channel over all spatial positions Produces one value per channel and avoids a large flatten layer CNN classifiers and image embeddings
Adaptive pooling Produces a configured output grid Handles varying input sizes with fixed output dimensions Models accepting multiple image resolutions

Pooling is not the only way to downsample. Strided convolutions can learn how to reduce spatial dimensions, and architectures such as ResNet and EfficientNet often use them instead of explicit pooling.

Limitations and Alternatives

Pooling permanently discards detailed spatial information. That can be problematic when the output requires precise object boundaries, pixel-level labels, or keypoint locations.

  • Image classification: Usually benefits from aggressive downsampling because the target is a single label.

  • Object detection: Needs enough spatial resolution to predict accurate bounding boxes.

  • Semantic segmentation: Often preserves or restores resolution, using architectures such as U-Net, and may avoid aggressive pooling.

  • Instance segmentation: Requires accurate masks, so spatial detail is especially important.

  • Keypoint estimation: Precise coordinates can be harmed by excessive downsampling.

Global average pooling can replace a large flatten-and-dense layer in classifiers, reducing parameter count and limiting reliance on one feature-map location.

Common Pitfalls

  • Confusing pooling with feature learning: Pooling applies a fixed operation, while convolution learns filters.

  • Assuming exact translation invariance: Pooling provides only local robustness; nearby or larger shifts can still change the result.

  • Losing localization accuracy: Excessive pooling makes precise boxes, masks, and keypoints harder to predict.

  • Forgetting that channels remain separate: A standard pooling layer reduces only spatial dimensions unless its implementation explicitly does otherwise.

  • Ignoring output-size constraints: Detection and segmentation architectures commonly use feature pyramids, skip connections, or upsampling to retain spatial detail.

21

What is transfer learning in computer vision?

Transfer learning in computer vision reuses representations learned by a model on a large source dataset and adapts them to a target task. It reduces the labeled data, compute, and training time needed to obtain accurate models, especially when the target dataset is small. The main trade-off is that source features may not match the target domain, while aggressive fine-tuning on limited data can overfit.

Read full answer

Applies tocomputer vision transfer-learning concepts (all deep learning framework versions)

How It Works

  • Pretraining: A backbone learns useful visual representations from a large dataset, commonly using classification, self-supervision, or image-text objectives. Common CNN backbones include ResNet and EfficientNet, while ViT is a standard transformer architecture.

  • Backbone reuse: The pretrained backbone becomes the feature extractor for the new task. Earlier layers often represent general patterns such as edges and textures, while deeper layers tend to encode more specialized features.

  • Feature extraction: Freeze the backbone and train only a new task-specific head. This is fast and stable for very small datasets.

  • Fine-tuning: Unfreeze some or all backbone layers and train them with a smaller learning rate than the new head. Progressive unfreezing is a common way to limit catastrophic forgetting.

  • Task adaptation: Replace or adapt the output head to match the target objective. Common model families include Faster R-CNN, YOLO, and DETR for detection; U-Net, Mask R-CNN, and promptable SAM for segmentation; and CLIP for image-text embeddings.

Common Vision Tasks

Task Typical adaptation Output
Image classification Replace or adapt the classifier One class label
Object detection Adapt the backbone and detection heads Bounding boxes and labels
Semantic segmentation Adapt the encoder and pixel-level classifier One label per pixel
Instance segmentation Adapt the backbone and mask heads Pixel masks and instance identities
Keypoint or pose estimation Adapt the backbone and add a keypoint head Keypoint coordinates or heatmaps
Tracking Transfer detection or embedding models and add association logic Consistent identities across frames
OCR Adapt recognition encoders and text-decoding heads Recognized text
Image generation Fine-tune a pretrained generative or diffusion model Generated images

When to Apply It

Situation Recommended approach
Small labeled dataset Use transfer learning as the default
Limited compute or time Start with a frozen backbone and train only the head
Similar source and target domains Fine-tune later layers gradually
Very different domains, such as medical or satellite imagery Compare deeper fine-tuning with domain-specific pretraining using validation performance
Large dataset and ample compute Training from scratch is viable; full fine-tuning of a pretrained model is another option

Preprocessing, Evaluation, and Deployment

  • Use the input size, channel order, normalization mean, and standard deviation expected by the pretrained model. OpenCV loads images in BGR order, whereas most models expect RGB.

  • Apply flips, crops, and color jitter only to training data. Keep validation and test preprocessing deterministic, and split the data before fitting any learned preprocessing to prevent leakage.

  • Use accuracy or top-k accuracy for balanced classification. For imbalanced classification, report precision, recall, F1, ROC-AUC, and PR-AUC because accuracy alone can hide poor performance on rare classes.

  • Use IoU and mAP for detection, and mIoU and Dice for semantic or instance segmentation.

  • Select the number of unfrozen layers and learning rates using a validation set or cross-validation rather than the test set.

  • Consider inference latency, model size, memory use, quantization, and ONNX export for edge deployment. Face-related systems also require explicit privacy, consent, fairness, and bias assessments.

Common Pitfalls

  • Incorrect preprocessing: Using the wrong normalization or image size can substantially degrade pretrained performance.

  • Data leakage: Augmenting validation data or tuning repeatedly on the test set produces unreliable results.

  • Overwriting pretrained features: Fine-tuning every layer at a large learning rate can cause overfitting or catastrophic forgetting.

  • Ignoring domain shift: Natural-image pretraining may not transfer effectively to medical, industrial, or scientific imagery.

  • Wrong metric: Accuracy can be misleading for imbalanced datasets, and unsuitable surrogate data can produce meaningless validation results.

Interview Tips

  • Be ready to compare feature extraction with fine-tuning and explain how dataset size and domain similarity affect the choice.

  • Discuss how transfer learning differs for classification, detection, segmentation, tracking, and generation.

  • Mention the trade-offs among accuracy, training cost, inference latency, model size, and deployment environment.

22

Why are pre-trained models useful in computer vision?

A pre-trained model is a neural network whose weights were learned on a large dataset before being reused for a new task. It solves the common problem of having limited labeled data by transferring useful visual features, reducing training time and improving generalization, although the source data may not match the target domain and fine-tuning can still be computationally expensive.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

How Pre-Trained Models Work

Pre-trained models are trained once on large datasets and then reused through two main approaches:

  • Feature extraction: Freeze the model and use its output as input to a new classifier or regressor. This is fast and appropriate when the target task is closely related to the original training task.

  • Transfer learning: Replace the task-specific output layer and fine-tune some or all model parameters on the target dataset. This adapts higher-level representations when the new domain differs from the original dataset.

For example, an ImageNet-trained ResNet or EfficientNet can be adapted to classify product images. Its early layers commonly detect basic patterns such as edges and textures, while deeper layers represent increasingly abstract features.

Common Computer Vision Uses

Task Typical pre-trained approach Model output
Image classification ResNet, EfficientNet, ViT Class probabilities or labels
Object detection Faster R-CNN, YOLO, DETR Bounding boxes, labels, confidence scores
Semantic segmentation U-Net, DeepLab, SegFormer One class label per pixel
Instance segmentation Mask R-CNN, YOLO segmentation models Pixel masks for individual objects
Keypoint or pose estimation ResNet- or ViT-based pose models Keypoint or skeleton coordinates
Image-text retrieval CLIP Joint image and text embeddings
Promptable segmentation SAM A mask selected using points, boxes, or text-related prompts

Pre-trained models may be trained for one task and adapted to another. For instance, a classification backbone can initialize an object detector, while a segmentation backbone can support instance segmentation.

Advantages and Limitations

  • Less labeled data: Models can perform well when only a small target dataset is available. Fine-tuning a pre-trained network usually requires fewer labels than training from scratch.

  • Faster convergence: Useful features are already learned, so optimization begins from a better starting point.

  • Lower compute requirements: Training from scratch on a large dataset requires substantially more GPU time and memory. Fine-tuning may use fewer epochs, although the entire model is usually still stored in memory.

  • Better generalization: Representations learned from large and diverse datasets often capture robust patterns that are difficult to discover from a small dataset.

  • Practical scalability: Teams can reuse established backbones instead of designing and training every layer from the beginning.

  • Reproducible baselines: Public checkpoints and architectures provide a reliable foundation for experiments and comparisons.

Consideration Advantage or limitation
Data availability Strong performance with limited labeled target data
Compute Usually cheaper than training from scratch
Features Learns useful low-level and high-level visual patterns
Domain mismatch Source features may not transfer well to a specialized domain
Bias Checkpoint behavior may reproduce biases in its training data
Licensing Dataset, weights, and code licenses must all be reviewed
Fine-tuning cost Backpropagation through the model still requires meaningful memory and compute

Transfer learning is generally the default for small computer vision datasets because it usually offers the best balance between performance, development time, and cost.

Fine-Tuning and Preprocessing

Pre-trained models expect input preprocessing compatible with their training setup. With PyTorch and torchvision, use the documented preprocessing pipeline, including the model’s required normalization mean and standard deviation. Images should usually be converted from OpenCV’s BGR format to RGB before being passed to models trained with RGB inputs.

Common augmentations include random resized crops, horizontal flips, rotation, and color jitter. These should reflect expected deployment conditions without changing the meaning of the target label. For example, horizontal flipping may be inappropriate if left-right orientation determines the class.

The amount of fine-tuning depends on domain similarity:

  • Similar domain: Freeze most layers and train only the new output head.

  • Moderate difference: Unfreeze the final block or several higher layers.

  • Large difference: Fine-tune more layers or use a larger adaptation strategy, with careful validation and regularization.

Common Pitfalls

  • Data leakage: Do not fine-tune or select models using validation or test data. Fit preprocessing and model selection using training data only.

  • Wrong metric: Use task-appropriate evaluation. Classification commonly uses accuracy or top-k accuracy; detection uses IoU-based mAP; segmentation uses mIoU or Dice. Accuracy can be misleading with imbalanced classes.

  • Incorrect preprocessing: Mismatched normalization, color order, or image resizing can severely degrade results.

  • Overfitting: A small fine-tuning dataset can still cause overfitting. Use validation-based early stopping, regularization, and appropriate augmentation.

  • Ignoring deployment constraints: Compare latency, model size, memory use, quantization, and ONNX export requirements—not only offline accuracy.

  • Unchecked bias and privacy: Face, medical, and surveillance applications require careful review of demographic performance, consent, data retention, and security.

Interview Tips

Be ready to explain the difference between feature extraction and fine-tuning, identify the target task’s output format, and describe how you would prevent leakage during cross-validation. Common follow-up questions include when transfer learning fails, how learning rates should differ between frozen and unfrozen layers, how preprocessing must match the checkpoint, and whether the model can be deployed under a latency or privacy constraint.

23

How do object detection and image classification differ?

Image classification assigns one or more labels to an entire image, while object detection identifies objects and predicts their locations using bounding boxes. Classification is generally simpler and faster, whereas detection provides more detailed information at the cost of greater model complexity, latency, and annotation requirements.

Read full answer

Applies toall versions

Core Differences

  • Image classification answers: “What is in this image?”

  • The output is typically a single class label, such as cat, car, or tumor.

  • Multi-label classification can predict several classes for one image.

  • It does not indicate where each object appears.

  • Object detection answers: “What objects are present, and where are they?”

  • The output includes a bounding box, class label, and confidence score for every detected object.

  • The number of predictions can vary by image.

  • Overlapping or densely packed objects can be difficult to represent with one box each.

Related computer-vision tasks produce different outputs:

  • Semantic segmentation: assigns a class label to every pixel.

  • Instance segmentation: assigns a pixel-level mask to each individual object instance.

  • Keypoint or pose estimation: predicts body joints, facial landmarks, or other keypoint locations.

  • Tracking: associates detections across frames to produce object trajectories and identities.

  • OCR: detects and recognizes text, often returning text strings and their locations.

  • Image generation: synthesizes new images, usually from text, other images, or random latent inputs.

For example, classifying a street image might return street. Detecting the same image might return boxes labeled car, person, and traffic light.

Comparison

Aspect Image classification Object detection
Main output Image-level class label or labels Bounding boxes, labels, and confidence scores
Spatial information None Location and approximate size of each object
Common tasks Recognize scenes, products, animals, or diseases Locate multiple objects in photographs or video
Annotation One label per image, sometimes multiple Bounding boxes and class labels per object
Model complexity Usually lower Usually higher
Inference speed Typically faster Often slower because locations must be predicted
Evaluation Accuracy, precision, recall, F1, top-k accuracy IoU, precision, recall, mAP
Typical approaches ResNet, EfficientNet, Vision Transformers Faster R-CNN, YOLO family, DETR

How Object Detection Works

Modern detectors commonly use one of two architectural styles:

  1. Two-stage detection, such as Faster R-CNN:

  2. First identifies candidate regions.

  3. Then classifies each region and refines its bounding box.

  4. Often provides strong accuracy but can be computationally expensive.

  5. Single-stage detection, such as many YOLO models:

  6. Predicts object classes and bounding boxes directly from image features.

  7. Usually supports faster real-time inference.

  8. DETR is another architecture that frames detection as a set-prediction problem.

Detection models commonly use CNN backbones such as ResNet or EfficientNet, while transformer-based detectors use vision transformers or hybrid backbones.

Evaluation Metrics

Classification metrics compare predicted labels with ground-truth labels:

  • Accuracy: For single-label classification, the fraction of correctly classified images. It can be misleading when classes are imbalanced.

  • Precision: Fraction of predicted positives that are correct.

  • Recall: Fraction of actual positives that were found.

  • F1-score: Harmonic mean of precision and recall.

  • Top-k accuracy: Correct if the true class appears among the model’s top k predictions.

Detection additionally evaluates localization quality:

  • Intersection over Union (IoU): Measures overlap between predicted and ground-truth boxes.

  • IoU = intersection area / union area

  • mAP: Mean average precision across classes. VOC-style evaluation commonly uses mAP@0.50, while COCO-style evaluation commonly averages over IoU thresholds from 0.50 to 0.95.

  • Under the chosen convention, a prediction is correct when its label matches and its box meets or exceeds the selected IoU threshold.

  • Precision and recall also remain important, especially when small objects, background clutter, or class imbalance are present.

Practical Considerations

  • Data requirements: Object detection requires bounding-box annotations, which are more expensive to create than image-level labels.

  • Preprocessing: Images are usually resized and normalized using the pretrained model’s expected mean and standard deviation.

  • Augmentation: Random resized crops, horizontal flips, color jitter, and related transformations can improve robustness. Boxes must be transformed consistently with the image.

  • Small datasets: Transfer learning with pretrained backbones is usually the best starting point; full training from scratch generally requires much more data.

  • Deployment: Detection models may need optimization for latency- or memory-constrained systems through smaller backbones, pruning, quantization, or ONNX export.

  • Multi-stage tasks: Object detection is often a component of instance segmentation, tracking, pose estimation, and autonomous-driving systems.

Common Pitfalls

  • Confusing detection with segmentation: Bounding boxes are rectangular and include background; segmentation produces pixel-level masks.

  • Evaluating with classification accuracy: Detection requires localization-aware metrics such as mAP and IoU.

  • Ignoring small objects: Resizing, feature-map resolution, and the detector’s input configuration can significantly affect small-object recall.

  • Using mismatched image formats: OpenCV loads images in BGR order, whereas most pretrained vision models expect RGB.

  • Training on the validation set: Use separate validation and test sets, and apply cross-validation when the dataset is small.

  • Overlooking class imbalance: Measure precision, recall, PR-AUC, and per-class performance rather than relying only on overall accuracy.

24

Which algorithms enable real-time object detection?

Real-time object detection algorithms identify multiple objects in an image or video while meeting a strict latency budget. Modern YOLO variants are often the best default for speed, while Faster R-CNN and optimized DETR-family models can provide additional accuracy at higher computational cost. The main trade-off is latency versus detection quality, model size, and hardware requirements.

Read full answer

Applies tocurrent YOLO, EfficientDet, RetinaNet, SSD, Faster R-CNN, and DETR-family implementations

Object Detection Tasks and Outputs

Task Typical output
Image classification One or more class labels
Object detection Bounding boxes, labels, and confidence scores
Semantic segmentation One class label per pixel
Instance segmentation A separate pixel mask and box per object
Keypoint or pose estimation Points and connections for people or animals
Tracking Object identities maintained across video frames
OCR Detected text regions and recognized characters
Image generation A newly generated image

Object detection differs from classification because it must determine where each object is located. A detector normally returns x_min, y_min, x_max, y_max, class label, and confidence for every prediction.

Leading Detection Algorithms

Algorithm How it works Strengths Weaknesses Typical use
Modern YOLO family One-stage detector; many current variants are anchor-free Very high throughput, strong pretrained models, broad deployment support Small or heavily occluded objects may require tuning; duplicate suppression adds latency Video, mobile, edge, and general-purpose detection
EfficientDet Combines an EfficientNet backbone, BiFPN feature fusion, and weighted loss Efficient scaling across model sizes; strong TensorFlow ecosystem Framework-specific implementations may limit portability Resource-constrained and production systems
RetinaNet One-stage detector using focal loss to address class imbalance Strong accuracy and a mature baseline Dense prediction heads can be slower than lightweight YOLO models Accurate detection where moderate latency is acceptable
SSD Predicts objects from feature maps using default boxes Simple, fast, and relatively small models Older designs generally trail newer detectors in accuracy Controlled environments and embedded applications
Faster R-CNN Generates region proposals, then classifies and refines each proposal High accuracy and strong feature reuse Two-stage inference increases latency and memory use Server-side detection and accuracy-sensitive systems
DETR and RT-DETR Transformer-based set prediction; real-time variants improve efficiency End-to-end formulation and global context reasoning Original DETR can be expensive; very small objects remain challenging Advanced real-time detection and research systems

“Real-time” is not an algorithmic property alone. The same model may run at high FPS on a GPU and miss its latency budget on a CPU or edge accelerator.

Evaluation and Selection

Evaluate both detection quality and runtime:

  • mAP@[0.50:0.95]: Common aggregate detection metric across IoU thresholds.

  • mAP@0.50: Measures localization at a less strict IoU threshold.

  • Precision and recall: Important when false positives and missed objects have different costs.

  • IoU: Measures bounding-box overlap; common thresholds include 0.5 and 0.75.

  • Latency and FPS: Measure preprocessing, inference, postprocessing, and data transfer—not only the neural network.

  • Memory and model size: Determine whether deployment fits the target device.

A pipeline sustaining 30 FPS has roughly a 33 ms end-to-end budget per frame. Batch size, input resolution, detector size, non-maximum suppression, accelerator, and runtime implementation all affect this budget.

Practical Training Workflow

  1. Start with pretrained weights: Transfer learning is the default for small datasets because training a detector from scratch requires substantially more data and compute.

  2. Use model-specific preprocessing: Apply the pretrained backbone’s required RGB normalization, including its documented mean and standard deviation.

  3. Augment only training data: Common choices include horizontal flips, resized crops, and color jitter. Box-aware augmentation must keep labels aligned with transformed objects.

  4. Prevent leakage: Create train, validation, and test splits before tuning preprocessing or training on validation data.

  5. Tune detection thresholds: Confidence thresholds and non-maximum suppression affect precision, recall, duplicate boxes, and latency.

  6. Handle imbalance carefully: Rare objects, small objects, and difficult backgrounds may require balanced sampling, hard-negative mining, or loss designs such as focal loss.

  7. Benchmark reproducibly: Record the random seed, software versions, hardware, batch size, precision mode, and warm-up procedure.

Deployment Considerations

  • GPU versus CPU: GPUs maximize throughput, while CPUs simplify deployment but usually reduce performance. Edge accelerators can provide a better latency-energy balance.

  • Optimization: Exporting to ONNX provides an interoperable graph, after which optimized runtimes or vendor-specific tools can improve latency. Quantization reduces memory and compute, but calibration and accuracy testing are required.

  • Model size: Large models may download and cache slowly, consume battery, or exceed device memory limits.

  • Video optimization: Reuse detections across frames or add a tracker when the application permits temporal smoothing.

  • Responsible use: Face-related and people-detection systems require privacy controls, representative evaluation, bias analysis, and clear limits on identification claims.

Common Pitfalls

  • Claiming a model is “real-time” without stating the hardware, resolution, precision mode, and end-to-end latency.

  • Reporting mAP without identifying the IoU range, dataset, input resolution, and postprocessing settings.

  • Increasing input resolution to improve small-object recall without measuring the resulting latency.

  • Training from scratch on a small dataset instead of fine-tuning a pretrained backbone.

  • Evaluating on data used for threshold selection or early stopping.

  • Treating detection as tracking or instance segmentation; those tasks produce different labels and require different metrics.

  • Exporting to ONNX or applying quantization without comparing accuracy against the original model.

Interviewers commonly follow up with: “How is latency measured?”, “Why not use one-stage detection?”, “How would you improve small-object recall?”, “How does non-maximum suppression affect results?”, and “What happens after export to ONNX?”

25

How does YOLO perform object detection in one forward pass?

YOLO (You Only Look Once) predicts candidate object detections in one neural-network forward pass, avoiding the proposal-and-per-region processing pipeline of classic two-stage detectors. Its main trade-off is speed and straightforward deployment versus the potentially higher localization accuracy of systems such as Faster R-CNN; “one pass” does not mean that all post-processing is eliminated.

Read full answer

Applies toYOLO concepts across all versions; Python 3.12, PyTorch 2.5, torchvision 0.20, NumPy 2.x, and OpenCV 4.x for the example

How It Works

  1. A CNN backbone extracts visual features, usually at several resolutions.

  2. A neck, such as a feature pyramid, combines information from those scales.

  3. A detection head predicts bounding-box coordinates and objectness or class scores at each feature-map location. Depending on the YOLO version, assignments may use anchors or be anchor-free.

  4. Training jointly optimizes classification, object-presence, and localization losses.

  5. Inference filters low-confidence predictions and, in most YOLO models, applies non-maximum suppression (NMS). NMS-free variants handle duplicate suppression differently.

A YOLO-style head usually stores box targets relative to the assigned cell. Modern variants may use decoupled heads or distribution-based localization, but the dense one-stage principle remains.

Outputs and Evaluation

Each detection contains a bounding box (x_min, y_min, x_max, y_max), a class label, and a confidence score. The standard overlap formula is:

IoU = intersection_area / union_area

  • Intersection area is the region shared by two boxes.

  • Union area is the region covered by either box.

Detection systems commonly report mAP, precision, recall, and F1. Image-level accuracy is unsuitable because it does not directly measure localization, missed objects, duplicate detections, or class errors.

Vision Tasks and Alternatives

Task Output Common approaches
Image classification Image label ResNet, EfficientNet, ViT
Object detection Boxes, labels, scores YOLO, Faster R-CNN, DETR
Semantic segmentation Class label per pixel U-Net, DeepLab
Promptable segmentation Mask for a prompted region SAM
Instance segmentation Separate mask per object Mask R-CNN, YOLO-seg
Keypoint or pose estimation Object and keypoint locations HRNet, pose YOLO
Tracking Persistent IDs and trajectories ByteTrack, DeepSORT
OCR Text and character or word boxes CRNN, transformer OCR
Image generation Generated images Diffusion models
Detector How it works Strengths Weaknesses Typical profile
YOLO Dense prediction in one network pass Low latency, efficient GPU use Small or crowded objects remain difficult Real-time and edge use; dense output is O(H × W)
Faster R-CNN Proposals followed by per-region heads Strong accuracy, mature tooling More latency and memory Accuracy-focused inference; cost grows with proposal count
DETR Transformer predicts an object set Global context, no anchors High-resolution attention can be costly; long training Batch and general detection; vanilla attention is O((H × W)^2)

Code Example

This compact PyTorch example trains a tiny YOLO-style grid detector on synthetic rectangles. It demonstrates cell-relative targets and one neural forward pass; it is not a production YOLO implementation.

import cv2
import numpy as np
import torch
import torch.nn.functional as F
from torch import nn
from torchvision.ops import nms

torch.manual_seed(42)
rng = np.random.default_rng(42)


def make_data(count):
    images, boxes, labels = [], [], []
    for _ in range(count):
        image = np.zeros((64, 64, 3), dtype=np.uint8)
        label = int(rng.integers(0, 2))
        width = int(rng.integers(12, 28))
        height = int(rng.integers(12, 28))
        x = int(rng.integers(1, 64 - width))
        y = int(rng.integers(1, 64 - height))
        color = (0, 0, 255) if label == 0 else (0, 255, 0)
        cv2.rectangle(image, (x, y), (x + width, y + height), color, -1)

        rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        images.append(torch.from_numpy(rgb).permute(2, 0, 1).float() / 255)
        boxes.append([x, y, x + width, y + height])
        labels.append(label)

    return (
        torch.stack(images),
        torch.tensor(boxes, dtype=torch.float32),
        torch.tensor(labels, dtype=torch.long),
    )


class TinyYOLO(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 16, 3, 2, 1), nn.BatchNorm2d(16), nn.ReLU(),
            nn.Conv2d(16, 32, 3, 2, 1), nn.BatchNorm2d(32), nn.ReLU(),
            nn.Conv2d(32, 64, 3, 2, 1), nn.BatchNorm2d(64), nn.ReLU(),
        )
        # Objectness, cell-relative x/y/width/height, and two class logits.
        self.head = nn.Conv2d(64, 7, 1)

    def forward(self, x):
        return self.head(self.features(x))


images, boxes, labels = make_data(100)
indices = torch.randperm(len(images))
train_idx, test_idx = indices[:80], indices[80:]

model = TinyYOLO()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

model.train()
for epoch in range(30):
    for start in range(0, len(train_idx), 20):
        idx = train_idx[start:start + 20]
        outputs = model(images[idx])
        objectness = torch.zeros(len(idx), 8, 8)
        box_targets = torch.zeros(len(idx), 8, 8, 4)
        class_targets = torch.zeros(len(idx), 8, 8, dtype=torch.long)

        for j, (box, label) in enumerate(zip(boxes[idx], labels[idx])):
            center_x = ((box[0] + box[2]) / 2) / 8
            center_y = ((box[1] + box[3]) / 2) / 8
            grid_x, grid_y = min(int(center_x), 7), min(int(center_y), 7)
            objectness[j, grid_y, grid_x] = 1
            box_targets[j, grid_y, grid_x] = torch.tensor([
                center_x - grid_x,
                center_y - grid_y,
                (box[2] - box[0]) / 64,
                (box[3] - box[1]) / 64,
            ])
            class_targets[j, grid_y, grid_x] = label

        positive = objectness.bool()
        loss = (
            F.binary_cross_entropy_with_logits(outputs[:, 0], objectness)
            + F.smooth_l1_loss(outputs[:, 1:5][positive], box_targets[positive])
            + F.cross_entropy(outputs[:, 5:][positive], class_targets[positive])
        )
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

model.eval()
with torch.inference_mode():
    raw = model(images[test_idx[0]:test_idx[0] + 1])[0]
    objectness = raw[0].sigmoid()
    box_values = raw[1:5].sigmoid()
    probabilities = raw[5:].softmax(dim=0)

grid_x, grid_y = torch.meshgrid(torch.arange(8), torch.arange(8), indexing="ij")
center_x = (grid_x + box_values[0]) / 8
center_y = (grid_y + box_values[1]) / 8
width, height = box_values[2], box_values[3]
xyxy = torch.stack((
    (center_x - width / 2) * 64,
    (center_y - height / 2) * 64,
    (center_x + width / 2) * 64,
    (center_y + height / 2) * 64,
), dim=-1).reshape(-1, 4)

class_confidence, class_ids = probabilities.max(dim=0)
scores = objectness * class_confidence
candidates = torch.where(scores >= 0.05)[0]
keep = candidates[nms(xyxy[candidates], scores[candidates], 0.5)]

print("Raw output shape:", tuple(raw.shape))
print("Detected boxes:", xyxy[keep].tolist())
print("Detected classes:", class_ids[keep].tolist())

Common Pitfalls

  • Leakage and validation misuse: Split by source, recording, or time before training, and keep related video frames in the same fold. Do not repeatedly tune against the test set.

  • Wrong metrics: Use mAP and class-aware detection matching; accuracy can hide localization errors.

  • Poor small-data training: Fine-tune pretrained weights by default. Use flips, crops, and color jitter when appropriate, while preserving tiny objects.

  • Preprocessing errors: Resize or letterbox as required by the checkpoint, convert BGR OpenCV images to RGB, and normalize with that model’s documented mean and standard deviation.

  • Deployment assumptions: Benchmark latency on the target CPU or GPU and consider model size, quantization, ONNX export, and edge-device support. Face-related systems also require explicit privacy, consent, demographic-bias, and false-match safeguards.

26

What are the R-CNN family’s object detection approaches?

The R-CNN family is a lineage of two-stage object detectors that first proposes candidate image regions and then classifies and localizes each proposal. It returns bounding boxes, class labels, and confidence scores; Mask R-CNN also returns per-instance segmentation masks. Its main trade-off is strong localization accuracy for added latency, training complexity, and deployment cost compared with many one-stage detectors.

Read full answer

Applies toR-CNN family architectures (all versions)

Core Pipeline

Object detection must determine both what appears in an image and where it appears. An R-CNN-style detector generally follows these steps:

  1. Generate candidate regions expected to contain objects, using selective search or a learned Region Proposal Network.

  2. Extract a fixed-size feature representation for each proposal.

  3. Predict an object category, confidence score, and bounding-box offsets.

  4. Remove duplicate detections using non-maximum suppression.

The output is one or more detections consisting of a bounding box, class label, and confidence score. Mask R-CNN additionally outputs a binary segmentation mask for each detected instance.

Family and Alternatives

R-CNN evolved by sharing more computation and progressively replacing hand-designed components:

Method How it works Strengths Weaknesses When to use or typical scale
R-CNN Uses selective search, then runs a CNN separately on every proposal Strong region-level image features Extremely slow because of repeated CNN computation Historical reference; roughly one backbone pass per proposal
Fast R-CNN Computes the backbone once, then applies RoI pooling to proposals on the shared feature map Much faster than R-CNN External proposal generation remains a bottleneck Legacy research and moderate workloads
Faster R-CNN Adds a Region Proposal Network that shares backbone features with the detection heads Mature, accurate, and end-to-end trainable Two-stage proposal processing and NMS add latency General-purpose detection and production systems where accuracy matters
Mask R-CNN Extends Faster R-CNN with a mask branch and RoIAlign Produces boxes, classes, scores, and instance masks More compute and more expensive mask annotations Instance segmentation and precise object-level analysis
Cascade R-CNN Adds detection heads with progressively higher IoU thresholds Improves localization of crowded or overlapping objects Additional heads increase inference cost Datasets with strict localization requirements
YOLO family Uses one-stage dense prediction rather than region classification Low latency and simple deployment path May require more careful tuning of localization and small-object handling Real-time, edge, and high-throughput applications; not an R-CNN-family member
Original DETR Predicts a set of objects with a transformer and does not require NMS Global context and end-to-end set prediction Transformer computation and training demands can be substantial Research and diverse detection scenarios; not an R-CNN-family member

Training and Evaluation

A common detector backbone is a pretrained ResNet or EfficientNet. Images should be normalized using the pretrained model’s documented channel means and standard deviations. Any dataset-specific normalization statistics must be estimated from the training split only, never from validation or test data. Training augmentation can include resized crops, horizontal flips, scale variation, and mild photometric changes.

Intersection over Union (IoU) measures bounding-box overlap as the intersection area divided by the union area. A prediction matches a ground-truth object when its IoU reaches the chosen threshold, commonly 0.5 in PASCAL VOC-style evaluation. Average Precision (AP) summarizes the precision-recall curve for a class and IoU threshold, while mAP averages AP across classes and, in COCO evaluation, across IoU thresholds from 0.50 to 0.95.

Precision measures how many detections are correct, while recall measures how many real objects were found. Accuracy can mislead because background proposals and duplicate detections may dominate the counts. Background proposals can also create class imbalance during training, so hard-negative mining and careful sampling may help.

Deployment Considerations

  • Latency: Benchmark the complete pipeline, including proposal generation, per-region heads, and NMS, rather than timing only the backbone.

  • Model size: Larger backbones generally require more memory and inference time, but this must be validated for the target hardware.

  • Quantization: FP16 or INT8 can improve throughput, but small-object accuracy and box-coordinate precision require careful evaluation.

  • ONNX export: It can simplify deployment on servers, mobile platforms, and edge devices when all required operators are supported.

  • Small objects: Use suitable input resolution, feature pyramids, and high-resolution proposals because aggressive downsampling can erase small objects.

  • Hardware and privacy: Compare CPU and GPU latency for the actual workload. Face-related systems also require consent, representative training data, subgroup error analysis, and controls against harmful surveillance use.

Interview Tips

Common mistakes include calling R-CNN a one-stage detector, confusing classification accuracy with mAP, evaluating only at one IoU threshold, and overlooking proposal-generation costs. Strong follow-up questions ask why Fast R-CNN is faster than R-CNN, how Faster R-CNN generates proposals, why Mask R-CNN uses RoIAlign, and when YOLO or a DETR-based model would be a better deployment choice.

27

How do image recognition models handle occlusion?

Occlusion occurs when another object hides part of a target, so the model must combine visible evidence with context, prior frames, or explicit hidden-shape reasoning. Common defenses are occlusion-aware augmentation, robust feature learning, attention, temporal reasoning, and amodal prediction; the trade-off is greater robustness versus more training complexity, compute cost, and possible loss of clean-image accuracy.

Read full answer

Applies tomachine learning concepts (Python 3.12); PyTorch 2.x example; scikit-learn 1.5+ cross-validation guidance

How Models Respond

Because hidden pixels cannot be observed directly, models infer the target from incomplete evidence:

  • Visible evidence: Edges, textures, colors, and contours identify the exposed portion.

  • Context: Surrounding objects and scene relationships help infer what is hidden.

  • Hierarchical features: CNN backbones such as ResNet and EfficientNet combine evidence across receptive-field sizes. ViTs use patch tokens and attention, while CLIP adds image-text semantic supervision.

  • Detection and segmentation: Faster R-CNN, YOLO-family models, and DETR normally use visible-region evidence. U-Net, Mask R-CNN, and promptable SAM are common segmentation approaches; amodal variants additionally estimate hidden regions.

  • Attention: Transformers connect separated visible regions, but attention alone does not guarantee occlusion robustness.

  • Temporal reasoning: Tracking systems combine appearance history, motion trajectories, and re-identification across frames.

  • Amodal reasoning: Specialized models predict an object’s visible mask and infer its complete hidden shape.

Common Approaches

Approach How it works Strengths Weaknesses Best use Relative cost
Random erasing Covers random image regions during training Simple and broadly useful Synthetic masks may be unrealistic Small or medium datasets Low
Copy-paste augmentation Places masked objects over other images Can create realistic occluders Requires suitable masks, scales, and placement Detector training Medium
Context and attention Relates visible parts to the whole scene Exploits partial evidence Cannot recover a completely hidden target reliably Recognition and detection Medium to high
Amodal segmentation Predicts visible and hidden object regions Estimates complete geometry Needs specialized models or labels Robotics and scene understanding High
Multi-view or tracking Combines cameras or frames Useful when another view becomes available Requires extra hardware or scene history Vehicles and surveillance High

Task Outputs and Evaluation

  • Image classification: A class label and confidence score.

  • Object detection: Bounding boxes, class labels, and confidence scores, normally for visible regions.

  • Semantic segmentation: A label per pixel; an occluder normally receives its own label.

  • Instance segmentation: A visible mask per object; amodal variants also predict hidden regions.

  • Keypoint and pose estimation: Joint coordinates and often visibility or occlusion flags.

  • Tracking: Object identities maintained across frames.

  • OCR: Text inferred from visible characters and linguistic context.

  • Image generation: Synthesized pixels, potentially including occluders or inpainted content.

Use accuracy or top-k accuracy for classification, mAP and IoU-based localization results for detection, and mIoU and Dice for segmentation. For imbalanced classification, also report precision, recall, F1, ROC-AUC, and PR-AUC because accuracy can be dominated by the majority class and hide poor minority-class performance.

Practical Guidance

Transfer learning is the default for small datasets. Use the pretrained model’s required mean and standard deviation, and combine realistic occluders with crops, flips, and color augmentation. Keep clean and occluded variants of the same source image in the same cross-validation fold; use StratifiedKFold for classification or StratifiedGroupKFold when grouping by source image, and reserve an untouched test set. Deployment decisions should balance latency, model size, quantization, ONNX export, CPU/edge/GPU hardware, and privacy and demographic-bias risks in face-related systems.

Code Example

This example trains a small CNN with random occluders and compares paired clean and occluded test images. The evaluation seed is different from the training seed to prevent overlap.

import numpy as np
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

SEED = 42
torch.manual_seed(SEED)


def draw_shape(label: int, rng: np.random.Generator,
               occluded: bool) -> np.ndarray:
    image = rng.normal(8, 3, size=(32, 32, 3))
    center_x, center_y = rng.integers(10, 23, size=2)
    yy, xx = np.ogrid[:32, :32]

    if label == 0:  # Square
        mask = (
            (np.abs(xx - center_x) <= 5)
            & (np.abs(yy - center_y) <= 5)
        )
    else:  # Triangle
        half_width = (yy - (center_y - 5)) * 0.5
        mask = (
            (yy >= center_y - 5)
            & (yy <= center_y + 5)
            & (np.abs(xx - center_x) <= half_width)
        )

    image[mask] = 235

    # Always consume these draws so clean and occluded versions stay paired.
    width = int(rng.integers(5, 9))
    x = int(rng.integers(8, 24))
    y = int(rng.integers(0, 12))
    if occluded:
        image[y:y + 20, x:x + width, :] = 25

    return np.clip(image, 0, 255).astype(np.float32)


def make_dataset(count: int, seed: int, mode: str) -> TensorDataset:
    rng = np.random.default_rng(seed)
    images = np.empty((count, 3, 32, 32), dtype=np.float32)
    labels = np.empty(count, dtype=np.int64)

    for index in range(count):
        label = int(rng.integers(0, 2))
        occluded = mode == "occluded" or (
            mode == "train" and rng.random() < 0.7
        )
        image = draw_shape(label, rng, occluded)

        # Normalize to [-1, 1] and convert HWC to CHW.
        images[index] = image.transpose(2, 0, 1) / 127.5 - 1.0
        labels[index] = label

    return TensorDataset(torch.from_numpy(images), torch.from_numpy(labels))


class ShapeCNN(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(16, 32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(32, 64, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d((4, 4)),
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(64 * 4 * 4, 64),
            nn.ReLU(),
            nn.Linear(64, 2),
        )

    def forward(self, inputs: torch.Tensor) -> torch.Tensor:
        return self.classifier(self.features(inputs))


train_data = make_dataset(1200, SEED, "train")
clean_data = make_dataset(300, SEED + 1, "clean")
occluded_data = make_dataset(300, SEED + 1, "occluded")

loader_generator = torch.Generator().manual_seed(SEED)
train_loader = DataLoader(
    train_data,
    batch_size=128,
    shuffle=True,
    generator=loader_generator,
)

model = ShapeCNN()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

for epoch in range(3):
    model.train()
    running_loss = 0.0
    for inputs, labels in train_loader:
        optimizer.zero_grad()
        loss = criterion(model(inputs), labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

    mean_loss = running_loss / len(train_loader)
    print(f"Epoch {epoch + 1}, mean loss: {mean_loss:.4f}")


@torch.no_grad()
def evaluate(loader: DataLoader) -> float:
    model.eval()
    correct = 0
    total = 0
    for inputs, labels in loader:
        predictions = model(inputs).argmax(dim=1)
        correct += (predictions == labels).sum().item()
        total += labels.size(0)
    return correct / total


clean_loader = DataLoader(clean_data, batch_size=128)
occluded_loader = DataLoader(occluded_data, batch_size=128)

clean_accuracy = evaluate(clean_loader)
occluded_accuracy = evaluate(occluded_loader)

print(f"Clean accuracy: {clean_accuracy:.3f}")
print(f"Occluded accuracy: {occluded_accuracy:.3f}")
print(f"Accuracy drop: {clean_accuracy - occluded_accuracy:.3f}")

Common Pitfalls

  • Training only on clean images: The model may become dependent on features that deployment regularly hides.

  • Using unrealistic cutouts: Models may learn the synthetic occluder rather than general occlusion robustness.

  • Confusing visible and amodal outputs: Standard segmentation predicts visible geometry, not the complete hidden shape.

  • Leaking related images: Clean and occluded variants of one source must not be split across training and testing folds.

  • Ignoring complete occlusion: A fully hidden target requires context, another view, or prior information.

  • Overfitting the validation set: Select augmentation and thresholds on validation folds, then evaluate once on the untouched test set.

Interviewers commonly ask which features remain reliable under occlusion, how amodal segmentation differs from semantic segmentation, and why attention does not guarantee robustness.

28

How do one-stage and two-stage object detectors differ?

A one-stage detector predicts bounding boxes, object classes, and confidence scores directly from backbone features. A two-stage detector first generates region proposals, then classifies and refines each proposal. The main trade-off is that one-stage models are generally simpler and faster, while two-stage models often provide stronger region-level classification and localization at greater computational cost.

Read full answer

Applies tomachine learning concepts (all versions)

Architecture Comparison

Approach How It Works Strengths Weaknesses When to Use Typical Scale and Complexity
One-stage Applies detection heads directly to dense feature maps or object queries Simple pipeline, low latency, effective for real-time and edge applications Dense training can suffer foreground-background imbalance; small or crowded objects can be difficult Video analytics, robotics, edge devices, high-frame-rate systems Cost generally scales with feature-map locations or query count and the candidates evaluated per location
Two-stage Generates candidate regions, then classifies and refines each region Strong localization, region-level classification, flexible proposal filtering More stages, additional latency, proposal storage and processing Accuracy-sensitive detection, varied object sizes, systems with sufficient compute Proposal generation scans the image, followed by per-proposal classification over a relatively small proposal set

Examples include the YOLO family for one-stage detection and Faster R-CNN for two-stage detection. DETR is best described as an end-to-end set-prediction alternative rather than forced into the traditional proposal-based taxonomy.

Detection Pipeline

A one-stage detector commonly:

  1. Extracts a multi-scale feature hierarchy with a CNN or vision-transformer backbone.

  2. Creates dense locations or object queries for potential detections.

  3. Predicts a box, class probability, and confidence for each candidate.

  4. Removes duplicate predictions using non-maximum suppression or an NMS-free alternative.

Some one-stage detectors use anchors, while others use anchor-free grid or query representations. RetinaNet uses anchors and focal loss to address foreground-background imbalance in dense detection.

A two-stage detector commonly:

  1. Extracts backbone features.

  2. Uses a region proposal network to identify candidate regions.

  3. Pools each region into fixed-size features, commonly with RoI Align.

  4. Applies classification, box regression, and optionally additional detection heads.

  5. Filters proposals and suppresses overlapping duplicates.

The first stage locates candidate objects; the second determines each candidate’s class and more precise boundaries.

Performance Factors

  • Small objects: Two-stage detectors can benefit from detailed per-proposal feature pooling. One-stage models can also perform well with feature pyramids, high-resolution inputs, or specialized small-object heads.

  • Large class counts: Two-stage classifiers evaluate a relatively small proposal set, while dense one-stage models must make predictions across many locations.

  • Background imbalance: Focal loss, proposal sampling, and positive-sample balancing can reduce this problem.

  • Inference settings: Confidence and duplicate-suppression thresholds affect precision and recall and should be tuned on validation data.

  • Backbone and resolution: Input resolution, feature-pyramid levels, and proposal count often affect performance as much as the conceptual number of stages.

  • Accuracy: Modern one-stage detectors can match or exceed older two-stage systems, so one-stage does not automatically mean less accurate.

Evaluation Metrics

Object detectors output bounding boxes, labels, and confidence scores, so evaluation must assess both classification and localization.

  • Intersection over Union (IoU): IoU = intersection area / union area, where the areas come from the predicted and ground-truth boxes.

  • Precision: The proportion of predicted objects that are correct.

  • Recall: The proportion of ground-truth objects that are detected.

  • mAP@50: Mean average precision across classes at an IoU threshold of 0.5.

  • mAP@50:95: Mean average precision across classes and IoU thresholds from 0.5 to 0.95, providing a stricter localization measure.

Overall accuracy is inappropriate because it does not adequately measure box localization or performance on rare classes. Per-class precision, recall, and average precision are more informative.

Practical Selection

  • Choose one-stage detection when latency, throughput, pipeline simplicity, or edge deployment is the priority.

  • Choose two-stage detection when localization quality or region-level processing is more important.

  • Consider DETR-style detection when training resources and large datasets justify transformer-based set prediction.

  • Use pretrained backbones for small datasets, normalizing inputs with the pretrained model’s specified mean and standard deviation.

  • Use flips, crops, and carefully chosen color jitter while preserving annotation validity.

  • Measure model size, quantization, ONNX export compatibility, hardware support, and end-to-end latency.

  • For face-related systems, evaluate privacy, demographic bias, false identification risks, and the consequences of missed detections.

Common Pitfalls

  • Assuming one-stage detectors are always less accurate or two-stage detectors are always faster.

  • Evaluating only one confidence threshold instead of precision-recall behavior and mAP.

  • Ignoring small or crowded objects that require suitable feature-map resolution.

  • Using an inappropriate duplicate-suppression strategy; verify whether it is class-aware and tune thresholds on validation data.

  • Tuning thresholds or hyperparameters repeatedly on the test set.

  • Neglecting annotation quality, dataset leakage, class imbalance, and differences in operating conditions.

Interviewers commonly ask which architecture you would deploy, how proposal count affects speed, how small objects are handled, and how you would diagnose poor recall for a specific class.

29

What is the difference between semantic and instance segmentation?

Semantic segmentation assigns a class label to every pixel, while instance segmentation identifies and separates individual objects by assigning a distinct mask to each instance. The main trade-off is that semantic segmentation is simpler and often more efficient, but it merges all objects of the same class, whereas instance segmentation preserves object-level boundaries at the cost of greater model and annotation complexity.

Read full answer

Applies tomachine learning and computer vision concepts

Core Difference

Aspect Semantic segmentation Instance segmentation
Output One class label per pixel A class label and distinct mask for each object instance
Typical output shape height × width label map List of masks, bounding boxes, labels, and confidence scores
Identifies object categories Yes Yes
Separates objects of the same class No Yes
Example Every road pixel receives the label road Each car receives a separate mask
Main complexity Lower Higher because objects must be detected and separated
Common metrics mIoU, Dice Mask AP, IoU, Dice, and detection AP

For example, if an image contains three people, semantic segmentation labels every visible person pixel as person. Instance segmentation produces three separate person masks, allowing each person’s shape and area to be measured independently.

How Each Approach Works

Semantic segmentation treats segmentation as dense pixel classification:

  • The model receives an image and predicts a class probability for every pixel.

  • The highest-scoring class becomes that pixel’s label.

  • Connected regions of the same class are not treated as separate objects.

  • Standard approaches include FCN-based networks, U-Net, DeepLab, and transformer-based models such as SegFormer.

Instance segmentation combines object-level localization with pixel-level masks:

  • The model detects candidate objects and predicts a class, bounding box, and pixel mask for each instance.

  • Overlapping or adjacent objects of the same class can remain separate.

  • Standard approaches include Mask R-CNN, YOLO models with segmentation heads, and other mask-based detectors.

  • Promptable models such as SAM can generate masks from points or boxes, but additional components may be required for classification and stable instance organization.

Evaluation Metrics

  • mIoU: Mean Intersection over Union, calculated separately for each class and then averaged. It measures how accurately the predicted region overlaps the corresponding ground-truth region.

  • Dice score: Measures region overlap using 2 × intersection / (predicted pixels + ground-truth pixels). It is especially useful when foreground objects occupy only a small portion of an image.

  • Mask IoU: Measures overlap between one predicted instance mask and its matched ground-truth instance mask.

  • Average Precision (AP): Evaluates instance predictions by considering class confidence and mask overlap. Detection AP and mask AP may be reported separately.

Pixel accuracy can be misleading for semantic segmentation because a model that predicts the background everywhere can achieve a high score when most pixels belong to the background. Prefer mIoU and Dice, and inspect per-class results.

Choosing an Approach

Use semantic segmentation when the task only requires categories such as road, building, vegetation, or sky. It is appropriate for land-cover mapping, medical region labeling, and autonomous-driving drivable-area segmentation.

Use instance segmentation when object-level information matters, such as counting vehicles, measuring individual product dimensions, or separating touching cells. It is particularly useful for robot grasping, video object tracking, and analyzing individual animals or people.

Practical Considerations

  • Data annotation: Semantic masks are simpler to label. Instance annotation must identify and assign a separate mask to every individual object; some formats also store bounding boxes, which can otherwise be derived from the masks.

  • Preprocessing: Resize images consistently with the model and normalize them using the pretrained model’s expected mean and standard deviation.

  • Augmentation: Random resized crops, horizontal flips, and carefully chosen color augmentation can improve generalization. Flips may be invalid for tasks where orientation has semantic meaning.

  • Class imbalance: Background pixels can dominate the loss. Class weighting, balanced sampling, or Dice and focal losses may help.

  • Inference cost: Instance segmentation generally requires more computation because the model must locate and mask multiple objects.

  • Deployment: Consider latency, model size, quantization, ONNX export, and whether the target device is a GPU or edge device.

  • Privacy and bias: Person and face segmentation processes sensitive biometric imagery. Evaluate demographic performance and avoid retaining unnecessary image data.

Common Pitfalls

  • Confusing semantic labels with instance identities: semantic segmentation cannot distinguish two adjacent cars.

  • Reporting only pixel accuracy on images dominated by background.

  • Mixing semantic IoU with instance AP when comparing results; they evaluate different outputs.

  • Assuming a promptable segmentation mask automatically provides class labels or stable instance identities.

  • Applying the same augmentation to images with directional or medically meaningful orientation.

  • Using full training data for preprocessing before cross-validation, which can cause data leakage.

Interviewers commonly ask whether the task requires category-level or object-level results, how overlapping instances are handled, which matching metric is used for evaluation, and how mask quality would be validated on a real dataset.

30

How does a Fully Convolutional Network enable semantic segmentation?

A Fully Convolutional Network (FCN) is a neural network architecture that replaces the fully connected layers of a classification CNN with convolutional layers so it can produce a dense, pixel-wise prediction map instead of a single label. It solves the problem of semantic segmentation, where each pixel in an image must be assigned a class label (for example, road, person, or background). Its main trade-off is that aggressive downsampling makes predictions spatially coarse, while skip connections and learned upsampling recover detail at the cost of added complexity and memory.

Read full answer

Applies todeep learning for computer vision (architecture concepts; current stable practice with PyTorch 2.x or TensorFlow 2.16+)

How FCN Works

  • Convolutional encoder: A backbone CNN (often adapted from a classifier such as VGG-style or ResNet-style networks) applies convolutions and pooling to extract hierarchical features. Spatial resolution shrinks while semantic information becomes stronger.

  • Fully connected to convolutional conversion: Final dense layers are rewritten as 1x1 (or equivalent) convolutions. This preserves spatial structure and allows the network to accept inputs of arbitrary height and width.

  • Upsampling for dense output: The low-resolution feature map is enlarged to the input size using learned upsampling, commonly transposed convolution (sometimes called deconvolution). The output has C channels for C classes at each pixel.

  • Skip connections (FCN variants): To reduce coarseness, FCN fuses deep, semantic features with shallower, higher-resolution features.

  • FCN-32s: upsamples the deepest prediction by 32×.

  • FCN-16s: combines a 2× upsampled deep prediction with a pool-layer prediction, then upsamples.

  • FCN-8s: adds an even earlier feature map for finer boundaries.

Variant Fusion detail Typical output quality Cost
FCN-32s Deep features only Coarsest boundaries Lowest
FCN-16s Deep + mid-level skip Better localization Moderate
FCN-8s Deep + mid + shallow skips Finer detail Higher memory/compute

Role in Semantic Segmentation

  • Dense prediction: FCN established the standard pattern for semantic segmentation: learn features, predict a class score per pixel, and train with a per-pixel loss (commonly cross-entropy over pixels).

  • End-to-end training: The whole network is trained jointly from input images to segmentation maps, unlike earlier pipelines that relied on separate region proposals or handcrafted post-processing.

  • Foundation for later models: FCN influenced architectures such as U-Net (symmetric encoder-decoder with dense skips), DeepLab variants (atrous/dilated convolutions and context modules), and modern transformer-based segmenters. For instance segmentation, models such as Mask R-CNN extend detection with per-instance masks.

Evaluation and Practical Considerations

  • Metrics: Pixel accuracy can mislead when background dominates. IoU (Intersection over Union) measures overlap between predicted and ground-truth pixels per class; mIoU averages IoU across classes. Dice (F1-like overlap) is common in medical imaging and emphasizes foreground agreement.

  • Preprocessing and augmentation: Normalize inputs with the pretrained backbone’s mean and standard deviation, and use augmentations such as flips, crops, and color jitter. Transfer learning is the default for small datasets: start from pretrained encoder weights and fine-tune.

  • Efficiency: Transposed convolutions and high-resolution outputs increase memory use. For deployment, consider latency, model size, quantization, ONNX export, and edge constraints. Be cautious with face or person segmentation because of privacy and bias risks.

Interview Tips

  • Common pitfalls: Confusing semantic segmentation (one label per pixel, no instance IDs) with instance segmentation; reporting pixel accuracy on imbalanced classes instead of mIoU/Dice; using mismatched input normalization for a pretrained backbone; ignoring checkerboard artifacts that can arise from poorly configured transposed convolutions.

  • Typical follow-ups: “How do FCN skip connections differ from U-Net skips?”, “Why use mIoU instead of accuracy?”, “How would you handle class imbalance in segmentation?”, and “What changes for instance segmentation versus semantic segmentation?”

31

What are the applications of scene understanding in computer vision?

Scene understanding is the process of interpreting an image or video by identifying objects, their spatial relationships, scene layout, and surrounding context. It enables decisions based on scene meaning rather than isolated pixels, supporting autonomous driving, robotics, healthcare, retail, surveillance, augmented reality, remote sensing, and video analytics. The main trade-off is that richer contextual reasoning generally requires more data, computation, and rigorous validation than basic image classification.

Read full answer

Applies tocomputer vision concepts (all versions)

Core Tasks and Outputs

Task Typical output Example application
Image classification One or more labels Classifying an urban or residential scene
Object detection Bounding boxes, labels, and confidence scores Detecting pedestrians and vehicles
Semantic segmentation One class label per pixel Separating roads, sidewalks, vegetation, and buildings
Instance segmentation Separate per-object masks Measuring individual tumors or separating overlapping objects
Keypoint and pose estimation Keypoint coordinates or skeletons Detecting unsafe worker postures or analyzing sports poses
Tracking Persistent object IDs and trajectories Estimating traffic flow or following objects through video
OCR Recognized text and its location Reading signs, labels, or scanned documents
Image generation Images generated from text or layout prompts Producing synthetic scenes for training data

Scene-level reasoning adds relationships, temporal context, and affordances—for example, inferring that a pedestrian is crossing in front of a vehicle or that an object is partially occluded by a table.

Applications

  • Autonomous driving: Detect road users and drivable space while identifying hazards and scene layout.

  • Robotics: Support object grasping, indoor navigation, and interpretation of human instructions.

  • Healthcare: Locate anatomical structures and assess spatial relationships in radiology, pathology, and surgical video.

  • Retail: Locate products, read prices, and monitor shelves or checkout activity.

  • Security and surveillance: Detect intrusions, abandoned objects, or unusual crowd activity subject to privacy controls.

  • Augmented reality: Anchor virtual objects to real surfaces, people, and furniture using depth, segmentation, and pose.

  • Remote sensing: Identify buildings, roads, land cover, and changes in satellite or drone imagery.

  • Video analytics: Combine recognition, detection, and tracking to produce alerts and operational statistics.

Modern Approaches

Approach How it works Strength Limitation Typical use
ResNet and EfficientNet Extract hierarchical local features Efficient transfer-learning backbones Limited native long-range reasoning Classification and small-to-medium datasets
Vision Transformers Model interacting image-patch tokens Capture global context Quadratic token attention; often data-hungry Large datasets and multimodal systems
Faster R-CNN Generates proposals, then classifies and refines regions High-quality region predictions Two-stage processing adds latency Accuracy-sensitive detection
YOLO family Uses a one-stage detector with one network pass Fast and suitable for real-time use Crowded or tiny objects may require tuning Edge, video, and real-time applications
DETR Predicts objects as a set with a transformer End-to-end and anchor-free Can be computationally heavy General detection and research
U-Net and Mask R-CNN Produce dense pixel-level predictions Support spatial measurements Need dense annotations or additional stages Medical, satellite, and industrial imagery
SAM Produces masks from point, box, or mask prompts Flexible promptable segmentation Depends on prompt quality and domain match Interactive tools and annotation assistance
CLIP Maps images and text into a shared embedding space Supports zero-shot classification and retrieval Does not directly provide geometry or pixel masks Image search and multimodal retrieval
Diffusion models Generate images through iterative denoising from conditioning inputs High-fidelity, controllable outputs Training and inference can be compute-intensive Synthetic training data

Evaluation

No single metric evaluates complete scene understanding because its outputs have different forms:

  • Classification: Accuracy and top-k accuracy; use precision, recall, macro-F1, and PR-AUC for imbalanced classes. Accuracy can conceal failures on rare but important categories.

  • Detection: Intersection over Union (IoU) and mean Average Precision (mAP), including separate results for small and medium objects.

  • Semantic segmentation: Mean Intersection over Union (mIoU) and Dice score, with rare classes checked separately.

  • Instance segmentation: Mask IoU and class-specific Average Precision.

  • Pose estimation: Percentage of Correct Keypoints (PCK) or mean joint position error.

  • Tracking: IDF1, HOTA, and identity-switch counts across sequences.

  • OCR: Character Error Rate and Word Error Rate.

Contextual correctness also matters: strong overlap metrics do not guarantee that inferred relationships are plausible, so scenario-level testing remains necessary.

Preprocessing and Deployment

  • Normalize inputs with the pretrained model’s required mean and standard deviation rather than generic averages.

  • Use suitable flips, crops, scale changes, or color jitter without changing the task’s semantics.

  • Prefer transfer learning for small datasets, fitting preprocessing and model adaptation only on training data.

  • Keep nearby video frames or spatially correlated samples in the same split to prevent leakage.

  • Evaluate latency, model size, peak memory, quantization, and ONNX export on the target CPU, GPU, or edge device.

  • Apply privacy safeguards, bias testing, human oversight, and regulatory compliance to face-related and surveillance systems.

Common Pitfalls

  • Treating object detection as equivalent to complete scene understanding.

  • Using accuracy for heavily imbalanced or safety-critical datasets.

  • Splitting nearby video frames across training and test sets.

  • Applying augmentations that invalidate text, medical, or pose labels.

  • Ignoring occlusion, small objects, domain shift, and contextual hallucinations.

  • Optimizing benchmark scores without testing real deployment failures.

Interviewers may follow up by asking how relationships are represented, how models avoid hallucinating objects, which metrics expose rare-class failures, and how systems handle domain shifts and ambiguous scenes.

32

How does depth improve semantic segmentation?

Depth-aware semantic segmentation combines color imagery with per-pixel depth information to predict a semantic class for every pixel. Depth supplies geometry—such as distance, surface continuity, and object boundaries—that can disambiguate regions with similar appearance, but it also introduces sensor, calibration, missing-data, and computational costs.

Read full answer

Applies tosemantic-segmentation concepts (all versions)

Why Depth Helps

Depth can improve segmentation by providing information that RGB alone may not capture:

  • Class disambiguation: Two surfaces may have similar colors and textures but lie at different distances.

  • Boundary detection: Depth discontinuities often align with object boundaries, helping separate touching objects.

  • Geometric context: Distance and surface orientation reveal whether a region belongs to a road, wall, floor, or another structure.

  • Scale reasoning: Apparent size becomes more meaningful when combined with metric distance.

  • Occlusion reasoning: Depth can help distinguish foreground objects from surfaces behind them.

Depth does not automatically improve every model. Noisy, sparse, misregistered, or out-of-distribution measurements can make performance worse than an RGB-only approach.

Semantic segmentation differs from image classification, which produces one image label; object detection, which produces boxes and labels; instance segmentation, which produces separate masks; keypoint and pose estimation, which produce points; tracking, which assigns identities over time; OCR, which produces text; and image generation, which produces new pixels.

How Models Use Depth

  • RGB-D fusion: Supply RGB and depth together, either by concatenating them as channels or by processing them in separate branches before combining their features.

  • Auxiliary supervision: Train a shared model with a segmentation objective and a weighted depth-estimation objective. The weighting controls how much the depth task contributes relative to segmentation.

  • Geometric regularization: Encourage segmentation predictions to agree with depth boundaries, surface orientation, or reconstructed 3D geometry.

  • 3D representation: Use camera calibration to back-project depth pixels into 3D points, then process the resulting point cloud or projected features.

  • Depth completion: Estimate a denser depth map when sensor measurements contain holes, allowing more pixels to participate in fusion.

Fusion Strategies

Approach How it works Strengths Weaknesses When to use
Channel concatenation Adds normalized depth as one or more input channels Simple and inexpensive May underrepresent modality-specific differences Small datasets and latency-sensitive systems
Separate encoders Uses distinct CNN or transformer branches Accommodates modality-specific features More parameters and training complexity Strong RGB and depth sensors with sufficient data
Attention-based fusion Uses one modality to guide features from the other Models conditional relationships Higher compute and implementation cost Large datasets or difficult cross-modal ambiguities
3D projection or point-cloud model Back-projects pixels into 3D and reasons geometrically Exploits explicit scene geometry Requires calibration and efficient point processing Tasks where 3D relationships are central

Evaluation and Training

  • Split the data before fitting anything, and compute depth-normalization statistics from training data only. Use image-level validation and test partitions; stratified image-level folds can help with class imbalance.

  • Apply geometric augmentations consistently to RGB, depth, and masks. Use nearest-neighbor or otherwise appropriate resampling for depth so that it is not blurred across sharp depth boundaries.

  • Evaluate semantic segmentation with mIoU, Dice score, and per-class IoU. Pixel accuracy can be misleading when background pixels dominate.

  • Use transfer learning when labeled data is limited, and normalize RGB inputs using the pretrained checkpoint’s mean and standard deviation.

  • Represent missing or invalid depth explicitly rather than silently treating it as a valid zero-depth measurement.

Deployment and Responsible Use

Additional depth processing can increase latency, memory use, and model size. For edge deployment, evaluate quantization and ONNX export, and consider whether a separate depth sensor is practical. Privacy and bias require particular attention when depth is used in face-related or people-tracking systems.

Common Pitfalls

  • Misregistering RGB and depth sensors.

  • Normalizing depth with full-dataset statistics.

  • Applying crops, flips, or resizing independently to the modalities.

  • Assuming valid sensor depth where measurements are missing or corrupted.

  • Reporting only pixel accuracy on imbalanced datasets.

  • Adding depth without comparing it against a strong RGB-only baseline.

Common interview follow-ups ask how RGB and depth are fused, how missing depth is handled, why geometric consistency helps, and how depth affects latency, model size, and deployment cost.

33

What is pose estimation in computer vision?

Pose estimation is a computer vision task that locates anatomical keypoints (such as joints or facial landmarks) on people or objects in an image or video and connects them into a skeletal structure. It solves the problem of understanding body position and movement without specialized motion-capture hardware, with the main trade-off between accuracy and speed: richer models and 3D estimation capture more detail but cost more compute and data.

Read full answer

Applies tocomputer vision concepts (examples in Python 3.12)

How It Works

  • Keypoint representation: The model predicts (x, y) coordinates and a confidence score for each keypoint (for example, 17 COCO body joints: nose, shoulders, elbows, wrists, hips, knees, ankles). 3D pose estimation additionally predicts depth z.

  • Single-person vs. multi-person: Single-person methods assume one subject. Multi-person methods must detect and separate individuals.

  • Top-down: A person detector finds bounding boxes, then a pose network estimates keypoints per crop. It is usually accurate but scales roughly with the number of people.

  • Bottom-up: The model detects all keypoints in the image, then groups them into people (for example, with part affinity fields). It scales better to crowds but grouping can fail in occlusion.

  • Modern approaches: CNN-based methods (HRNet-style high-resolution representations) and vision transformers (ViT-based pose models such as ViTPose) are common. For promptable or general segmentation context, SAM is used separately; pose models remain specialized for keypoints.

Applications

Domain Use Output Used
Healthcare & rehab Gait analysis, fall detection, exercise form feedback Joint angles, trajectories
Sports analytics Technique assessment, performance tracking Keypoint sequences over time
Human-computer interaction / AR Gesture control, avatar animation, virtual try-on Real-time 2D/3D keypoints
Robotics & safety Human-aware navigation, workplace ergonomics monitoring Person pose and proximity
Media & animation Markerless motion capture for films and games 3D skeletal motion

Evaluation Metrics

  • OKS (Object Keypoint Similarity): Measures keypoint distance normalized by object scale and per-keypoint visibility, analogous to IoU for keypoints.

  • mAP@OKS: Mean Average Precision computed over OKS thresholds; the standard detection-style metric for multi-person pose.

  • PCK (Percentage of Correct Keypoints): Fraction of predicted keypoints within a distance threshold of ground truth, often used in single-person settings.

  • Accuracy/top-k is less common for pose than for image classification because the output is spatial coordinates, not a single label.

Common Pitfalls

  • Occlusion and truncation: Hidden joints produce low-confidence or misplaced keypoints; filter by confidence and handle missing points explicitly.

  • Wrong coordinate space: Models expect specific input sizes/normalization; resizing without mapping keypoints back to original image coordinates breaks downstream use.

  • Privacy and bias: Face/body analysis can enable surveillance or perform unevenly across demographics; evaluate fairness, obtain consent, and minimize retained data.

  • Latency assumptions: Top-down pipelines can slow down with many people; for real-time or edge use, consider lighter models, quantization, or ONNX export.

  • Interview follow-ups: Difference between top-down and bottom-up, 2D vs. 3D pose, how OKS differs from IoU, and how to track poses across video frames.

34

How does optical flow reveal motion in video sequences?

Optical flow estimates the apparent motion of pixels between consecutive video frames, producing a 2D velocity field that shows where image patterns move over time. It reveals motion for tasks such as tracking, action analysis, video stabilization, and motion segmentation, but it measures image motion rather than object identity. Its main trade-off is accuracy versus robustness: classical methods are efficient but rely on assumptions that break under occlusion, lighting changes, blur, or large motion.

Read full answer

Applies tocomputer vision concepts (all versions)

How Optical Flow Works

Optical flow assigns each pixel a velocity vector:

flow[x, y] = (u, v)

  • u: horizontal displacement in pixels between frames.

  • v: vertical displacement in pixels between frames.

  • Magnitude: sqrt(u*u + v*v), the estimated motion speed in image space.

  • Direction: atan2(v, u), the direction of motion.

Common approaches include:

Approach How it works Strengths Weaknesses When to use
Lucas-Kanade Tracks selected feature points under local motion assumptions Fast, simple, good for sparse tracking Fails with large motion or poor texture Tracking corners/features
Horn-Schunck Estimates dense flow with a global smoothness constraint Full motion field Can over-smooth boundaries Dense motion with smooth scenes
Farneback Uses polynomial image approximation in a coarse-to-fine scheme Practical dense flow Sensitive to noise, blur, and lighting changes Classic dense-flow baselines
Learned flow models Predict flow from learned image features Better on complex motion in many settings Higher compute and data/training needs Modern video pipelines with sufficient resources

What It Reveals

  • Object movement: Vectors indicate where regions shift from one frame to the next.

  • Camera motion: Global flow patterns can reveal panning, zooming, or shaking.

  • Motion boundaries: Sudden changes in flow direction or magnitude can separate moving objects from background.

  • Temporal cues: Consistent motion patterns support action recognition, interpolation, and stabilization.

  • Not semantic understanding: Flow shows motion, but identifying a person, car, or gesture usually requires a separate recognition model.

Key Assumptions and Limitations

  • Brightness constancy: A point is assumed to keep similar appearance over time; shadows and exposure changes violate this.

  • Small or gradual motion: Large displacements between frames can cause wrong correspondences unless handled by pyramids or learned models.

  • Spatial coherence: Neighboring pixels are often assumed to move similarly; this can blur boundaries.

  • Visible texture: Flat, textureless regions provide weak evidence for matching.

  • Occlusion: Pixels that appear or disappear have no true frame-to-frame correspondence.

Common Pitfalls

  • Confusing pixel motion with physical speed: Flow is measured in pixels per frame; real-world speed needs frame rate and camera calibration.

  • Ignoring camera motion: A moving camera can make stationary objects appear to move.

  • Treating flow as identity: Motion vectors do not label what an object is.

  • Using dense flow when sparse is enough: Sparse tracking is often faster when only selected points matter.

  • Interview follow-ups: Be ready to compare sparse vs. dense flow, explain brightness constancy, discuss occlusion and large motion, and describe how optical flow differs from object detection or tracking-by-detection.

35

How do CNNs recognize human activities in video?

Convolutional neural networks (CNNs) recognize human activity by extracting spatial and temporal patterns from video frames and mapping them to labels such as walking, running, or sitting. A 3D CNN learns motion directly across frames, while a 2D CNN encodes each frame separately and then aggregates the features over time. The main trade-off is that stronger temporal modeling improves action discrimination, but video models generally require more computation and training data than image models.

Read full answer

Applies toPyTorch 2.x and scikit-learn 1.5+ (Python 3.12)

How It Works

A typical video-classification pipeline:

  1. Decode the video into a sequence of frames.

  2. Resize and normalize the frames using the mean and standard deviation expected by the selected model.

  3. Encode spatial features with convolutional filters that detect objects, body parts, textures, and shapes.

  4. Model temporal relationships to determine how those features change over time.

  5. Classify the sequence into one activity label using a final linear or softmax layer.

For a 3D CNN, the input commonly has shape N × C × T × H × W, where N is batch size, C is channels, T is frames, and H and W are height and width. The output has one score per activity class; the highest-scoring class becomes the prediction.

Architecture Options

Approach How it works Strengths Weaknesses Typical use
3D CNN Convolution kernels move across space and time Learns motion patterns directly High compute and memory cost Moderate-sized datasets and real-time applications
2D CNN plus temporal head A pretrained image CNN encodes frames; pooling, an RNN, or a Transformer combines them Reuses image backbones such as ResNet or EfficientNet Temporal patterns must be learned separately Small datasets and transfer learning
Two-stream network Processes RGB frames and optical flow in separate streams Captures appearance and motion explicitly More complex preprocessing and training Motion-sensitive activity recognition
Video Transformer Attention models relationships between frame or patch tokens Captures long-range temporal dependencies Usually needs substantial pretraining and compute Large-scale video datasets

For small datasets, transfer learning with a pretrained 2D backbone and temporal aggregation is usually the best starting point.

Training and Evaluation

  • Split before training: Split by subject, recording session, or time period rather than randomly splitting individual clips, which can leak near-duplicate frames across sets.

  • Use grouped cross-validation: Prefer StratifiedGroupKFold when subjects are available. Use StratifiedKFold only when samples are independent.

  • Augment training videos: Random crops, color jitter, frame dropping, temporal jitter, and label-safe flips improve robustness.

  • Use the checkpoint’s normalization: Pretrained ResNet, EfficientNet, ViT, and video models have specific input normalization requirements.

  • Evaluate appropriately: Report accuracy or top-k accuracy, macro F1, per-class precision and recall, and a confusion matrix. For imbalanced activities, include PR-AUC and ROC-AUC because accuracy can hide poor minority-class performance.

  • Track the subject split: A high random-split score may not represent performance on previously unseen people, locations, or cameras.

Code Example

This self-contained example trains a small 3D CNN on synthetic videos representing stationary, horizontal, and vertical movement.

import numpy as np
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report

SEED = 42
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.set_num_threads(1)

NUM_VIDEOS = 120
NUM_FRAMES = 8
HEIGHT = WIDTH = 32
CLASS_NAMES = ["stationary", "horizontal", "vertical"]


rng = np.random.default_rng(SEED)


def make_video(activity: int) -> np.ndarray:
    """Create an 8-frame grayscale video (C x T x H x W) with a moving square."""
    video = np.zeros((1, NUM_FRAMES, HEIGHT, WIDTH), dtype=np.float32)
    yy, xx = np.mgrid[:HEIGHT, :WIDTH]

    for frame in range(NUM_FRAMES):
        if activity == 0:
            center_x, center_y = 16, 16
        elif activity == 1:
            center_x = 5 + (22 * frame // (NUM_FRAMES - 1))
            center_y = 16
        else:
            center_x = 16
            center_y = 5 + (22 * frame // (NUM_FRAMES - 1))

        mask = (np.abs(xx - center_x) <= 3) & (np.abs(yy - center_y) <= 3)
        video[0, frame] = mask * 255.0

    # Shared generator: every clip gets different noise, so no duplicates.
    video += rng.normal(0, 8, video.shape)
    return (np.clip(video, 0, 255) / 255.0).astype(np.float32)


videos = np.stack([make_video(activity) for activity in range(3)
                   for _ in range(NUM_VIDEOS // 3)])
labels = np.repeat(np.arange(3), NUM_VIDEOS // 3).astype(np.int64)

# Split before training; normalization above uses fixed constants only.
indices = np.arange(len(labels))
train_idx, test_idx = train_test_split(
    indices,
    test_size=0.25,
    stratify=labels,
    random_state=SEED,
)

train_data = TensorDataset(
    torch.from_numpy(videos[train_idx]),
    torch.from_numpy(labels[train_idx]),
)
test_data = TensorDataset(
    torch.from_numpy(videos[test_idx]),
    torch.from_numpy(labels[test_idx]),
)

train_loader = DataLoader(train_data, batch_size=16, shuffle=True)
test_loader = DataLoader(test_data, batch_size=16)


class ActivityCNN(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv3d(1, 8, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=(1, 2, 2)),
            nn.Conv3d(8, 16, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=(2, 2, 2)),
            nn.AdaptiveAvgPool3d((1, 1, 1)),
        )
        self.classifier = nn.Linear(16, 3)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.features(x)
        return self.classifier(torch.flatten(x, 1))


model = ActivityCNN()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

for epoch in range(4):
    model.train()
    for inputs, targets in train_loader:
        optimizer.zero_grad()
        loss = criterion(model(inputs), targets)
        loss.backward()
        optimizer.step()

model.eval()
predictions = []
true_labels = []

with torch.no_grad():
    for inputs, targets in test_loader:
        predictions.extend(model(inputs).argmax(dim=1).tolist())
        true_labels.extend(targets.tolist())

print(f"Accuracy: {accuracy_score(true_labels, predictions):.3f}")
print(classification_report(
    true_labels,
    predictions,
    target_names=CLASS_NAMES,
    zero_division=0,
))

Deployment Considerations

  • Latency: Reduce frame count, resolution, or model depth when processing many streams concurrently.

  • Model size: Quantization can reduce memory usage and improve CPU or edge-device performance, with a trade-off in accuracy.

  • Export: ONNX Runtime and TensorRT can accelerate supported PyTorch models in production.

  • Hardware: 3D CNNs benefit substantially from GPUs; lightweight 2D CNNs may be more practical for edge devices.

  • Privacy and bias: Surveillance and face-related activity recognition require consent, access controls, retention limits, demographic testing, and human review for consequential decisions.

Common Pitfalls

  • Frame leakage: Adjacent clips from the same recording can appear in both training and test sets.

  • Ignoring subjects: Random clip splitting can make performance look better than generalization to new people.

  • Wrong augmentation: Horizontal flips may invalidate activities with a strong left-right direction.

  • Wrong metrics: Overall accuracy may conceal failures on rare activities; inspect per-class recall and the confusion matrix.

  • Excessive model size: A large video model may overfit a small dataset and fail to meet deployment latency requirements.

  • Expected follow-ups: Interviewers commonly ask which temporal architecture you would choose, how you would prevent leakage, how you would handle class imbalance, and how you would optimize the model for edge deployment.

36

How do you implement edge detection with the Canny filter in Python?

The Canny filter detects intensity transitions in an image, producing a binary edge map whose nonzero pixels represent detected boundaries. It improves on simple gradient methods through noise reduction, gradient-direction tracking, non-maximum suppression, and hysteresis thresholding, with the main trade-off being sensitivity versus false edges and lost detail.

Read full answer

Applies toOpenCV 4.x (Python 3.12)

How It Works

Canny edge detection follows these stages:

  1. Grayscale conversion: Color is reduced to intensity because the algorithm operates on single-channel brightness values.

  2. Noise reduction: A Gaussian blur suppresses small fluctuations that could otherwise be mistaken for edges.

  3. Gradient calculation: The algorithm estimates the direction and magnitude of intensity changes using derivatives such as Sobel.

  4. Non-maximum suppression: Pixels that are not the strongest gradient along their edge direction are removed, thinning the edge map.

  5. Hysteresis thresholding: threshold1 is the lower threshold and threshold2 is the higher threshold. Strong edges above the higher threshold seed the result, while weak edges above the lower threshold are retained only when connected to a strong edge.

The two thresholds control the trade-off:

  • Lower thresholds detect more edges but also produce more noise.

  • Higher thresholds retain only prominent, continuous boundaries.

  • A ratio of roughly 2:1 or 3:1 between the higher and lower thresholds is a common starting point.

Code Example

import cv2
import numpy as np


def detect_edges(image: np.ndarray) -> np.ndarray:
    """Return a binary Canny edge map for a single-channel image."""
    if image.ndim != 2:
        raise ValueError("image must be a single-channel grayscale array")

    blurred = cv2.GaussianBlur(image, (5, 5), sigmaX=1.5)
    return cv2.Canny(blurred, threshold1=50, threshold2=150)


def main() -> None:
    # Create a deterministic synthetic image containing a bright rectangle.
    image = np.zeros((240, 320), dtype=np.uint8)
    image[60:180, 80:240] = 220

    edges = detect_edges(image)

    print("Input shape:", image.shape)
    print("Edge map shape:", edges.shape)
    print("Detected edge pixels:", np.count_nonzero(edges))
    print("Edge values:", np.unique(edges).tolist())


if __name__ == "__main__":
    main()

The resulting edges array has the same height and width as the input. Its values are 0 for non-edge pixels and 255 for detected edge pixels.

Practical Guidance

  • Use grayscale input: cv2.imread(..., cv2.IMREAD_GRAYSCALE) loads an image directly as one channel. If loading color with cv2.IMREAD_COLOR, convert it using cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) because OpenCV uses BGR, not RGB, channel order.

  • Reduce noise first: Blur can prevent texture and sensor noise from producing fragmented edges, although excessive blurring can remove thin structures.

  • Tune thresholds on sample images: Start near 50 and 150, then lower them for faint boundaries or raise them for cleaner but less sensitive results.

  • Use edges as features: Canny output can support contour extraction, segmentation, document scanning, or as an input channel to another model.

  • Preserve input dimensions: Canny returns a single-channel image matching the input size, so it can be overlaid on the original image for visualization.

Common Pitfalls

  • Applying Canny directly to noisy images creates many false edges; blur first.

  • Treating the lower threshold as the minimum gradient required for every edge ignores Canny’s hysteresis behavior.

  • Using identical low and high thresholds weakens hysteresis and can produce discontinuous results.

  • Forcing Canny onto every computer-vision task is inappropriate when a learned segmentation model can infer semantic boundaries better.

  • When the edges look too thick, adjust blur and thresholds; when they look faint, avoid excessive smoothing or lower the thresholds.

37

How do I fine-tune a pretrained CNN for image classification in TensorFlow?

Transfer learning reuses a pretrained CNN, replaces its original classification head, and trains that head—or selected layers—on a target dataset. It provides strong results with limited labeled data and reduces training cost, but performance can suffer when the target images differ substantially from the pretraining domain.

Read full answer

Applies toTensorFlow 2.16+ with Keras 3, Python 3.12

How It Works

A common workflow is:

  1. Load a pretrained backbone such as MobileNetV2 with include_top=False.

  2. Freeze the backbone so its learned features are not overwritten.

  3. Add a global average pooling layer and a new classification head.

  4. Train on the training split with validation-based monitoring.

  5. Unfreeze the upper backbone layers with a smaller learning rate if more data is available.

Classification predicts one label and associated probabilities. Related computer-vision tasks produce different outputs:

Task Common approaches Output
Image classification ResNet, EfficientNet, ViT Class label and scores
Object detection Faster R-CNN, YOLO, DETR Bounding boxes, labels, scores
Semantic segmentation U-Net, DeepLab Label for every pixel
Instance segmentation Mask R-CNN, promptable SAM Separate mask and label per object
Pose estimation HRNet, ViTPose Joint or facial keypoints
Tracking DeepSORT, ByteTrack Object identities and trajectories
OCR CRNN, Transformer OCR Extracted text
Image generation Diffusion models, GANs Generated pixels
Image-text matching CLIP Shared embedding vectors

Evaluation Strategy

  • Classification: Use top-1 accuracy and, for many classes, top-5 accuracy. For imbalanced data, also report macro precision, recall, F1, and PR-AUC because accuracy can hide poor minority-class performance.

  • Detection: Evaluate localization with IoU and summarize detections with mAP.

  • Segmentation: Use mIoU and Dice score, accounting for both class coverage and overlap.

  • Hold out a test set that is used only after model and hyperparameter selection. Never tune decisions repeatedly against test results.

Code Example

This example fine-tunes MobileNetV2 on a small synthetic color-classification dataset. The first run downloads the ImageNet weights, which should be cached in repeatable environments.

import numpy as np
import tensorflow as tf
from sklearn.model_selection import train_test_split

SEED = 42
tf.keras.utils.set_random_seed(SEED)
tf.config.experimental.enable_op_determinism()

# Synthetic RGB images: each class has a distinct dominant color.
def create_synthetic_dataset(per_class=80, image_size=64):
    rng = np.random.default_rng(SEED)
    palette = np.array([
        [1.0, 0.1, 0.1], [0.1, 1.0, 0.1], [0.1, 0.1, 1.0],
        [1.0, 1.0, 0.1], [1.0, 0.1, 1.0], [0.1, 1.0, 1.0],
        [0.8, 0.4, 0.1], [0.4, 0.1, 0.8], [0.1, 0.8, 0.4],
        [0.8, 0.8, 0.8],
    ], dtype=np.float32)

    images, labels = [], []

    for class_id, color in enumerate(palette):
        for _ in range(per_class):
            image = rng.normal(0.0, 0.04, (image_size, image_size, 3))
            image += color * rng.uniform(0.30, 0.55)

            # Add a randomly positioned block so images are not identical.
            height, width = 24, 24
            top = rng.integers(0, image_size - height + 1)
            left = rng.integers(0, image_size - width + 1)
            image[top:top + height, left:left + width] += color * 0.20

            images.append(np.clip(image, 0.0, 1.0).astype(np.float32))
            labels.append(class_id)

    return np.stack(images), np.asarray(labels, dtype=np.int64)


images, labels = create_synthetic_dataset()

# Split before training; no test or validation information is used for fitting.
train_x, remaining_x, train_y, remaining_y = train_test_split(
    images, labels, test_size=0.30, random_state=SEED, stratify=labels
)
val_x, test_x, val_y, test_y = train_test_split(
    remaining_x, remaining_y, test_size=0.50,
    random_state=SEED, stratify=remaining_y
)

# Images are in [0, 1]; MobileNetV2's ImageNet preprocessing expects [-1, 1].
train_x = train_x * 2.0 - 1.0
val_x = val_x * 2.0 - 1.0
test_x = test_x * 2.0 - 1.0

# Augment training data only.
def augment(image):
    image = tf.image.random_flip_left_right(image)
    return tf.image.random_brightness(image, max_delta=0.05)

train_ds = (
    tf.data.Dataset.from_tensor_slices((train_x, train_y))
    .shuffle(len(train_x), seed=SEED, reshuffle_each_iteration=True)
    .map(lambda image, label: (augment(image), label))
    .batch(32)
    .prefetch(tf.data.AUTOTUNE)
)
val_ds = tf.data.Dataset.from_tensor_slices((val_x, val_y)).batch(32)
test_ds = tf.data.Dataset.from_tensor_slices((test_x, test_y)).batch(32)

# Load and initially freeze the pretrained CNN.
base_model = tf.keras.applications.MobileNetV2(
    input_shape=(64, 64, 3),
    include_top=False,
    weights="imagenet",
)
base_model.trainable = False

inputs = tf.keras.Input(shape=(64, 64, 3))
features = base_model(inputs, training=False)
features = tf.keras.layers.GlobalAveragePooling2D()(features)
features = tf.keras.layers.Dropout(0.2)(features)
outputs = tf.keras.layers.Dense(10, activation="softmax")(features)

model = tf.keras.Model(inputs, outputs)
model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"],
)

model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=5,
    verbose=2,
)

test_loss, test_accuracy = model.evaluate(test_ds, verbose=0)
print(f"Test accuracy: {test_accuracy:.3f}")

Best Practices

  • Start frozen: It reduces overfitting and compute requirements. Unfreeze only the upper layers after establishing a baseline.

  • Normalize correctly: Use the mean and standard deviation required by the selected checkpoint; avoid applying normalization twice.

  • Augment appropriately: Use flips when orientation does not change the label. More aggressive color transformations may remove medically or diagnostically relevant information.

  • Manage class imbalance: Consider weighted loss, resampling, or macro metrics rather than optimizing accuracy alone.

  • Control leakage: Fit learned preprocessing only on training data and do not repeatedly evaluate on the test set.

  • Plan deployment: Compare latency, model size, quantization, and ONNX export against the target device. Face-related systems also require privacy controls and bias testing.

Common Pitfalls

  • OpenCV loads images in BGR order, while most pretrained models expect RGB.

  • A synthetic example demonstrates the API but does not establish real-world accuracy.

  • Using the validation set for many tuning decisions gradually turns it into a training set.

  • Unfreezing the full backbone too early can overfit a small dataset.

  • Reporting only accuracy can conceal severe minority-class failures.

  • Forgetting to cache or provision pretrained weights makes CI and offline deployments unreliable.

38

How can a function automatically crop images around the main object?

Object-centered cropping locates the most prominent foreground object, estimates its bounding region, and extracts an image patch centered on that region. The implementation below uses OpenCV GrabCut and connected-component analysis, requiring no labels or downloaded weights. Its main trade-off is simplicity versus semantic accuracy: it works well for visually distinct, central objects, while diverse scenes require a learned detector or segmenter.

Read full answer

Applies toOpenCV 4.x (Python 3.12, NumPy 2.x)

How It Works

  1. Initialize GrabCut with a central rectangle, establishing the probable foreground region.

  2. Extract foreground pixels from the resulting segmentation mask.

  3. Apply morphological operations to remove small noise and fill holes.

  4. Select the largest connected component as the main object.

  5. Compute its bounding box and center.

  6. Extract a square region centered on that point, add context, and resize it to the requested output.

The returned bounding box uses (x, y, width, height) coordinates in the original image.

Object Localization Alternatives

A production cropper can use a learned model when “main object” means more than visual prominence.

Task Output Typical approaches Common metrics
Image classification Image-level label ResNet, EfficientNet, ViT Accuracy, top-k accuracy
Object detection Boxes, labels, confidence scores Faster R-CNN, YOLO family, DETR IoU, mAP
Semantic segmentation One class per pixel U-Net, DeepLab mIoU, Dice
Instance segmentation Pixel masks and instance identities Mask R-CNN, SAM mIoU, Dice
Keypoint/pose estimation Keypoint coordinates HRNet, ViTPose OKS, PCK
Tracking Object IDs and trajectories ByteTrack, DeepSORT IDF1, HOTA
OCR Text and text-region geometry CRNN, Transformer OCR CER, WER
Image generation Generated images Diffusion models, GANs FID

CLIP image-text embeddings can also identify a crop by matching it against a textual description, although it does not directly produce a precise pixel mask.

Code Example

import cv2
import numpy as np


def crop_to_main_object(
    image: np.ndarray,
    output_size: int = 224,
    context: float = 0.25,
) -> tuple[np.ndarray, tuple[int, int, int, int]]:
    """Return an object-centered crop and the main object's original bbox."""
    if not isinstance(image, np.ndarray):
        raise TypeError("image must be a NumPy array")
    if image.ndim != 3 or image.shape[2] != 3:
        raise ValueError("image must have shape (height, width, 3)")
    if image.dtype != np.uint8:
        raise TypeError("image must use dtype uint8")
    if output_size <= 0:
        raise ValueError("output_size must be positive")
    if not 0.0 <= context <= 1.0:
        raise ValueError("context must be between 0 and 1")

    height, width = image.shape[:2]

    # Assume the main object is near the image center.
    roi_width = max(2, int(width * 0.60))
    roi_height = max(2, int(height * 0.70))
    roi_x = (width - roi_width) // 2
    roi_y = (height - roi_height) // 2
    roi_x2 = min(width, roi_x + roi_width)
    roi_y2 = min(height, roi_y + roi_height)

    mask = np.zeros((height, width), dtype=np.uint8)
    background_model = np.zeros((1, 65), dtype=np.float64)
    foreground_model = np.zeros((1, 65), dtype=np.float64)

    # GrabCut expects the rectangle as (x, y, width, height).
    rectangle = (roi_x, roi_y, roi_x2 - roi_x, roi_y2 - roi_y)
    cv2.grabCut(
        image,
        mask,
        rectangle,
        background_model,
        foreground_model,
        5,
        cv2.GC_INIT_WITH_RECT,
    )

    foreground = np.where(
        (mask == cv2.GC_FGD) | (mask == cv2.GC_PR_FGD),
        1,
        0,
    ).astype(np.uint8)

    # Clean the mask before identifying the main connected component.
    kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
    foreground = cv2.morphologyEx(
        foreground, cv2.MORPH_CLOSE, kernel
    )
    foreground = cv2.morphologyEx(
        foreground, cv2.MORPH_OPEN, kernel
    )

    count, _, stats, _ = cv2.connectedComponentsWithStats(
        foreground, connectivity=8
    )
    if count <= 1:
        raise ValueError("No foreground object was detected")

    component = 1 + int(np.argmax(stats[1:, cv2.CC_STAT_AREA]))
    x, y, box_width, box_height = (
        int(stats[component, column]) for column in range(4)
    )
    bbox = (x, y, box_width, box_height)
    center_x = x + box_width // 2
    center_y = y + box_height // 2

    # Build a square crop large enough to retain context around the object.
    crop_side = max(box_width, box_height)
    crop_side = int(round(crop_side * (1.0 + 2.0 * context)))
    crop_side = max(crop_side, box_width, box_height, 2)
    if crop_side % 2:
        crop_side += 1

    left = center_x - crop_side // 2
    top = center_y - crop_side // 2
    right = left + crop_side
    bottom = top + crop_side

    source_left = max(0, left)
    source_top = max(0, top)
    source_right = min(width, right)
    source_bottom = min(height, bottom)

    patch = image[
        source_top:source_bottom,
        source_left:source_right,
    ].copy()

    # Preserve the requested center when the crop extends beyond an edge.
    patch = cv2.copyMakeBorder(
        patch,
        source_top - top,
        bottom - source_bottom,
        source_left - left,
        right - source_right,
        cv2.BORDER_REPLICATE,
    )

    interpolation = (
        cv2.INTER_AREA
        if crop_side > output_size
        else cv2.INTER_CUBIC
    )
    cropped = cv2.resize(
        patch,
        (output_size, output_size),
        interpolation=interpolation,
    )
    return cropped, bbox


def make_synthetic_scene() -> np.ndarray:
    rng = np.random.default_rng(42)
    image = np.full((240, 320, 3), (60, 70, 80), dtype=np.uint8)

    # Add reproducible low-level noise.
    noise = rng.normal(0, 4, image.shape)
    image = np.clip(
        image.astype(np.int16) + noise, 0, 255
    ).astype(np.uint8)

    # Two overlapping shapes form one visually prominent object.
    cv2.rectangle(image, (115, 70), (205, 170), (20, 180, 240), -1)
    cv2.circle(image, (160, 120), 38, (240, 90, 30), -1)
    return image


scene = make_synthetic_scene()
cropped, bbox = crop_to_main_object(scene)

print("Original shape:", scene.shape)       # (240, 320, 3)
print("Main-object bbox:", bbox)
print("Cropped shape:", cropped.shape)       # (224, 224, 3)

Evaluation and Deployment

  • Measure crop localization using the IoU between the crop’s covered object region and a ground-truth mask or bounding box.

  • Also measure center error and the downstream model’s accuracy after cropping.

  • Avoid accuracy for severely imbalanced datasets; use precision, recall, F1, PR-AUC, and ROC-AUC as appropriate.

  • OpenCV loads images as BGR, while most pretrained vision models expect RGB. Convert with cv2.cvtColor(image, cv2.COLOR_BGR2RGB).

  • For learned localization, use the model’s prescribed normalization and transfer learning when training data is limited. Augment with flips, crops, and color jitter during training.

  • GrabCut is CPU-oriented and lightweight. Learned detectors may benefit from GPUs but introduce model-size and latency costs.

  • For edge deployment, consider ONNX export, quantization, and smaller backbones.

  • Face-related applications require particular attention to privacy, demographic bias, and false identity attribution.

Common Pitfalls

  • Incorrect central-object assumption: GrabCut may miss an off-center object; use detector or segmentation output instead.

  • Poor foreground separation: Low contrast, shadows, and cluttered backgrounds can corrupt the mask.

  • Wrong color order: Mixing BGR and RGB silently degrades model predictions.

  • Excessive context: A large context ratio can retain distracting background; too little can crop important object parts.

  • Validation overfitting: Do not repeatedly tune crop thresholds against the test set.

  • Common follow-up questions: How would you select the most relevant object from multiple detections, preserve aspect ratio, evaluate crop quality, or replace GrabCut with YOLO, Mask R-CNN, or SAM?

39

How do you segment an image into superpixels with SLIC in Python?

SLIC (Simple Linear Iterative Clustering) groups nearby pixels with similar color into compact, roughly uniform regions called superpixels, reducing millions of pixels to a few hundred meaningful units for tasks like segmentation, detection, and feature extraction. It solves the problem of expensive per-pixel processing by providing a perceptually aligned over-segmentation. Its main trade-off is that n_segments and compactness control the balance between color adherence and spatial regularity: too few or overly compact superpixels can miss object boundaries.

Read full answer

Applies toscikit-image 0.24+ (Python 3.12, NumPy 2.x)

How It Works

  • Initialization: SLIC places n_segments cluster centers on a regular grid over the image.

  • Color-space clustering: For a color image, it converts to CIELAB (L, a, b) and clusters in a 5D space combining color and position (L, a, b, x, y).

  • Local search: Each center is refined by assigning pixels within a limited neighborhood (about 2S x 2S, where S is the grid step) to the nearest center using a combined distance, which keeps complexity low.

  • Iteration and cleanup: Assignments and centers are updated for max_num_iter iterations, then disconnected or small regions are merged to enforce connectivity.

  • Time complexity: Roughly O(N) per iteration over N pixels because of the limited search region, with modest memory proportional to the image and label map.

Key Parameters and Practical Guidance

Parameter Effect Practical tip
n_segments Requested number of superpixels (approximate) Start near 100-500 for typical images; increase for fine boundaries.
compactness Weight of spatial distance vs. color distance Higher values give squarer, more regular regions; lower values follow color edges better.
sigma Gaussian smoothing before clustering Use a small value (e.g., 1) for noisy images; keep 0 to preserve edges.
channel_axis Axis of color channels Use channel_axis=-1 for (H, W, C) images; pass None for grayscale.
start_label First label value Set explicitly (e.g., 0) for predictable indexing.
  • Preprocessing: SLIC converts input with img_as_float, so pass floats in [0, 1] or standard integer images, and keep the range consistent across images; convert color spaces deliberately (OpenCV loads BGR, while most pipelines and SLIC examples assume RGB ordering).

  • Use case: Superpixels are an intermediate representation, not a final semantic segmentation; pair them with classifiers, graph cuts, or region merging.

Code Example

import numpy as np
from skimage.segmentation import slic, mark_boundaries

# Deterministic synthetic RGB image (H, W, C) in [0, 1]
image = np.zeros((200, 200, 3), dtype=np.float64)
image[:100, :100] = [1.0, 0.0, 0.0]   # red region
image[:100, 100:] = [0.0, 1.0, 0.0]   # green region
image[100:, :100] = [0.0, 0.0, 1.0]   # blue region
image[100:, 100:] = [1.0, 1.0, 0.0]   # yellow region

# Segment into approximately n_segments superpixels
segments = slic(
    image,
    n_segments=20,        # requested superpixel count (actual count may differ slightly)
    compactness=10.0,     # balance between color similarity and spatial proximity
    sigma=0,              # no pre-smoothing for this clean synthetic image
    start_label=0,        # labels start at 0
    channel_axis=-1,      # channels are in the last axis
)

# Overlay boundaries for inspection (returns an RGB image)
boundary_image = mark_boundaries(image, segments)

print(f"Image shape: {image.shape}")                 # Image shape: (200, 200, 3)
print(f"Segments shape: {segments.shape}")           # Segments shape: (200, 200)
print(f"Number of superpixels: {len(np.unique(segments))}")  # around 20
print(f"Boundary image shape: {boundary_image.shape}")        # Boundary image shape: (200, 200, 3)
print(f"Label range: {segments.min()} to {segments.max()}")

Common Pitfalls

  • Wrong channel axis: Omitting or mis-setting channel_axis can make SLIC treat channels as spatial dimensions or fail on (H, W, C) images; always set it explicitly for color images.

  • Assuming an exact count: n_segments is a request, not a guarantee; use np.unique(segments).size when the exact number matters.

  • BGR/RGB confusion: If the image came from OpenCV (cv2.imread), convert with cv2.cvtColor(img, cv2.COLOR_BGR2RGB) before SLIC so color distances are meaningful.

  • Over-smoothing or wrong scale: Large sigma or unexpected value ranges can blur boundaries; verify image.dtype and range.

  • Interview follow-ups: How does SLIC differ from k-means on pixels, why use CIELAB, how to choose compactness, and how superpixels compare to learned segmentation (U-Net, SAM) for final masks.

40

How do I detect facial landmarks with Dlib and OpenCV?

Facial landmark detection estimates the positions of important facial features such as the eyes, nose, mouth, and jaw. Dlib provides a pretrained 68-point landmark model, while OpenCV handles image loading, preprocessing, and visualization. The main trade-off is speed and simplicity versus reduced accuracy on profile faces, occlusions, or low-quality images.

Read full answer

Applies toDlib 19.24+ and OpenCV 4.x (Python 3.12)

How It Works

The implementation uses two separate pretrained models:

  1. Face detector: Dlib’s frontal-face detector locates one or more face bounding boxes.

  2. Shape predictor: For each detected rectangle, the 68-point shape predictor estimates landmark coordinates relative to the original image.

  3. Visualization: OpenCV draws the detected points and facial contours on a copy of the image.

The resulting landmarks have the following standard groups:

Landmark indices Facial region
0–16 Jawline
17–26 Eyebrows
27–35 Nose (bridge 27–30, lower nose 31–35)
36–47 Eyes (36–41 and 42–47)
48–59 Outer lips
60–67 Inner lips

Implementation Requirements

Install the dependencies:

python -m pip install dlib opencv-python

Dlib does not bundle the trained landmark predictor. Download the official shape_predictor_68_face_landmarks.dat model and decompress it if the downloaded file is shape_predictor_68_face_landmarks.dat.bz2.

The script expects:

  • An input image containing one or more detectable faces.

  • The path to shape_predictor_68_face_landmarks.dat.

  • A writable directory for the annotated output.

Code Example

import argparse
from pathlib import Path

import cv2
import dlib
import numpy as np


def detect_facial_landmarks(image_path: Path, predictor_path: Path):
    image_bgr = cv2.imread(str(image_path))
    if image_bgr is None:
        raise FileNotFoundError(f"Could not read image: {image_path}")

    if not predictor_path.is_file():
        raise FileNotFoundError(f"Landmark model not found: {predictor_path}")

    # Dlib's image functions expect RGB pixel ordering.
    image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)

    face_detector = dlib.get_frontal_face_detector()
    landmark_predictor = dlib.shape_predictor(str(predictor_path))

    # Upsample the image once so smaller faces can be detected.
    detections, _, _ = face_detector.run(image_rgb, 1)

    all_landmarks = []

    for face in detections:
        shape = landmark_predictor(image_rgb, face)
        points = np.asarray(
            [
                (shape.part(index).x, shape.part(index).y)
                for index in range(68)
            ],
            dtype=np.int32,
        )
        all_landmarks.append(points)

    annotated = image_bgr.copy()

    for face_number, points in enumerate(all_landmarks, start=1):
        # Draw the main facial contours.
        for region in (
            points[0:17],   # Jaw
            points[17:22],  # Right eyebrow
            points[22:27],  # Left eyebrow
            points[36:42],  # Right eye
            points[42:48],  # Left eye
            points[48:60],  # Outer mouth
            points[60:68],  # Inner mouth
        ):
            cv2.polylines(
                annotated,
                [region],
                isClosed=False,
                color=(0, 255, 0),
                thickness=2,
                lineType=cv2.LINE_AA,
            )

        # Draw every estimated landmark.
        for x, y in points:
            cv2.circle(annotated, (int(x), int(y)), 2, (0, 0, 255), -1)

        xs, ys = points[:, 0], points[:, 1]
        print(
            f"Face {face_number}: "
            f"x={xs.min()} to {xs.max()}, "
            f"y={ys.min()} to {ys.max()}"
        )

    output_path = image_path.with_name(
        f"{image_path.stem}_landmarks{image_path.suffix}"
    )
    if not cv2.imwrite(str(output_path), annotated):
        raise RuntimeError(f"Could not write output image: {output_path}")

    print(f"Detected faces: {len(all_landmarks)}")
    print(f"Saved annotated image: {output_path}")

    return annotated, all_landmarks


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Detect 68 facial landmarks with Dlib."
    )
    parser.add_argument("image", type=Path, help="Input image path")
    parser.add_argument(
        "predictor",
        type=Path,
        help="Path to shape_predictor_68_face_landmarks.dat",
    )
    args = parser.parse_args()

    detect_facial_landmarks(args.image, args.predictor)

Run it with:

python landmarks.py face.jpg shape_predictor_68_face_landmarks.dat

The printed number of faces and coordinate ranges depend on the input image.

Practical Considerations

  • Image quality: Blur, small faces, extreme lighting, and heavy occlusion reduce landmark accuracy.

  • Face orientation: The Dlib model is primarily intended for frontal or near-frontal faces.

  • Multiple faces: Each detected face receives its own independent array of 68 (x, y) coordinates.

  • Performance: Landmark prediction is relatively fast for a few faces, but can become a bottleneck for high-resolution images or real-time multi-person video.

  • Deployment: Consider smaller face detectors, image resizing, batching, or alternative lightweight landmark models when strict latency is required.

  • Preprocessing: OpenCV reads images as BGR, so convert them to RGB before passing them to Dlib.

  • Privacy and bias: Facial geometry is biometric information; obtain consent, restrict retention, and evaluate performance across demographic groups.

Common Pitfalls

  • Using BGR directly: Convert the image with cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB).

  • Forgetting the model file: dlib.shape_predictor cannot work without the trained .dat file.

  • Expecting segmentation masks: These points represent geometry only; they do not label individual pixels.

  • Ignoring detection failures: An empty result often means the face detector did not find a sufficiently frontal or visible face.

  • Confusing landmarks with exact anatomy: The predictions are statistical estimates and may shift because of expression, glasses, facial hair, or masks.

  • Measuring only detection accuracy: Evaluate landmark quality with mean Euclidean error or normalized landmark error when comparing models.

41

What are GANs and their role in computer vision?

Generative Adversarial Networks (GANs) are deep learning models that train two neural networks—a generator that creates synthetic images and a discriminator that tries to distinguish real images from generated ones—in a competitive game. They solve the problem of learning complex image distributions to produce realistic samples, but their main trade-off is powerful generative ability versus training instability and difficulty evaluating output quality.

Read full answer

Applies todeep learning concepts for computer vision (current practice; framework examples typically use PyTorch 2.x or TensorFlow 2.16+ with Python 3.12)

Key Objective

The classic GAN uses a minimax objective: min_G max_D V(D, G) = E_{x ~ p_data(x)}[log D(x)] + E_{z ~ p_z(z)}[log(1 - D(G(z)))] - G: the generator network that maps noise to an image. - D: the discriminator network that outputs a probability that an input is real. - V(D, G): the value (objective) function of the game. - x: a real image sampled from the true data distribution p_data(x). - z: random noise sampled from a prior distribution p_z(z) (often Gaussian or uniform). - G(z): the synthetic image produced by the generator from noise z. - D(x): the discriminator’s estimated probability that x comes from real data. - E[...]: the expectation (average) over the indicated distribution. - log D(x) and log(1 - D(G(z))): terms that reward D for correctly classifying real and fake images, while G aims to make D(G(z)) large.

How GANs Work

  • Adversarial training: D is updated to maximize V(D, G) (better detection), and G is updated to minimize it (more convincing fakes). Training alternates between these updates.

  • Architectures: Early convolutional GANs (DCGAN-style) use CNN backbones; image-to-image models such as Pix2Pix (paired data) and CycleGAN (unpaired data) use encoder-decoder or residual generators with patch-based discriminators.

  • Conditioning: Conditional GANs add labels, masks, or text to guide generation, enabling controlled image synthesis.

Roles in Computer Vision

Task / Role Output Typical GAN Approach Notes
Image generation Synthetic images Unconditional or conditional GANs Used for data synthesis and creative tools
Image-to-image translation Transformed image Pix2Pix, CycleGAN Style transfer, domain adaptation, sketch-to-photo
Super-resolution Higher-resolution image SR-oriented GANs Adds realistic texture beyond interpolation
Inpainting / editing Completed or edited image Context or mask-conditioned GANs Fills missing regions; supports semantic edits
Segmentation / augmentation support Masks or extra training images Paired translation or synthetic data pipelines Can supplement labeled data when used carefully

In current practice, diffusion-based models are also widely used for high-fidelity image generation and editing; GANs remain relevant where fast sampling or specific image-to-image setups are important.

Evaluation and Practical Concerns

  • Metrics: Quality is often assessed with Fréchet Inception Distance (FID) or Inception Score, alongside task-specific measures (e.g., mIoU/Dice when GAN outputs support segmentation). Human evaluation is still common because no single metric fully captures realism and diversity.

  • Preprocessing: Normalize images to the range expected by the model, apply augmentation (flips, crops, color jitter) when appropriate, and keep train/validation splits strict to avoid leakage.

  • Deployment: Consider latency, model size, quantization, and export formats (e.g., ONNX) for edge use. Face-related applications raise privacy and bias concerns and require careful data governance.

Common Pitfalls

  • Mode collapse: The generator produces limited varieties of images; monitor sample diversity and training curves.

  • Training instability: Loss values can oscillate; stabilize with careful learning rates, normalization choices, and architecture tuning.

  • Misleading evaluation: Low training loss does not guarantee good images; use held-out evaluation and appropriate metrics.

  • Data leakage in augmentation pipelines: Fit any learned preprocessing only on training data.

  • Interview follow-ups: How GANs differ from VAEs and diffusion models; why the discriminator is discarded at inference; how conditional GANs work; what FID measures and its limitations.

42

What is few-shot learning in computer vision?

Few-shot learning enables a computer vision model to recognize or localize new concepts from only a few labeled examples. It reduces the cost of collecting and annotating training data for rare objects, medical images, and specialized domains. Its main trade-off is better data efficiency versus increased sensitivity to biased or unrepresentative support examples.

Read full answer

Applies tomachine learning concepts in computer vision (all versions)

How It Works

Few-shot experiments are commonly organized as an N-way K-shot episode:

  • N-way: The model distinguishes N classes in the episode.

  • K-shot: Each class has K labeled support examples.

  • Support set: The small labeled set used for adaptation or comparison.

  • Query set: Separate examples used to evaluate the model.

The main strategies are:

  • Metric learning: An encoder maps images into embeddings. A prototypical network, for example, represents each class by the mean embedding of its support examples and assigns a query to the nearest prototype.

  • Transfer learning and fine-tuning: A pretrained model is adapted using the support examples. A frozen encoder with a new classifier head is less prone to overfitting than fine-tuning every layer.

  • Meta-learning: A model is trained across simulated episodes to learn an initialization or adaptation strategy that works on new tasks.

Query labels must not be used for adaptation, augmentation, or model selection, because doing so leaks evaluation information into training.

Common Approaches

Few-shot systems use task-specific models and pretrained foundation models:

  • CNN backbones: ResNet and EfficientNet provide effective visual features, particularly when the target domain resembles the pretraining data.

  • Vision transformers: ViT-based models capture global relationships and can perform well after careful fine-tuning.

  • CLIP: CLIP produces joint image-text embeddings. Zero-shot classification differs from few-shot classification, but labeled examples can train a lightweight classifier or guide prompt tuning.

  • Detectors: Faster R-CNN, the YOLO family, and DETR can be adapted for few-shot object detection.

  • Segmentation models: U-Net, Mask R-CNN, and promptable models such as SAM can support few-shot segmentation workflows, although their success depends on domain similarity and adaptation quality.

For small datasets, transfer learning with a pretrained backbone is usually the first approach to evaluate.

Tasks and Evaluation

Task Typical output Appropriate metrics
Image classification Class label or top-k probabilities Accuracy, top-k accuracy, F1; ROC-AUC or PR-AUC for imbalanced classes
Object detection Bounding boxes, labels, and confidence scores IoU and mAP
Semantic segmentation Class label for every pixel mIoU and Dice score
Instance segmentation Separate pixel-level mask for each instance Mask mIoU, Dice, and instance AP
Keypoint or pose estimation Joint, landmark, or body-keypoint coordinates PCK or normalized error
Tracking Object identities and trajectories across frames IDF1 and HOTA
OCR Detected and transcribed text CER or WER
Image generation An image conditioned on text, examples, or other guidance FID or other fidelity and diversity measures

Few-shot classification should be evaluated over many independently sampled episodes, reporting mean performance and variation across episodes or random seeds. Results should be compared with zero-shot and full-supervised baselines, while disclosing how much pretraining data each model received.

Practical Considerations

  • Preprocessing: Resize or crop images to the backbone’s expected input dimensions. Normalize using the pretrained model’s training mean and standard deviation rather than statistics computed from the small support set. When using OpenCV, convert images from BGR to RGB for models that expect RGB input.

  • Augmentation: Flips, crops, and color jitter can regularize adaptation, but they should be label-preserving. Avoid transformations that remove small or medically important objects or change clinically relevant color.

  • Overfitting control: Prefer a frozen encoder or partial fine-tuning, use a low learning rate and regularization, and preserve a validation set separate from the final query episodes.

  • Deployment: Evaluate latency, model size, quantization, ONNX export, and edge-device compatibility rather than relying only on benchmark accuracy. Assess privacy, demographic bias, and error severity in sensitive applications such as face analysis.

Key Challenges

  • Limited support data: A few images may omit variation in lighting, pose, scale, background, or appearance.

  • Within-class variation: Examples of one class may differ more from one another than from another class.

  • Domain shift: Generic pretraining features may transfer poorly to medical, industrial, scientific, or low-light imagery.

  • Overfitting and miscalibration: Fine-tuning can memorize the support set and produce overconfident errors.

  • Base-to-novel generalization: Meta-learning may perform well on classes resembling its training classes but poorly on genuinely novel categories.

  • Ambiguous or noisy labels: Fine-grained distinctions may require domain knowledge unavailable from a few examples.

  • Reproducibility: Episode sampling, random seeds, backbone selection, and augmentation can materially affect results.

Common Pitfalls

  • Reusing query examples during adaptation or hyperparameter selection.

  • Reporting one favorable episode instead of averages across many episodes.

  • Comparing methods without specifying pretraining data, support-set size, and augmentation.

  • Applying aggressive crops or color jitter that invalidate labels.

  • Fine-tuning the entire network without regularization or learning-rate control.

  • Evaluating an imbalanced task with accuracy alone.

  • Assuming a larger foundation model automatically solves domain-specific adaptation.

  • Ignoring deployment constraints, privacy, demographic bias, and error severity.

During an interview, a strong discussion contrasts few-shot learning with zero-shot and full-supervised learning, explains the support/query split, and identifies overfitting, domain shift, leakage, and appropriate evaluation as the central issues.

43

What is zero-shot learning for image recognition?

Zero-shot learning enables an image classifier to recognize categories for which it received no labeled training examples. It transfers knowledge from related, previously learned classes using semantic information such as attributes, descriptions, or text-image embeddings, trading precise task-specific adaptation for broader category coverage.

Read full answer

Applies toall versions

Core Idea

A conventional image classifier learns a decision boundary from labeled images of each category. A zero-shot classifier instead learns how visual features relate to semantic concepts and then applies that relationship to unseen categories.

For example, a model may learn the visual appearance of:

  • Seen classes: dog, cat, and horse

  • Semantic attributes: has fur, has four legs, and has a mane

Given attributes for an unseen class such as zebra, it can predict whether an image contains a zebra even though no labeled zebra images were used to train the classifier.

The main trade-off is coverage versus specialization. Zero-shot learning supports rapidly changing or long-tail categories without collecting examples for each one, but its predictions may be less reliable than those of a model trained specifically on the target classes.

Common Approaches

Approach How it works Strengths Weaknesses Typical complexity or scale
Attribute-based Predicts semantic attributes and matches them to class descriptions Interpretable; works with small datasets Attributes may not capture appearance accurately Small or medium curated datasets; matching cost grows with attribute and class counts
Word-vector methods Represents classes and visual features in a shared semantic space Simple transfer across datasets Older embeddings have weaker contextual understanding Traditional benchmarks with moderate-size class vocabularies
Vision-language models Compares image embeddings with text embeddings for class prompts Strong generalization and natural class definitions Can inherit pretraining biases and prompt sensitivity Large pretrained models; candidate matching is typically linear in the number of classes
Large multimodal models Uses broader visual-language knowledge and richer instructions Handles diverse concepts and tasks Higher compute cost; less predictable behavior Large models with substantial inference cost

CLIP is a prominent vision-language example. Its zero-shot classifier converts prompts for candidate class names into text embeddings, compares them with the image embedding, and selects the most similar class. Prompt templates and class descriptions can materially affect predictions, so they should be validated.

Across computer-vision tasks, classification outputs a label, object detection outputs boxes and labels, semantic segmentation outputs a label per pixel, and instance segmentation outputs a class and pixel mask for each object. Keypoint or pose estimation outputs keypoint locations, tracking assigns persistent track IDs, OCR outputs text, and image generation produces an image from a prompt. Representative families include ResNet, EfficientNet, and ViT for classification; Faster R-CNN, the YOLO family, and DETR for detection; U-Net and Mask R-CNN for segmentation; and SAM for promptable segmentation. These architectures are not inherently zero-shot; open-vocabulary variants or vision-language bridges provide the semantic mechanism for unseen categories.

Training and Inference

A typical zero-shot workflow has two stages:

  1. Learn relationships between images and semantic representations during training.

  2. At inference time, describe the candidate categories and compare them with the image representation.

Candidate labels do not need labeled images during training, but they still need semantic representations. Zero-shot learning does not mean that the model has no prior data: a pretrained vision-language model may have seen large amounts of image-text data during pretraining. The relevant distinction is that the target task or unseen category has no labeled downstream examples.

Evaluation

Evaluation requires separate seen-class and unseen-class test sets. For classification, use accuracy and top-k accuracy, plus per-class recall or macro-averaged accuracy when classes are imbalanced. For detection, use intersection over union (IoU) and mean average precision (mAP); for semantic segmentation, use mean intersection over union (mIoU) and Dice score.

Generalized zero-shot accuracy measures performance across seen and unseen classes together. Accuracy can be misleading for large candidate sets or imbalanced classes, so confusion matrices and per-class metrics are useful. Zero-shot results should also be compared with a supervised baseline trained on labeled target-class examples.

Advantages and Limitations

  • Advantages: No target-class image-labeling requirement; easy to add categories; useful for long-tail datasets and open-vocabulary recognition.

  • Limitations: Predictions depend on semantic-description quality; similar classes may be confused; prompt wording can change results.

  • Data leakage risk: Designing prompts after inspecting test images or their ground-truth annotations leaks evaluation information.

  • Benchmark limitations: Standard test sets may contain distribution gaps, annotation errors, and unrealistic class vocabularies that no longer represent deployments.

Common Pitfalls and Interview Tips

  • Do not describe zero-shot learning as learning without data; it learns without labeled examples of the target classes.

  • Evaluate unseen classes separately and check whether pretraining data overlaps with the evaluation dataset.

  • Convert inputs to RGB when required by the pretrained model and normalize them with that checkpoint’s mean and standard deviation.

  • For small labeled target datasets, transfer learning or linear probing is the default adaptation strategy; it often improves zero-shot results, but compare both approaches empirically.

  • During adaptation, use training-fold-only augmentation such as flips, crops, and color jitter where appropriate.

  • Keep prompts, preprocessing, and label definitions consistent between evaluation and deployment.

  • In production, consider hybrid retrieval, latency, model size, quantization, ONNX export, privacy, and demographic bias.

  • Typical follow-up questions ask how CLIP creates embeddings, why prompts matter, how unseen classes are evaluated, and when fine-tuning is preferable to zero-shot inference.

44

How does reinforcement learning apply to computer vision tasks?

Reinforcement learning (RL) trains an agent to choose image-dependent actions that maximize long-term reward. In computer vision, observations can be images, video frames, detections, or masks, while actions might move a camera, select a region, refine a segmentation mask, or edit a generated image. Its main trade-off is adaptability to sequential decisions versus high sample cost and sensitivity to reward design.

Read full answer

Applies toreinforcement learning and computer vision concepts (examples in Python 3.12)

How It Works

At each time step, the agent receives a state s_t, selects an action a_t, receives a reward r_(t+1), and transitions to s_(t+1). It learns a policy that maximizes the expected discounted return:

G_t = sum(k = 0 to infinity) gamma^k * r_(t+k+1)

  • t is the current time step.

  • r_(t+k+1) is the immediate reward received at step t+k+1, such as reaching a target or improving a mask.

  • gamma is the discount factor, controlling how strongly future rewards matter.

  • G_t is the total discounted reward the policy aims to maximize.

In vision settings, a CNN or vision transformer (such as ResNet, EfficientNet, or ViT) often encodes the image into features, and the RL policy learns from those representations. DQN is commonly used for discrete actions, while PPO is often used for continuous control such as robotic movement.

Common Vision Tasks

Task Standard output Typical approach and RL opportunity
Image classification Class label ResNet, EfficientNet, ViT; RL can select informative samples or test-time actions
Object detection Boxes, labels, scores Faster R-CNN, YOLO, DETR; RL can control scanning or viewpoint
Semantic segmentation Label per pixel U-Net, SAM; RL can refine masks through sequential edits
Instance segmentation Mask per object Mask R-CNN, SAM; RL can select regions and refine boundaries
Keypoint or pose estimation Joint or landmark coordinates CNN/ViT pose models; RL can guide viewpoint or correction
Tracking Object identities across frames DeepSORT, ByteTrack; RL can support target selection and association
OCR Text and confidence CRNN or Transformer OCR; RL can guide region selection and decoding
Image generation Generated pixels Diffusion models; preference optimization or RLHF can improve outputs

Evaluation and Deployment

Evaluate the learning process using held-out episodic return, success rate, and sample efficiency. Evaluate the resulting vision outputs with task-specific metrics: accuracy or top-k accuracy for classification, IoU and mAP for detection, and mIoU or Dice for segmentation. For imbalanced classification, precision, recall, F1, and PR-AUC are often more informative than accuracy.

For small datasets, transfer learning is the default: initialize an encoder from a pretrained ResNet, EfficientNet, ViT, or CLIP-based model, normalize using its documented mean and standard deviation, and use flips, crops, or color jitter where appropriate. Keep training, validation, and test environments separate; repeatedly tuning against test rewards causes test-set leakage.

Deployment also requires attention to inference latency, model size, quantization, ONNX export, and edge-device support. Face-related applications require particular care around privacy, consent, demographic bias, and dataset representativeness.

Common Pitfalls

  • Reward hacking: The agent exploits a proxy metric instead of completing the intended task.

  • Sparse or misleading rewards: Training stalls, or the agent optimizes an irrelevant shortcut.

  • Weak evaluation: A good training return does not guarantee good detection, segmentation, or generalization.

  • Environment overfitting: The policy memorizes fixed training scenes rather than learning transferable behavior.

  • Weak reproducibility: Record package versions, random seeds, action definitions, and environment configuration.

  • Uncontrolled deployment: Latency and memory requirements can make an accurate policy unsuitable for robots or edge devices.

45

What are Siamese networks and their computer vision applications?

A Siamese network uses two branches with shared weights to encode two related inputs and compare their embeddings. It solves comparison-oriented problems such as image matching, retrieval, duplicate detection, face verification, and change detection; its main trade-off is strong task-specific comparison learning versus dependence on representative pairs, hard negatives, and threshold calibration.

Read full answer

Applies tomachine learning concepts (version-independent)

How It Works

  • Shared encoder: Both inputs pass through the same architecture and parameters, producing embeddings in a common space. Common backbones include ResNet, EfficientNet, and vision transformers such as ViT.

  • Comparison measure: The embeddings may be compared with Euclidean distance, cosine similarity, or a learned metric. Cosine similarity measures directional similarity rather than Euclidean distance.

  • Training objective: A loss encourages matching examples to have similar embeddings and non-matching examples to be separated.

A common contrastive loss using Euclidean distance is:

L = mean(y * d^2 + (1 - y) * max(0, m - d)^2)

  • L is the average loss over the batch.

  • y is 1 for a matching pair and 0 for a non-matching pair.

  • d is the Euclidean distance between the two embeddings.

  • m is the margin required for non-matching pairs.

  • mean averages the loss across the batch.

Other valid objectives include triplet loss, binary cross-entropy over predicted similarity, and InfoNCE. Identical inputs normally receive identical embeddings in deterministic evaluation, but different augmentations or stochastic operations can produce different results.

Computer Vision Applications

Task Typical output Siamese-network use
Image classification Class label Compare a query with support images for one-shot or few-shot classification
Object detection Boxes, labels, and confidence scores Associate detections across related images or frames
Semantic segmentation Label per pixel Compare support and query scenes or identify changed regions
Instance segmentation Instance masks and labels Match individual objects across images
Keypoint and pose estimation Keypoint locations Compare poses, match body parts, or retrieve similar actions
Tracking Object identities across frames Match a target template with current-frame candidates
OCR Recognized text Match printed instances, document regions, or scanned text
Image generation Generated pixels or tokens Supply perceptual embeddings or consistency losses; the generator itself is usually not Siamese

Siamese components can be used within systems built around Faster R-CNN, the YOLO family, or DETR for detection, and around U-Net, Mask R-CNN, or promptable SAM for segmentation. Those architectures are not inherently Siamese; the defining feature is the shared-weight comparison branches. CLIP is also not a classic Siamese network because its image and text branches have different modalities and separate parameters.

Other established uses include image retrieval, duplicate and logo detection, visual place recognition, signature verification, medical-image comparison, and face verification. Face verification compares two faces, whereas identification searches a gallery for a known identity.

Training and Evaluation

  • Prepare pairs: Include genuine matches, subtle non-matches, and hard negatives. Easy negatives may allow high training loss without teaching useful distinctions.

  • Augment carefully: Use flips, crops, and color jitter only when the desired invariance permits them. For aligned comparisons, avoid transformations that erase meaningful differences.

  • Use transfer learning: For small datasets, initialize the shared encoder with pretrained weights and normalize inputs using that backbone’s required mean and standard deviation.

  • Prevent leakage: Split by subject or identity before training, checkpoint selection, gallery construction, or threshold tuning. Use identity-disjoint stratified folds for pair classification and time-ordered splits for temporal data.

  • Choose suitable metrics: Classification uses accuracy or top-k accuracy. For balanced verification, report precision, recall, F1, and accuracy; for imbalanced pairs, ROC-AUC and PR-AUC are more informative because accuracy can be misleading. Retrieval uses Recall@K, mean average precision, and precision at K.

  • Evaluate downstream tasks: Detection uses IoU and mAP; semantic and instance segmentation use mIoU and Dice; tracking commonly uses identity-aware metrics such as IDF1.

Best Practices and Deployment

  • Tune the matching threshold on validation data, not test data, and keep a final untouched test set.

  • Cache gallery embeddings when one input remains fixed. Although the branches share parameters, processing two uncached inputs still requires two encoder passes.

  • GPU acceleration helps training and large-batch inference, while smaller backbones and optimized runtimes may suit CPU or edge deployment.

  • Measure latency, memory use, and model size on the target device; quantization and ONNX export can reduce deployment cost but require accuracy validation.

  • Handle missing or corrupt images explicitly, inspect hard negatives for mislabeled examples, and assess demographic and operational bias.

Common Pitfalls

  • No shared weights: Two independently parameterized encoders do not form a classic Siamese network.

  • Poor negatives: Random unrelated pairs are usually less informative than plausible non-matches.

  • Leaking identities or galleries: Pairwise random splits can place images of the same subject in training and evaluation.

  • Over-augmentation: Independent crops or color changes may remove the evidence being compared.

  • Misusing evaluation metrics: Aggregate embedding scores do not establish fairness or downstream detection and segmentation performance.

  • Test-set threshold tuning: Choosing the threshold on test results invalidates the final evaluation.

46

How do you reduce overfitting in computer vision models?

Overfitting occurs when a computer vision model memorizes training-specific patterns and generalizes poorly to unseen images. It can be reduced with representative data, transfer learning, training-only augmentation, regularization, reduced model capacity, early stopping, and robust validation. The main trade-off is underfitting, because controls that are too restrictive can prevent the model from learning useful features.

Read full answer

Applies toPyTorch 2.2+ with matching torchvision 0.17+, and scikit-learn 1.5+ (Python 3.12)

Diagnosing Overfitting

Compare training and validation error using the same metric:

generalization gap = validation error - training error

A large positive gap means the model fits the training data much better than unseen data. Choose metrics for the actual task:

Task Model output Suitable metrics
Image classification Class label Accuracy, top-k accuracy, macro F1
Object detection Boxes and labels mAP, IoU
Semantic segmentation Label per pixel mIoU, Dice
Instance segmentation Masks, boxes, and labels Mask AP, IoU
Keypoint or pose estimation Keypoint locations OKS, PCKh
Tracking Identities and trajectories IDF1, HOTA
OCR Recognized text Character error rate, word error rate
Image generation Generated images FID, CLIP similarity

Accuracy can mislead on imbalanced classes because predicting the majority class may produce a high score while missing rare classes.

Techniques That Reduce Overfitting

  • Improve the dataset: Cover rare classes, lighting conditions, viewpoints, backgrounds, and edge cases. Correct labels and remove duplicates, including related images that would otherwise cross splits.

  • Use transfer learning: For small datasets, start with pretrained ResNet, EfficientNet, or ViT weights. Train the classification head first, then fine-tune selected backbone layers with a smaller learning rate.

  • Use training-only augmentation: Random resized crops, flips, rotations, color jitter, and blur improve robustness. Avoid transformations that invalidate the task, such as flipping text.

  • Regularize the model: Use weight decay and dropout. Weight decay limits parameter magnitude, while dropout randomly disables activations during training.

  • Control capacity: Smaller or shallower networks are less able to memorize when data is limited.

  • Stop early: Track validation loss, retain the best checkpoint, and stop after validation performance stops improving.

  • Prevent preprocessing leakage: Use the pretrained model’s documented mean and standard deviation. If normalization statistics are estimated from the dataset, calculate them from training folds only.

  • Validate reliably: Use StratifiedKFold when data is limited, construct augmentations independently for each fold, and keep the test set untouched until model selection is complete.

Code Example

This example creates synthetic shapes and demonstrates training-only augmentation, dropout, weight decay, and early stopping.

from copy import deepcopy

import numpy as np
import torch
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split
from torch import nn
from torch.utils.data import DataLoader, Dataset
from torchvision import transforms

SEED = 42
IMAGE_SIZE = 32
NUM_IMAGES = 240

np.random.seed(SEED)
torch.manual_seed(SEED)


def make_shapes(count: int):
    rng = np.random.default_rng(SEED)
    images = np.zeros((count, 3, IMAGE_SIZE, IMAGE_SIZE), dtype=np.float32)
    labels = np.zeros(count, dtype=np.int64)
    yy, xx = np.mgrid[:IMAGE_SIZE, :IMAGE_SIZE]

    for index in range(count):
        label = index % 2
        center_x = rng.integers(8, IMAGE_SIZE - 8)
        center_y = rng.integers(8, IMAGE_SIZE - 8)
        width = rng.integers(9, 15)
        height = rng.integers(9, 15)

        if label == 0:
            mask = (
                (np.abs(xx - center_x) <= width / 2)
                & (np.abs(yy - center_y) <= height / 2)
            )
        else:
            mask = (
                ((xx - center_x) / (width / 2)) ** 2
                + ((yy - center_y) / (height / 2)) ** 2
            ) <= 1.0

        color = rng.uniform(0.25, 0.75, size=3)
        images[index, :, mask] = color[:, None]
        labels[index] = label

    return torch.from_numpy(images), torch.from_numpy(labels)


class ShapeDataset(Dataset):
    def __init__(self, images, labels, transform=None):
        self.images = images
        self.labels = labels
        self.transform = transform

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, index):
        image = self.images[index]
        if self.transform is not None:
            image = self.transform(image)
        return image, self.labels[index]


class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 16, 3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.AdaptiveAvgPool2d((4, 4)),
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(32 * 4 * 4, 64),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(64, 2),
        )

    def forward(self, images):
        return self.classifier(self.features(images))


images, labels = make_shapes(NUM_IMAGES)
train_indices, validation_indices = train_test_split(
    np.arange(NUM_IMAGES),
    test_size=0.25,
    random_state=SEED,
    stratify=labels.numpy(),
)

normalize = transforms.Normalize([0.5] * 3, [0.5] * 3)
train_transform = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomCrop(IMAGE_SIZE, padding=4),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    normalize,
])
validation_transform = transforms.Normalize([0.5] * 3, [0.5] * 3)

train_dataset = ShapeDataset(
    images[train_indices], labels[train_indices], train_transform
)
validation_dataset = ShapeDataset(
    images[validation_indices],
    labels[validation_indices],
    validation_transform,
)

train_loader = DataLoader(
    train_dataset,
    batch_size=32,
    shuffle=True,
    generator=torch.Generator().manual_seed(SEED),
)
validation_loader = DataLoader(validation_dataset, batch_size=32)

model = SmallCNN()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-3)

best_loss = float("inf")
best_epoch = 0
best_state = deepcopy(model.state_dict())
epochs_without_improvement = 0

for epoch in range(1, 41):
    model.train()
    for batch_images, batch_labels in train_loader:
        optimizer.zero_grad(set_to_none=True)
        loss = criterion(model(batch_images), batch_labels)
        loss.backward()
        optimizer.step()

    model.eval()
    validation_loss = 0.0
    with torch.no_grad():
        for batch_images, batch_labels in validation_loader:
            loss = criterion(model(batch_images), batch_labels)
            validation_loss += loss.item() * batch_images.size(0)

    validation_loss /= len(validation_dataset)

    if validation_loss < best_loss - 1e-4:
        best_loss = validation_loss
        best_epoch = epoch
        best_state = deepcopy(model.state_dict())
        epochs_without_improvement = 0
    else:
        epochs_without_improvement += 1

    if epochs_without_improvement >= 6:
        break

model.load_state_dict(best_state)
model.eval()
predictions = []
true_labels = []

with torch.no_grad():
    for batch_images, batch_labels in validation_loader:
        predictions.extend(model(batch_images).argmax(dim=1).tolist())
        true_labels.extend(batch_labels.tolist())

print(f"Best epoch: {best_epoch}")
print(f"Best validation loss: {best_loss:.4f}")
print(f"Validation accuracy: {accuracy_score(true_labels, predictions):.4f}")
print(f"Validation macro F1: {f1_score(true_labels, predictions, average='macro'):.4f}")

Deployment and Practical Considerations

  • Latency and model size: Smaller backbones, pruning, and quantization can help edge deployment, but quantized models must be re-evaluated.

  • Export: Compatible PyTorch models can often be exported to ONNX, although dynamic operations may require changes.

  • Hardware: GPUs accelerate CNN and ViT training; smaller models may run efficiently on CPUs or specialized accelerators.

  • Privacy and bias: Face-related systems need privacy safeguards, representative evaluation, and testing across demographic groups.

Common Pitfalls

  • Fitting data-dependent preprocessing before splitting causes leakage.

  • Random augmentation is not inherently leakage, but related or duplicate images can cross split boundaries; keep augmentation inside training folds.

  • Repeatedly tuning against the test set produces an optimistic evaluation.

  • Accuracy alone hides minority-class failures in imbalanced datasets.

  • Unrealistic augmentation or excessive regularization can cause underfitting.

  • Optimizing training loss alone ignores generalization.

Common follow-up questions include which regularization method to choose, how to detect leakage, when to fine-tune the entire backbone, and how to choose a stopping point without using the test set.

47

What metrics evaluate computer vision system performance?

No single metric captures every aspect of a computer vision system; the appropriate choice depends on whether the system performs classification, detection, segmentation, tracking, OCR, or image generation. Metrics should reflect the task’s costs and failure modes—for example, accuracy can conceal poor performance on minority classes, while IoU is more informative than exact pixel agreement for segmentation. The main trade-off is that simple aggregate metrics are easy to communicate but may hide class imbalance, localization errors, or operational problems.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Metrics by Computer Vision Task

Task Typical output Common metrics
Image classification One or more class labels Accuracy, top-k accuracy, precision, recall, F1, ROC-AUC, PR-AUC
Object detection Bounding boxes, labels, confidence scores IoU, precision, recall, mAP
Semantic segmentation One class label per pixel mIoU, pixel accuracy, Dice
Instance segmentation A separate class label and mask per object Mask IoU, mask AP, Dice
Keypoint or pose estimation Coordinates for body joints or facial landmarks PCK, MPJPE, OKS
Multi-object tracking Object identities across video frames MOTA, IDF1, HOTA
OCR Extracted text Character error rate, word error rate
Image generation Generated images FID, KID, precision, recall, CLIP score

Classification Metrics

Accuracy is the number of correct predictions divided by the total number of predictions:

accuracy = correct predictions / total predictions

  • Top-k accuracy: Correct if the true class appears among the model’s k highest-scoring predictions. It is useful for large label sets, but k must be reported.

  • Precision: TP / (TP + FP), measuring how many predicted positives are correct.

  • Recall: TP / (TP + FN), measuring how many actual positives are found.

  • F1 score: The harmonic mean of precision and recall, useful when false positives and false negatives both matter.

  • ROC-AUC: Measures ranking performance across all classification thresholds.

  • PR-AUC: Often more informative for imbalanced datasets because it focuses on the positive class.

Here, TP means true positives, FP means false positives, and FN means false negatives. Accuracy alone can be misleading when one class dominates the dataset.

Detection and Segmentation Metrics

For object detection, IoU measures overlap between a predicted box and its ground-truth box:

IoU = intersection area / union area

A prediction is typically considered a true positive only when its class is correct and its IoU exceeds a threshold such as 0.5. Mean Average Precision (mAP) averages precision-recall performance across classes and, depending on the benchmark, across IoU thresholds such as AP50 or AP50:95.

For semantic and instance segmentation, mIoU averages IoU across semantic classes, while Dice score is:

Dice = 2 * intersection pixels / (predicted pixels + ground-truth pixels)

Dice and IoU emphasize overlapping regions and are generally more useful than exact pixel accuracy when small objects or boundary errors matter. For instance segmentation, mask-based AP is commonly used.

Specialized and Operational Metrics

  • PCK: Percentage of keypoints within a specified distance from the ground truth.

  • MPJPE: Mean average joint position error, measured in spatial units or after normalization.

  • OKS: Object Keypoint Similarity, which compares predicted keypoints while accounting for object scale.

  • MOTA: Measures tracking errors such as missed objects, false positives, and identity mismatches.

  • IDF1: Measures how consistently object identities are maintained across frames.

  • HOTA: Combines detection, association, and localization quality into one tracking metric.

  • CER: (substitutions + deletions + insertions) / ground-truth characters.

  • WER: A similar normalized error rate measured at the word level.

  • FID and KID: Compare generated-image distributions with reference images; lower values generally indicate closer distributions, but they can be sensitive to sample size and evaluation choices.

  • CLIP score: Measures semantic alignment between generated images and their prompts using CLIP embeddings.

For a deployed system, also report p50 and p95 latency, throughput, throughput under concurrent load, peak memory, model size, and hardware used. Quantization or ONNX export should be followed by re-evaluation to confirm that accuracy and task-specific metrics remain acceptable.

Common Pitfalls

  • Reporting accuracy for heavily imbalanced classification without precision, recall, F1, and PR-AUC.

  • Using mAP without stating the IoU thresholds, image size, confidence filtering, and evaluation protocol.

  • Evaluating on the test set while repeatedly tuning decisions against it.

  • Comparing metrics produced with different data splits or preprocessing pipelines.

  • Treating mean IoU as sufficient when rare classes or small objects need separate analysis.

  • Reporting only model quality while ignoring latency, memory, model size, and edge-device constraints.

  • Omitting fairness checks for face-related or other human-centered applications; report performance gaps across relevant demographic groups where lawful and appropriate.

48

Why is cross-validation important for vision models?

Cross-validation estimates how a vision model is likely to perform on unseen data by repeatedly training and evaluating it on different data partitions. It produces a more reliable estimate than a single validation split and reduces the risk of selecting a model that performs well only because of an unusually easy split. The main trade-off is higher computational cost, especially when every fold requires fine-tuning a deep model.

Read full answer

Applies tomachine learning concepts for computer vision (Python 3.12)

Why Cross-Validation Matters

  • Better generalization estimate: A single split can be unusually easy or difficult. Rotating through several folds gives a more representative assessment of expected performance.

  • Measures variability: Reporting the mean and standard deviation across folds shows whether performance is stable or sensitive to the training data.

  • Improves model selection: Cross-validation helps compare architectures, pretrained backbones, augmentation policies, and hyperparameter settings consistently.

  • Uses data efficiently: Each fold trains on a different subset, which is valuable when collecting and labeling images is expensive.

  • Reveals failure patterns: Fold-level results can expose sensitivity to class imbalance, underrepresented groups, noisy labels, or domain-specific images.

  • Supports ensembles: Models trained on different folds can sometimes be combined to improve robustness, although their errors may be correlated.

Cross-validation does not guarantee performance on the final deployment distribution. If production data differs from the training distribution, a separate external test set is still necessary.

Choosing the Right Validation Strategy

Data situation Recommended strategy Reason
Images with balanced or moderately imbalanced classes StratifiedKFold Preserves class proportions in each fold
Several images from the same subject, patient, or session GroupKFold Prevents related samples from appearing in both training and validation
Frames extracted from the same video Group all frames by video Prevents near-duplicate frames from leaking into validation
Images ordered by time Time-based or forward-chaining validation Simulates deployment on future data
Very small dataset Repeated stratified or grouped cross-validation Reduces dependence on one partition, although uncertainty remains high

For object detection, consider a multi-label-aware stratification strategy because images can contain several classes. Grouping usually matters more than maintaining exact class proportions.

Every image transformation that learns from data must be fitted inside the training fold. This includes computing normalization statistics, selecting features, imputing missing values, or learning augmentation parameters. Using validation data for those operations causes data leakage.

Metrics for Vision Tasks

Task Model output Appropriate evaluation metrics
Image classification One or more class labels Accuracy, top-k accuracy, F1, ROC-AUC, PR-AUC
Object detection Bounding boxes, labels, confidence scores IoU and mAP
Semantic segmentation One class per pixel mIoU and Dice
Instance segmentation Per-pixel masks with instance identities Mask AP and IoU-based measures
Keypoint or pose estimation Coordinates for body joints or other keypoints PCK, MPJPE, or normalized distance error
Tracking Object identities across frames HOTA, IDF1, and identity switches
OCR Recognized text Exact match, word accuracy, character error rate
Image generation Generated images Human evaluation plus distributional metrics such as FID or KID with caution

Accuracy can mislead for imbalanced classification because a model predicting the majority class may appear strong. PR-AUC is often more informative than ROC-AUC when the positive class is rare. Detection and segmentation should also be evaluated with task-specific thresholds because a single aggregate score can hide poor localization or minority-class performance.

Practical Evaluation Workflow

  1. Split the dataset according to the realistic deployment scenario.

  2. Create folds that preserve important groups and, when appropriate, class balance.

  3. Fit all preprocessing only on each fold’s training portion.

  4. Train the model using only training data and augmentation.

  5. Evaluate on the held-out fold using metrics appropriate to the task.

  6. Repeat for every fold and report the mean, dispersion, and worst-performing folds.

  7. Select the final configuration using the cross-validation results.

  8. Evaluate that configuration once on an untouched test set or external dataset.

For neural networks, initialize every fold independently rather than reusing weights trained on another fold’s validation data. If early stopping uses the validation fold, that fold is also influencing model selection, so an untouched final test set remains important. Nested cross-validation is appropriate when estimating the performance of the entire hyperparameter-selection procedure.

Common Pitfalls

  • Random image-level splitting of related data: Duplicate images, video frames, or subjects can leak information and inflate scores.

  • Preprocessing before splitting: Fitting a scaler, encoder, or feature selector on the full dataset exposes validation information.

  • Validation-set overfitting: Repeatedly tuning against one validation set turns it into a training resource.

  • Ignoring dispersion: Reporting only the average fold score hides unstable results and strong dependence on particular subsets.

  • Using unsuitable metrics: Accuracy alone can conceal severe imbalance, while aggregate mAP may conceal poor performance on rare categories.

  • Ignoring distribution shift: Random folds may mix camera types, locations, demographic groups, or acquisition conditions. Group-aware or external validation is needed when deployment differs from training.

  • Underestimating compute cost: Training five folds means approximately five full training runs; pretrained transfer learning usually makes cross-validation more practical.

  • Skipping reproducibility controls: Record dataset versions, fold assignments, random seeds, pretrained weights, augmentation settings, and software versions.

Interview Tips: A strong follow-up question is how you would prevent leakage when several images belong to the same video or subject. Explain that you would group those samples with GroupKFold, keep all preprocessing inside the training folds, and use a separate untouched test set for the final evaluation.

49

How does Intersection over Union work for object detection?

Intersection over Union (IoU) measures how much a predicted bounding box overlaps its corresponding ground-truth box. It solves the problem of evaluating localization quality, rather than only whether the predicted class is correct. Its main trade-off is that stricter thresholds require more precise localization, while looser thresholds allow less accurate matches.

Read full answer

Applies tomachine learning concepts (all versions)

How It Works

For a predicted bounding box (B_p) and ground-truth box (B_g):

IoU = area(B_p ∩ B_g) / area(B_p ∪ B_g)

  • B_p: the bounding box predicted by the detection model.

  • B_g: the ground-truth bounding box annotated for the object.

  • ∩: the overlapping region shared by both boxes.

  • ∪: the total area covered by either box.

  • area: the geometric area of a region.

The intersection and union areas must be calculated in the same coordinate system, usually pixels or normalized image coordinates. IoU ranges from 0.0 for no overlap to 1.0 for identical boxes.

For two valid boxes, their union area is:

union area = area(B_p) + area(B_g) - intersection area

The intersection is subtracted because the shared region was counted once in each box’s area.

Using IoU in Detection Evaluation

IoU itself does not determine whether a prediction is correct; it supplies a similarity value used with a threshold such as 0.5:

Comparison Typical detection outcome
Prediction IoU ≥ threshold Matched prediction, usually a true positive
Prediction IoU < threshold Unmatched prediction, usually a false positive
Ground-truth box remains unmatched False negative

Evaluation systems generally match predictions to ground-truth objects according to confidence scores and class labels. The matching rules can vary between frameworks, but IoU is the central localization criterion.

For overall detection quality, IoU is commonly incorporated into average precision (AP). COCO-style mAP averages precision-recall performance over multiple IoU thresholds, such as 0.50 to 0.95, making the evaluation stricter than a single threshold.

Interpreting IoU

  • 1.00: Perfect overlap.

  • 0.75: Strong localization.

  • 0.50: Moderate overlap and a traditional threshold for treating a detection as matched.

  • 0.00: No overlap.

For segmentation, related measures include mask IoU, mean IoU, and Dice score. These evaluate masks or predicted regions rather than only rectangular detection boxes.

Common Pitfalls

  • Confusing IoU with AP or mAP: IoU measures overlap between two boxes; AP and mAP summarize detection quality across matches, thresholds, and precision-recall behavior.

  • Using the wrong coordinate format: Ensure every box uses the same convention, such as [x1, y1, x2, y2], and that corner coordinates are correctly ordered.

  • Calculating union incorrectly: The union is the sum of both areas minus the intersection.

  • Ignoring class labels: A geometrically perfect box with the wrong class should not count as a correct detection.

  • Reporting only one threshold: A single IoU threshold can hide performance differences between precise and loose localization.

  • Confusing box IoU with segmentation metrics: Mask IoU and Dice operate on region masks and are not interchangeable with rectangular-box IoU.

50

How do you design an automatic license plate recognition system?

Automatic license plate recognition (ALPR) is a computer vision pipeline that detects vehicles or plates in images or video and reads the plate text to identify vehicles for tasks like tolling, parking, and access control. It solves the problem of extracting structured identity data from unconstrained scenes. Its main trade-off is accuracy versus operating conditions: high read rates require handling motion blur, glare, occlusion, varied plate formats, and viewpoint changes, which increases model and system complexity.

Read full answer

Applies tocomputer vision system design concepts (examples in Python 3.12, PyTorch 2.x, torchvision, OpenCV 4.x)

System Pipeline

  • Acquisition: capture frames with suitable resolution, shutter speed, and infrared illumination for night scenes.

  • Plate localization: run object detection to predict plate bounding boxes ((x, y, w, h) + confidence). This is an object detection task whose output is boxes plus labels.

  • Preprocessing: crop the plate region, resize, normalize contrast, and optionally rectify perspective before recognition.

  • Text recognition: read characters with OCR. Common outputs are a string and per-character confidences; sequence models may produce the full plate directly.

  • Postprocessing: apply plate syntax rules (country/state format, length, allowed characters), confidence thresholds, and temporal voting across video frames.

Models and Approaches

Approach How it works Strengths Weaknesses When to use
Two-stage (detector + OCR) Detect plate with a detector, then recognize text with a CRNN/CTC or transformer OCR model Modular, easier to debug and swap parts, works well with transfer learning on small datasets Errors propagate from detection to OCR; more latency Default for most production systems
End-to-end One model predicts plate regions and text jointly Fewer handoffs, can optimize jointly Needs more labeled data; harder to diagnose failures Large, curated datasets and tight latency budgets
Classic CV + OCR Edge/morphology heuristics find plates, traditional OCR reads them Low compute, no training for simple cases Fragile to lighting, angle, and plate design Controlled environments only

Current standard components include CNN backbones (ResNet, EfficientNet) or vision transformers (ViT) for features, detectors such as Faster R-CNN, the YOLO family, or DETR for localization, and segmentation models like U-Net or Mask R-CNN when pixel-accurate plate masks help rectification. CLIP-style image-text embeddings are less common for fixed-format plate reading but can support retrieval tasks.

Evaluation Metrics

  • Detection: IoU (intersection over union) measures box overlap; mAP summarizes precision/recall across IoU thresholds and classes.

  • Recognition: plate-level accuracy and character error rate (CER = edit_distance / reference_length) capture full-string and partial errors. Top-k accuracy is useful if the system returns candidate reads.

  • System: measure end-to-end read rate, latency per frame, and performance by condition (day/night, angle, speed). Report precision and recall when false accepts matter.

Code Example

import cv2
import numpy as np
import torch
from torchvision.models.detection import fasterrcnn_resnet50_fpn

# Synthetic BGR image (OpenCV default); draw a bright rectangle as a "plate"
rng = np.random.default_rng(42)
bgr = (rng.integers(0, 80, size=(480, 640, 3), dtype=np.uint8))
cv2.rectangle(bgr, (220, 200), (420, 260), (220, 220, 220), thickness=-1)

# Preprocess: OpenCV uses BGR, most models expect RGB in [0, 1]
rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
plate_crop = rgb[200:260, 220:420]  # crop from detector box in practice
plate_gray = cv2.cvtColor(plate_crop, cv2.COLOR_RGB2GRAY)
plate_resized = cv2.resize(plate_gray, (128, 32))

# Detector skeleton (weights=None avoids downloads; use pretrained weights for transfer learning)
model = fasterrcnn_resnet50_fpn(weights=None)
model.eval()
tensor = torch.from_numpy(rgb).permute(2, 0, 1).float() / 255.0  # CHW, [0,1]
with torch.no_grad():
    output = model([tensor])[0]

print("crop shape:", plate_resized.shape)          # (32, 128)
print("boxes tensor shape:", output["boxes"].shape) # (N, 4)

For small datasets, use transfer learning from a pretrained detector and OCR model, with augmentation (flips where valid, crops, color jitter) and normalization matching the pretrained model’s mean and standard deviation.

Interview Tips

  • Common pitfalls: evaluating only cropped, clean plates (ignores detector errors); using accuracy alone when formats or classes are imbalanced; training on frames from the same video without a split (data leakage); forgetting BGR/RGB conversion; not normalizing with the pretrained model’s statistics.

  • Deployment concerns: optimize latency and model size with quantization and ONNX export for edge cameras or gateways; handle GPU versus CPU budgets; enforce privacy controls because plate data can enable location tracking, and audit for bias across regions, plate styles, and lighting conditions.

  • Follow-ups: How do you track one plate across frames? How do you handle multilingual plates or motorcycles? How would you design the data-labeling and active-learning loop?

51

What computer vision technologies power autonomous vehicle navigation?

Autonomous vehicle navigation uses computer vision to interpret camera images and estimate objects, drivable space, depth, and motion for safe driving. The central trade-off is between modular, interpretable perception pipelines and end-to-end systems that may generalize better but are harder to validate and explain.

Read full answer

Applies toautonomous-driving computer vision concepts (Python 3.12)

How Vision Fits the Navigation Stack

A typical autonomous-driving pipeline processes synchronized camera, LiDAR, and radar data through several stages:

  1. Sensor preparation: Calibrate cameras, correct distortion, synchronize timestamps, and convert images into a standard color space.

  2. Perception: Detect and classify traffic participants, estimate depth, segment drivable areas, and track objects over time.

  3. Localization and mapping: Estimate the vehicle’s pose with visual odometry or SLAM, often fused with IMU, GPS, and map data.

  4. Prediction: Forecast the likely behavior of pedestrians, cyclists, vehicles, and other actors.

  5. Planning and control: Generate a safe trajectory and convert it into steering, braking, and acceleration commands.

Vision usually operates alongside LiDAR and radar rather than replacing them. Radar can measure radial velocity directly and remains useful in poor visibility, although heavy precipitation, fog, snow, multipath, and interference can degrade it. LiDAR provides accurate 3D geometry but is expensive, while cameras provide dense color and semantic information.

Core Computer Vision Tasks

Task Output Typical role Evaluation metrics
Image classification One or more labels Identify scenes, signs, or traffic lights Accuracy, precision, recall, top-k accuracy
Object detection Boxes, labels, confidence scores Locate vehicles, pedestrians, cyclists, and obstacles IoU, precision-recall curves, mAP
Semantic segmentation Label for every pixel Separate road, sidewalk, lanes, vegetation, and obstacles mIoU, Dice, pixel accuracy
Instance segmentation Separate mask per object Distinguish overlapping vehicles or pedestrians Mask IoU, average precision
Keypoint and pose estimation Keypoint coordinates Estimate body, limb, or vehicle-part poses OKS, PCKh, normalized error
Tracking Object IDs and motion history Maintain identities and estimate velocity HOTA, MOTA, IDF1
Depth estimation Per-pixel distance or geometry Support obstacle avoidance and 3D planning Absolute depth error, scale-invariant error
OCR Text and text locations Read signs or navigation information Word accuracy, character error rate
Image generation Synthetic images or scenes Produce training data and simulation scenarios Human evaluation and task-specific realism metrics

Modern Modeling Approaches

Approach How it works Strengths Weaknesses When to use and typical scale
ResNet, EfficientNet CNN backbones learn hierarchical image features Efficient, established, strong baselines Limited global context; spatial compute grows with resolution Classification and perception backbones, from edge to server
ViT Splits images into patch tokens and applies self-attention Global context and flexible scaling Often needs substantial data and compute; attention cost grows roughly with the square of token count Large datasets and server-side perception
Faster R-CNN, Mask R-CNN Generate proposals, then classify objects or predict masks Strong box and mask accuracy Two-stage processing is comparatively heavy Accurate server-side detection and instance segmentation
YOLO family Predicts object locations and classes in one network pass Low latency and efficient deployment Small or densely packed objects can remain difficult, depending on the variant Real-time systems and edge devices
DETR Uses a transformer for set-based object prediction Global context and anchor-free prediction Can require more compute and longer training than compact one-stage detectors Server-side or large-data perception systems
U-Net, SAM U-Net uses an encoder-decoder with skip connections; SAM produces masks from prompts Effective dense or promptable segmentation Basic U-Net has limited global context; SAM is not itself a complete navigation model Road segmentation, labeling, and assisted annotation
CLIP Learns joint image-text embeddings Supports zero-shot and weakly supervised recognition Embeddings do not directly provide reliable depth, geometry, or tracking Auxiliary semantic classification or retrieval
Diffusion models Iteratively denoise noise to synthesize images Generate rare or dangerous driving scenarios Generation is expensive and may contain artifacts Synthetic training data and simulation, not direct navigation

Tracking commonly combines motion models such as Kalman filters with detection association methods such as DeepSORT and ByteTrack to preserve identities across frames.

Training and Preprocessing

  • Use transfer learning when labeled driving data is limited; fine-tune all or selected layers according to the required domain shift and available data.

  • Normalize images using the pretrained model’s required mean and standard deviation. OpenCV loads images as BGR, whereas most models expect RGB.

  • Common augmentations include horizontal flips, carefully selected crops, weather or lighting changes, and limited color jitter. Preserve physically plausible scene geometry.

  • Balance day, night, weather, geographic, and rare-event coverage to reduce domain bias.

  • Use geographic, temporal, or grouped splits for evaluation because ordinary random cross-validation can leak information from near-duplicate driving frames.

  • Use sensor fusion when vision confidence is low, especially for depth, small objects, and severe weather.

Deployment Considerations

  • Measure end-to-end latency from image capture to control output, not only individual model inference time.

  • Consider model size, power, thermal limits, memory bandwidth, CPU versus GPU execution, and edge-accelerator support.

  • Quantize models when useful and deploy them through runtimes such as ONNX Runtime or vendor-optimized stacks, then verify accuracy after conversion.

  • Safety systems need redundancy, uncertainty estimation, out-of-distribution detection, and conservative fallbacks when perception confidence is insufficient.

  • Validate in simulation, controlled courses, public roads, and progressively expanded operating domains.

  • Driver-monitoring and other face-related applications require privacy, consent, demographic-bias, and surveillance safeguards.

Common Pitfalls

  • Data leakage: Do not tune preprocessing, augmentation, or model choices on data reserved for final testing.

  • Wrong metrics: Overall accuracy can hide dangerous false negatives; inspect per-class recall and failure severity.

  • Ignoring 3D geometry: A correct 2D detection may still have incorrect depth, size, position, or orientation.

  • Overfitting the validation set: Repeated optimization against one validation set can produce misleading results; keep a separate test set.

  • Weak temporal evaluation: Frame-level accuracy does not reveal broken tracking, unstable object IDs, or delayed hazard detection.

  • Domain shift: Models trained on clear daytime footage may fail under night, rain, glare, construction zones, or unfamiliar road markings.

52

What is a practical computer vision approach for medical image analysis?

A practical task-specific deep-learning pipeline converts de-identified medical images into a clearly defined clinical output, such as a malignancy label, lesion mask, or anatomical landmark. The system should target one clinical endpoint, validate at the patient level, and support human review. Its main trade-off is predictive performance versus explainability, generalizability, and deployment complexity.

Read full answer

Applies tomedical computer vision concepts (current practice)

Selecting the Clinical Task

Begin by matching the computer-vision task to the required output rather than attempting end-to-end diagnosis.

Task Output Common approaches Relevant metrics
Image classification One label, such as benign or malignant ResNet, EfficientNet, ViT Accuracy, top-k, sensitivity, specificity
Object detection Bounding boxes and labels Faster R-CNN, YOLO family, DETR IoU, mAP
Semantic segmentation A class label per pixel U-Net, SegFormer mIoU, Dice
Instance segmentation A separate mask per lesion or object Mask R-CNN Mask AP, Dice
Promptable segmentation Masks guided by points, boxes, or text prompts SAM IoU, Dice (task-dependent)
Keypoint or pose estimation Anatomical landmark coordinates HRNet, pose-estimation networks PCK, MPJPE
Tracking Object trajectories across frames DeepSORT, ByteTrack IDF1, HOTA
OCR Recognized text and confidence CRNN, Tesseract CER, WER
Image generation Synthetic images or augmentations Diffusion models FID plus clinical review

A practical first project is often binary classification of suspicious findings, followed by segmentation when precise localization or measurement is clinically required.

Recommended Workflow

  1. Define the endpoint: Specify the clinical use, reference standard, intended population, acceptable error costs, and operating threshold.

  2. Curate data: Load DICOM studies correctly, preserve relevant metadata, normalize orientation when appropriate, and de-identify patient information.

  3. Split by patient: Keep all images from one patient in a single partition. Use grouped stratified splitting when multiple studies exist per patient; use stratified splitting only for patient-independent data.

  4. Preprocess: Resize appropriately, use the pretrained model’s normalization statistics, and apply clinically safe augmentation such as flips, crops, rotations, and limited intensity variation. Avoid transformations that alter diagnostic evidence, such as aggressive color or intensity changes for modalities where appearance is clinically meaningful.

  5. Train a baseline: Start with transfer learning from a pretrained CNN backbone (ResNet or EfficientNet) or a vision transformer (ViT) when data and compute allow. Compare with a simpler baseline and a segmentation-first approach when localization is essential.

  6. Evaluate and calibrate: Select thresholds using validation data, not the test set, and report clinically relevant metrics and calibration.

Evaluation Strategy

  • Imbalanced classification: Accuracy can mislead because a model that predicts the majority class may miss most positive cases. Report sensitivity, specificity, precision, F1, ROC-AUC, PR-AUC, and calibration where appropriate.

  • Localization tasks: Use IoU and mAP for detection, and mIoU and Dice for segmentation.

  • Subgroup analysis: Report performance by scanner, site, demographic group, disease prevalence, and image quality to detect hidden failure modes.

  • External validation: Keep a final test set untouched until model and threshold selection are complete, and evaluate on external data when possible.

Deployment and Governance

  • Latency and scale: GPUs accelerate training, while CPU or edge inference may be preferable for portable or point-of-care devices.

  • Model optimization: Consider model compression, quantization, and ONNX export only after measuring their effect on accuracy, calibration, and latency.

  • Privacy: Encrypt data in transit and at rest, restrict access, and audit de-identification. Face or biometric data requires explicit consent, governance, and bias review.

  • Human oversight: Present predictions as decision support, surface uncertainty or flagged cases, and preserve audit logs.

  • Monitoring: Track scanner changes, input drift, calibration, subgroup performance, and clinical outcomes after deployment.

Common Pitfalls

  • Data leakage: Splitting images rather than patients can place related studies in both training and test sets.

  • Wrong metrics: Accuracy alone hides poor sensitivity, while ROC-AUC can obscure low precision in rare-disease settings.

  • Test-set tuning: Repeated threshold or model selection against the test set produces optimistic results.

  • Uncontrolled augmentation: Transformations that change anatomy or disease evidence can invalidate labels.

  • Weak generalization: Random-split results may not represent performance at another hospital or on a different scanner.

  • Insufficient documentation: Record dataset composition, preprocessing, subgroup results, calibration, intended clinical use, and known limitations. Interviewers commonly ask about leakage, threshold selection, external validation, explainability, and human-in-the-loop deployment.

53

How does computer vision support product recognition and tracking in retail?

Computer vision enables retailers to identify products, locate them on shelves, and follow them across video frames without manual scanning. Product recognition combines classification, detection, and image embeddings with retail catalog data, while tracking maintains temporary identities through spatial and temporal association. The main trade-off is recognition and tracking accuracy versus latency, compute cost, model size, and privacy risk.

Read full answer

Applies tomachine learning concepts (examples in Python 3.12)

Retail Vision Pipeline

A typical system processes images from fixed cameras, handheld devices, or robots:

  1. Preprocess images: Correct orientation, crop relevant regions, resize to the model input, and normalize using the pretrained model’s mean and standard deviation.

  2. Detect or segment products: Locate individual items, including partially visible or tightly packed products.

  3. Recognize products: Assign a category, SKU, or product variant using classification, embeddings, or catalog matching.

  4. Track objects across frames: Associate detections using position, motion, appearance, and re-identification features.

  5. Apply business rules: Count stock, measure shelf coverage, detect missing products, or trigger replenishment.

For example, a system can recognize a branded beverage bottle, maintain its tracking ID as a shopper moves it, and compare its shelf position with the expected planogram.

Tasks and Outputs

Computer vision task Typical output Retail application
Image classification Category or SKU label Identify the dominant product in an image
Object detection Bounding boxes, labels, and confidence scores Locate products on shelves or conveyor belts
Semantic segmentation Label for every pixel Mark all areas belonging to product classes
Instance segmentation Pixel-level mask per product Separate tightly packed or overlapping items
Keypoint or pose estimation Keypoints and body structure Analyze shopper reach, grasp, and placement behavior
Tracking Object identities and trajectories over time Count movements and monitor products temporarily
OCR Text, numbers, and structured fields Read prices, labels, dates, and shelf tags
Image generation Synthetic or edited images Create marketing scenes, although it is not core recognition

Product recognition often uses a two-stage design: detection or segmentation first, followed by recognition of each isolated product. This is generally more reliable than classifying a cluttered shelf image directly.

Modeling Approaches

Approach How it works Strengths Weaknesses Best use
CNNs such as ResNet and EfficientNet Extract hierarchical visual features Efficient, strong transfer-learning results May need substantial labeled data SKU and category classification
Vision Transformers such as ViT Uses attention over image patches Effective with large, diverse datasets Often more compute-intensive Large-scale classification
Faster R-CNN, YOLO, or DETR families Detect objects with boxes and labels Supports localization and counting Errors from occlusion and visual similarity Shelf and inventory detection
U-Net or Mask R-CNN Produce pixel-level masks Handles precise product boundaries More annotation and compute cost Dense or overlapping products
CLIP-style embeddings Maps images and text into a shared space Supports image-text matching and open-vocabulary queries Less reliable for visually similar exact SKUs Search and broad product discovery
ByteTrack or DeepSORT-style tracking Associates detections across frames Adds temporal consistency IDs can switch or be lost under occlusion Movement and interaction analysis

For small retail datasets, transfer learning should normally be the starting point. Freeze or fine-tune a pretrained backbone, train only on the catalog categories required, and augment carefully with crops, flips, brightness changes, and simulated occlusion. Excessive color jitter can be harmful when package color is a SKU-defining feature.

Evaluation

  • Classification: Use accuracy and top-k accuracy, but also precision, recall, and F1 because rare products and visually similar classes can make accuracy misleading.

  • Detection: Evaluate with mAP at appropriate IoU thresholds and inspect errors such as duplicate boxes, missed products, and false shelf items.

  • Segmentation: Use mIoU and Dice, especially when precise boundaries or heavily packed products matter.

  • Tracking: Use IDF1, HOTA, ID switches, and count error in addition to visual detection metrics.

  • Business performance: Measure planogram compliance, shelf availability, replenishment precision, and inventory count error.

Split training and validation data by store, date, or physical shelf location when possible. Randomly splitting near-duplicate product images can cause leakage and unrealistically high results.

Deployment and Operations

  • Latency: Real-time shelf monitoring may require low inference latency, while batch auditing can tolerate slower processing.

  • Model size and hardware: Smaller backbones and quantized models are useful on edge devices and cameras with limited compute.

  • ONNX export: Exporting supported PyTorch models to ONNX can simplify deployment across inference engines; validate numerical outputs after conversion.

  • Robustness: Handle glare, blur, low light, occlusions, packaging changes, and camera drift.

  • Privacy: Blur faces, restrict retention, and encrypt stored footage where shopper behavior is analyzed.

  • Bias and auditability: Evaluate performance across stores, lighting conditions, brands, and customer groups rather than relying on one average score.

Common Pitfalls

  • Catalog drift: New packaging and promotions can invalidate a model trained only on historical images.

  • Visual similarity: Different flavors or sizes may look nearly identical, requiring OCR, package metadata, or catalog embeddings.

  • Tracking confusion: Similar products, rapid movement, and occlusion cause ID switches; appearance features should complement motion constraints.

  • Data leakage: Images from the same shelf or store should not appear across training and validation splits.

  • Wrong metric: High classification accuracy can conceal poor localization, stock counting, or performance on rare products.

  • Overfitting the validation set: Repeatedly tuning against one store or campaign weakens its reliability as an independent test set.

  • Expected follow-up questions: How would you build the dataset? Which task requires segmentation rather than detection? How would you handle occlusion? Which metrics support shelf replenishment, and how would you deploy and monitor the system?

54

How do computer vision advances improve augmented reality applications?

Computer vision gives augmented reality (AR) systems the ability to understand the real world from camera input so virtual content can be placed, aligned, and rendered convincingly. It solves the core AR problem of knowing where surfaces, objects, and people are in 3D space and how the camera moves through that space. Its main trade-off is that higher accuracy and richer scene understanding usually increase latency, compute cost, and sensitivity to lighting, motion blur, and occlusion.

Read full answer

Applies tocomputer vision and AR concepts (current approaches)

Key Capabilities That Benefit AR

  • Camera tracking and localization: Feature matching and visual-inertial odometry estimate the device pose frame by frame, keeping virtual objects stable as the user moves.

  • Plane and depth estimation: Detecting floors, walls, and tables enables realistic placement and scaling of virtual objects; monocular or stereo depth models improve occlusion handling.

  • Object detection and recognition: Detectors such as Faster R-CNN, the YOLO family, and DETR (boxes + labels) let AR apps attach information or effects to real objects.

  • Segmentation: Semantic segmentation (a label per pixel, e.g., U-Net) and instance segmentation (e.g., Mask R-CNN) separate people, hands, or objects from the background; promptable segmentation such as SAM supports interactive selection.

  • Keypoint and pose estimation: Body, hand, and face landmarks enable try-on, gesture control, and expressive filters.

  • Image-text understanding: CLIP-style embeddings help AR systems match visual scenes to text prompts or retrieve relevant overlays.

How Advances Translate to Better AR

CV Capability How it works (typical approach) AR benefit Main limitation
Tracking / SLAM Tracks visual features across frames; fuses with IMU data Stable, low-drift anchoring of virtual content Fails in low texture, fast motion, or glare
Depth estimation CNN/ViT models predict per-pixel depth from RGB or sensor fusion Realistic occlusion and physics-aware placement Scale ambiguity (monocular) and edge artifacts
Object detection Backbone (e.g., ResNet, EfficientNet, ViT) + detection head Context-aware overlays and interactions Misses small or novel objects; latency on device
Segmentation / matting Pixel-wise classification or mask prediction Clean background removal and object insertion Boundary errors around hair, motion blur
Pose estimation Predicts keypoints for hands/body/face Natural gesture and avatar control Occlusion and unusual poses reduce accuracy

Evaluation and Deployment Considerations

  • Metrics: AR features inherit CV metrics such as accuracy/top-k for classification, IoU and mAP for detection, and mIoU and Dice for segmentation. For AR quality, tracking stability and end-to-end latency matter as much as offline accuracy.

  • Preprocessing: Inputs are typically resized and normalized using the pretrained model’s mean/std; augmentation (flips, crops, color jitter) and transfer learning are common when AR-specific data is limited. OpenCV (cv2) handles classic steps on NumPy images (BGR), while PyTorch/torchvision models expect RGB tensors.

  • Deployment: Mobile and headset AR require low latency and small models, so teams use quantization, pruning, and ONNX export for edge devices. GPU acceleration helps, but CPU-only fallbacks are often needed.

Common Pitfalls

  • Ignoring latency budgets: A model with strong offline mAP can feel unusable in AR if inference plus rendering exceeds the frame budget.

  • Poor lighting and domain shift: Models trained on clean images may fail indoors, outdoors, or under motion blur; test in real AR conditions.

  • Privacy and bias risks: Face and body analysis can expose sensitive data and show uneven performance across demographics; collect consent, minimize data retention, and audit fairness.

  • Interview tips: Be ready to explain how tracking, depth, and segmentation work together in one AR scene, why mAP alone is insufficient for AR, and how you would reduce latency (smaller backbone, quantization, lower input resolution) without breaking user experience.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 54 Computer Vision answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects