What is computer vision and how does it compare with human vision?
Computer vision is the field of enabling computers to interpret images and video, identifying objects, patterns, spatial relationships, and semantic meaning. It resembles human vision because both transform light into structured information, but computer systems typically depend on learned models and explicit training data, while human vision combines biological processing, context, experience, and rapid adaptation. The main trade-off is that machines can analyze images consistently and at scale, whereas humans generally generalize better to unfamiliar situations and use broader common-sense reasoning.
Read full answerHide full answer
Applies tomachine learning and computer vision concepts (all versions)
How Computer Vision Works
A typical computer-vision system converts visual input into numerical representations and then analyzes those representations for a specific task:
-
Acquire an image or video: Data may come from cameras, medical scanners, satellites, or stored files.
-
Preprocess the input: Common operations include resizing, color conversion, normalization, cropping, noise removal, and augmentation.
-
Extract features: Traditional methods use hand-designed features, while modern systems usually learn representations directly from pixels.
-
Train or run a model: The model learns patterns during training and later performs inference on new data.
-
Produce task-specific outputs: Examples include a class label, bounding boxes, pixel masks, object trajectories, or recognized text.
Pretrained models are often used with transfer learning, especially when the available dataset is small. For models pretrained on natural images, preprocessing normally follows the model’s expected normalization, such as its documented channel mean and standard deviation.
Core Tasks and Modern Approaches
| Task | Typical output | Common approaches |
|---|---|---|
| Image classification | One or more labels for the whole image | ResNet, EfficientNet, Vision Transformers |
| Object detection | Bounding boxes, labels, and confidence scores | Faster R-CNN, YOLO family, DETR |
| Semantic segmentation | One class label per pixel | U-Net and other encoder-decoder architectures |
| Instance segmentation | Pixel masks for individual objects | Mask R-CNN, DETR-based models |
| Keypoint or pose estimation | Locations of joints, facial landmarks, or other keypoints | CNNs, Vision Transformers, pose-specific networks |
| Tracking | Object identities and trajectories across frames | Detection plus matching, optical flow, transformer trackers |
| OCR | Text, words, characters, and sometimes layout | Detection and recognition pipelines |
| Image generation | New or modified images | Diffusion models, GANs |
Other important representations include CLIP embeddings, which place images and text in a shared vector space for tasks such as zero-shot classification and image-text retrieval.
Relationship to Human Vision
Both human and machine vision rely on related principles: detecting edges, contrasts, textures, shapes, motion, and spatial relationships. However, their implementations and limitations differ substantially:
| Human vision | Computer vision |
|---|---|
| Uses the retina, optic nerves, and visual cortex | Uses cameras, sensors, algorithms, and learned models |
| Learns continuously from everyday experience | Usually learns from explicitly prepared training data |
| Combines vision with memory, reasoning, and common sense | Often performs a narrowly defined task |
| Handles occlusion, changing viewpoints, and unusual lighting well in many cases | May struggle with distribution shifts and unfamiliar contexts |
| Automatically focuses, adapts, and uses depth cues | Focus, exposure, depth, and alignment usually require explicit handling |
| Has substantial energy and computational cost | Can process large volumes consistently and often in parallel |
Computer vision therefore does not reproduce human sight in full. It provides specialized approximations of particular abilities.
Evaluation
Metrics must match the output of the task:
-
Classification: Accuracy, precision, recall, F1, top-k accuracy, ROC-AUC, and PR-AUC.
-
Object detection: Intersection over Union (IoU) and mean Average Precision (mAP).
-
Semantic and instance segmentation: Mean Intersection over Union (mIoU) and Dice score.
-
Keypoint estimation: Normalized keypoint error or percentage of correctly located points.
-
Tracking: Identity consistency, trajectory quality, and fragmentation or identity-switch rates.
Accuracy can be misleading for imbalanced datasets: a model predicting the majority class every time may achieve high accuracy while failing to detect rare but important cases.
Practical Considerations
-
Data quality: Label errors, inconsistent annotations, poor lighting, and underrepresented groups can reduce real-world performance.
-
Augmentation: Flips, crops, color jitter, and other transformations improve robustness when they represent plausible deployment conditions.
-
Overfitting: Validation data should not be repeatedly used for manual tuning, and final evaluation should use untouched test data.
-
Deployment: Latency, memory, model size, quantization, and hardware determine whether a model is suitable for cloud, mobile, or edge devices. ONNX export can improve interoperability with supported inference runtimes.
-
GPU versus CPU: GPUs accelerate large neural-network training and inference, while CPUs may be sufficient for compact models or low-throughput edge workloads.
-
Privacy and bias: Face recognition, medical imaging, and surveillance systems require careful attention to consent, demographic performance, security, and legal obligations.
Common Pitfalls
-
Confusing computer vision with image processing: preprocessing transforms pixels, whereas vision models infer higher-level meaning.
-
Evaluating on data similar to the training set and then claiming real-world generalization.
-
Ignoring class imbalance, localization quality, or per-class performance.
-
Using inappropriate augmentation, such as horizontal flips when text or anatomical direction makes them invalid.
-
Forgetting that OpenCV loads images in BGR order, while most image models expect RGB.
-
Reporting accuracy alone for safety-critical or imbalanced tasks.
Common interview follow-ups include: How do convolutional networks learn hierarchical features? Why use transfer learning? How would you detect data leakage? Which metrics matter for a medical-imaging system? How would you choose a deployment-friendly model?