Computer Vision in Practice

What Is Computer Vision?

Computer vision is the field of AI concerned with enabling machines to interpret and understand visual information from the world: photographs, video, medical scans, satellite imagery, and more. Where a language model processes sequences of tokens, a computer vision model processes grids of pixel values and learns to extract meaning from spatial patterns.

Humans do this effortlessly. You glance at a street scene and instantly recognise pedestrians, road signs, distances, and potential hazards. For decades, this was considered uniquely biological. The breakthrough of deep learning, specifically convolutional neural networks (covered in Lesson 4.3), changed that assumption.

Today, computer vision systems match or exceed human performance on many narrow tasks. They can spot tumours in chest X-rays that radiologists miss on first review, count vehicles across thousands of traffic cameras simultaneously, and detect manufacturing defects at a speed no human inspector could match.

Core Computer Vision Tasks

Computer vision is not a single problem. It encompasses a family of related tasks, each with its own output format and level of difficulty.

Each task increases in complexity and output specificity. A classification model needs to understand the image globally; a segmentation model must make a precise decision for every one of millions of pixels. The right task depends on what your application actually requires.

Object Detection and YOLO

Object detection is one of the most commercially important CV tasks. Autonomous vehicles, retail analytics, security cameras, and medical imaging all rely on it. The core challenge is that a single image may contain dozens of objects at different scales and positions, and the model must find them all at once.

How Bounding Boxes Work

A bounding box is defined by four numbers: the x and y coordinates of the top-left corner, plus the width and height of the box. A detection model predicts these four numbers along with a class probability distribution for each detected object. At inference time, a confidence threshold filters out low-probability detections, and a technique called Non-Maximum Suppression (NMS) removes duplicate boxes that refer to the same object.

YOLO: You Only Look Once

Early detection models like R-CNN (Region-based CNN, Girshick et al. 2014) used a two-stage approach: first propose candidate regions, then classify each one. This was accurate but slow, making real-time video processing impractical.

YOLO (Redmon et al. 2016) reframed detection as a single regression problem. Instead of proposing regions separately, YOLO divides the image into a grid and predicts bounding boxes and class probabilities for each grid cell in a single forward pass through the network. This made real-time detection possible on standard hardware.

The YOLO Family

Image Segmentation

Object detection tells you where something is with a rectangle. Segmentation tells you exactly which pixels belong to it. This precision matters enormously in surgery planning, autonomous driving, and satellite land-cover mapping.

The U-Net Architecture

U-Net (Ronneberger et al. 2015) was originally designed for biomedical image segmentation and has become one of the most widely used CV architectures in medicine. Its distinctive feature is a symmetric encoder-decoder structure with skip connections that pass high-resolution spatial information directly from the encoder to the corresponding decoder layer, enabling precise pixel-level localisation.

OpenCV in Practice

OpenCV (Open Source Computer Vision Library) is the foundational toolkit for CV in Python. Released in 2000 and maintained by the OpenCV Foundation, it provides hundreds of image processing functions that sit below the neural network level: loading and resizing images, colour space conversion, edge detection, contour finding, and camera access.

Most CV pipelines start with OpenCV for preprocessing, feed processed images into a deep learning model, and then use OpenCV again to overlay results on the original image for visualisation.

Basic Image Operations

import cv2
import numpy as np

# Load an image (BGR colour order by default in OpenCV)
img = cv2.imread("street.jpg")
print(img.shape)  # (height, width, channels) e.g. (720, 1280, 3)

# Convert to RGB for display in matplotlib
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

# Convert to greyscale
grey = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# Resize to 224x224 for a model that expects that input size
resized = cv2.resize(img, (224, 224))

# Apply Gaussian blur to reduce noise before edge detection
blurred = cv2.GaussianBlur(grey, (5, 5), 0)

# Canny edge detection
edges = cv2.Canny(blurred, threshold1=50, threshold2=150)

# Normalise pixel values to [0, 1] for model input
img_norm = resized.astype(np.float32) / 255.0

Drawing Detection Results

# After a model returns detections, draw them back onto the original image
# detections: list of (label, confidence, x, y, w, h)
for label, confidence, x, y, w, h in detections:
    # Draw bounding box rectangle
    cv2.rectangle(img, (x, y), (x + w, y + h), (0, 255, 0), 2)

    # Compose label string
    text = f"{label}: {confidence*100:.1f}%"

    # Draw filled background for the text so it is legible
    (tw, th), _ = cv2.getTextSize(text, cv2.FONT_HERSHEY_SIMPLEX, 0.6, 1)
    cv2.rectangle(img, (x, y - th - 8), (x + tw, y), (0, 255, 0), -1)
    cv2.putText(img, text, (x, y - 4),
                cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 0, 0), 1)

# Save the annotated image
cv2.imwrite("annotated.jpg", img)

A Typical CV Pipeline

Using Pre-Trained Models

You rarely need to train a CV model from scratch. Pre-trained models that have already learned rich visual representations from millions of images are available through torchvision, Hugging Face, and the TensorFlow Hub.

Object Detection with Hugging Face

The Hugging Face Transformers library provides a high-level pipeline for object detection using modern transformer-based detectors like DETR (Detection Transformer, Carion et al., FAIR 2020) and DETA, which bring the self-attention mechanism from Lesson 4.4 into the vision domain.

from transformers import pipeline
from PIL import Image

# Load a pre-trained DETR model (Facebook Research, fine-tuned on COCO)
detector = pipeline(
    "object-detection",
    model="facebook/detr-resnet-50",
    threshold=0.7   # only return detections with confidence > 70%
)

# Load image from disk
image = Image.open("street.jpg")

# Run detection
results = detector(image)

# Each result is a dict: {'score', 'label', 'box': {'xmin','ymin','xmax','ymax'}}
for r in results:
    box = r["box"]
    print(f"{r['label']:15s}  score={r['score']:.2f}  "
          f"box=({box['xmin']},{box['ymin']},{box['xmax']},{box['ymax']})")

Image Classification with torchvision

import torch
from torchvision import models, transforms
from PIL import Image
import json, urllib.request

# Load ResNet-50 pre-trained on ImageNet-1k (1000 classes)
model = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
model.eval()   # switch off dropout and batchnorm training behaviour

# Standard ImageNet preprocessing
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                          std=[0.229, 0.224, 0.225]),
])

# Load and preprocess the image
img = Image.open("cat.jpg")
batch = preprocess(img).unsqueeze(0)  # add batch dimension: (1, 3, 224, 224)

# Run inference (no gradient computation needed)
with torch.no_grad():
    logits = model(batch)          # shape: (1, 1000)
    probs  = torch.softmax(logits, dim=1)

# Get the top 3 predictions
top3 = probs.topk(3)
top3_indices = top3.indices[0].tolist()
top3_probs   = top3.values[0].tolist()

# Map indices to human-readable labels using ImageNet class list
labels_url = "https://raw.githubusercontent.com/anishathalye/imagenet-simple-labels/master/imagenet-simple-labels.json"
labels = json.loads(urllib.request.urlopen(labels_url).read())
for idx, prob in zip(top3_indices, top3_probs):
    print(f"{labels[idx]:30s} {prob*100:.2f}%")

Medical Imaging AI

Healthcare is one of the most consequential domains for computer vision. Medical images, including X-rays, CT scans, MRI, histopathology slides, and retinal fundus photographs, contain patterns that are difficult, time-consuming, and expensive for human experts to interpret at scale. AI is beginning to change that.

Key Applications

What Makes Medical Imaging Different

Medical CV has unique challenges compared to general CV:

DICOM and Medical Image Formats

Medical images are typically stored in DICOM (Digital Imaging and Communications in Medicine) format, which bundles the pixel data with rich metadata: patient information, scanner parameters, acquisition settings, and standardised anatomical labels. The pydicom library is the standard Python tool for reading DICOM files. Many CV pipelines convert DICOM images to PNG or NumPy arrays before feeding them into models.

import pydicom
import numpy as np

# Load a DICOM chest X-ray file
dcm = pydicom.dcmread("chest_xray.dcm")

# Access metadata
print(dcm.PatientAge)       # e.g. '045Y'
print(dcm.Modality)         # 'CR' (computed radiography)
print(dcm.StudyDescription) # 'CHEST AP'

# Extract pixel array and apply stored windowing for display
pixel_array = dcm.pixel_array.astype(np.float32)
print(pixel_array.shape)    # (2048, 2048) for a high-resolution chest X-ray

# Normalise to [0, 255] for model input or OpenCV display
pixel_min, pixel_max = pixel_array.min(), pixel_array.max()
img_8bit = ((pixel_array - pixel_min) / (pixel_max - pixel_min) * 255).astype(np.uint8)

Real-World Applications

Computer vision is embedded in systems that billions of people encounter every day. Here is a survey of key domains.

Ethics in Computer Vision

Computer vision raises some of the most pressing ethical questions in AI, partly because it interacts with the physical world and involves the identification of individual people in ways that other AI applications do not.

Facial Recognition

Facial recognition is one of the highest-stakes CV applications. Systems matching faces against watchlists are deployed in airports, stadiums, retail stores, and public spaces, often without the knowledge of the people being scanned. Several studies have documented significant accuracy disparities across demographic groups. Joy Buolamwini and Timnit Gebru's Gender Shades study (referenced in Lesson 5.1) showed error rates for dark-skinned women were up to 34 percentage points higher than for light-skinned men in commercial facial analysis systems.

In response to accuracy and civil liberties concerns, IBM, Amazon, and Microsoft all paused or restricted sales of facial recognition technology to law enforcement between 2020 and 2021. Several cities in the United States, including San Francisco and Boston, have banned government use of the technology outright.

Key Takeaways

• Computer vision covers a family of tasks: classification (what?), detection (what and where?), segmentation (which pixels?), and pose estimation.
• YOLO (Redmon et al. 2016) reformulated object detection as a single regression problem, enabling real-time detection in one forward pass. The family has continued to evolve through YOLOv8 and YOLOv11.
• Segmentation assigns class labels to individual pixels. U-Net's encoder-decoder architecture with skip connections is the standard for medical image segmentation tasks.
• OpenCV is the foundational toolkit for image pre-processing, colour conversion, and result overlay. Nearly all CV pipelines combine OpenCV with a deep learning framework.
• Pre-trained models on COCO (detection) and ImageNet (classification) are available through Hugging Face and torchvision. For most custom tasks, fine-tuning these is far more practical than training from scratch.
• Medical imaging AI must contend with data scarcity, class imbalance, distribution shift between hospitals, and strict regulatory requirements. Benchmark accuracy alone is an insufficient measure of clinical readiness.
• Facial recognition, surveillance, and biometric identification applications require careful attention to demographic fairness, consent, and transparency. Accuracy disparities across demographic groups are well documented and must be addressed before deployment.

Practice Notebook

Run this lesson's code in Google Colab

OpenCV, DETR object detection, ResNet-50 · Free GPU · No setup needed