Unit 1: Foundations of Computer Vision and Deep Learning

1. Introduction to Computer Vision and the Deep Learning Paradigm

1.1 The Nature and Scope of Computer Vision

Computer Vision (CV) is the interdisciplinary field of artificial intelligence and computer science focused on developing mathematical models and computational algorithms that enable digital systems to extract high-level, meaningful, and actionable insights from multidimensional visual data (images and video streams).

• Core Objective: Map an input visual signal $\mathbf{x} \in \mathbb{R}^{H \times W \times C}$ to a structured semantic output $\mathbf{y}$, which may range from discrete categorical labels (image classification) to spatial coordinates (object localization and bounding box detection), pixel-dense label arrays (semantic and instance segmentation), or high-dimensional synthesized images (generative modeling).
• Inherent Challenges:
  • Semantic Gap: The profound conceptual disparity between low-level numerical pixel values (e.g., integer intensity arrays $[0, 255]$) and the rich semantic interpretation humans naturally perceive (e.g., distinguishing a domestic cat from a dog).
  • Viewpoint Variation: Rotations, translations, and perspective transformations drastically alter individual pixel coordinates while preserving underlying semantic identity.
  • Illumination Changes: Variations in scene lighting (specular reflections, shadows, color temperature) alter absolute pixel values across orders of magnitude.
  • Deformation and Scale: Non-rigid bodies (animals, clothing) change shape dynamically, and subjects appear at wildly differing spatial scales depending on camera proximity.
  • Occlusion and Background Clutter: Objects are frequently partially obscured by foreground obstacles or blend into visually intricate surroundings.

1.2 The Paradigm Shift: Hand-Crafted Features vs. Learned Representations

Historically, classical computer vision relied on a two-stage pipeline:

1. Manual Feature Engineering: Domain experts designed heuristic feature extractors to detect localized geometric or statistical primitives invariant to specific photometric and geometric perturbations:
   • SIFT (Scale-Invariant Feature Transform): Uses Difference-of-Gaussians (DoG) scale-space pyramids to identify scale-invariant keypoints, computing gradient orientation histograms within local patches.
   • SURF (Speeded-Up Robust Features): Accelerates scale-space detection using box filters and integral images.
   • HOG (Histogram of Oriented Gradients): Evaluates normalized local histograms of image gradient orientations in dense grids, widely adopted for human pedestrian detection.
   • Haar-like Features & AdaBoost: Integral-image box difference filters used in real-time face detection cascades (Viola-Jones).
2. Shallow Machine Learning Classifiers: The extracted feature descriptors were aggregated (e.g., Bag-of-Visual-Words) and fed into classical classifiers such as Support Vector Machines (SVM) or Random Forests.

• The Modern Deep Learning Shift:
  Beginning with AlexNet in the 2012 ImageNet challenge, End-to-End Deep Learning superseded manual feature design. Neural networks optimize continuous parameterized functions that learn hierarchical visual representations directly from raw pixel arrays through gradient descent:
  • Low-Level Layers: Learn universal spatial primitives (Gabor-like edge detectors, color gradients, corners).
  • Mid-Level Layers: Synthesize edges into textures, motifs, contours, and object parts.
  • High-Level Layers: Compose parts into holistic semantic object classes and scene abstractions.

2. Digital Image Formation, Representation, and Color Spaces

2.1 From Continuous Light to Numerical Tensors

Visual imaging begins with electromagnetic radiation reflecting off physical surfaces. An optical lens focuses incident light onto a photosensitive sensor array (CCD or CMOS):

1. Spatial Sampling: Discretizes the continuous spatial domain $(x, y)$ into a regular two-dimensional grid of finite picture elements (pixels) of size $H \times W$.
2. Quantization: Converts the continuous analog electrical charge accumulated per sensor photodiode into a discrete numerical integer level. In standard 8-bit unsigned integer encoding (uint8), the continuous radiometric luminance is mapped to $2^8 = 256$ discrete levels: $[0, 255]$.

Tensor Formats in Deep Learning Frameworks

In deep learning frameworks, image collections are organized as 4-dimensional numerical tensors:

• NCHW Format (Channels-First): Shape (Batch_Size, Channels, Height, Width). Default format for PyTorch and NVIDIA cuDNN, maximizing memory cache spatial locality during matrix convolution routines.
• NHWC Format (Channels-Last): Shape (Batch_Size, Height, Width, Channels). Default format for TensorFlow/Keras and standard image processing libraries (OpenCV, PIL).

2.2 Color Spaces and Representations

• Grayscale (Single Channel): Represents monochrome intensity $I \in [0, 255]^{H \times W}$. Standard weighted luminance conversion from linear RGB follows ITU-R Recommendation BT.601:
  $Y = 0.299 \cdot R + 0.587 \cdot G + 0.114 \cdot B$
  (Reflecting the human eye's peak photoreceptor sensitivity to green wavelengths).
• RGB Color Space: An additive color model where red, green, and blue primary wavelengths combine to produce any visible color. Each pixel is a vector $(R, G, B) \in [0, 255]^3$. While intuitive for display hardware, RGB channels are highly correlated, meaning illumination changes affect all three channels simultaneously.
• HSV / HSL Color Space:
  • Hue ($H \in [0, 360^\circ)$): Identifies the dominant spectral wavelength (color tint).
  • Saturation ($S \in [0, 1]$): Denotes color purity or vibrancy (distance from neutral gray).
  • Value ($V \in [0, 1]$): Represents overall brightness or intensity.
  • Advantage: Separates chromaticity (Hue and Saturation) from luminance (Value), allowing color-based thresholding and tracking invariant to lighting shifts.
• $\text{YC}_b\text{C}_r$ and LAB Spaces:
  • $\text{YC}_b\text{C}_r$: Used in JPEG compression and digital broadcasting; isolates luma ($Y$) from blue-difference chroma ($C_b$) and red-difference chroma ($C_r$), facilitating chroma subsampling (e.g., 4:2:0).
  • CIE $L^*a^*b^*$: Perceptually uniform color space where Euclidean distances directly correspond to perceived perceptual color differences.

3. Image Preprocessing, Scaling, and Augmentation Pipelines

3.1 Geometric Transformations and Spatial Interpolation

Deep neural networks require fixed-dimension tensor inputs across mini-batches. Converting raw, arbitrarily sized images requires spatial resizing governed by interpolation algorithms:

• Nearest-Neighbor Interpolation: Assigns each output pixel the intensity of the nearest input coordinate. Extremely fast ($\mathcal{O}(1)$), but produces severe blocky pixelation and jagged staircase artifacts.
• Bilinear Interpolation: Computes an intensity value using a distance-weighted linear combination of the $2 \times 2 = 4$ nearest neighborhood pixels:
  $I(x, y) \approx (1 - \alpha)(1 - \beta) I(x_1, y_1) + \alpha(1 - \beta) I(x_2, y_1) + (1 - \alpha)\beta I(x_1, y_2) + \alpha\beta I(x_2, y_2)$
  Provides smooth edge transitions with low computational cost; default for standard vision networks.
• Bicubic Interpolation: Uses a cubic spline weighting over a $4 \times 4 = 16$ pixel neighborhood. Preserves high-frequency details and fine textures, widely utilized in super-resolution preprocessing.

3.2 Radiometric Normalization and Standardization

Raw integer pixel inputs in $[0, 255]$ present severe optimization challenges for gradient descent: large input magnitudes create steep, ill-conditioned loss surfaces, causing saturated activations or exploding gradients.

Min-Max Rescaling

Linearly scales values into a normalized floating-point range:
$\hat{x} = \frac{x - x_{\min}}{x_{\max} - x_{\min}} \implies \hat{x} \in [0.0, 1.0] \quad \text{or mapped to } [-1.0, 1.0]$

Channel-Wise Z-Score Standardization

Transforms the data to have zero mean ($\mu = 0$) and unit variance ($\sigma = 1$) per color channel:
$\hat{x}_{c} = \frac{x_c - \mu_c}{\sigma_c}$
For models pretrained on ImageNet:

• $\mu_{\text{RGB}} = [0.485, 0.456, 0.406]$
• $\sigma_{\text{RGB}} = [0.229, 0.224, 0.225]$

Standardization centers the optimization landscape around the origin, accelerating gradient descent convergence and mitigating internal covariate shift.

3.3 Data Augmentation Strategies

Data augmentation artificially expands the effective size and diversity of training datasets to prevent overfitting and enforce geometric and photometric invariance:

• Spatial / Geometric Augmentations: Random horizontal flipping ($p=0.5$), random affine scaling, random rotations ($[-15^\circ, +15^\circ]$), random cropping (e.g., RandomResizedCrop to $224 \times 224$), and perspective distortion.
• Photometric / Color Augmentations: Color jitter (randomly altering brightness, contrast, saturation, and hue by $\pm 10-20\%$), random grayscale conversion, Gaussian blur, and motion blur.
• Advanced Compositional Augmentations:
  • Cutout: Randomly zeros out square rectangular regions, forcing networks to utilize diverse context rather than relying on isolated visual features.
  • MixUp: Generates synthetic training samples by linearly interpolating both images and their one-hot target labels:
    $\tilde{\mathbf{x}} = \lambda \mathbf{x}_i + (1 - \lambda) \mathbf{x}_j, \quad \tilde{\mathbf{y}} = \lambda \mathbf{y}_i + (1 - \lambda) \mathbf{y}_j \quad \text{where } \lambda \sim \text{Beta}(\alpha, \alpha)$
  • CutMix: Replaces a randomly cut-out bounding patch of image $\mathbf{x}_i$ with the corresponding patch from $\mathbf{x}_j$, mixing class labels proportional to patch area.

4. Feedforward Neural Networks and the Multilayer Perceptron (MLP)

4.1 Biological Inspiration to Artificial Neurons

The fundamental computational unit of deep learning is the artificial neuron (perceptual unit), which computes a linear combination of its $d$-dimensional input vector $\mathbf{x} = [x_1, \dots, x_d]^T$ weighted by learnable parameters $\mathbf{w} = [w_1, \dots, w_d]^T$ and an additive scalar bias $b$, followed by an activation function $\sigma(\cdot)$:
$z = \mathbf{w}^T \mathbf{x} + b = \sum_{i=1}^d w_i x_i + b, \qquad a = \sigma(z)$

Matrix Formulation of a Fully Connected Layer

For an input vector $\mathbf{x} \in \mathbb{R}^{d_{\text{in}}}$ mapped to an output layer of dimension $d_{\text{out}}$:
$\mathbf{z} = \mathbf{W} \mathbf{x} + \mathbf{b}, \qquad \mathbf{a} = \sigma(\mathbf{z})$
where $\mathbf{W} \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ is the weight matrix and $\mathbf{b} \in \mathbb{R}^{d_{\text{out}}}$ is the bias vector.
In mini-batch matrix form with batch size $N$ ($\mathbf{X} \in \mathbb{R}^{N \times d_{\text{in}}}$):
$\mathbf{Z} = \mathbf{X} \mathbf{W}^T + \mathbf{b}^T \in \mathbb{R}^{N \times d_{\text{out}}}$

4.2 Why Flat MLPs Fail on High-Dimensional Image Data

Attempting to process raw digital images directly with Multilayer Perceptrons (MLPs) suffers from three catastrophic structural limitations:

1. Destruction of 2D Spatial Locality: Flattening an image tensor $(H, W, C)$ into a 1D vector of length $H \cdot W \cdot C$ discards spatial neighborhood topology. Pixels that were directly adjacent vertically are separated by $W$ elements in the flattened vector.
2. Exploding Parameter Complexity: A modest $256 \times 256 \times 3$ RGB image produces an input vector of length $196,608$. Connecting this input to a single hidden layer of $1,024$ neurons requires:
   $196,608 \times 1,024 \approx 201 \times 10^6 \text{ parameters (over 200 million weights!)}$
   This causes extreme memory consumption, slow computation, and massive overfitting.
3. Lack of Translation Invariance: In an MLP, each weight connects to a fixed pixel coordinate. If an object shifts by even a few pixels, its values activate completely different weights, preventing the network from recognizing objects across different positions without millions of redundant training examples.

(This structural inadequacy motivated the creation of Convolutional Neural Networks, which enforce weight sharing and local receptive fields).

5. Non-Linear Activation Functions and Gradient Dynamics

5.1 The Mathematical Necessity of Non-Linearity

If all activation functions in a deep neural network are linear identity mappings ($\sigma(z) = z$), a multi-layer composition of $L$ layers:
$\mathbf{h}_1 = \mathbf{W}_1 \mathbf{x} + \mathbf{b}_1, \quad \mathbf{h}_2 = \mathbf{W}_2 \mathbf{h}_1 + \mathbf{b}_2, \quad \dots, \quad \mathbf{y} = \mathbf{W}_L \mathbf{h}_{L-1} + \mathbf{b}_L$
collapses algebraically into a single affine transformation:
$\mathbf{y} = \left(\prod_{l=1}^L \mathbf{W}_l\right) \mathbf{x} + \tilde{\mathbf{b}} = \mathbf{W}_{\text{effective}} \mathbf{x} + \tilde{\mathbf{b}}$
Regardless of depth, an un-activated network cannot represent non-linear decision boundaries or solve problems as simple as XOR. Non-linear activations empower networks to act as Universal Function Approximators.

5.2 Comparative Analysis of Activation Functions

1. Sigmoid (Logistic Function)

$\sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \sigma'(z) = \sigma(z)(1 - \sigma(z))$

• Output Range: $(0, 1)$ — naturally interprets activations as probabilities for binary classification.
• Critical Flaws:
  • Vanishing Gradients: The derivative $\sigma'(z)$ has a maximum value of $0.25$ at $z=0$ and rapidly approaches $0$ for $|z| > 4$. In deep networks, multiplying these fractional derivatives through backpropagation causes gradients at earlier layers to shrink exponentially to zero.
  • Non-Zero Centered Outputs: Since $\sigma(z) > 0$, all inputs to downstream layers are strictly positive, causing weight gradients during backpropagation to take either all-positive or all-negative steps (zigzagging optimization dynamics).

2. Hyperbolic Tangent (Tanh)

$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} = 2\sigma(2z) - 1, \qquad \tanh'(z) = 1 - \tanh^2(z)$

• Output Range: $(-1, 1)$ — zero-centered, mitigating the zigzag gradient dynamics of Sigmoid.
• Limitation: Still suffers from vanishing gradients when saturated ($|z| \gg 0$).

3. Rectified Linear Unit (ReLU)

$\text{ReLU}(z) = \max(0, z), \qquad \text{ReLU}'(z) = \begin{cases} 1, & z > 0 \\ 0, & z < 0 \end{cases}$

• Advantages:
  • Constant gradient of $1$ for all positive inputs ($z > 0$), eliminating the vanishing gradient problem along active paths.
  • Fast computation: evaluated with a simple hardware threshold comparison.
  • Enforces biological sparsity: roughly $50\%$ of neurons produce zero activations.
• The Dying ReLU Problem: If a large gradient updates weights such that a neuron outputs $z < 0$ for all training inputs, its gradient becomes permanently $0$, leaving the neuron inactive for the rest of training.

4. Leaky ReLU and Parametric ReLU (PReLU)

$\text{LeakyReLU}(z) = \max(\alpha z, z) = \begin{cases} z, & z \ge 0 \\ \alpha z, & z < 0 \end{cases}$

• Fixes the dying ReLU issue by assigning a small non-zero slope ($\alpha = 0.01$) to negative inputs.
• In PReLU, $\alpha$ is treated as a learnable parameter updated via backpropagation.

5. Exponential Linear Unit (ELU) & Gaussian Error Linear Unit (GELU)

• ELU: Smooths negative inputs using $\alpha(e^z - 1)$ for $z < 0$, bringing mean unit activations closer to zero.
• GELU: Weights inputs by their cumulative probability under a normal distribution: $\text{GELU}(z) = z \cdot \Phi(z) = z \cdot \frac{1}{2}\left[1 + \text{erf}\left(\frac{z}{\sqrt{2}}\right)\right]$. Standard activation for modern Vision Transformers (ViT) and state-of-the-art vision models.

6. Softmax Activation (Multi-Class Probability Distribution)

For a vector of $K$ unnormalized logits $\mathbf{z} = [z_1, \dots, z_K]^T$:
$\text{Softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}}$

• Maps arbitrary real logits into a normalized categorical probability distribution: $\sum_{i=1}^K \text{Softmax}(\mathbf{z})_i = 1$ with all values $\in (0, 1)$.

6. Loss Functions, Optimization, and Generalization

6.1 Loss Functions for Computer Vision

1. Mean Squared Error (MSE / $L_2$ Loss)

$\mathcal{L}_{\text{MSE}}(\mathbf{y}, \hat{\mathbf{y}}) = \frac{1}{N} \sum_{i=1}^N \|\mathbf{y}_i - \hat{\mathbf{y}}_i\|_2^2$

• Used primarily in continuous regression tasks: bounding box coordinate regression, keypoint estimation, and autoencoder reconstruction. Heavily penalizes large outlier errors due to quadratic squaring.

2. Smooth $L_1$ Loss (Huber Loss)

$\text{Smooth}_{L_1}(x) = \begin{cases} 0.5 x^2, & \text{if } |x| < 1 \\ |x| - 0.5, & \text{otherwise} \end{cases}$

• Combines the smooth behavior of $L_2$ near zero with the outlier robustness of $L_1$ for large errors; standard loss in object detection systems (Fast R-CNN, Faster R-CNN).

3. Binary Cross-Entropy (BCE Loss)

$\mathcal{L}_{\text{BCE}}(y, \hat{y}) = - \left[ y \log \hat{y} + (1 - y) \log (1 - \hat{y}) \right]$

• Standard for binary classification and multi-label classification tasks.

4. Categorical Cross-Entropy Loss (Softmax Loss)

For mutually exclusive multi-class classification with $C$ classes and one-hot target vector $\mathbf{y}$:
$\mathcal{L}_{\text{CE}}(\mathbf{y}, \hat{\mathbf{y}}) = - \sum_{c=1}^C y_c \log \hat{y}_c = - \log \hat{y}_{\text{target}}$

• Gradient with respect to logit $z_i$:
  $\frac{\partial \mathcal{L}_{\text{CE}}}{\partial z_i} = \hat{y}_i - y_i$
  Yields an intuitive error signal: the gradient is the difference between predicted probability and ground-truth indicator.

6.2 Backpropagation and Optimization Dynamics

Backpropagation via the Multivariate Chain Rule

Backpropagation recursively calculates gradients of scalar loss $\mathcal{L}$ with respect to all network weights $\mathbf{W}^{(l)}$ by traversing the computational graph in reverse (Reverse-Mode Automatic Differentiation):
$\frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(l)}} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{(l)}} \cdot \frac{\partial \mathbf{z}^{(l)}}{\partial \mathbf{W}^{(l)}} = \boldsymbol{\delta}^{(l)} (\mathbf{a}^{(l-1)})^T$
where the upstream error delta satisfies:
$\boldsymbol{\delta}^{(l)} = \left( (\mathbf{W}^{(l+1)})^T \boldsymbol{\delta}^{(l+1)} \right) \odot \sigma'(\mathbf{z}^{(l)})$

Gradient Descent Optimization Algorithms

• Stochastic Gradient Descent (SGD) with Momentum:
  $v_t = \beta v_{t-1} + \eta \nabla_\theta \mathcal{L}(\theta_t), \qquad \theta_{t+1} = \theta_t - v_t$
  Accumulates velocity in directions of consistent gradients, dampening high-frequency oscillations across steep ravine loss surfaces.
• RMSprop:
  Maintains an exponentially decaying average of squared gradients $s_t = \beta s_{t-1} + (1 - \beta) g_t^2$ and updates parameters by $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{s_t + \epsilon}} g_t$, providing adaptive per-parameter learning rates.
• Adam (Adaptive Moment Estimation):
  Combines momentum (first moment $m_t$) and RMSprop (second moment $v_t$) with bias correction:
  $\hat{m}_t = \frac{\beta_1 m_{t-1} + (1 - \beta_1) g_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{\beta_2 v_{t-1} + (1 - \beta_2) g_t^2}{1 - \beta_2^t}, \quad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$
• AdamW (Decoupled Weight Decay):
  Applies weight decay directly to weights rather than mixing it into the gradient moments, preventing $L_2$ regularization distortion in adaptive optimizers.

6.3 Regularization and Evaluation Metrics

• $L_2$ Regularization (Weight Decay): Penalizes large weights by adding $\frac{1}{2} \lambda \|\mathbf{W}\|_2^2$ to the loss, driving weights closer to zero and smoothing decision surfaces.
• Dropout: Randomly deactivates neurons with probability $p$ during each training forward pass, preventing co-adaptation and approximating an ensemble of exponentially many sub-networks.
• Core Vision Evaluation Metrics:
  $\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}, \qquad \text{Precision} = \frac{TP}{TP + FP}$
  $\text{Recall} = \frac{TP}{TP + FN}, \qquad F_1\text{-Score} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$