Unit 1: Foundations of Generative AI

1. Introduction to Generative Artificial Intelligence

Generative Artificial Intelligence represents a transformative paradigm shift in machine learning. While classical machine learning focused primarily on analysis, classification, and pattern recognition, generative AI enables computers to synthesize completely novel, high-dimensional digital artifacts—spanning natural language, computer code, photorealistic images, synthetic audio, and complex molecular structures.

CODECopy Code
+---------------------------------------------------------------------------------------------------+
|                                 DISCRIMINATIVE VS GENERATIVE PARADIGMS                            |
+--------------------------------------------------+------------------------------------------------+
| DISCRIMINATIVE MODELING                          | GENERATIVE MODELING                            |
+--------------------------------------------------+------------------------------------------------+
| - Estimates conditional distribution P(Y | X)    | - Estimates data distribution P(X) or P(X, Y)  |
| - Identifies decision boundaries between classes | - Models the underlying data distribution      |
| - Answers: "Is this image a cat or a dog?"       | - Answers: "Generate a novel image of a cat"   |
| - Maps high-dimensional input to discrete label  | - Samples from learned latent probability space|
| - Deterministic boundary estimation              | - Probabilistic synthesis of novel instances   |
+--------------------------------------------------+------------------------------------------------+

1.1 Mathematical Formulation of Generative Modeling

Formally, let $\mathcal{X}$ represent a high-dimensional data space (such as the space of all possible $1024 \times 1024$ RGB images or sequences of natural language text tokens). Real-world data instances are drawn from an unknown, underlying data distribution:

$x \sim p_{\text{data}}(x)$

The fundamental objective of a generative model parameterized by learnable weight vectors $\theta$ is to construct a parametric probability distribution $p_\theta(x)$ that closely approximates the true distribution:

$p_\theta(x) \approx p_{\text{data}}(x)$

Once the model has converged, novel synthetic instances $x_{\text{new}}$ are generated by sampling directly from the learned parametric distribution:

$x_{\text{new}} \sim p_\theta(x)$

1.2 The Foundation Model Paradigm

Modern generative AI is built upon the Foundation Model architecture—a term coined by the Stanford Institute for Human-Centered Artificial Intelligence (HAI). A foundation model is an exceptionally large deep neural network (often containing tens to hundreds of billions of parameters) trained on vast, uncurated multimodal datasets using self-supervised learning at scale.

Rather than engineering isolated models for specific downstream tasks (such as sentiment analysis, translation, or named entity recognition), a single unified foundation model develops a broad, generalized representation of world knowledge and language syntax, which can subsequently be adapted to countless specialized downstream applications through prompt engineering, parameter-efficient fine-tuning (PEFT), or reinforcement learning.

2. Mathematical Mechanisms: How Generative AI Works

Generative AI systems operate through a structured pipeline: transforming raw data into discrete numerical tokens, projecting tokens into dense vector embeddings, learning statistical dependencies via deep neural architectures, and generating novel sequences using stochastic decoding strategies.

CODECopy Code
+---------------------------------------------------------------------------------------------------+
|                                  THE GENERATIVE MODELING PIPELINE                                 |
+---------------------------------------------------------------------------------------------------+
|  [ Raw Input Data ]                                                                               |
|         |                                                                                         |
|         v                                                                                         |
|  [ Tokenization ] --------> Splitting into subwords (Byte-Pair Encoding, WordPiece)               |
|         |                                                                                         |
|         v                                                                                         |
|  [ Embedding Space ] -----> Mapping discrete tokens to continuous geometric vectors (d_model)    |
|         |                                                                                         |
|         v                                                                                         |
|  [ Deep Architecture ] ---> Transformers / U-Net: Modeling contextual dependencies & likelihood   |
|         |                                                                                         |
|         v                                                                                         |
|  [ Decoding & Sampling ] -> Softmax logits scaled by Temperature (T), Top-k, Top-p (Nucleus)      |
|         |                                                                                         |
|         v                                                                                         |
|  [ Synthesized Output ] --> High-fidelity text, image, code, audio, or 3D asset                   |
+---------------------------------------------------------------------------------------------------+

2.1 Tokenization and Discrete Encodings

Deep neural networks cannot process continuous text strings directly; text must be converted into discrete numerical tokens:

• Character-Level Tokenization: Minimal vocabulary size, but sequences become excessively long, and models struggle to capture semantic abstractions over distant characters.
• Word-Level Tokenization: Retains semantic integrity, but suffers from massive vocabulary bloat (millions of words) and fails entirely when encountering Out-Of-Vocabulary (OOV) terms.
• Subword Tokenization (Industry Standard): Algorithms such as Byte-Pair Encoding (BPE), WordPiece, and Unigram iteratively merge frequent character pairs. Common words remain single tokens (e.g., "learning" $\to$ ["learning"]), while rare or compound words are decomposed into subword units (e.g., "unconditionally" $\to$ ["un", "condition", "ally"]), balancing vocabulary efficiency with robust OOV handling.

2.2 Vector Embeddings and Semantic Geometry

Once tokenized, integer token IDs are mapped via an embedding lookup matrix $E \in \mathbb{R}^{|V| \times d_{\text{model}}}$ into continuous, dense vector representations. In this high-dimensional latent space:

• Semantic relationships are encoded as geometric distances and directional vectors.
• A classic property of word embeddings is vector arithmetic demonstrating relational analogies:

$\vec{v}_{\text{king}} - \vec{v}_{\text{man}} + \vec{v}_{\text{woman}} \approx \vec{v}_{\text{queen}}$

• Proximity and semantic similarity between two vector embeddings $\vec{u}$ and $\vec{v}$ are measured using Cosine Similarity:

$\text{Cosine Similarity}(\vec{u}, \vec{v}) = \frac{\vec{u} \cdot \vec{v}}{\|\vec{u}\| \|\vec{v}\|} = \frac{\sum_{i=1}^{d} u_i v_i}{\sqrt{\sum_{i=1}^{d} u_i^2} \sqrt{\sum_{i=1}^{d} v_i^2}}$

2.3 The Autoregressive Training Objective

In autoregressive text models (such as the GPT family, LLaMA, and Gemini), generation is formulated as sequential next-token prediction. Given a sequence of tokens $x = (x_1, x_2, \dots, x_T)$, the joint probability distribution decomposes factorially via the probability chain rule:

$p_\theta(x) = \prod_{t=1}^{T} p_\theta(x_t \mid x_1, x_2, \dots, x_{t-1}) = \prod_{t=1}^{T} p_\theta(x_t \mid x_{<t})$

The network is trained using Maximum Likelihood Estimation (MLE) by minimizing the empirical Cross-Entropy Loss across billions of tokens:

$\mathcal{L}_{\text{NLL}}(\theta) = - \frac{1}{T} \sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t})$

2.4 Decoding and Sampling Strategies

During inference, the neural network outputs unnormalized log-probabilities (called logits $z \in \mathbb{R}^{|V|}$) over the vocabulary for the next token position. These logits are converted into a normalized probability distribution using the Softmax function with Temperature scaling:

$p(x_i) = \frac{\exp(z_i / T)}{\sum_{j=1}^{|V|} \exp(z_j / T)}$

Where $T > 0$ represents the Temperature parameter:

• Low Temperature ($T \to 0$): Sharpens the distribution, driving probabilities toward the single most likely token. When $T = 0$, generation collapses to deterministic Greedy Search ($\text{argmax}$), producing coherent, factual, but repetitive and robotic text.
• High Temperature ($T > 1.0$): Flattens the distribution, giving lower-probability tokens a higher chance of selection, increasing novelty, creativity, and linguistic diversity, but raising the risk of incoherence or hallucination.

CODECopy Code
+---------------------------------------------------------------------------------------------------+
|                                  INFERENCE DECODING STRATEGIES                                    |
+---------------------------------+--------------------------------+--------------------------------+
| GREEDY SEARCH                   | TOP-K SAMPLING                 | TOP-P (NUCLEUS) SAMPLING       |
+---------------------------------+--------------------------------+--------------------------------+
| - Selects argmax P(x_i)         | - Truncates vocabulary to top  | - Selects smallest subset of   |
| - Deterministic & repetitive    |   K tokens with highest prob.  |   tokens whose cumulative sum  |
| - Tends to get stuck in loops   | - Ignores long tail of noise   |   exceeds probability threshold|
| - No diversity or creativity    | - Fixed K regardless of shape  | - Dynamically adapts to shape  |
+---------------------------------+--------------------------------+--------------------------------+

• Top-k Sampling: Filters the vocabulary to retain only the $k$ most probable tokens before renormalizing and sampling. Limits the risk of sampling bizarre outliers, but uses a rigid cutoff regardless of whether the distribution is peaked or flat.
• Top-p (Nucleus) Sampling: Selects the dynamic, minimal set of tokens whose cumulative probability exceeds threshold $p \in (0, 1]$:

$\sum_{i \in V^{(p)}} p(x_i) \ge p$

This dynamically expands the candidate pool when many plausible options exist and contracts it to one or two tokens when context strictly determines the next word.

2.5 The Transformer Self-Attention Mechanism

The core engine enabling modern generative models to condition on long-range context without the vanishing gradient limitations of Recurrent Neural Networks (RNNs) is the Scaled Dot-Product Attention mechanism:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V$

Where:

• $Q \in \mathbb{R}^{n \times d_k}$ represents the Query matrix (what tokens are searching for).
• $K \in \mathbb{R}^{m \times d_k}$ represents the Key matrix (what tokens advertise about their content).
• $V \in \mathbb{R}^{m \times d_v}$ represents the Value matrix (the actual contextual information retrieved).
• $\sqrt{d_k}$ is the scaling factor preventing dot-product magnitudes from growing excessively large in high dimensions, which would push softmax gradients into regions with vanishingly small derivatives.

3. Primary Generative AI Model Architectures

Generative AI architectures differ fundamentally in how they represent, approximate, and sample from the data probability distribution:

CODECopy Code
+---------------------------------------------------------------------------------------------------+
|                                 CORE GENERATIVE ARCHITECTURE FAMILIES                             |
+---------------------+---------------------+-----------------------+-------------------------------+
| ARCHITECTURE        | MATHEMATICAL BASIS  | PRIMARY ADVANTAGE     | PRIMARY LIMITATION            |
+---------------------+---------------------+-----------------------+-------------------------------+
| GANs                | Minimax Game        | Sharp, photorealistic | Training instability, mode    |
|                     | Generator vs Discrim| single-pass synthesis | collapse, no density estimate |
+---------------------+---------------------+-----------------------+-------------------------------+
| VAEs                | Probabilistic Latent| Smooth, interpolatable| Blurry visual outputs due     |
|                     | Maximizing ELBO     | latent representations| to pixel-wise loss objectives |
+---------------------+---------------------+-----------------------+-------------------------------+
| Diffusion Models    | Denoising Markov    | Unsurpassed fidelity, | Slow multi-step iterative     |
|                     | Forward/Reverse SDE | high sample diversity | reverse sampling latency      |
+---------------------+---------------------+-----------------------+-------------------------------+
| Autoregressive      | Next-Token Predict. | Exceptional reasoning,| Compute-intensive sequential  |
| Transformers        | Causal Self-Attn    | long-range coherence  | inference, hallucinations     |
+---------------------+---------------------+-----------------------+-------------------------------+

3.1 Generative Adversarial Networks (GANs)

Introduced by Ian Goodfellow et al. (2014), GANs formulate generative modeling as a two-player zero-sum game between two competing neural networks:

1. The Generator $G(z)$: Takes a random noise vector $z \sim p_z(z)$ sampled from a standard Gaussian or uniform distribution and maps it into synthetic data space $G(z)$.
2. The Discriminator $D(x)$: A binary classifier that evaluates whether an input sample $x$ is drawn from real empirical training data ($D(x) \approx 1$) or synthesized by the generator ($D(G(z)) \approx 0$).

The networks are trained simultaneously by solving the Minimax Objective:

$\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))]$

Critical GAN Challenges:

• Mode Collapse: The generator discovers a narrow subset of realistic outputs that consistently fool the discriminator and repeatedly generates only those samples, failing to capture the full diversity of the real data distribution.
• Training Instability: If the discriminator becomes too strong too early, its gradients vanish, providing zero learning signal to the generator. Conversely, if the discriminator is too weak, the generator receives chaotic feedback.
• Wasserstein GAN (WGAN): Solves vanishing gradients by replacing Jensen-Shannon divergence with the Earth Mover's (Wasserstein-1) Distance, enforcing a 1-Lipschitz continuity constraint via weight clipping or gradient penalty.

3.2 Variational Autoencoders (VAEs)

Introduced by Kingma and Welling (2013), Variational Autoencoders are directed probabilistic graphical models that introduce explicit continuous latent variables $z$:

1. The Probabilistic Encoder (Recognition Model) $q_\phi(z \mid x)$: Maps an observed high-dimensional data point $x$ into the parameters of a latent Gaussian distribution: mean vector $\mu(x)$ and covariance matrix $\Sigma(x) = \text{diag}(\sigma^2(x))$.
2. The Probabilistic Decoder (Generative Model) $p_\theta(x \mid z)$: Reconstructs the original data point from a sampled latent vector $z$.

Because the true posterior $p(z \mid x)$ is mathematically intractable, VAEs optimize the Evidence Lower Bound (ELBO) on the marginal log-likelihood:

$\log p_\theta(x) \ge \mathcal{L}_{\text{ELBO}}(\theta, \phi; x) = \underbrace{\mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)]}_{\text{Reconstruction Term}} - \underbrace{D_{\text{KL}}(q_\phi(z \mid x) \parallel p(z))}_{\text{Regularization Term}}$

• Reconstruction Term: Measures how accurately the decoder reconstructs the input from latent vector $z$ (e.g., mean squared error or binary cross-entropy).
• Kullback-Leibler (KL) Divergence Term: Penalizes deviations of the approximate posterior $q_\phi(z \mid x)$ from a standard Gaussian prior $p(z) = \mathcal{N}(0, I)$, enforcing a smooth, continuous, and sampleable latent space without gaps.
• The Reparameterization Trick: Standard random sampling $z \sim \mathcal{N}(\mu, \sigma^2)$ is stochastic and prevents gradient backpropagation. The reparameterization trick isolates stochasticity into an independent random noise variable $\epsilon \sim \mathcal{N}(0, I)$:

$z = \mu(x) + \sigma(x) \odot \epsilon$

Now, $z$ is expressed as a deterministic, differentiable function of $\mu$, $\sigma$, and input $x$, allowing gradients to backpropagate cleanly through the encoder parameters $\phi$.

3.3 Diffusion Models (Denoising Diffusion Probabilistic Models - DDPM)

Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) represent the state of the art in high-fidelity perceptual synthesis (powering systems like Stable Diffusion, Midjourney, and DALL-E 3). They model generation through two complementary Markov chains:

CODECopy Code
+---------------------------------------------------------------------------------------------------+
|                                  THE DIFFUSION PROCESS DYNAMICS                                   |
+---------------------------------------------------------------------------------------------------+
|  FORWARD (NOISING) PROCESS [ q(x_t | x_{t-1}) ]: Systematically injects Gaussian noise             |
|                                                                                                   |
|  [ x_0 (Clean Data) ] ---> [ x_1 ] ---> [ x_2 ] ---> ... ---> [ x_T (Pure Gaussian Noise) ]       |
|                                                                                                   |
|  <---------------------------------------------------------------------------------------------   |
|  REVERSE (DENOISING) PROCESS [ p_\theta(x_{t-1} | x_t) ]: Neural network iteratively removes noise |
+---------------------------------------------------------------------------------------------------+

1. Forward (Noising) Process $q(x_{1:T} \mid x_0)$: An analytical, parameter-free Markov chain that systematically destroys structure by adding tiny amounts of Gaussian noise at each timestep $t \in \{1, \dots, T\}$ according to a variance schedule $\beta_1, \dots, \beta_T$:

$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t I)$

Using the substitution $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^t \alpha_s$, any arbitrary step $x_t$ can be sampled directly from $x_0$ in closed form without iterating intermediate steps:

$x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)$

2. Reverse (Denoising) Process $p_\theta(x_{0:T})$: A learned parameterized reverse Markov chain. A deep neural network (typically a U-Net equipped with self-attention and cross-attention layers) is trained to estimate the noise $\epsilon_\theta(x_t, t)$ that was added at step $t$:

$\mathcal{L}_{\text{simple}}(\theta) = \mathbb{E}_{t, x_0, \epsilon}\left[ \|\epsilon - \epsilon_\theta(x_t, t)\|^2 \right]$

3. Latent Diffusion Models (LDMs): Standard diffusion in high-resolution pixel space (e.g., $1024 \times 1024 \times 3$) requires enormous computational memory. Latent Diffusion Models (such as Stable Diffusion) train a pre-trained VAE encoder to compress high-resolution images into a low-dimensional perceptual latent space. The diffusion denoising process operates entirely within this compressed latent space, reducing compute requirements by orders of magnitude while preserving fine visual textures upon final VAE decoding.
4. Classifier-Free Guidance (CFG): During text-to-image synthesis, the model generates an unconditional prediction $\epsilon_\theta(x_t, \emptyset)$ alongside a text-conditioned prediction $\epsilon_\theta(x_t, c)$. The final guided noise estimate interpolates between them using guidance scale $w$:

$\hat{\epsilon}_\theta(x_t, c) = \epsilon_\theta(x_t, \emptyset) + w \cdot (\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \emptyset))$

Higher values of $w$ force strict adherence to the prompt text at the expense of sample diversity.

4. Multimodal Generative AI & Diverse Modalities

Generative AI extends far beyond pure natural language processing, integrating multiple sensory modalities into cohesive reasoning and synthesis engines:

4.1 Vision-Language Models (VLMs) and Contrastive Alignment

Modern multimodal generation bridges language and vision using architectures pioneered by OpenAI's Contrastive Language-Image Pretraining (CLIP):

• CLIP trains an image encoder (Vision Transformer - ViT) and a text encoder simultaneously on hundreds of millions of (image, caption) pairs.
• By maximizing the cosine similarity of matching pairs while minimizing the similarity of mismatched pairs using a symmetric cross-entropy contrastive loss, CLIP constructs a shared multimodal embedding space where textual concepts and visual features map to identical geometric vectors.
• This shared space provides the conditioning signals used by diffusion models to generate photorealistic images from descriptive text prompts.

4.2 Image Editing Paradigms

• Inpainting: Replacing or synthesizing a specified masked region within an existing image while maintaining seamless contextual and lighting continuity with unmasked surroundings.
• Outpainting (Uncropping): Extending an image's canvas beyond its original borders, synthesizing new surroundings that logically match the artistic style, perspective, and contents of the central frame.

4.3 Code Synthesis and Software Engineering

Generative code models (e.g., Code LLaMA, StarCoder) are pretrained on billions of lines of open-source software repositories:

• Utilize specialized tokenizers preserving indentation, whitespace, and programming syntax delimiters.
• Implement Fill-In-the-Middle (FIM) training objectives: documents are partitioned into prefix, middle, and suffix chunks; the middle chunk is moved to the end, training the model to synthesize code that cleanly bridges existing prefixes and suffixes.

4.4 Scientific and Molecular Generation

In computational chemistry and structural biology:

• Diffusion models and generative transformers synthesize novel 3D molecular structures, crystal lattices, and candidate pharmaceutical compounds conditioned on target binding affinities.
• Systems like AlphaFold generate 3D protein structures directly from 1D amino acid sequences, revolutionizing drug discovery timelines.

5. Foundational Challenges, Limitations & Risks

Deploying generative AI systems requires addressing fundamental technological limitations and socio-technical risks:

5.1 The Hallucination Phenomenon

Hallucination refers to the generation of fluent, logically structured, and authoritative-sounding statements that are factually false, ungrounded, or completely fabricated.

• Root Cause: Large language models optimize statistical token likelihood ($p_\theta(x_t \mid x_{<t})$) based on surface patterns in training corpora; they possess no intrinsic world model, sensory perception, or ground-truth verification engine.
• Mitigation Techniques: Grounding models via Retrieval-Augmented Generation (RAG), tool-use execution (calculators, code interpreters, web search), and calibrating uncertainty estimation.

5.2 The "Stochastic Parrot" Critique

Articulated by Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell (2021), the Stochastic Parrot thesis posits that large language models perform statistical pattern synthesis by recombining linguistic forms without any genuine communicative intent, semantic comprehension, or real-world understanding:

• Models manipulate syntactic tokens based on probabilistic distributions, creating the illusion of reasoning without grasping the underlying meanings of the concepts they discuss.

5.3 Algorithmic Bias and Representation Harm

Generative models inherit and amplify the systemic societal, racial, gender, cultural, and historical biases embedded within raw, uncurated web training corpora:

• Text models may associate certain professions with specific genders or generate stereotypical characterizations.
• Image generators may default to producing images of specific demographics unless explicitly prompted otherwise.

5.4 Compute Footprint and Scaling Dynamics

• Training modern frontier foundation models requires tens of thousands of specialized accelerators (NVIDIA H100s/B200s, Google TPUs) operating continuously for months, consuming megawatts of electrical power and millions of gallons of cooling water.
• Chinchilla Scaling Laws (Hoffmann et al., 2022): Demonstrated that many early LLMs were significantly undertrained; for compute-optimal training, model parameter count and training token volume should scale in equal proportions (approximately 20 tokens per model parameter).