Unit 1: Foundations of LLMs and Agentic Systems

1. Paradigm Shift: From Statistical Language Models to Agentic AI

1.1 Evolution of Natural Language Processing

The evolution of natural language processing (NLP) and artificial intelligence can be traced through four fundamental eras:

Rule-Based Symbolic AI (1950s–1980s): Expert systems and formal grammars (e.g., context-free grammars, Chomsky hierarchies) relying on handcrafted heuristics and rigid symbolic manipulation.
Statistical Language Modeling (1990s–2000s): $N$ -gram language models computing conditional word probabilities using Markov assumptions:
$P(w_1, w_2, \dots, w_T) = \prod_{t=1}^T P(w_t \mid w_{t-n+1}, \dots, w_{t-1})$
Constrained by the curse of dimensionality and inability to capture long-range semantic dependencies beyond $n=3$ or $n=5$ .
Neural Representation Learning (2010s): Distributed vector representations (Word2Vec, GloVe) followed by recurrent neural architectures (RNNs, LSTMs, GRUs). While LSTMs introduced gating mechanisms to mitigate vanishing gradients, sequential token processing prevented parallelization across GPUs and suffered from information bottlenecks over long context windows.
Transformer-Based Large Language Models (2017–Present): Introduced by Vaswani et al. (2017), the self-attention mechanism discarded recurrence entirely, enabling massive parallelization across billions of parameters trained on multi-terabyte web corpora.

CODE
       CHRONOLOGICAL NLP PARADIGM EVOLUTION
  +------------------+     +--------------------+     +---------------------+
  |   Symbolic AI    | --> |    Statistical     | --> |    Recurrent /      |
  |  Expert Systems  |     |   N-Gram Models    |     |  LSTM Gated Networks|
  |  (Handcrafted)   |     | (Markov Chains)    |     |  (Sequential Bottl.)|
  +------------------+     +--------------------+     +---------------------+
                                                                 |
                                                                 v
  +------------------+     +--------------------+     +---------------------+
  |    Agentic AI    | <-- | Instruction-Tuned  | <-- | Transformer-Based   |
  | Action-Reasoning |     | Conversational LLM |     | Foundation LLMs     |
  | Decision Loops   |     | (RLHF Alignment)   |     | (Self-Attention)    |
  +------------------+     +--------------------+     +---------------------+

1.2 Defining Agentic AI: Generation vs. Autonomous Action

A foundational distinction exists between a standalone Large Language Model and an Agentic AI System:

CODE
+-----------------------------------------------------------------------------------------+
|                    STANDALONE LLM VS. AGENTIC AI DECISION SYSTEM                        |
+-----------------------------------------------------------------------------------------+
| Characteristic      | Standalone Large Language Model    | Agentic AI System            |
+---------------------+------------------------------------+------------------------------+
| Primary Objective   | Next-token sequence prediction     | Multi-step goal achievement  |
| Execution Topology  | Single-turn / Feed-forward         | Iterative Closed-Loop        |
| Environmental State | Stateless (Static context prompt)  | Stateful (Persistent memory) |
| Agency & Action     | Generates text passively           | Invokes tools, executes code |
| Feedback Handling   | None during inference generation   | Evaluates tool observation   |
| Error Recovery      | Hallucinates or produces errors    | Self-corrects via reflection |
+-----------------------------------------------------------------------------------------+

An agentic AI system embeds a foundation model within a continuous sense-plan-act loop:
$\text{Percept} \longrightarrow \text{State Update} \longrightarrow \text{Reasoning (Thought)} \longrightarrow \text{Action Selection} \longrightarrow \text{Environment Execution} \longrightarrow \text{Observation}$

2. Theory of Intelligent Agents: The PEAS Framework and Agent Taxonomy

In classical artificial intelligence (Russell & Norvig), an agent is defined as an entity that perceives its environment through sensors and acts upon that environment through actuators:
$f: \mathcal{P}^* \longrightarrow \mathcal{A}$
Where $\mathcal{P}^*$ represents the sequence of all historical percepts (percept history) and $\mathcal{A}$ denotes the set of possible actions.

2.1 The PEAS Framework

Systematic agent design requires formalizing the PEAS descriptor:

P — Performance Measure: The objective, quantifiable criterion used to evaluate the success of an agent's behavior (e.g., code compilation success, accuracy of flight bookings, task latency).
E — Environment: The external context and domain within which the agent operates (e.g., web browser DOM, bash terminal, SQL database).
A — Actuators: The mechanisms through which the agent alters environmental state (e.g., API requests, mouse clicks, SQL INSERT commands, bash execution).
S — Sensors: The data streams through which the agent observes environmental state (e.g., HTTP response payloads, stdout/stderr streams, DOM tree elements).

CODE
                      THE PERCEPT-ACTION AGENT LOOP
                     +-----------------------------+
                     |         ENVIRONMENT         |
                     +-----------------------------+
                        |                       ^
                 Percepts (Sensors)       Actions (Actuators)
                        |                       |
                        v                       |
                     +-----------------------------+
                     |      INTELLIGENT AGENT      |
                     |  - Internal State & Memory  |
                     |  - LLM Reasoning Engine     |
                     |  - Goal / Utility Function  |
                     +-----------------------------+

2.2 Russell & Norvig Agent Taxonomy

CODE
+-----------------------------------------------------------------------------------------+
|                              INTELLIGENT AGENT TAXONOMY                                 |
+-----------------------------------------------------------------------------------------+
| Agent Class         | Decision Mechanism                 | Environmental Requirement    |
+---------------------+------------------------------------+------------------------------+
| Simple Reflex       | Direct Condition-Action rules      | Fully observable             |
| Model-Based Reflex  | Internal state tracks unobserved   | Partially observable         |
| Goal-Based          | Evaluates actions towards a goal   | Planning / Search capability |
| Utility-Based       | Maximizes continuous utility curve | Trade-offs & preference opt  |
| Learning Agent      | Self-improves via critic feedback  | Dynamic / Uncertain domains  |
+-----------------------------------------------------------------------------------------+

Simple Reflex Agents: Act strictly based on the current percept, ignoring percept history. Driven by hardcoded IF condition THEN action rules. Fails completely in partially observable environments.
Model-Based Reflex Agents: Maintain an internal state to track aspects of the environment that cannot be viewed in the current percept (handling partial observability).
Goal-Based Agents: Combine internal state with explicit goal descriptions to evaluate which actions achieve the desired objective, enabling search and forward planning.
Utility-Based Agents: Map world states to real-valued utility functions ( $U: \mathcal{S} \to \mathbb{R}$ ). When multiple action sequences achieve a goal, utility functions balance trade-offs (e.g., speed vs. cost vs. safety).
Learning Agents: Divided into a learning element (improves performance over time), a critic (evaluates behavior against a performance standard), a learning goals generator (proposes exploratory actions), and an execution element (selects actions).

3. Environment Classifications and Interaction Dynamics

The complexity of designing an agent is directly governed by the properties of its operational environment:

CODE
+-----------------------------------------------------------------------------------------+
|                           ENVIRONMENTAL TAXONOMY MATRIX                                 |
+-----------------------------------------------------------------------------------------+
| Dimension           | Discrete Case                      | Continuous Case              |
+---------------------+------------------------------------+------------------------------+
| Observability       | Fully Observable (Chess board)     | Partially Observable (Poker) |
| Determinism         | Deterministic (Virtual compiler)   | Stochastic (Weather, Markets)|
| Temporal Dependency | Episodic (Image classification)    | Sequential (Chess, Robotics) |
| State Dynamism      | Static (Crossword puzzle)          | Dynamic (Stock trading floor)|
| Continuity          | Discrete (Turn-based game)         | Continuous (Self-driving car)|
| Agent Cardinality   | Single-Agent (Sudoku solver)       | Multi-Agent (E-commerce auc) |
+-----------------------------------------------------------------------------------------+

Fully vs. Partially Observable: If an agent's sensors provide access to the complete state of the environment at each point in time, it is fully observable; otherwise, sensor noise or hidden states introduce partial observability.
Deterministic vs. Stochastic: If the next environmental state is determined entirely by current state and agent action, it is deterministic. Stochastic environments involve inherent probabilistic unpredictability.
Episodic vs. Sequential: In episodic tasks, each action choice is isolated; current actions do not impact future episodes. In sequential tasks, current decisions have lasting ramifications on future states.
Static vs. Dynamic: A static environment does not change while the agent is deliberating; dynamic environments evolve continuously during agent reasoning loops.

4. Transformer Architecture: Sequence Modeling Foundations

Introduced by Vaswani et al. in "Attention Is All You Need" (2017), the Transformer discarded recurrence and convolution, relying entirely on attention mechanisms to model global dependencies between input and output tokens.

CODE
                  TRANSFORMER DECODER-ONLY ARCHITECTURE
                                +--------------+
                                | Output Token |
                                +--------------+
                                       ^
                                       | Linear Projection (LM Head)
                                +--------------+
                                | RMSNorm / LN |
                                +--------------+
                                       ^
                 +---------------------+---------------------+
                 |                                           |
                 |      +-----------------------------+      |
                 |      | SwiGLU / MLP Feed-Forward   |      |
                 |      +-----------------------------+      |
                 |                     ^                     |
                 |                     | Residual Connection |
                 |      +-----------------------------+      |
                 |      | RMSNorm / Pre-LayerNorm     |      |
                 |      +-----------------------------+      |
                 |                     ^                     |
                 |      +-----------------------------+      |
                 |      | Masked Causal Self-Attention|      |
                 |      +-----------------------------+      |
                 |                     ^                     |
                 |                     | Residual Connection |
                 |      +-----------------------------+      |
                 |      | RMSNorm / Pre-LayerNorm     |      |
                 |      +-----------------------------+      |
                 +---------------------+---------------------+  (x N Layers)
                                       ^
                                       |
                       +-------------------------------+
                       | Input Tokens + Positional Enc |
                       | (RoPE / Learned Embeddings)   |
                       +-------------------------------+

4.1 Input Embeddings and Positional Encoding

Because self-attention operations are permutation-equivariant (they compute pairwise token relationships independent of order), the architecture must inject explicit positional information:

Sinusoidal Positional Embeddings (Original Transformer): $PE_{(pos, 2i)} &= \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) \\ PE_{(pos, 2i+1)} &= \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right) \end{aligned}$$$
Rotary Position Embedding (RoPE):
Adopted by modern foundation models (LLaMA, Mistral, Gemma). Rather than adding a position vector to word embeddings, RoPE rotates the Query and Key vectors in complex space by an angle proportional to their sequence position $m$ :
$R_{\Theta, m}^d = \text{diag}\left( R_{\theta_1, m}, R_{\theta_2, m}, \dots, R_{\theta_{d/2}, m} \right)$
Where the inner product between Query at position $m$ and Key at position $n$ depends strictly on relative distance $m - n$ :
$\langle R_m q, R_n k \rangle = g(q, k, m - n)$
Enabling seamless extrapolation to long context windows.

4.2 Normalization Topologies: Post-LN vs. Pre-LN vs. RMSNorm

Post-LayerNorm (Original 2017 Transformer): $\text{Output} = \text{LayerNorm}(x + \text{SubLayer}(x))$ . Suffered from severe gradient vanishing/exploding during deep initialization, requiring warmups.
Pre-LayerNorm: $\text{Output} = x + \text{SubLayer}(\text{LayerNorm}(x))$ . Stabilizes gradient flow, enabling training of models with hundreds of layers.
RMSNorm (Root Mean Square Normalization): Omits mean-centering from standard LayerNorm, scaling activations strictly by root-mean-square magnitude:
$\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^d x_i^2 + \epsilon}} \odot \gamma$
Saves ~7% of computational runtime per layer without degrading training convergence.

5. Attention Mechanics: Mathematical Formulations and Variants

5.1 Scaled Dot-Product Attention

Given input representations, the model projects activations into Queries ( $Q$ ), Keys ( $K$ ), and Values ( $V$ ) via learned weight matrices $W^Q, W^K, W^V \in \mathbb{R}^{d_{\text{model}} \times d_k}$ :

$\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} + M \right) V$

The Scaling Factor $1/\sqrt{d_k}$ : When dimensionality $d_k$ is large, dot products $q \cdot k$ grow large in magnitude, pushing the softmax function into regions with extremely small gradients ( $\approx 0$ ), causing gradient vanishing. Dividing by $\sqrt{d_k}$ preserves unit variance.
Causal Mask ( $M$ ): In autoregressive decoder-only models, future tokens are masked out to preserve causality:
$M_{i,j} = \begin{cases} 0 & \text{if } i \ge j \\ -\infty & \text{if } i < j \end{cases}$

5.2 Attention Variants and Memory Optimization

CODE
+-----------------------------------------------------------------------------------------+
|                               ATTENTION TOPOLOGY MATRIX                                 |
+-----------------------------------------------------------------------------------------+
| Topology                | Query Heads | Key/Value Heads | KV-Cache Memory | Inference Lat|
+-------------------------+-------------+-----------------+-----------------+--------------+
| Multi-Head (MHA)        | $H$ heads   | $H$ heads       | $1.0\times$     | Baseline     |
| Multi-Query (MQA)       | $H$ heads   | $1$ single head | $1/H\times$     | Fastest      |
| Grouped-Query (GQA)     | $H$ heads   | $G$ groups      | $G/H\times$     | Optimal Bal. |
+-----------------------------------------------------------------------------------------+

CODE
       MHA (Multi-Head)              GQA (Grouped-Query)           MQA (Multi-Query)
   Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8       Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8       Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
   |  |  |  |  |  |  |  |        \  /  \  /  \  /  \  /        \   \   |   /   /   /
   K1 K2 K3 K4 K5 K6 K7 K8         K1    K2    K3    K4                     K1
   V1 V2 V3 V4 V5 V6 V7 V8         V1    V2    V3    V4                     V1
  (8 KV Heads for 8 Q)          (4 KV Heads for 8 Q)          (1 KV Head for 8 Q)

Multi-Head Attention (MHA): Each Query head has an independent Key and Value head. Requires massive VRAM for the KV cache during autoregressive generation.
Multi-Query Attention (MQA): All Query heads share a single Key and Value head, reducing KV-cache size by $H\times$ at a slight cost in capacity.
Grouped-Query Attention (GQA): Divides Query heads into $G$ groups, where each group shares a single Key/Value head. Adopted by LLaMA-2/3, Mistral, and Claude as the gold standard for high-throughput serving.
FlashAttention: An IO-aware exact attention algorithm (Dao et al.) that tiles matrix multiplication across GPU SRAM, avoiding slow High Bandwidth Memory (HBM) round trips and cutting memory footprint from $O(N^2)$ to $O(N)$ .

6. Text Tokenization: Algorithms and Vocabulary Design

Tokenization converts continuous natural language strings into discrete integer sequences processed by neural networks.

CODE
+-----------------------------------------------------------------------------------------+
|                           TOKENIZATION ALGORITHM COMPARISON                             |
+-----------------------------------------------------------------------------------------+
| Algorithm          | Vocabulary Building Strategy               | Primary Usage         |
+--------------------+--------------------------------------------+-----------------------+
| Byte-Pair Encoding | Bottom-up frequency-based character merge  | GPT-4, LLaMA, RoBERTa |
| WordPiece          | Bottom-up likelihood-maximization merge    | BERT, DistilBERT      |
| Unigram / SentenceP| Top-down pruning from large candidate set  | T5, ALBERT, Gemma     |
+-----------------------------------------------------------------------------------------+

6.1 Byte-Pair Encoding (BPE)

BPE begins at the character or byte level and iteratively merges the most frequently co-occurring adjacent token pair:

Initialize vocabulary $V$ with all unique atomic characters/bytes.
Segment the training corpus into individual characters.
Count frequency of all adjacent pairs $(t_i, t_{i+1})$ .
Identify pair with maximum frequency:
$(t_A, t_B) = \arg\max_{(u, v)} \text{Freq}(u, v)$
Add new merged token $t_{AB}$ to vocabulary $V$ .
Repeat until target vocabulary size $|V|$ (typically 32,000 to 128,000 tokens) is reached.

6.2 Tokenization Boundary Effects and Anomalies

Whitespace Sensitivity: In tokenizers like cl100k_base, " world" and "world" map to distinct token IDs, causing unexpected prompt variations.
Arithmetic Vulnerabilities: Standard tokenizers break numbers into irregular subword chunks (e.g., 123456 $\to$ 12, 34, 56), impairing multi-digit mathematical reasoning.
Multilingual Token Inefficiency: Tokenizers dominated by English text fragment non-Latin scripts (Hindi, Arabic, Chinese) into multiple bytes per character, inflating token counts and raising API inference costs.

7. LLM Pre-Training and Compute Scaling Laws

7.1 Autoregressive Causal Language Modeling

Foundation LLMs are pre-trained via self-supervised learning on massive web-scale corpora using the Causal Language Modeling (CLM) cross-entropy loss:

$\mathcal{L}_{\text{CLM}}(\theta) = -\frac{1}{T} \sum_{t=1}^T \log P(x_t \mid x_1, x_2, \dots, x_{t-1}; \theta)$

Where the conditional probability distribution is computed via the linear projection LM head and temperature-scaled softmax:
$P(x_t = w_i \mid x_{<t}) = \frac{\exp(z_i / \tau)}{\sum_{j=1}^{|V|} \exp(z_j / \tau)}$

7.2 Compute Scaling Laws: Kaplan vs. Chinchilla

CODE
+-----------------------------------------------------------------------------------------+
|                        SCALING LAWS: COMPUTE BUDGET ALLOCATION                          |
+-----------------------------------------------------------------------------------------+
| Formulation         | Finding / Rule of Thumb                    | Implication          |
+---------------------+--------------------------------------------+----------------------+
| Kaplan et al. (2020)| Loss scales primarily with parameter size; | Led to under-trained |
|                     | $N$ should scale faster than tokens $D$    | giants (GPT-3 175B)  |
+---------------------+--------------------------------------------+----------------------+
| Chinchilla (2022)   | Compute-optimal training: $N$ and $D$      | Models should train  |
| Hoffmann et al.     | must scale in equal 1:1 proportion         | on ~20 tokens/param  |
+---------------------+--------------------------------------------+----------------------+

Hoffmann et al. (DeepMind) demonstrated that for optimal compute budget $C \approx 6ND$ :
$N_{\text{opt}} \propto C^a, \quad D_{\text{opt}} \propto C^b \quad (\text{where } a \approx b \approx 0.5)$
Consequently, a 70-billion parameter model requires at least 1.4 trillion tokens to be compute-optimal. Modern foundation models (e.g., LLaMA-3) train on over 15 trillion tokens, confirming that over-training past the Chinchilla boundary significantly reduces inference latency costs.

8. Agentic Control Loops: ReAct, Planning, and Error Recovery

An agent translates raw next-token prediction into coordinated action through explicit reasoning loops.

8.1 The ReAct Framework (Reasoning + Acting)

Introduced by Yao et al. (2022), ReAct interleaves reasoning traces ("Thoughts") with environmental actions ("Actions") and tool execution outputs ("Observations"):

CODE
                      THE REACT EXECUTION LOOP
  [User Goal / Task Description]
                 |
                 v
  +---------------------------------------------------------+
  | Thought: Analyze current state and plan immediate step |
  +---------------------------------------------------------+
                 |
                 v
  +---------------------------------------------------------+
  | Action: Invoke external tool with arguments (e.g., API) |
  +---------------------------------------------------------+
                 |
                 v
  +---------------------------------------------------------+
  | Observation: Receive return payload from environment    |
  +---------------------------------------------------------+
                 |
                 v
  [Check: Is Goal Achieved?]
     |-- NO  --> Return to "Thought" with new context
     |-- YES --> Output Final Answer to User

Illustrative ReAct Execution Trace

text
Question: What is the current stock price of Apple divided by the price of Microsoft?
Thought 1: I need to find the current stock price of Apple (AAPL).
Action 1: StockSearch[AAPL]
Observation 1: Apple Inc. (AAPL) is currently trading at $180.00.
Thought 2: Now I need to find the current stock price of Microsoft (MSFT).
Action 2: StockSearch[MSFT]
Observation 2: Microsoft Corp. (MSFT) is currently trading at $400.00.
Thought 3: Now I will divide 180 by 400.
Action 3: Calculator[180 / 400]
Observation 3: 0.45
Thought 4: I have obtained the final numerical answer.
Action 4: Finish[0.45]

8.2 Loop Guards and Failure Modes

Autonomous agentic loops risk catastrophic failure without architectural safeguards:

Infinite Execution Loops: The agent repeatedly takes the same invalid action. Mitigated by maximum step limits ( $N \le 25$ ) and cycle detection hashes over state actions.
Context Window Exhaustion: Long tool observation dumps fill context memory. Mitigated by rolling summarization buffers and vector retrieval filters.
Cascading Hallucinations: An early hallucination pollutes observation history, derailing subsequent planning. Mitigated by self-reflection loops and critic verification nodes.

9. Major Foundation Model Ecosystems

CODE
+---------------------------------------------------------------------------------------------------------+
|                               FOUNDATION MODEL ECOSYSTEM COMPARISON                                     |
+---------------------------------------------------------------------------------------------------------+
| Model Family       | Primary Organization | Architecture Type     | Context Window | Access Model       |
+--------------------+----------------------+-----------------------+----------------+--------------------+
| GPT-4 / GPT-4o     | OpenAI               | Dense / MoE Mixture   | 128k tokens    | Proprietary API    |
| Claude 3 / 3.5     | Anthropic            | Dense Transformer     | 200k tokens    | Proprietary API    |
| Gemini 1.5 Pro     | Google DeepMind      | Mixture of Experts    | 2M+ tokens     | Proprietary API    |
| LLaMA 3 / 3.1      | Meta AI              | Dense Decoder (GQA)   | 128k tokens    | Open-Weights       |
| Mixtral 8x7B/8x22B | Mistral AI           | Sparse MoE (Top-2)    | 32k - 64k      | Open-Weights       |
+---------------------------------------------------------------------------------------------------------+

Modern agentic engineering leverages both proprietary frontier APIs (offering top reasoning benchmarks) and open-weights models (offering local deployment, fine-tuning, data privacy, and zero inference throttling).

Large Language Models And Agentic Ai