Unit 1: Introduction to NLP and Text Processing

1. Introduction to Natural Language Processing (NLP)

1.1 Definition, Scope, and Evolution

Natural Language Processing (NLP) is an interdisciplinary subfield of artificial intelligence, computer science, and computational linguistics concerned with enabling computational systems to understand, interpret, manipulate, and generate human language in a valuable, contextually appropriate manner.

Human language is inherently symbolic, unstructured, compositional, and nuanced. The core objective of NLP is bridging the divide between human communication (fuzzy, ambiguous, evolving) and computational processing (deterministic, structured, binary).

CODE
+----------------------------------------------------------------------------------------------------+
|                                    THE NLP INTERDISCIPLINARY INTERSECTION                          |
+----------------------------------------------------------------------------------------------------+
|                                                                                                    |
|            ARTIFICIAL INTELLIGENCE                             COMPUTATIONAL LINGUISTICS           |
|            (Machine Learning, Deep Learning,                   (Syntax, Morphology, Semantics,     |
|             Knowledge Representation)                           Phonetics, Discourse Analysis)     |
|                               \                                       /                            |
|                                \                                     /                             |
|                                 v                                   v                              |
|                           +-----------------------------------------------+                        |
|                           |         NATURAL LANGUAGE PROCESSING (NLP)     |                        |
|                           +-----------------------------------------------+                        |
|                                                 ^                                                  |
|                                                 |                                                  |
|                                     COMPUTER SCIENCE & STATISTICS                                  |
|                                     (Algorithms, Data Structures, Information                      |
|                                      Theory, High-Performance Computing)                          |
|                                                                                                    |
+----------------------------------------------------------------------------------------------------+

1.2 Historical Milestones in NLP

Rule-Based & Symbolic Era (1950s - 1980s):
- 1950: Alan Turing publishes "Computing Machinery and Intelligence," proposing the Turing Test as a benchmark for machine intelligence.
- 1954: The Georgetown-IBM experiment successfully translates over sixty Russian sentences into English using a hardcoded dictionary and rule set.
- 1966: Joseph Weizenbaum develops ELIZA, an early conversational agent simulating a Rogerian psychotherapist via pattern matching.
- Noam Chomsky's Syntactic Structures (1957): Introduces formal language theory, generative grammars, and the Chomsky hierarchy.
Statistical & Empirical Era (1990s - 2010s):
- Shift from hand-crafted expert linguistic rules to statistical language models trained on large text corpora.
- Introduction of Hidden Markov Models (HMMs) for Part-of-Speech (POS) tagging and speech recognition, alongside statistical machine translation (IBM Models).
Deep Learning & Transformer Revolution (2013 - Present):
- 2013: Mikolov et al. introduce Word2Vec, establishing dense distributed vector embeddings.
- 2014-2015: Recurrent Neural Networks (RNNs) and LSTMs dominate sequence modeling.
- 2017: Vaswani et al. publish "Attention Is All You Need," introducing the Transformer architecture.
- 2018-Present: Pre-trained Large Language Models (BERT, GPT, T5, LLaMA) redefine state-of-the-art NLP through self-supervised pre-training and fine-tuning.

2. Core Linguistic Levels and Challenges

CODE
+----------------------------------------------------------------------------------------------------+
|                                    LEVELS OF LINGUISTIC ANALYSIS                                   |
+-------------------+---------------------------------------+----------------------------------------+
| LEVEL             | FOCUS OF STUDY                        | COMPUTATIONAL TASK / CHALLENGE         |
+-------------------+---------------------------------------+----------------------------------------+
| 1. Phonetics &    | Physical sounds of speech and their   | Speech-to-text, acoustic modeling,     |
|    Phonology      | cognitive sound patterns.             | homophones ("two", "to", "too").       |
+-------------------+---------------------------------------+----------------------------------------+
| 2. Morphology     | Internal structure and formation of   | Stemming, lemmatization, morpheme      |
|                   | words from minimal meaningful units.  | segmentation (roots, prefixes, suffixes)|
+-------------------+---------------------------------------+----------------------------------------+
| 3. Syntax         | Structural rules governing how words  | Parsing, POS tagging, Context-Free     |
|                   | combine to form grammatical sentences.| Grammars (CFGs), dependency parsing.   |
+-------------------+---------------------------------------+----------------------------------------+
| 4. Semantics      | Literal meaning of individual words,  | Word Sense Disambiguation (WSD),       |
|                   | phrases, and complete sentences.      | semantic role labeling, named entities.|
+-------------------+---------------------------------------+----------------------------------------+
| 5. Pragmatics &   | How context, speaker intent, and real-| Sarcasm detection, coreference         |
|    Discourse      | world knowledge affect meaning.       | resolution, dialogue act modeling.     |
+-------------------+---------------------------------------+----------------------------------------+

2.1 The Pervasive Challenge of Ambiguity

The fundamental obstacle in automated natural language processing is Ambiguity across every linguistic level:

CODE
+------------------------------------------------------------------------------------+
|                               TYPES OF AMBIGUITY IN NLP                            |
+---------------------+--------------------------------------------------------------+
| AMBIGUITY TYPE      | EXPLANATORY EXAMPLE & CHALLENGE                              |
+---------------------+--------------------------------------------------------------+
| Lexical Ambiguity   | "I went to the bank." -> Financial institution vs. riverbank.|
| (Polysemy/Homonymy) | Requires Word Sense Disambiguation (WSD).                    |
+---------------------+--------------------------------------------------------------+
| Syntactic /         | "I saw the man with a telescope." -> Who has the telescope?  |
| Structural          | Generates multiple distinct parse trees.                     |
+---------------------+--------------------------------------------------------------+
| Semantic Ambiguity  | "Every student loves a teacher." -> One universal teacher,   |
| (Quantifier Scope)  | or does every student love a different teacher?              |
+---------------------+--------------------------------------------------------------+
| Referential /       | "The trophy didn't fit into the brown suitcase because it was|
| Anaphoric           | too big." -> Does "it" refer to trophy or suitcase? (Winograd)|
+---------------------+--------------------------------------------------------------+
| Pragmatic Ambiguity | "Can you pass the salt?" -> Literal inquiry into physical     |
| (Speech Acts)       | capability vs. polite social request for action.             |
+---------------------+--------------------------------------------------------------+

3. Linguistic Essentials: Morphology, Syntax & Semantics

3.1 Morphology

Morphology is the study of word formation and the internal structure of words:

Morpheme: The minimal, indivisible linguistic unit carrying grammatical or semantic meaning.
- Free Morpheme: Can stand alone as an independent word (e.g., dog, run, happy).
- Bound Morpheme: Cannot stand alone; must be attached to a root (e.g., prefixes like un-, suffixes like -ness, -ing, -s).
Inflectional vs. Derivational Morphology:
- Inflectional: Alters the grammatical form of a word (tense, number, person, case) without changing its fundamental core meaning or grammatical part of speech (e.g., walk $o$ walks, walked, walking; dog $o$ dogs).
- Derivational: Creates an entirely new lexeme, often changing the grammatical category (e.g., happy (adj) $o$ unhappy (adj) $o$ happiness (noun); compute (verb) $o$ computer (noun)).

3.2 Syntax and Formal Grammars

Syntax dictates the hierarchical structural arrangement of words into valid grammatical sentences.

Context-Free Grammar (CFG): A formal four-tuple $G = (V, \Sigma, R, S)$ $G = (V, Σ, R, S)$ where:
- $V$ : Finite set of non-terminal characters (e.g., $S, NP, VP, PP$ ).
- $\Sigma$ : Finite set of terminal characters / words (the lexicon: the, dog, barked).
- $R$ : Finite set of production rules of the form $A o lpha$ where $A \in V$ and $lpha \in (V \cup \Sigma)^*$ .
- $S$ : Designated start symbol ( $S \in V$ ).

CODE
Example CFG Rules:
  S  -> NP VP
  NP -> Det N | Det Adj N | NP PP
  VP -> V NP | V | VP PP
  PP -> P NP

Parse Trees: Hierarchical tree structures visualizing syntactic derivation:
- Constituency Parsing: Breaks sentences into nested phrase constituents (Noun Phrases, Verb Phrases).
- Dependency Parsing: Direct head-dependent grammatical relations (subject, direct object, modifier) between individual word tokens.

3.3 Lexical Semantics & WordNet Relations

Semantics concerns literal word and sentence meanings. Words relate to each other through structured lexical relations:

Synonymy: Words with identical or closely related senses (automobile / car).
Antonymy: Words with opposite meanings (hot / cold).
Hypernymy (Superordinate): An umbrella class category (animal is a hypernym of dog).
Hyponymy (Subordinate): A specific instance of a category (dog is a hyponym of animal).
Meronymy (Part-Whole): Part of a whole entity (wheel is a meronym of car).
Holonymy (Whole-Part): The complete entity containing parts (car is a holonym of wheel).

4. Text Preprocessing Pipeline

Raw textual data scraped from the web, databases, or social media is inherently noisy, inconsistent, and unstructured. Text Preprocessing transforms raw characters into standardized, clean linguistic tokens suitable for mathematical vectorization:

CODE
+----------------------------------------------------------------------------------------------------+
|                                    TEXT PREPROCESSING PIPELINE                                     |
+----------------------------------------------------------------------------------------------------+
|                                                                                                    |
|  [RAW TEXT]                                                                                        |
|  "The cats jumped over 3 lazzy dogs!! https://example.com"                                         |
|      |                                                                                             |
|      v                                                                                             |
|  1. TEXT CLEANING & NORMALIZATION                                                                  |
|     - Strip HTML tags, URLs, emojis, and non-ASCII characters.                                     |
|     - Case folding: lowercase conversion ("The" -> "the").                                         |
|      |                                                                                             |
|      v                                                                                             |
|  2. TOKENIZATION                                                                                   |
|     - Segment text into discrete lexical tokens:                                                   |
|       ["the", "cats", "jumped", "over", "lazzy", "dogs"]                                           |
|      |                                                                                             |
|      v                                                                                             |
|  3. STOPWORD REMOVAL                                                                               |
|     - Filter uninformative grammatical function words ("the", "over"):                             |
|       ["cats", "jumped", "lazzy", "dogs"]                                                          |
|      |                                                                                             |
|      v                                                                                             |
|  4. STEMMING / LEMMATIZATION                                                                       |
|     - Reduce inflections to canonical base forms:                                                  |
|       Stemming:      ["cat", "jump", "lazzi", "dog"]                                               |
|       Lemmatization: ["cat", "jump", "lazy",  "dog"]                                               |
|      |                                                                                             |
|      v                                                                                             |
|  [STRUCTURED TOKEN LIST / VECTOR SPACE MODEL INPUT]                                                |
|                                                                                                    |
+----------------------------------------------------------------------------------------------------+

4.1 Tokenization

Tokenization is the task of chopping character streams into discrete meaningful chunks called tokens (words, numbers, punctuation symbols):

Word Tokenization: Splitting by whitespace and punctuation delimiters.
- Challenges: Contractions (don't $o$ do + n't vs. dont), hyphenated compounds (state-of-the-art), abbreviations (U.S.A., Dr.), non-segmented languages (Chinese, Japanese).
Sentence Tokenization (Sentence Boundary Disambiguation): Splitting running paragraphs into sentences using period, exclamation, and question delimiters while avoiding mid-sentence abbreviation periods.
Subword Tokenization (Modern LLMs): Breaks words into frequent character n-grams to eliminate Out-Of-Vocabulary (OOV) tokens:
- Byte-Pair Encoding (BPE): Iteratively merges most frequent adjacent character pairs (GPT family).
- WordPiece: Maximizes likelihood of training data merges (BERT).
- SentencePiece: Language-agnostic subword tokenizer treating whitespace as a regular character (_).

4.2 Text Normalization & Cleaning

Case Folding: Converting text to lowercase. Reduces vocabulary sparsity ("Apple" and "apple" map to the same token). Exception: Case is preserved in Named Entity Recognition where capitalization distinguishes proper nouns ("Apple Inc." vs. "apple" fruit).
Noise Removal: Stripping HTML tags, URLs, email addresses, escape characters, and non-alphanumeric noise using Regular Expressions (Regex).

4.3 Stopword Filtering

Stopwords: High-frequency functional grammatical words (the, is, at, which, on, a, and) that carry negligible semantic domain information.
When to Remove: Beneficial in Information Retrieval (search index compression), topic modeling (LDA), and document classification to reduce dimensional noise.
When to Retain: Must never be removed in machine translation, question answering, conversational dialogue, or sentiment analysis (e.g., removing not completely inverts sentence sentiment: "not good" becomes "good").

4.4 Stemming vs. Lemmatization

Both techniques reduce inflectional forms of a word to a common base, but operate on fundamentally different algorithmic principles:

CODE
+----------------------------------------------------------------------------------------------------+
|                                    STEMMING vs. LEMMATIZATION                                      |
+-------------------+---------------------------------------+----------------------------------------+
| FEATURE           | STEMMING                              | LEMMATIZATION                          |
+-------------------+---------------------------------------+----------------------------------------+
| Mechanism         | Heuristic rule-based suffix stripping | Morphological analysis backed by       |
|                   | without vocabulary dictionaries.      | lexical dictionaries (e.g. WordNet).   |
+-------------------+---------------------------------------+----------------------------------------+
| Output            | Crude 'stem' (often NOT a valid       | Valid dictionary 'lemma' (canonical    |
|                   | dictionary root word).                | base form).                            |
+-------------------+---------------------------------------+----------------------------------------+
| POS Awareness     | Context-blind; ignores part of speech.| Context-aware; requires Part-of-Speech |
|                   |                                       | tag to resolve correctly.              |
+-------------------+---------------------------------------+----------------------------------------+
| Examples          | "running"  -> "run"                   | "running" (verb) -> "run"              |
|                   | "flies"    -> "fli"                   | "flies" (noun)   -> "fly"              |
|                   | "better"   -> "better"                | "better" (adj)   -> "good"             |
+-------------------+---------------------------------------+----------------------------------------+
| Computational     | Extremely fast, low memory footprint. | Slower, requires vocabulary lookup and |
| Cost              |                                       | morphological POS tagging.             |
+-------------------+---------------------------------------+----------------------------------------+
| Common Algorithms | Porter Stemmer, Snowball, Lancaster.  | WordNet Lemmatizer, spaCy lemmatizer.  |
+-------------------+---------------------------------------+----------------------------------------+

5. Statistical Representation: Bag of Words & TF-IDF

5.1 N-Grams

An N-gram is a contiguous sequence of $n$ items (words, characters, or syllables) extracted from a text string:

Unigram ( $n=1$ ): ["the", "data", "science", "pipeline"]
Bigram ( $n=2$ ): ["the data", "data science", "science pipeline"]
Trigram ( $n=3$ ): ["the data science", "data science pipeline"]
Utility: Bigrams and trigrams capture local context and word ordering that unigrams discard (e.g., capturing negation: "not happy" as a single bigram token).

5.2 The Bag of Words (BoW) Model

The Bag of Words (BoW) model represents text as an unordered collection of words, disregarding grammar, word order, and syntax, retaining solely frequency counts.

Constructing a Document-Term Matrix (DTM):
1. Build a global vocabulary $V$ containing all unique terms across a corpus of $N$ documents.
2. Represent each document $d$ as a vector of length $|V|$ where vector index $i$ stores the occurrence frequency of term $t_i$ .
Limitations:
- Extreme Sparsity: DTM vectors consist overwhelmingly of zeros (high dimensional waste).
- Complete Loss of Context: "Dog bites man" and "Man bites dog" yield identical BoW feature vectors.
- Frequency Bias: Long documents have inflated counts, and uninformative high-frequency terms dominate.

5.3 TF-IDF (Term Frequency - Inverse Document Frequency)

TF-IDF mitigates the shortcomings of raw Bag-of-Words by weighting terms according to how informative and discriminative they are across a document collection.

CODE
+----------------------------------------------------------------------------------------------------+
|                                    TF-IDF MATHEMATICAL FOUNDATIONS                                 |
+----------------------------------------------------------------------------------------------------+
|                                                                                                    |
|    1. TERM FREQUENCY (TF):                                                                         |
|       Measures how frequently term t appears in document d.                                        |
|                                                                                                    |
|                                   f(t, d)                                                          |
|                  TF(t, d) = --------------------                                                   |
|                             sum_{t' in d} f(t', d)                                                 |
|                                                                                                    |
|    2. INVERSE DOCUMENT FREQUENCY (IDF):                                                            |
|       Measures how rare or informative term t is across the entire corpus D of N documents.        |
|                                                                                                    |
|                                           N                                                        |
|                  IDF(t, D) = ln [ ------------------ ]                                             |
|                                   1 + |{d in D : t in d}|                                          |
|                                                                                                    |
|       * Terms appearing in EVERY document have near-zero IDF (e.g., ln(1) = 0).                    |
|       * Rare, discriminative keywords have high IDF.                                               |
|                                                                                                    |
|    3. COMPOSITE TF-IDF WEIGHT:                                                                     |
|                                                                                                    |
|                  TF-IDF(t, d, D) = TF(t, d) * IDF(t, D)                                            |
|                                                                                                    |
+----------------------------------------------------------------------------------------------------+

5.4 Worked Numerical Example of TF-IDF

Consider a corpus of $N = 3$ documents:

$D_1$ : "NLP is fascinating" (Length: 3 words)
$D_2$ : "NLP is powerful and NLP is fast" (Length: 7 words)
$D_3$ : "Machine learning is powerful" (Length: 4 words)

Task: Compute TF-IDF for the term "NLP" in document $D_2$ .

Term Frequency (TF):
- Count of "NLP" in $D_2$ : $f( ext{"NLP"}, D_2) = 2$ .
- Total words in $D_2$ : $7$ .
  $ext{TF}( ext{"NLP"}, D_2) = rac{2}{7} pprox 0.2857$
Inverse Document Frequency (IDF):
- Total documents: $N = 3$ .
- Documents containing "NLP": $D_1$ and $D_2$ ( $|\{d : t \in d\}| = 2$ ).
- Using standard smoothed formula $ext{IDF} = \ln\left( rac{3}{2} ight) pprox \ln(1.5) pprox 0.4055$ .
TF-IDF Score:
$ext{TF-IDF}( ext{"NLP"}, D_2) = 0.2857 imes 0.4055 pprox 0.1158$

6. Common Pitfalls and Exam Tips

Stemming vs. Lemmatization: Stemming strips suffixes heuristically (e.g., flies $o$ fli); Lemmatization uses a dictionary and POS tag to return a valid base word (e.g., flies (noun) $o$ fly).
Stopword Hazards: Never blindly remove stopwords in sentiment analysis or translation; eliminating not or never completely reverses polarity.
TF-IDF Weighting Logic: Common words (like the, is) get high TF but zero/near-zero IDF, yielding a low TF-IDF score. Rare, topic-specific words get moderate TF and high IDF, yielding the highest TF-IDF score.
Constituency vs. Dependency: Constituency trees group words into phrasal blocks ( $NP, VP$ ); Dependency trees map direct head-dependent grammatical links between individual words.