Unit 1: Data Pre-processing

1. Introduction to Data Pre-processing Pipeline & The GIGO Axiom

Data pre-processing forms the foundational substrate upon which all predictive machine learning workflows are constructed. In industrial and enterprise environments, raw observational data is seldom collected in clean, structured, or mathematically consistent forms. Instead, it is characteristically corrupted by missing values, sensor failures, transmission latency, human recording discrepancies, non-standard scales, high cardinality categorical representations, and multi-scale distributional skewness.

1. The Principle of "Garbage In, Garbage Out" (GIGO)

The GIGO axiom articulates an immutable rule of computational learning theory: The statistical upper bound on a machine learning model's predictive performance is fundamentally governed by the informational quality, integrity, and representation of its training data. Regardless of algorithmic complexity—whether deploying regularized linear classifiers, gradient boosted decision ensembles, or deep multilayer perceptrons—a model trained on corrupted, biased, unscaled, or noisy data will fail to generalize to unseen test instances.

CODE
+-----------------------------------------------------------------------------------------+
|                         END-TO-END DATA PRE-PROCESSING PIPELINE                         |
+-----------------------------------------------------------------------------------------+
|                                                                                         |
|   +-----------------------+        +-----------------------+                            |
|   |   Raw Data Ingestion  |  ===>  |     Data Cleaning     |                            |
|   |  (CSV, SQL, NoSQL,    |        | - Missing Value Imput.|                            |
|   |   Sensors, Streams)   |        | - Outlier Rectificat. |                            |
|   +-----------------------+        | - Deduplication       |                            |
|                                    +-----------------------+                            |
|                                                |                                        |
|                                                v                                        |
|   +-----------------------+        +-----------------------+                            |
|   | Feature Engineering   |  <===  |   Data Transformation |                            |
|   | - Interaction Terms   |        | - Standardization/Norm|                            |
|   | - Domain Synthesis    |        | - Categorical Encod.  |                            |
|   | - Polynomial Features |        | - Power Transforms    |                            |
|   +-----------------------+        +-----------------------+                            |
|               |                                                                         |
|               v                                                                         |
|   +-----------------------+        +-----------------------+                            |
|   |   Data Reduction &    |  ===>  |  Model-Ready Feature  |                            |
|   |   Class Balancing     |        |   Matrix (X, y)       |                            |
|   | - SMOTE / Sampling    |        |  Passed to Estimator  |                            |
|   | - PCA / Feature Selec.|        +-----------------------+                            |
|   +-----------------------+                                                             |
+-----------------------------------------------------------------------------------------+

2. Core Stages of the Pre-processing Pipeline

Data Cleaning: Identification and remediation of erroneous records, missing field imputation, resolution of contradictory entries, and removal of duplicate or uninformative observations.
Data Integration: Merging schemas, resolving entity identification conflicts across heterogeneous relational tables, and aligning timestamps.
Data Transformation: Numerical scaling, non-linear variance stabilization (e.g., logarithmic and power transformations), and mapping non-numeric attributes into vector spaces.
Data Reduction: Compressing dimensional volume via feature selection and extraction while preserving primary variance components.
Class Imbalance Rectification: Adjusting sample distributions or algorithmic loss functions to counter severe class disparity.

2. Stevens' Typology of Measurement Levels

In 1946, psychologist Stanley Smith Stevens formulated a foundational classification framework categorizing empirical data into four distinct levels of measurement: Nominal, Ordinal, Interval, and Ratio. Selecting appropriate exploratory visualizations, encoding pipelines, distance metrics, and statistical learning algorithms depends strictly on the level of measurement of each attribute.

CODE
Measurement Scales Hierarchy:
[Nominal] < [Ordinal] < [Interval] < [Ratio]
(Lowest Mathematical Structure  -->  Highest Mathematical Structure)

1. Qualitative (Categorical) Scales

Qualitative data captures attributes describing identity, state, or categorical group membership rather than quantitative magnitudes.

A. Nominal Scale

Definition: Categorical labels devoid of intrinsic mathematical order, rank, or measurable distance between classes.
Valid Mathematical Operations: Equivalence testing ( $=$ or $\neq$ ). Arithmetic operations ( $+$ , $-$ , $\times$ , $\div$ ) are undefined.
Permissible Central Tendency: Mode (frequency-based majority class). Mean and median are mathematically meaningless.
Real-World Examples: Country of origin, blood group (A, B, AB, O), vehicle color, postal codes, server hostnames.
Machine Learning Treatment: Requires One-Hot Encoding, Binary Encoding, or Target/Frequency Encoding. Integer label assignment without one-hot expansion improperly induces false linear orderings.

B. Ordinal Scale

Definition: Categorical variables possessing an explicit, ordered sequence or rank hierarchy, but lacking uniform or quantifiable distances between consecutive ranks.
Valid Mathematical Operations: Comparative relations ( $<$ , $>$ , $\le$ , $\ge$ , $=$ , $\neq$ ).
Permissible Central Tendency: Median, percentiles, and mode. Mean calculations remain technically invalid due to non-equidistant intervals.
Real-World Examples: Education level (High School < Bachelor's < Master's < Ph.D.), Likert satisfaction ratings (Strongly Disagree to Strongly Agree), customer churn risk tiers (Low < Medium < High), credit rating grades (AAA, AA, A, BBB).
Machine Learning Treatment: Encoded via monotonic integer maps (e.g., OrdinalEncoder) preserving structural rank.

2. Quantitative (Numerical) Scales

Quantitative scales represent objective, numerical measurements capable of parametric mathematical computation.

A. Interval Scale

Definition: Numerical measurements where intervals between consecutive units are precisely uniform and equidistant, but the scale lacks an absolute, non-arbitrary true physical zero.
Zero Point: Arbitrary zero. The value $0$ does not represent the complete absence of the measured physical property.
Valid Mathematical Operations: Addition and subtraction. Ratios and multiplication are invalid (e.g., $40^\circ\text{C}$ is not "twice as hot" as $20^\circ\text{C}$ ).
Permissible Central Tendency: Arithmetic mean, standard deviation, median, mode.
Real-World Examples: Temperature in Celsius or Fahrenheit, calendar years (Gregorian calendar), IQ scores.

B. Ratio Scale

Definition: Numerical measurements possessing equidistant intervals and a distinct, non-arbitrary true physical zero point signifying complete absence of the measured quantity.
Zero Point: Absolute true zero ( $0\text{ kg}$ indicates absence of mass; $\$ 0$ indicates absence of financial capital).
Valid Mathematical Operations: All arithmetic operations ( $+$ , $-$ , $\times$ , $\div$ ), ratios, geometric means, harmonic means, and logarithms.
Real-World Examples: Annual salary, physical mass, distance traveled, processing latency in milliseconds, engine displacement volume, absolute temperature in Kelvin.

3. Summary of Measurement Properties

Scale	Ranking	Equal Intervals	True Zero	Permissible Central Tendency	Valid Mathematical Operations	Recommended ML Encoding
Nominal	No	No	No	Mode	$=$ , $\neq$	One-Hot / Target / Frequency
Ordinal	Yes	No	No	Median, Mode	$=$ , $\neq$ , $<$ , $>$	Ordinal Integer Map
Interval	Yes	Yes	No	Mean, Median, Mode	$+$ , $-$	Standardization / RobustScaler
Ratio	Yes	Yes	Yes	Geometric/Arithmetic Mean	$+$ , $-$ , $\times$ , $\div$	Min-Max, Box-Cox, Log, Scaling

3. Data Leakage: Mechanisms, Consequences, and Prevention

Data leakage represents one of the most insidious failure modes in machine learning systems. It occurs when extraneous information originating outside the training partition contaminates the model-building pipeline, enabling the algorithm to access signals during training that will not exist at inference time.

1. The Dual Impact of Leakage

Artificially Inflated Offline Metrics: Cross-validation error drops to near-zero; accuracy, precision, and recall appear exceptional during development.
Catastrophic Online Performance: The model experiences profound failure when evaluated against real-world production distributions because the unobserved predictive crutches vanish.

CODE
LEAKAGE SCENARIO (INCORRECT):
Raw Data -----------------> [Global Imputation & Scaling] ---------> [Train / Test Split]
                                                                        ^
                                                                        |
                       Test distribution parameters leak into training! -+

CORRECT PROTOCOL:
Raw Data -----------------> [Train / Test Split]
                                  |
              +-------------------+-------------------+
              |                                       |
              v                                       v
      [Training Data]                           [Testing Data]
              |                                       |
     fit_transform(Pipeline)                  transform(Pipeline)
              |                                       |
              v                                       v
      [Model Training]  ===================>  [Pure Generalization Eval]

2. Primary Leakage Vectors

A. Train-Test Contamination via Global Transformations

Occurs when transformations calculating distributional parameters (e.g., global mean, sample variance, minimum, maximum, target encoding priors, or PCA projection matrices) are fitted across the entire dataset prior to partitioning into training and testing folds.
Consequence: The test fold's mean and variance subtly alter the training feature space, artificially deflating generalization risk estimates.

B. Lookahead and Temporal Leakage

Occurs in longitudinal and time-series data when future observations inform feature generation for historical prediction targets.
Example: Including rolling 30-day customer transaction aggregates calculated from dates occurring after the prediction cutoff point $T$ .

C. Proxy and Direct Target Contamination

Occurs when a feature included in the predictor matrix is a direct mathematical consequence, duplicate, or administrative artifact of the target label.
Example: Predicting whether a patient contracted an infectious disease while utilizing "Antibiotic Prescription Code" as an input feature.

3. Prevention via Scikit-Learn Pipelines

The primary defense against data leakage is strict architectural encapsulation using sklearn.pipeline.Pipeline or sklearn.compose.ColumnTransformer.

python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

# 1. Split BEFORE any transformation
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

# 2. Define modular pre-processing pipelines
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder(handle_unknown='ignore', drop='first'))
])

preprocessor = ColumnTransformer(transformers=[
    ('num', numeric_transformer, numeric_features),
    ('cat', categorical_transformer, categorical_features)
])

# 3. Encapsulate preprocessor and estimator
model_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', LogisticRegression(solver='lbfgs'))
])

# fit calculates parameters strictly on X_train; transform applies them to X_test
model_pipeline.fit(X_train, y_train)
test_accuracy = model_pipeline.score(X_test, y_test)

4. Missing Data: Mechanisms and Imputation Paradigms

Missing data arises ubiquitously across real-world datasets due to human error, equipment failure, dropped network packets, survey non-response, and selective reporting. Imputing or dropping missing values without understanding the underlying generative mechanism leads to biased parameter estimators and inaccurate inferences.

1. Donald Rubin's Taxonomy of Missingness

CODE
                                  MISSING DATA MECHANISMS
                                             |
            +--------------------------------+-------------------------------+
            |                                |                               |
            v                                v                               v
     [MCAR] Completely At Random       [MAR] At Random              [MNAR] Not At Random
   P(M | Y_obs, Y_mis) = P(M)     P(M | Y_obs, Y_mis) = P(M|Y_obs)   P(M | Y_obs, Y_mis) depends on Y_mis
  Missingness entirely random     Missingness depends on observed    Missingness depends on unobserved value

A. Missing Completely at Random (MCAR)

Condition: The probability that a variable is missing is totally independent of both observed variables and unobserved missing values:
$P(M \mid Y_{obs}, Y_{mis}) = P(M)$
Consequence: Deleting missing rows (complete-case analysis) yields unbiased parameter estimates, though it causes statistical power loss.
Example: A laboratory test tube randomly shatters on the floor; packet drops caused by random hardware thermal fluctuation.

B. Missing at Random (MAR)

Condition: The propensity for missingness is systematically related to observed variables in the dataset, but unrelated to the value of the missing attribute itself once observed variables are controlled for:
$P(M \mid Y_{obs}, Y_{mis}) = P(M \mid Y_{obs})$
Example: In health surveys, male respondents report mental health consultation frequencies less often than female respondents. Within the male demographic, however, missingness occurs at random.
Treatment: Conditional multivariate imputation (e.g., MICE, KNN) generates unbiased estimates.

C. Missing Not at Random (MNAR)

Condition: The probability of missingness depends directly on the unobserved value of the missing variable itself:
$P(M \mid Y_{obs}, Y_{mis}) \text{ depends on } Y_{mis}$
Example: High-net-worth individuals refuse to disclose annual personal income on financial surveys; patients experiencing severe medication side-effects drop out of clinical trials.
Treatment: Standard imputation methods produce biased estimates; requires joint pattern-mixture modeling, selection models, or appending missingness indicator indicators.

2. Missing Value Remediation Strategies

A. Deletion Paradigms

Listwise Deletion (Complete-Case Analysis): Discard every sample row containing at least one missing entry.
- Risk: Drastic sample reduction. If data is MAR or MNAR, listwise deletion introduces severe selection bias.
Pairwise Deletion: Utilize all available non-missing pairs of variables when computing bivariate statistics (e.g., covariance matrices).
- Risk: May yield non-positive-definite covariance matrices incompatible with optimization.

B. Univariate Imputation Techniques

Mean Imputation: Replaces missing entries with the arithmetic mean of the observed values.
- Risk: Artificially compresses feature variance, attenuates covariance structures with other variables, and distorts confidence intervals.
Median Imputation: Replaces missing entries with the 50th percentile. Preferred over mean for heavily skewed numerical distributions (e.g., income, house prices).
Mode Imputation: Substitutes the most frequent category. Standard baseline for discrete categorical attributes.
Missingness Indicator Flagging: Creating an auxiliary binary indicator column ( $1$ if missing, $0$ otherwise) alongside numerical imputation to preserve potential informative missingness signals:

python
from sklearn.impute import MissingIndicator, SimpleImputer
indicator = MissingIndicator()
missing_flags = indicator.fit_transform(X_train)

C. Advanced Multivariate Imputation

K-Nearest Neighbors (KNN) Imputation (KNNImputer):
- Computes distance metrics (typically nan-Euclidean distance) between the sample with missing values and all complete samples.
- Imputes missing coordinates using the unweighted or distance-weighted average of the $k$ nearest neighbors:
  $d(\mathbf{u}, \mathbf{v}) = \sqrt{\frac{p}{p_{common}} \sum_{i \in \text{common}} (u_i - v_i)^2}$
Multivariate Imputation by Chained Equations (MICE / IterativeImputer):
- Formulates imputation as a round-robin regression problem.
- Each incomplete feature is modeled conditionally as a function of all other features in the dataset.
- Cycles through features iteratively until convergence, preserving covariance structures and realistic distributions.

python
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
from sklearn.linear_model import BayesianRidge

mice_imputer = IterativeImputer(
    estimator=BayesianRidge(),
    max_iter=15,
    random_state=42
)
X_imputed = mice_imputer.fit_transform(X_train)

5. Outlier Detection and Rectification Techniques

An outlier is an empirical observation that diverges significantly from the prevailing structural pattern or probability density of the remainder of the dataset. Outliers skew sample means, inflate variance estimates, bias ordinary least squares regression planes, and destabilize distance-metric algorithms.

CODE
                         OUTLIER DETECTION METHODOLOGIES
                                        |
              +-------------------------+-------------------------+
              |                                                   |
              v                                                   v
    [Univariate Approaches]                            [Multivariate Approaches]
    - Z-Score (|z| > 3)                                - Mahalanobis Distance
    - Modified Z-Score (MAD)                           - Isolation Forest
    - Tukey's IQR Fences                               - Local Outlier Factor (LOF)

1. Univariate Statistical Detection

A. Z-Score Method (Parametric)

Quantifies the signed distance between an observation $x_i$ and the feature mean $\mu$ measured in units of standard deviation $\sigma$ :
$z_i = \frac{x_i - \mu}{\sigma}$
Threshold Rule: Observations where $|z_i| > 3.0$ are classified as outliers.
Underlying Assumption: Assumes data follows a Gaussian (normal) distribution. When applied to multimodal or heavily skewed distributions, the sample mean and standard deviation themselves become contaminated by the outliers.

B. Modified Z-Score via Median Absolute Deviation (MAD)

Replaces non-robust parameters ( $\mu, \sigma$ ) with robust statistics (median and MAD):
$\text{MAD} = \text{median}(|x_i - \text{median}(\mathbf{x})|)$
$M_i = \frac{0.6745 \cdot (x_i - \text{median}(\mathbf{x}))}{\text{MAD}}$
Observations with $|M_i| > 3.5$ are flagged. Highly resilient to multiple extreme anomalies.

C. Tukey's Interquartile Range (IQR) Fences (Non-Parametric)

Derives quartile boundaries $Q_1$ (25th percentile) and $Q_3$ (75th percentile).
Computes Interquartile Range: $\text{IQR} = Q_3 - Q_1$ .
Inner Fences (Mild Outliers):
$\text{Lower Fence} = Q_1 - 1.5 \times \text{IQR}, \quad \text{Upper Fence} = Q_3 + 1.5 \times \text{IQR}$
Outer Fences (Extreme Outliers):
$\text{Lower Extreme} = Q_1 - 3.0 \times \text{IQR}, \quad \text{Upper Extreme} = Q_3 + 3.0 \times \text{IQR}$
Requires no distributional assumptions; functions effectively on skewed real-world metrics.

2. Multivariate Anomaly Detection

Mahalanobis Distance:
- Measures distance from sample point $\mathbf{x}$ to distribution centroid $\boldsymbol{\mu}$ , incorporating the feature covariance matrix $\boldsymbol{\Sigma}$ :
  $D_M(\mathbf{x}) = \sqrt{(\mathbf{x} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})}$
- Identifies samples that are univariate inliers but multivariate anomalies due to violating inter-feature correlations.
Isolation Forest (IsolationForest):
- Builds ensembles of randomized binary decision trees.
- Exploits the mathematical fact that anomalies are isolated close to the root node (requiring fewer random axis-aligned splits) compared to normal density clusters.
- Anomaly score derived from average isolation path length $E(h(\mathbf{x}))$ .

3. Outlier Remediation Strategies

Trimming / Deletion: Discarding identified outlier instances from the training corpus. Valid only when anomalies are verified sensor or logging errors.
Winsorization (Capping): Clamping extreme values at predetermined percentile boundaries (e.g., replacing values below the 1st percentile with the 1st percentile value, and values exceeding the 99th percentile with the 99th percentile value).
Non-Linear Monotonic Transformations: Applying concave mathematical transformations to compress long right tails:
- Logarithmic: $y = \ln(x + 1)$ (defined for $x \ge 0$ ).
- Square Root: $y = \sqrt{x}$ .
- Box-Cox Transformation: Strictly positive data ( $x > 0$ ):
  $y^{(\lambda)} = \begin{cases} \frac{x^\lambda - 1}{\lambda} & \text{if } \lambda \neq 0 \\ \ln(x) & \text{if } \lambda = 0 \end{cases}$
- Yeo-Johnson Transformation: Extends Box-Cox functionality to accommodate negative, zero, and positive values.

6. Categorical Feature Encoding

Statistical estimators, gradient optimizers, and matrix decompositions require numerical tensor representations. Converting categorical strings into mathematical representations demands careful architectural consideration to avoid injecting spurious metric ordering or causing exponential dimensional explosion.

1. One-Hot Encoding (OHE)

Mechanism: Projects a categorical variable with $K$ unique cardinality states into $K$ distinct binary indicator dimensions $\{0, 1\}$ .
The Dummy Variable Trap:
- In linear models containing an intercept term ( $\beta_0$ ), creating $K$ indicator columns introduces perfect linear multicollinearity:
  $\sum_{j=1}^K x_j = 1 \implies \mathbf{X}^T \mathbf{X} \text{ is singular (non-invertible)}.$
- Resolution: Set drop='first', reducing dimensions to $K-1$ . The excluded category serves as the baseline reference.
Limitations: In high-cardinality features (e.g., zip codes, user IDs with thousands of levels), OHE induces severe sparsity and the curse of dimensionality.

2. Ordinal and Label Encoding

Ordinal Encoding: Assigns integers $\{0, 1, \dots, K-1\}$ based on the explicit domain hierarchy of the feature (e.g., {'Junior': 0, 'Mid-Level': 1, 'Senior': 2, 'Executive': 3}).
Label Encoding: Arbitrarily assigns alphabetical or index integers to classes.
Critical Pitfall: Applying Label Encoding to nominal attributes (e.g., Red=0, Green=1, Blue=2) induces a false Euclidean distance metric ( $|2 - 0| > |1 - 0|$ ) that tricks linear models, SVMs, and neural networks into assuming Blue is "greater than" Red.

3. High-Cardinality Encoding Paradigms

CODE
               HIGH-CARDINALITY CATEGORICAL ENCODING STRATEGIES
                                       |
            +--------------------------+--------------------------+
            |                                                     |
            v                                                     v
   [Target / Mean Encoding]                               [Frequency Encoding]
Replaces category with E[y | Category]               Replaces category with relative
Must use smoothing to prevent leakage!               empirical frequency n_c / N

A. Target (Mean) Encoding

Replaces each category level with the empirical expected value of the target label conditioned on that category:
$\hat{x}_c = \mathbb{E}[y \mid x = c] = \frac{1}{n_c} \sum_{i: x_i = c} y_i$
Vulnerability to Overfitting & Leakage: If a category has very few samples (e.g., $n_c = 1$ ), target encoding creates extreme memorization and severe test-set generalization failure.
Regularized Smoothing (m-Estimate / Additive Smoothing):
Blends the category-specific target mean with the global prior target mean $\bar{y}_{global}$ :
$S_c = \frac{n_c \cdot \bar{y}_c + m \cdot \bar{y}_{global}}{n_c + m}$
where $m$ is a smoothing hyperparameter controlling shrinkage toward the global prior.

B. Frequency / Count Encoding

Replaces each categorical category with its normalized empirical frequency or raw sample count:
$f_c = \frac{n_c}{N}$
Preserves cardinality ranking and frequency dynamics without expanding dimensionality. Risk: Different categories with identical counts collide.

C. Binary & Hashing Encoding

Binary Encoding: Maps categories to integers, converts integers to binary bit strings, and splits bits into individual binary columns. Scales logarithmically: $O(\log_2 K)$ columns instead of $O(K)$ .
Feature Hashing (The Hashing Trick): Applies a hash function (e.g., MurmurHash3) to project arbitrary categorical strings into a fixed-dimensional array ( $B$ bins). Robust to unseen streaming categories; trade-off involves potential hash collisions.

7. Numerical Feature Scaling and Normalization

Numerical feature scaling aligns variables measured across disparate units, magnitudes, and ranges onto unified numerical frameworks. Unscaled features cause severe pathologies in distance-based estimators and gradient descent optimizers.

1. Algorithmic Sensitivity to Scale

CODE
+-----------------------------------------------------------------------------------------+
|                              SCALE SENSITIVITY SPECTRUM                                 |
+-----------------------------------------------------------------------------------------+
|                                                                                         |
|   HIGHLY SENSITIVE                   SENSITIVE                     COMPLETELY INVARIANT |
|  (Scaling is mandatory)         (Scaling accelerates conv.)      (Scaling has 0 impact) |
|                                                                                         |
|  - K-Nearest Neighbors (KNN)    - Logistic Regression            - Decision Trees       |
|  - Support Vector Machines      - Linear Regression (Ridge/Lasso)- Random Forests       |
|  - Principal Component Analysis - Multi-Layer Perceptrons (ANN)  - Gradient Boosted     |
|  - K-Means Clustering           - Stochastic Gradient Descent      Trees (XGBoost,      |
|                                                                    LightGBM, CatBoost)  |
+-----------------------------------------------------------------------------------------+

Distance-Based Algorithms: In algorithms evaluating Euclidean distance $d(\mathbf{u}, \mathbf{v}) = \sqrt{\sum (u_j - v_j)^2}$ , an unscaled feature varying in thousands (e.g., Income: $\$ 20,000 $to$ $200,000 $) completely dominates a feature varying across fractions (e.g., Age/100:$ 0.18 $to$ 0.85$).
Gradient Optimizers: Unscaled features produce elongated, eccentric elliptical loss surface contours. Gradient vectors oscillate perpendicular to the optimal trajectory, slowing learning rates. Rescaling transforms contours into spherical geometries, enabling rapid, direct convergence.
Tree-Based Ensembles: Split evaluation considers rank order within individual features independently. Splitting $x_j > \theta$ is invariant under any strictly monotonic transformation.

2. Primary Rescaling Methodologies

A. Standardization (Z-Score Scaling)

Rescales data to possess an empirical mean $\mu = 0$ and standard deviation $\sigma = 1$ :
$x' = \frac{x - \mu}{\sigma}$
Characteristics: Unbounded output range $(-\infty, +\infty)$ . Retains the relative magnitude of extreme outliers without truncating them. Preserves zero-mean centering optimal for neural networks and regularized regressions.

B. Min-Max Normalization

Linearly compresses the feature range into a fixed interval, typically $[0, 1]$ :
$x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$
Characteristics: Highly sensitive to outliers. If an extreme maximum value exists, inlier observations are compressed into a narrow band near $0.0$ .

C. Robust Scaling (`RobustScaler`)

Scales data using robust order statistics—the median and the Interquartile Range (IQR):
$x' = \frac{x - \text{median}}{Q_3 - Q_1}$
Characteristics: Resilient against extreme outliers. Centered at $0$ with interquartile spread normalized to $1.0$ .

D. Maximum Absolute Scaling (`MaxAbsScaler`)

Scales each feature by its maximum absolute value:
$x' = \frac{x}{\max(|x|)}$
Characteristics: Maps values to $[-1, 1]$ . Crucially preserves exact sparsity structures in sparse CSR matrices (does not destroy zero entries).

8. Handling Imbalanced Datasets

Class imbalance manifests when the target label distribution is severely skewed across classes (e.g., credit card fraud where $99.8\%$ of transactions are legitimate and $0.2\%$ are fraudulent; rare tumor detection).

1. The Accuracy Paradox

When evaluating imbalanced datasets, standard classification accuracy is misleading. A naive classifier predicting the majority class exclusively achieves $99.8\%$ accuracy while failing to identify a single fraudulent transaction (0% recall on the minority class). Evaluating imbalanced models requires Precision-Recall AUC, Balanced Accuracy, F1-Score, or Cost-Weighted Utility Matrices.

CODE
                        CLASS BALANCING METHODOLOGIES
                                      |
            +-------------------------+-------------------------+
            |                                                   |
            v                                                   v
   [Data-Level Resampling]                         [Algorithmic Adaptations]
   - Random Undersampling (RUS)                    - Cost-Sensitive Weighting
   - Random Oversampling (ROS)                     - Focal Loss
   - SMOTE (Synthetic Minority)                    - Decision Threshold Tuning
   - ADASYN / Borderline-SMOTE

2. Resampling Paradigms

A. Random Undersampling (RUS)

Randomly removes instances from the majority class until parity with the minority class is achieved.
Limitation: Discards potentially vital informational samples, degrading decision boundary resolution.

B. Random Oversampling (ROS)

Randomly duplicates existing instances from the minority class with replacement.
Limitation: Promotes severe overfitting because the classifier memorizes tight, duplicated decision regions around individual minority instances.

C. Synthetic Minority Over-sampling Technique (SMOTE)

Generates synthetic minority instances via convex linear interpolation along line segments connecting k-nearest minority neighbors.
Mathematical Procedure:
1. For each minority sample $\mathbf{x}_i$ , identify its $k$ nearest neighbors within the minority class.
2. Select one neighbor $\mathbf{x}_{zi}$ at random from the $k$ neighbors.
3. Compute the feature difference vector: $\boldsymbol{\delta} = \mathbf{x}_{zi} - \mathbf{x}_i$ .
4. Multiply $\boldsymbol{\delta}$ by a random scalar $\lambda \sim U(0, 1)$ .
5. Generate synthetic point:
  $\mathbf{x}_{\text{new}} = \mathbf{x}_i + \lambda (\mathbf{x}_{zi} - \mathbf{x}_i)$

CODE
SMOTE Synthesis Mechanism:
Minority Point A (x_i)  o-----------------x-----------------o  Minority Point B (x_zi)
                                  Generated Synthetic
                                  Sample (x_new)

D. Advanced Synthetic Variants

Borderline-SMOTE: Identifies minority samples residing along the dangerous decision boundary (where $50\%$ to $100\%$ of their nearest neighbors belong to the majority class) and focuses synthetic generation strictly on these vulnerable boundary instances.
ADASYN (Adaptive Synthetic): Calculates an adaptive density distribution over minority samples, generating proportionally more synthetic samples for minority instances that are harder to learn (surrounded by more majority class samples).

3. Algorithmic Cost-Sensitive Adaptation

Instead of artificially modifying data distributions, cost-sensitive learning adapts the empirical risk objective function by assigning higher loss penalties to misclassifications of the minority class.

Balanced Class Weights:
$w_c = \frac{N}{K \cdot n_c}$
where $N$ is total sample count, $K$ is number of classes, and $n_c$ is frequency of class $c$ .

Scikit-Learn Implementation:

python
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(class_weight='balanced', random_state=42)

Focal Loss: Adds a dynamic modulating factor $(1 - p_t)^\gamma$ to standard cross-entropy loss, down-weighting the loss contribution of easy, well-classified majority instances and directing gradient updates toward hard minority instances.

Applied Machine Learning