Unit 1: Introduction and Data Preparation

1. Orientation: Learning Patterns from Data

1.1 The Discipline of Predictive Analytics

Predictive Analytics is the branch of advanced data analytics that utilizes historical data, statistical modeling, machine learning algorithms, and mathematical optimization to estimate unknown past, present, or future outcomes. Rather than merely describing what has occurred, predictive analytics extracts underlying functional relationships $\hat{f}(X) \approx Y$ to generalize to unseen, incoming observations.

CODECopy Code
+-------------------------------------------------------------------------+
|                  Analytics Maturity Spectrum Taxonomy                   |
+-------------------+--------------------+--------------------------------+
| Analytics Domain  | Guiding Question   | Core Methodologies             |
+-------------------+--------------------+--------------------------------+
| **Descriptive**   | What happened?     | Summary statistics, OLAP cubes,|
|                   |                    | BI dashboards, aggregate KPIs  |
| **Diagnostic**    | Why did it happen? | Root-cause analysis, drill-down|
|                   |                    | correlation, variance analysis |
| **Predictive**    | What will happen?  | Supervised learning, regression|
|                   |                    | time series, neural networks   |
| **Prescriptive**  | What should we do? | Optimization algorithms, linear|
|                   |                    | programming, simulation, RL    |
+-------------------+--------------------+--------------------------------+

Rendering diagram...

1.2 Mathematical Foundations of Supervised Learning

In supervised predictive modeling, the underlying phenomena is formalized as an unknown target function:
$Y = f(X) + \epsilon$
Where:

• $X = [x_1, x_2, \dots, x_p]^T \in \mathbb{R}^p$ represents the $p$-dimensional input feature vector (covariates, independent variables, or predictors).
• $Y \in \mathcal{Y}$ represents the ground-truth target variable (response or dependent variable).
  • In Regression Tasks, $\mathcal{Y} \subseteq \mathbb{R}$ (continuous real values, e.g., real estate pricing, hospital readmission duration).
  • In Classification Tasks, $\mathcal{Y} = \{C_1, C_2, \dots, C_K\}$ (discrete qualitative labels, e.g., loan default status, benign/malignant tumor).
• $\epsilon$ is an irreducible random error term satisfying $\mathbb{E}[\epsilon] = 0$ and $\text{Var}(\epsilon) = \sigma^2$, representing noise, unobserved latent variables, and stochastic measurement limits.
• $\hat{f}(X)$ is the estimated model hypothesis learned from training data $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$.

The expected prediction error at an unseen test point $x_0$ using Mean Squared Error (MSE) decomposes into three distinct terms:
$\mathbb{E}[(Y - \hat{f}(x_0))^2] = \underbrace{\left(\text{Bias}[\hat{f}(x_0)]\right)^2}_{\text{Underfitting}} + \underbrace{\text{Var}(\hat{f}(x_0))}_{\text{Overfitting}} + \underbrace{\sigma^2}_{\text{Irreducible Error}}$

CODECopy Code
+-------------------------------------------------------------------------+
|                    The Bias-Variance Tradeoff Curve                     |
+-------------------------------------------------------------------------+
| Error                                                                   |
|   ▲                                                                     |
|   │         High Bias                Optimum             High Variance  |
|   │       (Underfitting)             Balance             (Overfitting)  |
|   │                                                                     |
|   │   \                                                   / Total Error |
|   │    \                                                 /              |
|   │     \                     _---_                     /               |
|   │      \                   /     \                   /  Variance      |
|   │       \                 /       \                 /                 |
|   │        \               /         \               /                  |
|   │  Bias²  \             /           \             /                   |
|   │          \           /             \           /                    |
|   │           \_________/               \_________/                     |
|   │                                                                     |
|   │  -------------------------------------------------- Irreducible (σ²)|
|   └──────────────────────────────────────────────────────────────►      |
|   Low Model Complexity                          High Model Complexity   |
+-------------------------------------------------------------------------+

1.3 End-to-End Predictive Analytics Workflow

The rigorous lifecycle of predictive analytics follows the standard CRISP-DM (Cross-Industry Standard Process for Data Mining) methodology:

CODECopy Code
+-------------------------------------------------------------------------+
|                  CRISP-DM Predictive Analytics Cycle                    |
+-------------------------------------------------------------------------+
|                                                                         |
|  [ Business Understanding ] ──> Formulate business goal as statistical  |
|               │                 optimization objective                  |
|               ▼                                                         |
|  [ Data Understanding ]     ──> Ingestion, metadata audit, schema check |
|               │                                                         |
|               ▼                                                         |
|  [ Exploratory Data (EDA) ] ──> Distributions, missingness, outliers,   |
|               │                 correlations, skewness analysis         |
|               ▼                                                         |
|  [ Data Preparation ]       ──> Cleaning, imputation, feature scaling,  |
|               │                 encoding, transformation, selection     |
|               ▼                                                         |
|  [ Modeling & Training ]    ──> Estimator fitting, hyperparameter tuning|
|               │                 cross-validation                        |
|               ▼                                                         |
|  [ Evaluation ]             ──> Verification on unseen holdout test set |
|               │                 (ROC-AUC, RMSE, F1, calibration)        |
|               ▼                                                         |
|  [ Deployment & MLOps ]     ──> Serving, drift detection, monitoring    |
+-------------------------------------------------------------------------+

2. Exploratory Data Analysis (EDA) & Data Characteristics

2.1 Taxonomy of Data Types

Prior to applying statistical transformations, data attributes must be classified according to Stevens’ Scales of Measurement:

CODECopy Code
+--------------------------------------------------------------------------+
|                     Measurement Scales & Variable Types                  |
+-------------------+--------------------+---------------------------------+
| Level / Scale     | Mathematical Ops   | Canonical Examples              |
+-------------------+--------------------+---------------------------------+
| **Nominal**       | Equality ($=, \neq$)| Blood type ($A, B, AB, O$),     |
| (Categorical)     | Mode only          | Zip codes, marital status       |
| **Ordinal**       | Order ($<, >$)     | Education (High School < BSc    |
| (Categorical)     | Median, Percentiles| < MSc < PhD), customer rating   |
| **Interval**      | Difference ($+, -$)| Temperature in Celsius/Fahr,    |
| (Continuous)      | Mean, Std Dev      | Calendar dates (arbitrary zero) |
| **Ratio**         | Ratio ($\times, \div$)| Salary, age, weight, distance,  |
| (Continuous)      | Geometric Mean     | transaction value (true zero)   |
+-------------------+--------------------+---------------------------------+

2.2 Univariate Statistical Measures: Central Tendency & Dispersion

Given a feature vector $\mathbf{x} = [x_1, x_2, \dots, x_n]^T$:

1. Arithmetic Mean ($\mu$ or $\bar{x}$):
   $\bar{x} = \frac{1}{n} \sum_{i=1}^n x_i$
   Sensitivity: Highly non-robust; a single extreme outlier corrupts the mean.
2. Median ($M$ or $\tilde{x}$): The 50th percentile ($Q_2$). Robust to outliers.
3. Variance ($\sigma^2$ or $s^2$) & Standard Deviation ($s$):
   $s^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2, \quad s = \sqrt{s^2}$
4. Interquartile Range ($IQR$):
   $IQR = Q_3 - Q_1$
   Where $Q_1$ is the 25th percentile and $Q_3$ is the 75th percentile. $IQR$ forms the foundation of Tukey's robust outlier detection.

CODECopy Code
+-------------------------------------------------------------------------+
|                  Anatomy of a Tukey Box Plot (Five-Number Summary)      |
+-------------------------------------------------------------------------+
|                                                                         |
|    Outlier                                                 Outlier      |
|       *                                                       *         |
|             |───────[=========|=========]───────|                       |
|                     |         |         |                               |
|            Lower    Q1     Median      Q3     Upper                     |
|           Whisker  (25%)    (50%)     (75%)  Whisker                    |
|                                                                         |
|  - Lower Whisker: max(min(x), Q1 - 1.5 * IQR)                           |
|  - Upper Whisker: min(max(x), Q3 + 1.5 * IQR)                           |
|  - Data points beyond whiskers are flagged as potential outliers (*)   |
+-------------------------------------------------------------------------+

2.3 Higher-Order Moments: Skewness and Kurtosis

To determine whether continuous features conform to normal Gaussian distributions:

Skewness ($\gamma_1$)

Measures distributional asymmetry about the mean:
$\gamma_1 = \frac{\frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^3}{\left(\frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2\right)^{3/2}}$

• $\gamma_1 = 0$: Perfectly symmetric distribution (e.g., standard normal).
• $\gamma_1 > 0$: Positive / Right-Skewed (long right tail; Mean $>$ Median $>$ Mode; e.g., personal wealth, insurance claim costs).
• $\gamma_1 < 0$: Negative / Left-Skewed (long left tail; Mode $>$ Median $>$ Mean; e.g., age at human mortality).

Kurtosis ($\gamma_2$ / Fisher Kurtosis)

Measures the "tailedness" and propensity for extreme outlier generation relative to a Gaussian distribution:
$\gamma_2 = \frac{\frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^4}{\left(\frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2\right)^2} - 3$

• Mesokurtic ($\gamma_2 = 0$): Normal distribution baseline.
• Leptokurtic ($\gamma_2 > 0$): Heavy tails, high peak; high probability of extreme outlier events.
• Platykurtic ($\gamma_2 < 0$): Light tails, flat peak; low probability of extreme values.

2.4 Bivariate & Multivariate Association Metrics

1. Pearson Product-Moment Correlation Coefficient ($r$):
   Measures linear co-variation between two continuous features $X$ and $Y$:
   $r_{XY} = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2} \sqrt{\sum_{i=1}^n (y_i - \bar{y})^2}} \in [-1, +1]$
   • $r = +1$: Perfect positive linear correlation.
   • $r = -1$: Perfect negative linear correlation.
   • $r = 0$: Absence of linear relationship (non-linear relationships may still exist!).
2. Spearman Rank Correlation ($\rho$):
   Measures monotonic relationships by calculating the Pearson correlation over rank-transformed values:
   $\rho = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)}$
   Where $d_i = \text{rank}(x_i) - \text{rank}(y_i)$. Highly robust against outliers and non-linear monotonic curves.

3. Missing Data Treatment & Imputation Strategies

3.1 Rubins Taxonomy of Missing Data Mechanisms

Understanding why values are missing dictates the validity of downstream statistical and machine learning inferences:

CODECopy Code
+--------------------------------------------------------------------------+
|                  Rubin's Missingness Mechanisms Matrix                   |
+-------------------+--------------------+---------------------------------+
| Mechanism         | Formal Definition  | Operational Remedy              |
+-------------------+--------------------+---------------------------------+
| **MCAR**          | $P(M|Y_{obs}, Y_{mis})| Complete Case Deletion is safe  |
| (Missing Complete | = P(M)$            | (unbiased), though statistical  |
| at Random)        | Probability of     | power is sacrificed. Simple     |
|                   | missingness is     | mean/median imputation valid.   |
|                   | independent of all |                                 |
|                   | data values.       |                                 |
+-------------------+--------------------+---------------------------------+
| **MAR**           | $P(M|Y_{obs}, Y_{mis})| Deletion introduces bias! Must  |
| (Missing at       | = P(M|Y_{obs})$    | use conditional imputation      |
| Random)           | Missingness depends| (MICE, KNN, regression-based    |
|                   | systematically on  | models) conditioned on observed |
|                   | observed variables.| covariates.                     |
+-------------------+--------------------+---------------------------------+
| **MNAR**          | $P(M|Y_{obs}, Y_{mis})| Most dangerous! Standard        |
| (Missing Not at   | depends on         | imputation causes structural    |
| Random / Non-Ign) | $Y_{mis}$ itself.  | bias. Requires explicit missing |
|                   | E.g., high earners | indicator features or pattern-  |
|                   | refusing to report | mixture models.                 |
|                   | income.            |                                 |
+-------------------+--------------------+---------------------------------+

3.2 Imputation Methodologies & Implementation

pythonCopy Code
import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

# Sample Data with Missing Values
df = pd.DataFrame({
    'Income': [45000, np.nan, 82000, 110000, 38000, np.nan],
    'Age': [25, 30, 45, 52, np.nan, 29],
    'CreditScore': [650, 710, np.nan, 800, 590, 720]
})

# 1. Univariate Statistical Imputation (Median is robust against income outliers)
median_imputer = SimpleImputer(strategy='median', add_indicator=True)
# add_indicator=True creates a binary flag column indicating whether value was imputed!
income_imputed = median_imputer.fit_transform(df[['Income']])

# 2. Multivariate Distance-Based Imputation (k-Nearest Neighbors)
# Finds k nearest neighbors based on Euclidean distance of observed features
knn_imputer = KNNImputer(n_neighbors=3, weights='distance')
df_knn_imputed = pd.DataFrame(knn_imputer.fit_transform(df), columns=df.columns)

# 3. MICE (Multivariate Imputation by Chained Equations / IterativeImputer)
# Models each feature with missing values as a function of all other features iteratively
mice_imputer = IterativeImputer(max_iter=10, random_state=42)
df_mice_imputed = pd.DataFrame(mice_imputer.fit_transform(df), columns=df.columns)

4. Outlier Detection and Treatment

4.1 Statistical Detection Rules

1. Z-Score Method (Parametric)

Assumes features follow an approximately Gaussian distribution $\mathcal{N}(\mu, \sigma^2)$:
$Z_i = \frac{x_i - \bar{x}}{s}$
A data point is flagged as an anomaly if $|Z_i| > 3.0$ (representing data points residing in the outer $0.27\%$ tails of the normal distribution).

2. Tukey's Interquartile Range (IQR) Method (Non-Parametric)

Does not assume normality. Uses empirical order statistics:
$\text{Lower Bound} = Q_1 - 1.5 \times IQR$
$\text{Upper Bound} = Q_3 + 1.5 \times IQR$

• Points outside $[Q_1 - 1.5 \cdot IQR, Q_3 + 1.5 \cdot IQR]$ are classified as mild outliers.
• Points outside $[Q_1 - 3.0 \cdot IQR, Q_3 + 3.0 \cdot IQR]$ are classified as extreme outliers.

4.2 Machine Learning Based Anomaly Detection

When data features exhibit complex multivariate dependencies, univariate rules fail to detect multidimensional outliers.

pythonCopy Code
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor

# 1. Isolation Forest (Tree-based ensemble isolation)
# Isolates anomalies by randomly selecting a feature and randomly splitting value.
# Outliers require significantly fewer recursive splits to be isolated in tree leaves.
iso_forest = IsolationForest(contamination=0.05, random_state=42)
outlier_preds = iso_forest.fit_predict(df_knn_imputed) # Returns -1 for outliers, 1 for inliers

# 2. Local Outlier Factor (LOF)
# Compares the local density of an observation to that of its k-nearest neighbors
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.05)
lof_preds = lof.fit_predict(df_knn_imputed)

4.3 Outlier Remediation Techniques

CODECopy Code
+-------------------------------------------------------------------------+
|                      Outlier Remediation Approaches                     |
+-------------------+-----------------------------------------------------+
| Treatment Strategy| Mechanics and Statistical Tradeoffs                 |
+-------------------+-----------------------------------------------------+
| **Trimming**      | Drops outlier rows. Only valid if outliers represent|
| (Deletion)        | proven recording errors or corrupted sensor data.   |
| **Winsorization** | Caps extreme values at predetermined percentiles    |
| (Capping)         | (e.g. clipping all values $> 99$th percentile to the|
|                   | 99th percentile value; $< 1$st to 1st percentile).  |
| **Robust Scaling**| Retains outliers but normalizes features using      |
|                   | median and IQR to prevent gradient distortion.      |
| **Log/Power Xform**| Compresses long right-hand tails mathematically.    |
+-------------------+-----------------------------------------------------+

pythonCopy Code
# Implementing Winsorization (Clipping) via Pandas
lower_limit = df['Income'].quantile(0.01)
upper_limit = df['Income'].quantile(0.99)

df['Income_Winsorized'] = df['Income'].clip(lower=lower_limit, upper=upper_limit)

5. Feature Scaling & Mathematical Transformations

5.1 Why Feature Scaling is Essential

Feature scaling is mandatory for any algorithm sensitive to numerical magnitudes and Euclidean distances:

CODECopy Code
+--------------------------------------------------------------------------+
|             Algorithm Sensitivity to Feature Scaling Matrix              |
+--------------------+---------------------+-------------------------------+
| Algorithm Family   | Scaling Sensitive?  | Operational Reason            |
+--------------------+---------------------+-------------------------------+
| **Distance-Based** | **CRITICAL**        | Features with large magnitudes|
| (KNN, K-Means,     |                     | dominate Euclidean distance   |
| SVM with RBF)      |                     | calculations completely.      |
| **Gradient-Based** | **MANDATORY**       | Unscaled features distort loss|
| (Linear/Logistic,  |                     | surface into elongated canyons|
| Neural Networks)   |                     | causing slow/failed descent.  |
| **Regularized**    | **MANDATORY**       | L1/L2 penalties penalize large|
| (Lasso, Ridge)     |                     | coefficients; unscaled features|
|                    |                     | are penalized unfairly!       |
| **Tree-Based**     | **IMMUNE**          | Decision trees split on single|
| (Random Forest,    |                     | features independently; scale |
| XGBoost, LightGBM) |                     | does not affect split orders. |
+--------------------+---------------------+-------------------------------+

5.2 Comparative Analysis of Scaling Algorithms

CODECopy Code
+--------------------------------------------------------------------------+
|                     Comparative Scaling Formulae                         |
+-------------------+----------------------------+-------------------------+
| Scaler            | Mathematical Formula       | Key Properties          |
+-------------------+----------------------------+-------------------------+
| **StandardScaler**| $z = \frac{x - \mu}{\sigma}$| Centers mean at 0, unit |
|                   |                            | variance ($\sigma = 1$).|
| **MinMaxScaler**  | $x_{norm} = \frac{x - x_{min}}{x_{max} - x_{min}}$ | Bounds data to $[0, 1]$.|
|                   |                            | Highly sensitive to outl|
| **RobustScaler**  | $x_{rob} = \frac{x - Q_2}{IQR}$ | Centers on median ($Q_2$)|
|                   |                            | scales by IQR; outlier-r|
| **MaxAbsScaler**  | $x_{abs} = \frac{x}{|x_{max}|}$ | Scales to $[-1, 1]$;    |
|                   |                            | preserves sparse zeros  |
+-------------------+----------------------------+-------------------------+

5.3 Non-Linear Distributional Transformations

When continuous features exhibit heavy positive skewness ($\gamma_1 > 1.0$), linear models violate homoscedasticity and normality of residuals assumptions.

CODECopy Code
+-------------------------------------------------------------------------+
|                  Mathematical Power Transformations                     |
+--------------------+----------------------------------------------------+
| Transformation     | Mathematical Definition & Operational Constraint   |
+--------------------+----------------------------------------------------+
| **Log (Natural)**  | $y = \log(x + 1)$ (or $\ln(x)$)                    |
|                    | Requires $x \ge 0$. Compresses large right tails.  |
| **Box-Cox**        | $y^{(\lambda)} = \frac{x^\lambda - 1}{\lambda}$    |
|                    | $(\text{for } \lambda \neq 0; \ln(x) \text{ if } 0)$|
|                    | **Strictly requires $x > 0$ strictly positive!**   |
|                    | Optimizes $\lambda$ via Maximum Likelihood.        |
| **Yeo-Johnson**    | Generalization of Box-Cox that **supports negative |
|                    | and zero values** seamlessly.                      |
+--------------------+----------------------------------------------------+

pythonCopy Code
from sklearn.preprocessing import PowerTransformer

# Automatically estimates optimal lambda parameters
pt_yeo = PowerTransformer(method='yeo-johnson')
income_normalized = pt_yeo.fit_transform(df[['Income_Winsorized']])

6. Categorical Variable Encoding

Machine learning models are mathematical optimization systems operating over linear algebra tensors; non-numeric categorical attributes must be encoded into numerical representations.

CODECopy Code
+--------------------------------------------------------------------------+
|                     Categorical Encoding Strategies                      |
+-------------------+--------------------+---------------------------------+
| Encoding Scheme   | Mechanics          | Optimal Application & Tradeoff  |
+-------------------+--------------------+---------------------------------+
| **One-Hot**       | Creates $k$ binary | Nominal variables with low      |
| **Encoding**      | indicator columns  | cardinality ($k < 15$). Expands |
| (OHE)             | (0 or 1).          | dimensionality (curse of dim).  |
| **Ordinal**       | Maps discrete ranks| Strictly for ordinal features   |
| **Encoding**      | to integers        | with inherent hierarchy         |
|                   | (e.g. 1, 2, 3).    | (e.g. Low=1, Med=2, High=3).   |
| **Target**        | Replaces category  | High cardinality nominals.      |
| **Encoding**      | with mean target   | **High risk of Data Leakage &   |
| (Mean Encoding)   | value: $\bar{y}_c$ | overfitting! Must use smoothing)|
| **Frequency / **  | Replaces category  | Preserves frequency signals;    |
| **Count Encoding**| with occurrence cnt| collision risk if counts match. |
+-------------------+--------------------+---------------------------------+

The Dummy Variable Trap in One-Hot Encoding

When all $k$ binary columns are included in a linear regression model containing an intercept term ($\beta_0$), the columns exhibit perfect multicollinearity because their sum equals 1:
$\sum_{j=1}^k D_j = 1$
This causes the feature matrix $X^T X$ to become singular and non-invertible. To avoid the dummy variable trap in linear models, drop one reference category by specifying drop='first', producing $k - 1$ indicator features.

7. Feature Engineering and Selection

7.1 Engineering Domain-Specific Features

1. Interaction Features: Multiplicative pairs $X_1 \times X_2$ modeling synergistic effects (e.g. Income * CreditScore).
2. Polynomial Expansion: Adding quadratic/cubic powers ($X_1^2, X_2^2$) to capture non-linear curvature in linear algorithms.
3. Cyclical Temporal Encoding: Encoding periodic temporal features (hours 0–23, months 1–12) into continuous circular coordinates using sine and cosine transformations:
   $x_{sin} = \sin\left(\frac{2\pi \times \text{hour}}{24}\right), \quad x_{cos} = \cos\left(\frac{2\pi \times \text{hour}}{24}\right)$
   This preserves the true circular distance (hour 23 is mathematically adjacent to hour 0).

7.2 Taxonomy of Feature Selection Methods

CODECopy Code
+--------------------------------------------------------------------------+
|                 Feature Selection Methodologies Comparison               |
+-------------------+--------------------+---------------------------------+
| Selection Family  | Operating Strategy | Representative Algorithms       |
+-------------------+--------------------+---------------------------------+
| **Filter**        | Evaluates features | Pearson $r$, ANOVA $F$-test,    |
| **Methods**       | individually using | Chi-Square ($\chi^2$), Mutual   |
|                   | statistical metrics| Information (Fast, ignores model|
|                   | independent of ML. | interactions).                  |
| **Wrapper**       | Uses a predictive  | Recursive Feature Elimination   |
| **Methods**       | model to evaluate  | (RFE), Forward Selection,       |
|                   | feature subsets    | Sequential Backward Selection.  |
|                   | iteratively.       | (High compute cost, overfits).  |
| **Embedded**      | Feature selection  | Lasso (L1 Regularization),      |
| **Methods**       | occurs internally  | Decision Tree Gini importance,  |
|                   | during model       | Random Forest Permutation       |
|                   | training.          | Feature Importance.             |
+-------------------+--------------------+---------------------------------+

8. Data Splitting & Preventing Data Leakage

8.1 The Data Partitioning Architecture

To obtain an unbiased estimate of generalization performance on unseen real-world data:

CODECopy Code
+-------------------------------------------------------------------------+
|                  Data Partitioning Strategy (Train-Val-Test)            |
+-------------------------------------------------------------------------+
|                                                                         |
|  [ Entire Historical Dataset (100%) ]                                   |
|  ┌───────────────────────────────────────────────────┬────────────────┐ |
|  │                 Development Set (80%)             │ Holdout (20%)  │ |
|  │  ┌─────────────────────────────┬────────────────┐ │                │ |
|  │  │      Training Set (64%)     │ Validation(16%)│ │ Test Set (20%) │ |
|  │  └─────────────────────────────┴────────────────┘ │                │ |
|  └───────────────────────────────────────────────────┴────────────────┘ |
|                                                                         |
|  - Training Set:   Fits model parameters (weights, coefficients).       |
|  - Validation Set: Tunes hyperparameters, feature selection iterations. |
|  - Test Set:       Strictly untouched until final model audit.          |
+-------------------------------------------------------------------------+

• Stratified Splitting (StratifiedKFold / stratify=y): Mandatory for imbalanced classification tasks (e.g. fraud detection with 1% positive class) to guarantee identical target class proportions across train, validation, and test partitions.
• Temporal Splitting: In time-series forecasting, random shuffling causes lookahead bias. Data must be split sequentially along the time index (e.g. train on 2021–2023, test on 2024).

8.2 The Menace of Data Leakage

[!CAUTION]
Data Leakage occurs when information from outside the training dataset (specifically from validation or test holdouts) contaminates the training pipeline. A leaked model exhibits artificially stellar cross-validation scores in development, but fails catastrophically in production.

Cardinal Rules to Prevent Leakage:

1. Split BEFORE Any Transformation: Perform train_test_split() before calculating imputations, scaling parameters, or outlier bounds.
2. fit_transform on Train, transform ONLY on Test:
   • Fit transformers strictly on the training partition: scaler.fit(X_train).
   • Apply learned parameters ($\mu_{train}, \sigma_{train}$) to test: scaler.transform(X_test).
3. Encapsulate Pipelines with scikit-learn Pipeline:

pythonCopy Code
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# 1. Split Data First
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

# 2. Define Sub-Pipelines by Feature Type
numeric_features = ['Age', 'Income', 'CreditScore']
numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_features = ['Education', 'Occupation']
categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder(drop='first', handle_unknown='ignore'))
])

# 3. Assemble Full Preprocessor via ColumnTransformer
preprocessor = ColumnTransformer(transformers=[
    ('num', numeric_transformer, numeric_features),
    ('cat', categorical_transformer, categorical_features)
])

# 4. Bind into an Atomic Pipeline
full_model_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', LogisticRegression(C=1.0, max_iter=1000))
])

# Fits ONLY on training partition; zero leakage!
full_model_pipeline.fit(X_train, y_train)

# Evaluates on holdout test set cleanly
accuracy = full_model_pipeline.score(X_test, y_test)

9. Comprehensive Review & Quick Reference Table

CODECopy Code
+-------------------------------------------------------------------------+
|                  Data Preparation Summary & Formulas                    |
+---------------------+---------------------------------------------------+
| Task                | Formula / Implementation Pattern                  |
+---------------------+---------------------------------------------------+
| IQR Outlier Rule    | $[Q_1 - 1.5 \cdot IQR, Q_3 + 1.5 \cdot IQR]$      |
| Z-Score Scaling     | $Z = \frac{x - \mu}{\sigma} \quad (\mu=0, \sigma=1)$|
| Min-Max Scaling     | $X_{norm} = \frac{x - x_{min}}{x_{max} - x_{min}} \in [0, 1]$|
| Robust Scaling      | $X_{rob} = \frac{x - Q_2}{IQR}$ (Outlier-safe)    |
| Log Transformation  | $y = \log(x + 1)$ (Compresses right skew)         |
| Box-Cox Power       | $y = \frac{x^\lambda - 1}{\lambda} \quad (x > 0)$ |
| Yeo-Johnson Power   | Generalized Box-Cox allowing zero and negative $x$|
| One-Hot Collinearity| Use `OneHotEncoder(drop='first')` to drop dummy   |
| Leakage Prevention  | Encapsulate in `Pipeline` with `ColumnTransformer`|
+---------------------+---------------------------------------------------+