Unit 1: Introduction and Data Preprocessing

1. Overview of Supervised Learning Paradigm

Supervised learning constitutes the foundational bedrock of applied machine learning, characterized by the presence of ground-truth supervisor signals during estimator training. Formally, let $\mathcal{X} \subseteq \mathbb{R}^d$ represent the input feature space of $d$-dimensional feature vectors, and let $\mathcal{Y}$ denote the target label space. The training corpus consists of $N$ independent and identically distributed (i.i.d.) observations drawn from an unknown joint probability distribution $\mathcal{D}$ over $\mathcal{X} \times \mathcal{Y}$:

$\mathcal{D}_{\text{train}} = \{(\mathbf{x}_i, y_i)\}_{i=1}^N, \quad \mathbf{x}_i \in \mathcal{X}, \; y_i \in \mathcal{Y}$

The objective of supervised learning is to induce a mapping hypothesis function $h: \mathcal{X} \to \mathcal{Y}$ from an admissible hypothesis class $\mathcal{H}$ such that the expected risk (generalization error) under loss function $L(y, h(\mathbf{x}))$ is minimized:

$R(h) = \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{D}} [L(y, h(\mathbf{x}))] = \int_{\mathcal{X} \times \mathcal{Y}} L(y, h(\mathbf{x})) \, dP(\mathbf{x}, y)$

Because the joint distribution $P(\mathbf{x}, y)$ is strictly unknown in practice, estimators minimize the Empirical Risk across observed samples, penalized by structural regularization terms:

$R_{\text{emp}}(h) = \frac{1}{N} \sum_{i=1}^N L(y_i, h(\mathbf{x}_i)) + \lambda \Omega(h)$

1.1 Supervised Learning Taxonomy: Regression vs. Classification

The structural configuration of $\mathcal{Y}$ categorizes supervised problems:

1. Regression: $\mathcal{Y} \subseteq \mathbb{R}$ (or $\mathbb{R}^m$ for multi-output regression). The objective is predicting continuous scalar quantities such as real estate valuation, temperature trajectories, or asset volatility. Standard loss functions include Mean Squared Error ($L_2$ loss: $L(y, \hat{y}) = (y - \hat{y})^2$), Mean Absolute Error ($L_1$ loss: $L(y, \hat{y}) = |y - \hat{y}|$), and Huber loss for outlier resilience.
2. Binary Classification: $\mathcal{Y} \in \{0, 1\}$ or $\{-1, +1\}$. Predicts qualitative class membership (e.g., credit default prediction, medical malignancy screening). Evaluated via binary cross-entropy (log-loss):
   $L(y, \hat{p}) = - [y \log \hat{p} + (1 - y) \log (1 - \hat{p})]$
3. Multiclass Classification: $\mathcal{Y} \in \{1, 2, \dots, K\}$ where $K > 2$. Instances belong to exactly one discrete category among $K$ mutually exclusive outcomes. Evaluated using categorical cross-entropy:
   $L(\mathbf{y}, \hat{\mathbf{p}}) = - \sum_{k=1}^K y_k \log \hat{p}_k$
4. Multilabel Classification: $\mathcal{Y} \in \{0, 1\}^K$. Each observation simultaneously possesses zero, one, or multiple independent non-exclusive labels (e.g., document tagging, image multi-object tagging).

1.2 Comparison Across Learning Paradigms

• Unsupervised Learning: Operates on unlabelled inputs $\{\mathbf{x}_i\}_{i=1}^N \subset \mathcal{X}$. Solves density estimation, latent manifold discovery, dimensionality reduction, and clustering without feedback signals.
• Semi-Supervised Learning: Leverages a small set of labeled observations paired with a voluminous corpus of unlabeled data, utilizing smoothness, cluster, or manifold assumptions to regularize decision boundaries.
• Reinforcement Learning (RL): Involves an autonomous agent exploring an interactive environment modeled as a Markov Decision Process (MDP). Receives scalar delayed reward feedback rather than immediate input-target pairings.

1.3 Inductive Bias and No Free Lunch Theorem

Without prior assumptions regarding the underlying target function, no learning algorithm can outperform uniform random guessing across all conceivable data distributions—a fundamental principle encapsulated by Wolpert's No Free Lunch (NFL) Theorem. Consequently, every practical machine learning algorithm exhibits an inductive bias: the collection of structural, topological, or mathematical assumptions made to extrapolate predictions to unobserved regions of feature space.

• Linear Models: Assume the target can be approximated as a linear combination of input dimensions (hyperplanar decision surface).
• Nearest Neighbors: Assume locality; points proximal in Euclidean or Manhattan metric space share identical target distributions.
• Decision Trees: Assume orthogonal axis-aligned hyper-rectangular partitioning of feature space.

2. Dataset Ingestion, Types, and Exploratory Data Analysis (EDA)

Real-world supervised learning workflows encounter heterogeneous data modalities requiring distinct computational pipelines.

2.1 Tabular Data Representation and Ingestion

Tabular datasets structure observations as rows and feature attributes as columns. High-performance ingestion leverages Pandas, Polars, and Apache Arrow.

pythonCopy Code
import pandas as pd
import numpy as np

# Ingestion with explicit data type specification and chunking for memory safety
dtype_spec = {
    'transaction_id': 'int64',
    'user_id': 'int32',
    'transaction_amount': 'float32',
    'merchant_category': 'category',
    'is_fraudulent': 'int8'
}

# Reading large datasets via chunking to prevent out-of-memory (OOM) exceptions
chunk_iterator = pd.read_csv(
    'large_financial_transactions.csv',
    dtype=dtype_spec,
    chunksize=50000,
    parse_dates=['timestamp'],
    low_memory=False
)

processed_chunks = []
for chunk in chunk_iterator:
    # Downcast floats and integers to minimal sufficient bitwidths
    chunk['transaction_amount'] = pd.to_numeric(chunk['transaction_amount'], downcast='float')
    processed_chunks.append(chunk)

df = pd.concat(processed_chunks, ignore_index=True)
print(f"Dataset memory footprint: {df.memory_usage(deep=True).sum() / (1024**2):.2f} MB")

2.2 Scikit-Learn Synthetic Dataset Generation

For algorithm benchmarking, synthetic generative utilities provide controlled parametric testing:

pythonCopy Code
from sklearn.datasets import make_classification, make_regression

# Generating non-linear, collinear, and imbalanced synthetic classification instances
X_clf, y_clf = make_classification(
    n_samples=10000,
    n_features=20,
    n_informative=12,
    n_redundant=4,
    n_repeated=0,
    n_classes=2,
    weights=[0.92, 0.08],  # 8% minority class imbalance
    flip_y=0.01,           # 1% label noise
    random_state=42
)

# Generating synthetic regression with controlled noise and multicollinearity
X_reg, y_reg, coef_reg = make_regression(
    n_samples=5000,
    n_features=15,
    n_informative=8,
    noise=1.5,
    coef=True,
    random_state=42
)

2.3 Comprehensive Exploratory Data Analysis (EDA)

EDA systematically identifies data anomalies, empirical distribution shapes, skewness, collinearity, and multimodal structures prior to statistical modeling.

1. Location and Dispersion Metrics: Arithmetic mean ($\bar{x}$), median ($\tilde{x}$), trimmed mean, sample variance ($s^2$), standard deviation ($s$), and Interquartile Range ($\text{IQR} = Q_3 - Q_1$).
2. Shape Parameters:
   • Skewness ($\gamma_1$): Quantifies distributional asymmetry around the sample mean:
     $\gamma_1 = \frac{\frac{1}{N}\sum_{i=1}^N (x_i - \bar{x})^3}{\left(\frac{1}{N}\sum_{i=1}^N (x_i - \bar{x})^2\right)^{3/2}}$
     $\gamma_1 > 0$ indicates a prolonged right (positive) tail; $\gamma_1 < 0$ denotes left (negative) tail skewness.
   • Kurtosis ($\gamma_2$): Quantifies tail thickness relative to a standard Gaussian distribution. Excess kurtosis $> 0$ (leptokurtic) signifies fat tails prone to extreme outliers; excess kurtosis $< 0$ (platykurtic) signifies thin tails.
3. Multicollinearity and Variance Inflation Factor (VIF):
   Multicollinearity destabilizes matrix inversion in Ordinary Least Squares ($(\mathbf{X}^T\mathbf{X})^{-1}$), inflating coefficient standard errors. The Variance Inflation Factor for the $j$-th predictor is:
   $\text{VIF}_j = \frac{1}{1 - R_j^2}$
   where $R_j^2$ is the coefficient of determination obtained from regressing feature $\mathbf{x}_j$ against all remaining $d-1$ predictors. Values of $\text{VIF}_j > 5 \text{ to } 10$ confirm severe multicollinearity requiring regularization or feature pruning.

pythonCopy Code
from statsmodels.stats.outliers_influence import variance_inflation_factor

def calculate_vif(dataframe, numeric_cols):
    X = dataframe[numeric_cols].dropna()
    vif_data = pd.DataFrame()
    vif_data["feature"] = numeric_cols
    vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
    return vif_data.sort_values(by="VIF", ascending=False)

3. Data Cleaning and Missing Value Imputation

Data generated by physical instrumentation, enterprise databases, or human survey interactions routinely contains corrupted entries, missing values, and record duplication.

3.1 Taxonomy of Missing Data Mechanisms (Rubin's Framework)

Statistically sound handling of missing observations requires categorizing the underlying missingness mechanism:

1. Missing Completely at Random (MCAR): The probability of missingness is strictly independent of both observed variables and unobserved latent factors:
   $P(M \mid Y_{\text{obs}}, Y_{\text{mis}}) = P(M)$
   Example: A laboratory test tube randomly shatters due to handling error. Dropping rows yields unbiased estimates, though it degrades statistical power.
2. Missing at Random (MAR): Missingness probability depends systematically on observed data, but is conditionally independent of the missing values themselves:
   $P(M \mid Y_{\text{obs}}, Y_{\text{mis}}) = P(M \mid Y_{\text{obs}})$
   Example: Male survey respondents under-report depression metrics, but conditional on gender and age, missingness is completely random. Controlled conditional imputation yields unbiased estimates.
3. Missing Not at Random (MNAR): The missingness mechanism depends directly on the unobserved missing value itself:
   $P(M \mid Y_{\text{obs}}, Y_{\text{mis}}) \neq P(M \mid Y_{\text{obs}})$
   Example: High-income individuals refuse to disclose annual wealth. Deletion or naive mean imputation introduces severe selection bias; joint modeling or missingness indicator flags are mandatory.

3.2 Imputation Methodologies in Scikit-Learn

• Univariate Imputation (SimpleImputer): Replaces missing cells via scalar column summaries (mean, median, mode, or arbitrary constant).
  • Mean: Preserves column mean; drastically underestimates variance and suppresses feature covariance. Applicable only for symmetric distributions under MCAR.
  • Median: Resistant to extreme outliers; ideal for skewed distributions.
  • Most Frequent (Mode): Standard fallback for discrete categorical features.
  • Constant: Injects fixed sentinel token (e.g., "missing", -999).
• Missingness Indicator Flag (MissingIndicator): Explicitly appends binary indicator columns $I(x_{ij} = \text{NaN})$ to preserve information regarding non-random missingness mechanisms.
• K-Nearest Neighbors Imputation (KNNImputer): Completes missing coordinates by computing a weighted average across the $k$ most proximal complete instances using Euclidean distance normalized across available coordinates:
  $d(\mathbf{u}, \mathbf{v}) = \sqrt{\frac{d_{\text{total}}}{d_{\text{observed}}} \sum_{i \in \text{observed}} (u_i - v_i)^2}$
• Multivariate Imputation by Chained Equations (MICE / IterativeImputer): Models each feature with missing coordinates as a regression function of all other features in a round-robin cycle until convergence.

pythonCopy Code
from sklearn.impute import SimpleImputer, KNNImputer, MissingIndicator
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
from sklearn.linear_model import BayesianRidge

# Simple Median Imputer with MissingIndicator concatenation
simple_imp = SimpleImputer(strategy='median', add_indicator=True)
X_imputed_simple = simple_imp.fit_transform(X_raw)

# Multivariate Imputation via Chained Equations (MICE) using Bayesian Ridge regression
mice_imp = IterativeImputer(
    estimator=BayesianRidge(),
    max_iter=15,
    tol=1e-3,
    imputation_order='ascending',
    random_state=42
)
X_imputed_mice = mice_imp.fit_transform(X_raw)

# KNN Imputer using 5 neighbors weighted by inverse distance
knn_imp = KNNImputer(n_neighbors=5, weights='distance')
X_imputed_knn = knn_imp.fit_transform(X_raw)

4. Outlier Detection and Treatment

Outliers represent observations deviating so markedly from the structural distribution of the dataset as to suggest generation by an alternative mechanism or measurement error.

4.1 Statistical Detection Techniques

1. Z-Score Rule (Parametric): Assumes an underlying Gaussian distribution $\mathcal{N}(\mu, \sigma^2)$:
   $z_i = \frac{x_i - \mu}{\sigma}$
   Observations where $|z_i| > 3.0$ (or $3.5$) are flagged as outliers. A severe limitation is masking: true outliers inflate $\mu$ and $\sigma$, thereby dampening their own calculated Z-scores.
2. Modified Z-Score via Median Absolute Deviation (MAD): A robust alternative immune to masking:
   $\text{MAD} = \text{median}(|x_i - \tilde{x}|), \quad M_i = \frac{0.6745 \cdot (x_i - \tilde{x})}{\text{MAD}}$
   Points with $|M_i| > 3.5$ constitute statistical anomalies.
3. Tukey’s Interquartile Range (IQR) Fences (Non-parametric):
   $\text{Lower Fence} = Q_1 - 1.5 \times \text{IQR}, \quad \text{Upper Fence} = Q_3 + 1.5 \times \text{IQR}$
   Points transcending extreme fences ($Q \pm 3.0 \times \text{IQR}$) are classified as extreme outliers.
4. Mahalanobis Distance (Multivariate Parametric): Measures distance from sample centroid $\boldsymbol{\mu}$ accounting for feature covariance matrix $\boldsymbol{\Sigma}$:
   $D_M(\mathbf{x}) = \sqrt{(\mathbf{x} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})}$
   Squared Mahalanobis distances follow a Chi-Square distribution $\chi_d^2$ with $d$ degrees of freedom.

4.2 Machine Learning Detection Algorithms

• Isolation Forest (IsolationForest): Constructs randomized isolation trees. Anomalous instances reside in sparse peripheral regions, requiring fewer axis-aligned splits to isolate than normal instances. Anomaly score derives from average path length $E(h(\mathbf{x}))$.
• Local Outlier Factor (LocalOutlierFactor): Computes local reachability density of an instance relative to its $k$-nearest neighbors. LOF values $\approx 1$ denote uniform cluster density; values $\gg 1$ flag isolated outliers.

pythonCopy Code
from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor

# Isolation Forest anomaly scoring
iso_forest = IsolationForest(contamination=0.03, random_state=42, n_jobs=-1)
outlier_pred_iso = iso_forest.fit_predict(X_data)  # Returns -1 for outliers, 1 for inliers

# Local Outlier Factor
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.03)
outlier_pred_lof = lof.fit_predict(X_data)

4.3 Outlier Remediation Strategies

• Trimming: Complete deletion of records containing verified measurement errors. Harmful when sample size is constrained.
• Winsorization (Capping): Clamping extreme tails to specified percentiles (e.g., setting values $< 1\text{st percentile}$ to $p_{0.01}$ and values $> 99\text{th percentile}$ to $p_{0.99}$).
• Power Transformations: Compresses extreme right-skewed heavy tails toward Gaussianity:
  • Logarithmic: $y = \log(x + 1)$ for strictly positive features.
  • Box-Cox Transformation: Requires $x > 0$:
    $y^{(\lambda)} = \begin{cases} \frac{x^\lambda - 1}{\lambda} & \text{if } \lambda \neq 0 \\ \ln(x) & \text{if } \lambda = 0 \end{cases}$
  • Yeo-Johnson Transformation: Generalizes Box-Cox to accommodate zero and negative values.

5. Class Imbalance Remediation

Class imbalance manifests when class priors are heavily skewed (e.g., fraud rates of 0.1%, rare disease prevalence of 0.5%). Models minimizing unweighted empirical loss predict the majority class uniformly, achieving deceptive 99.9% accuracy while exhibiting 0.0% recall on the minority class of interest.

5.1 Resampling Paradigms

1. Random Undersampling (RUS): Randomly discards majority class instances until balance is achieved.
   Risk: Loss of critical discriminant information in majority class regions.
2. Random Oversampling (ROS): Replicates existing minority samples randomly.
   Risk: Severe overfitting, as learning algorithms memorize identical duplicate feature vectors.
3. SMOTE (Synthetic Minority Over-sampling Technique):
   For every minority sample $\mathbf{x}_i$, computes its $k$-nearest minority neighbors. Randomly selects one neighbor $\mathbf{x}_{zi}$ and synthesizes a new point along the interconnecting line segment:
   $\mathbf{x}_{\text{synthetic}} = \mathbf{x}_i + \lambda (\mathbf{x}_{zi} - \mathbf{x}_i), \quad \lambda \sim U(0, 1)$
   Risk: Interpolation across noisy or boundary regions generates unrealistic synthetic artifacts.
4. Borderline-SMOTE: Identifies minority samples whose $k$-nearest neighbors are predominantly majority instances (borderline danger zone) and restricts synthetic generation strictly to these vulnerable boundary points.
5. ADASYN (Adaptive Synthetic): Evaluates the ratio of majority neighbors around each minority point and adaptively creates more synthetic samples in regions with high local majority concentration (difficult-to-learn regions).
6. Hybrid Methods (Over-sampling followed by Cleaning):
   • SMOTE + Tomek Links: Synthesizes minority instances via SMOTE, then searches for Tomek Links (pairs of nearest neighbors belonging to opposing classes) and strips majority instances from these ambiguous boundary pairs.
   • SMOTE + ENN (Edited Nearest Neighbors): Applies SMOTE, then strips any instance whose class contradicts the majority label of its 3 nearest neighbors.

pythonCopy Code
from imblearn.over_sampling import SMOTE, BorderlineSMOTE, ADASYN
from imblearn.combine import SMOTEENN, SMOTETomek
from collections import Counter

# Applying Borderline-SMOTE
bsmote = BorderlineSMOTE(sampling_strategy='auto', k_neighbors=5, random_state=42)
X_resampled, y_resampled = bsmote.fit_resample(X_train, y_train)

# Hybrid over-sampling and cleaning with SMOTEENN
smote_enn = SMOTEENN(smote=SMOTE(k_neighbors=5, random_state=42), random_state=42)
X_hybrid, y_hybrid = smote_enn.fit_resample(X_train, y_train)

5.2 Algorithmic Cost-Sensitive Learning

Rather than altering data distributions, cost-sensitive learning imposes asymmetric misclassification penalties. In Scikit-Learn, passing class_weight='balanced' assigns sample weight $w_k$ to class $k$:
$w_k = \frac{N}{K \cdot N_k}$
where $N$ is total sample count, $K$ is number of classes, and $N_k$ is count of samples in class $k$. Consequently, minority misclassifications contribute proportionally larger gradients during optimization.

6. Feature Scaling and Numerical Preprocessing

Supervised estimators compute distances, dot products, or gradient steps across numerical dimensions. If features possess disparate variance or measurement scales (e.g., Age in $[18, 90]$ vs. Income in $[15000, 1000000]$), models suffer severe convergence degradation or bias toward high-magnitude features.

6.1 Mathematical Scaling Formulations

6.2 Invariance vs. Sensitivity Across Estimator Families

• Distance-Based Estimators (KNN, SVM with RBF Kernel, K-Means): Critically Sensitive. Distances are dominated by features with largest variances.
• Gradient-Based Optimizers (Logistic Regression, Neural Networks, Linear Regression): Highly Sensitive. Disparate scales result in elliptical, poorly conditioned loss contours with high condition numbers, causing zig-zagging gradient oscillations.
• Tree-Based Ensembles (Decision Trees, Random Forest, XGBoost, LightGBM): Completely Invariant. Decision trees split features based on monotonic ordering inequalities ($x_j \le \theta$). Monotonic scaling transforms preserve ordinal relationships identically.

pythonCopy Code
from sklearn.preprocessing import StandardScaler, RobustScaler, MinMaxScaler, QuantileTransformer

# Robust scaling followed by quantile mapping to standard normal
robust = RobustScaler()
X_robust = robust.fit_transform(X_numeric)

quantile = QuantileTransformer(output_distribution='normal', random_state=42)
X_gaussian = quantile.fit_transform(X_robust)

7. Categorical Encoding Schemes

Machine learning models require numerical tensor inputs. Categorical features require mathematical mapping into real-valued coordinates.

7.1 Encoding Techniques

1. Ordinal Encoding (OrdinalEncoder): Maps discrete levels into ordered integer values: $\{c_1, c_2, \dots, c_m\} \to \{0, 1, \dots, m-1\}$.
   Applicability: Strictly appropriate for ordinal features with natural ranking (e.g., Education: High School=0, Bachelor's=1, Master's=2, PhD=3). Imposing ordinal integers on nominal categories (e.g., Country) introduces spurious distance metrics that distort linear and distance-based estimators.
2. One-Hot Encoding (OneHotEncoder): Constructs $m$ binary dummy variables for an $m$-level nominal attribute.
   Dummy Variable Trap: When an intercept term is present in linear regression, including all $m$ indicator columns induces exact multicollinearity ($\sum_{j=1}^m D_j = 1$). Dropping the baseline reference category (drop='first') resolves singularity.
   High Cardinality Bottleneck: If a categorical column contains 5,000 postal codes, One-Hot Encoding inflates feature dimensionality by 5,000 sparse dimensions, causing memory exhaustion and curse of dimensionality.
3. Target / Mean Encoding: Replaces each category level $c$ with the expected value of the target label $y$ conditioned on category $c$:
   $S_c = E[y \mid x = c] = \frac{\sum_{i \in c} y_i}{n_c}$
   Target Leakage Risk: High-cardinality categories with few observations overfit drastically (e.g., a single customer who defaulted gives $S_c = 1.0$). To prevent leakage, m-estimate empirical Bayesian smoothing is applied:
   $S_c^{\text{smooth}} = \frac{n_c \cdot \bar{y}_c + m \cdot \bar{y}_{\text{global}}}{n_c + m}$
   where $m$ is a smoothing regularization weight and $\bar{y}_{\text{global}}$ is the global target mean.
4. Binary Encoding: Converts integer category ranks into binary bits, projecting $m$ categories into $\lceil \log_2(m) \rceil$ feature columns. Highly effective for medium-to-high cardinality nominal features.
5. Frequency / Count Encoding: Replaces each categorical category with its normalized frequency or occurrence count in the training dataset.

pythonCopy Code
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
from category_encoders import TargetEncoder, BinaryEncoder

# OneHotEncoder dropping the first category to avoid dummy trap
ohe = OneHotEncoder(drop='first', sparse_output=False, handle_unknown='ignore')
X_ohe = ohe.fit_transform(df[['device_type', 'subscription_tier']])

# Smooth Target Encoding with out-of-fold smoothing
target_encoder = TargetEncoder(cols=['zip_code', 'merchant_id'], smoothing=10.0)
X_target_encoded = target_encoder.fit_transform(df[['zip_code', 'merchant_id']], df['target'])

8. Feature Engineering and Selection Paradigms

Feature engineering constructs domain-relevant representations to linearize relationships and expose signal; feature selection prunes redundant, noisy, and uninformative dimensions to improve generalization and computational efficiency.

8.1 Advanced Feature Construction

• Polynomial Features and Interaction Terms: Generates explicit interaction terms ($x_i \cdot x_j$) and nonlinear terms ($x_i^2$) enabling linear estimators to capture complex non-linear manifolds:
  $\phi(\mathbf{x}) = [1, x_1, x_2, x_1^2, x_1 x_2, x_2^2]$
• Discretization and Binning (KBinsDiscretizer): Partitions continuous features into discrete bins using uniform widths, quantiles, or K-Means clustering. Introduces non-linearity into linear models and isolates outlier spikes.
• Ratio and Domain Interaction Features: Constructing scale-invariant ratios such as Debt-to-Income, Price-to-Earnings, or Sensor Differential gradients.

pythonCopy Code
from sklearn.preprocessing import PolynomialFeatures, KBinsDiscretizer

# Polynomial features with degree 2 interaction terms only
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
X_interactions = poly.fit_transform(X_numeric)

# Quantile-based binning into 5 discrete ordinal categories
binner = KBinsDiscretizer(n_bins=5, encode='ordinal', strategy='quantile')
X_binned = binner.fit_transform(X_numeric[['income']])

8.2 Feature Selection Taxonomies

Feature selection techniques are categorized into three algorithmic families:

CODECopy Code
                   Feature Selection Paradigms
                                │
        ┌───────────────────────┼───────────────────────┐
        ▼                       ▼                       ▼
  Filter Methods          Wrapper Methods        Embedded Methods
  - Fast, statistical     - Model-dependent      - Regularization-driven
  - Independent of model  - Iterative search     - Built into training
  - Chi-Square, ANOVA     - RFE, Forward/Back    - Lasso (L1), Tree Gini
  - Mutual Information    - Computationally slow - ElasticNet

1. Filter Methods: Apply statistical hypothesis testing to evaluate feature relevance independently of model training:
   • Variance Thresholding: Discards low-variance features that remain constant across nearly all samples: $\text{Var}(X) \le \tau$.
   • Pearson Correlation: Measures bivariate linear association for continuous features.
   • ANOVA F-test (f_classif, f_regression): Evaluates whether group means differ significantly across class categories.
   • Mutual Information (mutual_info_classif, mutual_info_regression): Non-parametric metric quantifying statistical dependence based on Shannon entropy:
     $I(X; Y) = \iint p(x, y) \log \frac{p(x, y)}{p(x) p(y)} \, dx \, dy$
     Detects arbitrary non-linear relationships without parametric distributional assumptions.
2. Wrapper Methods: Treat model training as an evaluation oracle within a combinatorial search space:
   • Recursive Feature Elimination (RFE / RFECV): Fits an estimator, calculates feature importance attributes (coef_ or feature_importances_), and recursively strips the least important features, utilizing cross-validation to select the optimal subset cardinality.
   • Sequential Feature Selector (SFS): Forward selection progressively adds the best performing feature; backward selection starts with all features and iteratively prunes the worst contributor.
3. Embedded Methods: Perform feature selection inherently during model optimization:
   • Lasso Regularization ($L_1$ Penalty): Adds an $L_1$ norm penalty to the loss function ($\lambda \sum |\beta_j|$). The geometry of the $L_1$ diamond ball induces exact sparsity, driving redundant feature weights to zero.
   • Tree-based Feature Importances: Calculates total impurity reduction (Gini or Mean Decrease Impurity) aggregated across all split nodes.
   • Permutation Feature Importance: Measures the drop in validation score when values of a single feature column are randomly shuffled, breaking its relationship with the target. Avoids tree-based impurity biases toward high-cardinality features.

pythonCopy Code
from sklearn.feature_selection import (
    VarianceThreshold, SelectKBest, mutual_info_classif,
    RFE, SelectFromModel
)
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

# 1. Filter: Select top 15 features via Mutual Information
selector_mi = SelectKBest(score_func=mutual_info_classif, k=15)
X_mi_selected = selector_mi.fit_transform(X_train, y_train)

# 2. Wrapper: Recursive Feature Elimination with Cross-Validation
rfe = RFE(estimator=LogisticRegression(penalty='l2', max_iter=1000), n_features_to_select=10, step=2)
X_rfe_selected = rfe.fit_transform(X_train, y_train)

# 3. Embedded: Feature selection via L1-penalized Lasso / Logistic Regression
lasso_selector = SelectFromModel(
    estimator=LogisticRegression(penalty='l1', solver='liblinear', C=0.1),
    threshold='median'
)
X_lasso_selected = lasso_selector.fit_transform(X_train, y_train)

9. Data Splitting Strategies and Data Leakage Prevention

Rigorous model validation requires strict isolation of evaluation data from the training phase.

9.1 Data Partitioning Strategies

• Simple Train-Test Split (train_test_split): Randomly partitions data according to a specified ratio (e.g., 80% train, 20% test). Applicable only when data is independent and identically distributed with balanced classes.
• Stratified Partitioning (StratifiedKFold, train_test_split(stratify=y)): Enforces exact target class percentage proportions across both training and test partitions. Mandatory for imbalanced classification tasks.
• Group Partitioning (GroupKFold, GroupShuffleSplit): Prevents observations belonging to the same entity (e.g., multiple medical images from the same patient, multiple reviews from the same user) from spanning both train and test partitions.
• Temporal Splitting (TimeSeriesSplit): Enforces chronological forward-chaining splits where test folds always follow training folds chronologically, preventing lookahead bias.

CODECopy Code
Temporal Split (TimeSeriesSplit):
Fold 1: [Train: T1]             --> [Test: T2]
Fold 2: [Train: T1, T2]         --> [Test: T3]
Fold 3: [Train: T1, T2, T3]     --> [Test: T4]
Fold 4: [Train: T1, T2, T3, T4] --> [Test: T5]

9.2 The Anatomy of Data Leakage

Data leakage occurs when information from outside the training dataset is inadvertently utilized to build or train the model, resulting in overly optimistic validation scores and catastrophic test/production failure.

1. Pre-Processing Leakage: Fitting transformers (StandardScaler, SimpleImputer, PCA, TargetEncoder) on the entire dataset prior to train-test splitting. Validation sample means, standard deviations, and label distributions bleed into training folds.
   Resolution: Always fit transformers strictly on X_train and call transform() on X_test.
2. Target Leakage: Features constructed using knowledge of the target outcome that will not be available at real-world inference time (e.g., using "hospital discharge date" to predict "patient mortality").
3. Temporal Leakage: Shuffling time-series observations randomly, allowing past predictions to condition on future target states.

9.3 Scikit-Learn Pipeline Encapsulation

The Pipeline abstraction encapsulates preprocessing transformers and estimators into a single atomic object, ensuring all fitting occurs strictly on training folds during cross-validation.

pythonCopy Code
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score

numeric_features = ['age', 'income', 'credit_score']
categorical_features = ['employment_type', 'education_level']

numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder(drop='first', handle_unknown='ignore'))
])

preprocessor = ColumnTransformer(transformers=[
    ('num', numeric_transformer, numeric_features),
    ('cat', categorical_transformer, categorical_features)
])

full_pipeline = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', LogisticRegression(solver='liblinear', random_state=42))
])

# Cross-validation evaluates pipeline atomically without data leakage
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(full_pipeline, X, y, cv=cv, scoring='roc_auc')
print(f"5-Fold Cross-Validated Mean ROC-AUC: {scores.mean():.4f} (+/- {scores.std():.4f})")

10. Dimensionality Reduction Foundations: PCA and LDA

As feature dimensionality $d$ expands, the volume of feature space grows exponentially, rendering sample density sparse—a phenomenon termed the Curse of Dimensionality. High-dimensional spaces lead to distance concentration: the Euclidean distance between any two random points converges to a constant ratio, degrading nearest-neighbor and kernel methods.

10.1 Principal Component Analysis (PCA)

PCA is an unsupervised, orthogonal linear transformation that projects data onto a lower-dimensional coordinate system that maximizes sample variance.

Mathematical Formulation

Let $\mathbf{X} \in \mathbb{R}^{N \times d}$ be a centered data matrix ($\mathbb{E}[\mathbf{X}] = \mathbf{0}$). The sample covariance matrix is:
$\boldsymbol{\Sigma} = \frac{1}{N-1} \mathbf{X}^T \mathbf{X} \in \mathbb{R}^{d \times d}$
PCA seeks a unit projection vector $\mathbf{w}_1 \in \mathbb{R}^d$ ($\|\mathbf{w}_1\|_2 = 1$) maximizing projected variance:
$\max_{\|\mathbf{w}_1\|=1} \mathbf{w}_1^T \boldsymbol{\Sigma} \mathbf{w}_1$
Formulating the Lagrangian:
$\mathcal{L}(\mathbf{w}_1, \lambda_1) = \mathbf{w}_1^T \boldsymbol{\Sigma} \mathbf{w}_1 - \lambda_1 (\mathbf{w}_1^T \mathbf{w}_1 - 1)$
Setting the gradient with respect to $\mathbf{w}_1$ to zero:
$\nabla_{\mathbf{w}_1} \mathcal{L} = 2 \boldsymbol{\Sigma} \mathbf{w}_1 - 2 \lambda_1 \mathbf{w}_1 = \mathbf{0} \implies \boldsymbol{\Sigma} \mathbf{w}_1 = \lambda_1 \mathbf{w}_1$
Thus, $\mathbf{w}_1$ is the principal eigenvector of $\boldsymbol{\Sigma}$, and $\lambda_1$ represents the variance captured along that axis. In practice, modern implementations compute PCA via Singular Value Decomposition (SVD) of data matrix $\mathbf{X} = \mathbf{U} \boldsymbol{\Sigma}_s \mathbf{V}^T$, bypassing the direct computation and inversion of $\mathbf{X}^T\mathbf{X}$.

pythonCopy Code
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

# Center and scale data prior to PCA
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_numeric)

# Fit PCA retaining 95% of total cumulative variance
pca = PCA(n_components=0.95, random_state=42)
X_pca = pca.fit_transform(X_scaled)

explained_variance = pca.explained_variance_ratio_
cumulative_variance = np.cumsum(explained_variance)
print(f"Original dimensions: {X_scaled.shape[1]} -> Reduced dimensions: {X_pca.shape[1]}")
print(f"Variance explained per component: {explained_variance}")

10.2 Linear Discriminant Analysis (LDA)

Unlike PCA, LDA is a supervised dimensionality reduction technique that identifies projection axes maximizing class separability.

Fisher's Linear Discriminant

LDA defines two scatter matrices:

1. Within-Class Scatter Matrix ($\mathbf{S}_W$): Measures spread around class centroids:
   $\mathbf{S}_W = \sum_{k=1}^K \sum_{\mathbf{x} \in C_k} (\mathbf{x} - \boldsymbol{\mu}_k)(\mathbf{x} - \boldsymbol{\mu}_k)^T$
2. Between-Class Scatter Matrix ($\mathbf{S}_B$): Measures spread of class centroids around global centroid $\boldsymbol{\mu}$:
   $\mathbf{S}_B = \sum_{k=1}^K N_k (\boldsymbol{\mu}_k - \boldsymbol{\mu})(\boldsymbol{\mu}_k - \boldsymbol{\mu})^T$

LDA maximizes Fisher's criterion:
$J(\mathbf{w}) = \frac{\mathbf{w}^T \mathbf{S}_B \mathbf{w}}{\mathbf{w}^T \mathbf{S}_W \mathbf{w}}$
This corresponds to solving the generalized eigenvalue problem:
$\mathbf{S}_W^{-1} \mathbf{S}_B \mathbf{w} = \lambda \mathbf{w}$
Because $\mathbf{S}_B$ has a maximum rank of $K - 1$ (where $K$ is the number of distinct target classes), LDA can project data into at most $\min(d, K - 1)$ dimensional space.

pythonCopy Code
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

# Supervised projection using LDA into K-1 discriminant dimensions
lda = LinearDiscriminantAnalysis(n_components=min(X_train.shape[1], len(np.unique(y_train)) - 1))
X_lda_train = lda.fit_transform(X_train, y_train)
X_lda_test = lda.transform(X_test)

10.3 Comparative Summary: PCA vs. LDA

11. End-to-End Enterprise Data Preprocessing Architecture

Below is a complete enterprise-grade Python implementation unifying the concepts covered in this unit:

pythonCopy Code
import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.preprocessing import RobustScaler, OneHotEncoder, TargetEncoder
from sklearn.feature_selection import SelectKBest, mutual_info_classif
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score

class OutlierCapper(BaseEstimator, TransformerMixin):
    """Caps numerical features to 1st and 99th percentiles to mitigate extreme outliers."""
    def __init__(self, lower_percentile=1.0, upper_percentile=99.0):
        self.lower_percentile = lower_percentile
        self.upper_percentile = upper_percentile
        self.lower_bounds_ = {}
        self.upper_bounds_ = {}

    def fit(self, X, y=None):
        X_df = pd.DataFrame(X)
        for col in X_df.columns:
            self.lower_bounds_[col] = np.percentile(X_df[col].dropna(), self.lower_percentile)
            self.upper_bounds_[col] = np.percentile(X_df[col].dropna(), self.upper_percentile)
        return self

    def transform(self, X):
        X_df = pd.DataFrame(X).copy()
        for col in X_df.columns:
            X_df[col] = np.clip(X_df[col], self.lower_bounds_[col], self.upper_bounds_[col])
        return X_df.values

def build_enterprise_preprocessing_pipeline(numerical_cols, categorical_low_card, categorical_high_card):
    # Numerical pipeline: Outlier capping -> KNN Imputation -> Robust Scaling
    num_pipeline = Pipeline([
        ('capper', OutlierCapper(lower_percentile=1.0, upper_percentile=99.0)),
        ('imputer', KNNImputer(n_neighbors=5, weights='distance')),
        ('scaler', RobustScaler())
    ])

    # Low-cardinality nominal pipeline: Frequent imputation -> One-Hot Encoding
    cat_low_pipeline = Pipeline([
        ('imputer', SimpleImputer(strategy='most_frequent')),
        ('ohe', OneHotEncoder(drop='first', sparse_output=False, handle_unknown='ignore'))
    ])

    # High-cardinality categorical pipeline: Most frequent imputation -> Target Encoding
    cat_high_pipeline = Pipeline([
        ('imputer', SimpleImputer(strategy='most_frequent')),
        ('target_enc', TargetEncoder(cv=5, smooth='auto'))
    ])

    # ColumnTransformer combining feature pipelines
    preprocessor = ColumnTransformer(
        transformers=[
            ('num', num_pipeline, numerical_cols),
            ('cat_low', cat_low_pipeline, categorical_low_card),
            ('cat_high', cat_high_pipeline, categorical_high_card)
        ],
        remainder='drop'
    )

    # Full Model Pipeline: Preprocessor -> Feature Selection -> Estimator
    full_pipeline = Pipeline([
        ('preprocessor', preprocessor),
        ('selector', SelectKBest(score_func=mutual_info_classif, k='all')),
        ('classifier', RandomForestClassifier(n_estimators=100, class_weight='balanced', random_state=42, n_jobs=-1))
    ])

    return full_pipeline