Unit 1: Random Variables and Probability Distributions

1. Fundamentals of Random Variables

1.1 Formal Mathematical Definition

In probability theory, a random experiment is characterized by a probability space $(\Omega, \mathcal{F}, \mathbb{P})$, where:

• $\Omega$ represents the sample space containing all elementary outcomes $\omega$.
• $\mathcal{F}$ denotes a $\sigma$-algebra of events (subsets of $\Omega$).
• $\mathbb{P}: \mathcal{F} \to [0, 1]$ is a Kolmogorov probability measure satisfying non-negativity, unit measure ($\mathbb{P}(\Omega) = 1$), and countable additivity.

A Random Variable (RV) $X$ is a measurable mapping from the sample space $\Omega$ to the real line $\mathbb{R}$:
$X: \Omega \to \mathbb{R}$
such that for every Borel set $B \in \mathcal{B}(\mathbb{R})$, the pre-image belongs to $\mathcal{F}$:
$X^{-1}(B) = \{\omega \in \Omega : X(\omega) \in B\} \in \mathcal{F}$

CODECopy Code
   Sample Space (Omega)                         Real Number Line (R)
   +---------------------+                     +---------------------+
   |      omega_1        | ---- X(omega_1) --> |        x_1          |
   |      omega_2        | ---- X(omega_2) --> |        x_2          |
   |      omega_3        | ---- X(omega_3) --> |        x_3          |
   +---------------------+                     +---------------------+
          Event E                                   Borel Set B

1.2 Classification: Discrete vs. Continuous Random Variables

1. Discrete Random Variable: Takes values in a countable set $S_X \subseteq \mathbb{R}$ (finite or countably infinite, e.g., integers $\mathbb{Z}$ or natural numbers $\mathbb{N}$).
2. Continuous Random Variable: Takes an uncountably infinite continuum of values over an interval $(a, b) \subseteq \mathbb{R}$, where the probability of realizing any single point is identically zero: $\mathbb{P}(X = c) = 0$ for any $c \in \mathbb{R}$.

2. Probability Functions: PMF, PDF, and CDF

2.1 Probability Mass Function (PMF)

For a discrete random variable $X$ with support $\{x_1, x_2, \dots\}$, the Probability Mass Function (PMF) $p_X(x)$ assigns probability point masses:
$p_X(x) = \mathbb{P}(X = x)$

Axiomatic Conditions:

1. Non-negativity: $p_X(x_i) \ge 0, \quad \forall x_i \in S_X$
2. Total probability: $\sum_{x_i \in S_X} p_X(x_i) = 1$
3. Subset probability: For any event $A \subseteq \mathbb{R}$, $\mathbb{P}(X \in A) = \sum_{x_i \in A} p_X(x_i)$

2.2 Probability Density Function (PDF)

For a continuous random variable $X$, probabilities are described by an integrable function $f_X(x)$ known as the Probability Density Function (PDF):
$\mathbb{P}(a \le X \le b) = \int_a^b f_X(x) \, dx$

Axiomatic Conditions:

1. Non-negativity: $f_X(x) \ge 0, \quad \forall x \in \mathbb{R}$
2. Normalization: $\int_{-\infty}^{\infty} f_X(x) \, dx = 1$
3. Point mass vanishing: For any singleton $c \in \mathbb{R}$, $\mathbb{P}(X = c) = \int_c^c f_X(x) \, dx = 0$
4. Boundary equivalence: $\mathbb{P}(a \le X \le b) = \mathbb{P}(a < X \le b) = \mathbb{P}(a \le X < b) = \mathbb{P}(a < X < b)$

2.3 Cumulative Distribution Function (CDF)

The Cumulative Distribution Function (CDF), denoted $F_X(x)$, unifies discrete and continuous variables by defining accumulated probability up to a threshold $x$:
$F_X(x) = \mathbb{P}(X \le x), \quad \forall x \in \mathbb{R}$

Explicit Formulations:

• Discrete: $F_X(x) = \sum_{x_i \le x} p_X(x_i)$ (a monotonic right-continuous step function with jump discontinuities at support points of magnitude $p_X(x_i)$).
• Continuous: $F_X(x) = \int_{-\infty}^x f_X(t) \, dt$ (an absolutely continuous function).

Fundamental Properties of the CDF:

1. Bounded Range: $0 \le F_X(x) \le 1, \quad \forall x \in \mathbb{R}$
2. Monotonic Non-Decreasing: If $x_1 < x_2$, then $F_X(x_1) \le F_X(x_2)$
3. Asymptotic Limits:
   $\lim_{x \to -\infty} F_X(x) = 0, \quad \lim_{x \to \infty} F_X(x) = 1$
4. Right-Continuity: $F_X(x^+) = \lim_{h \to 0^+} F_X(x + h) = F_X(x)$
5. Interval Probability Rule:
   $\mathbb{P}(a < X \le b) = F_X(b) - F_X(a)$
6. Derivative Relation: At every point where $F_X(x)$ is differentiable:
   $f_X(x) = \frac{d}{dx} F_X(x)$

3. Mathematical Expectation & Variance

3.1 Definition of Expected Value

The mathematical expectation (mean) $\mathbb{E}[X]$ or $\mu$ represents the probability-weighted central value:
$\mathbb{E}[X] = \begin{cases} \sum_{i} x_i p_X(x_i) & \text{if } X \text{ is discrete} \\ \int_{-\infty}^{\infty} x f_X(x) \, dx & \text{if } X \text{ is continuous} \end{cases}$
Absolute Convergence Condition: The expectation exists if and only if $\mathbb{E}[|X|] < \infty$.

3.2 Law of the Unconscious Statistician (LOTUS)

For any measurable function $g: \mathbb{R} \to \mathbb{R}$:
$\mathbb{E}[g(X)] = \begin{cases} \sum_{i} g(x_i) p_X(x_i) & \text{discrete} \\ \int_{-\infty}^{\infty} g(x) f_X(x) \, dx & \text{continuous} \end{cases}$

3.3 Fundamental Properties of Expectation

1. Linearity: $\mathbb{E}[aX + bY + c] = a\mathbb{E}[X] + b\mathbb{E}[Y] + c$, for any random variables $X, Y$ (regardless of independence).
2. Monotonicity: If $X \le Y$ almost surely, then $\mathbb{E}[X] \le \mathbb{E}[Y]$.
3. Product Rule for Independent RVs: If $X$ and $Y$ are independent, $\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y]$.

3.4 Variance and Standard Deviation

The Variance $\text{Var}(X) = \sigma^2$ measures dispersion around the mean:
$\text{Var}(X) = \mathbb{E}[(X - \mu)^2] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$

Properties of Variance:

1. Non-negativity: $\text{Var}(X) \ge 0$.
2. Invariance to Shift: $\text{Var}(X + c) = \text{Var}(X)$.
3. Scaling Factor: $\text{Var}(aX) = a^2 \text{Var}(X) \implies \text{Var}(aX + b) = a^2 \text{Var}(X)$.
4. Sum of Two Random Variables:
   $\text{Var}(X \pm Y) = \text{Var}(X) + \text{Var}(Y) \pm 2\text{Cov}(X, Y)$
   If $X$ and $Y$ are uncorrelated ($\text{Cov}(X, Y) = 0$), then $\text{Var}(X \pm Y) = \text{Var}(X) + \text{Var}(Y)$.
5. Standard Deviation: $\sigma = \sqrt{\text{Var}(X)}$.

4. Higher Order Moments, Skewness & Kurtosis

4.1 Raw Moments vs. Central Moments

• $r$-th Raw Moment (about the origin):
  $\mu_r' = \mathbb{E}[X^r]$
  Note: $\mu_1' = \mathbb{E}[X] = \mu$.
• $r$-th Central Moment (about the mean):
  $\mu_r = \mathbb{E}[(X - \mu)^r]$
  Note: $\mu_1 = \mathbb{E}[X - \mu] = 0$, $\mu_2 = \mathbb{E}[(X - \mu)^2] = \sigma^2 = \text{Var}(X)$.

4.2 Inter-relation Between Moments

Using the binomial expansion $\mu_r = \mathbb{E}[(X - \mu)^r] = \sum_{k=0}^r \binom{r}{k} (-1)^{r-k} \mu_k' (\mu_1')^{r-k}$:
$\mu_2 = \mu_2' - (\mu_1')^2$
$\mu_3 = \mu_3' - 3\mu_2'\mu_1' + 2(\mu_1')^3$
$\mu_4 = \mu_4' - 4\mu_3'\mu_1' + 6\mu_2'(\mu_1')^2 - 3(\mu_1')^4$

4.3 Skewness (Asymmetry of Probability Mass)

Skewness measures the degree of asymmetry of the probability distribution around its mean:

• Moment Coefficient of Skewness:
  $\gamma_1 = \frac{\mu_3}{\sigma^3} = \frac{\mu_3}{(\mu_2)^{3/2}}$
• Pearson's $\beta_1$ coefficient: $\beta_1 = \gamma_1^2 = \frac{\mu_3^2}{\mu_2^3}$.

CODECopy Code
   Negative Skew (Left-Tailed)     Symmetric (Mesokurtic)     Positive Skew (Right-Tailed)
         Mean < Median < Mode           Mean = Median = Mode        Mode < Median < Mean
              /\                                /\                              /\
             /  \                              /  \                            /  \
      ______/    \                            /    \                          /    \______
     gamma_1 < 0                             gamma_1 = 0                     gamma_1 > 0

4.4 Kurtosis (Peakedness and Tail Heaviness)

Kurtosis measures the tail-weight and sharpness of the distribution peak relative to a Gaussian distribution:

• Pearson's $\beta_2$ Coefficient:
  $\beta_2 = \frac{\mu_4}{\mu_2^2} = \frac{\mu_4}{\sigma^4}$
• Excess Kurtosis ($\gamma_2$):
  $\gamma_2 = \beta_2 - 3 = \frac{\mu_4}{\sigma^4} - 3$

Kurtosis Classification:

5. Generating Functions: MGF & Characteristic Function

5.1 Moment Generating Function (MGF)

The Moment Generating Function (MGF) of a random variable $X$ is defined as:
$M_X(t) = \mathbb{E}[e^{tX}]$
provided the expectation exists in an open neighborhood around $t = 0$ ($|t| < h$ for some $h > 0$).

Formulas:

• Discrete: $M_X(t) = \sum_i e^{tx_i} p_X(x_i)$
• Continuous: $M_X(t) = \int_{-\infty}^{\infty} e^{tx} f_X(x) \, dx$

5.2 Generating Moments from MGF

Using the Maclaurin series expansion of $e^{tX} = \sum_{r=0}^{\infty} \frac{(tX)^r}{r!}$:
$M_X(t) = \sum_{r=0}^{\infty} \frac{t^r}{r!} \mathbb{E}[X^r] = 1 + t\mu_1' + \frac{t^2}{2!}\mu_2' + \frac{t^3}{3!}\mu_3' + \dots$
Differentiating $r$ times with respect to $t$ and setting $t = 0$:
$\mu_r' = \mathbb{E}[X^r] = \left. \frac{d^r}{dt^r} M_X(t) \right|_{t=0}$

5.3 Algebraic Properties of the MGF

1. Affine Transformation: $M_{aX + b}(t) = e^{bt} M_X(at)$.
2. Sum of Independent Random Variables:
   If $X_1, X_2, \dots, X_n$ are mutually independent, then:
   $M_{\sum X_i}(t) = \prod_{i=1}^n M_{X_i}(t)$
3. Uniqueness Theorem: If two random variables have MGFs that agree on an open interval containing zero, they share identical probability distributions everywhere.

5.4 The Characteristic Function

When $\mathbb{E}[e^{tX}]$ diverges for $t \neq 0$ (e.g., Cauchy distribution), the Characteristic Function $\phi_X(t)$ is utilized:
$\phi_X(t) = \mathbb{E}[e^{itX}] = \int_{-\infty}^{\infty} e^{itx} f_X(x) \, dx, \quad (i = \sqrt{-1})$
Because $|e^{itX}| = 1$, the integral converges absolutely for all $t \in \mathbb{R}$, guaranteeing existence for every probability distribution.

6. Joint & Multivariable Probability Distributions

6.1 Bivariate Distributions

When pairs of random variables $(X, Y)$ are observed simultaneously:

• Joint PMF: $p_{X, Y}(x, y) = \mathbb{P}(X = x, Y = y)$ with $\sum_x \sum_y p_{X, Y}(x, y) = 1$.
• Joint PDF: $\mathbb{P}((X, Y) \in A) = \iint_A f_{X, Y}(x, y) \, dx \, dy$ with $\int_{-\infty}^\infty \int_{-\infty}^\infty f_{X, Y}(x, y) \, dx \, dy = 1$.
• Joint CDF: $F_{X, Y}(x, y) = \mathbb{P}(X \le x, Y \le y) = \int_{-\infty}^x \int_{-\infty}^y f_{X, Y}(u, v) \, dv \, du$.

6.2 Marginal Distributions

Collapsing across the partner dimension:
$f_X(x) = \int_{-\infty}^{\infty} f_{X, Y}(x, y) \, dy, \quad f_Y(y) = \int_{-\infty}^{\infty} f_{X, Y}(x, y) \, dx$

6.3 Conditional Distributions & Independence

• Conditional PDF:
  $f_{Y \mid X}(y \mid x) = \frac{f_{X, Y}(x, y)}{f_X(x)}, \quad \text{for } f_X(x) > 0$
• Statistical Independence: $X$ and $Y$ are independent if and only if:
  $f_{X, Y}(x, y) = f_X(x) f_Y(y), \quad \forall (x, y) \in \mathbb{R}^2$

6.4 Covariance and Correlation

The Covariance measures linear co-dependence:
$\text{Cov}(X, Y) = \sigma_{XY} = \mathbb{E}[(X - \mu_X)(Y - \mu_Y)] = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y]$
The Pearson Correlation Coefficient normalizes covariance into $[-1, 1]$:
$\rho_{X, Y} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}$
If $X$ and $Y$ are independent, $\text{Cov}(X, Y) = 0$ and $\rho_{X, Y} = 0$. (Note: Zero correlation does not generally imply independence, except under bivariate normality).

7. Fundamental Inequalities: Markov & Chebyshev

7.1 Markov's Inequality

Let $Y$ be a non-negative random variable ($Y \ge 0$) with finite mean $\mathbb{E}[Y]$. For any constant $a > 0$:
$\mathbb{P}(Y \ge a) \le \frac{\mathbb{E}[Y]}{a}$

7.2 Chebyshev's Inequality

Let $X$ have finite mean $\mu$ and variance $\sigma^2$. Setting $Y = (X - \mu)^2$ and $a = k^2 \sigma^2$ in Markov's inequality:
$\mathbb{P}(|X - \mu| \ge k\sigma) \le \frac{1}{k^2}, \quad \forall k > 0$
Complementary form:
$\mathbb{P}(|X - \mu| < k\sigma) \ge 1 - \frac{1}{k^2}$
For any arbitrary distribution, at least $75\%$ of probability mass lies within $k = 2$ standard deviations of the mean, and at least $88.89\%$ lies within $k = 3$ standard deviations.

8. Summary Formulation Table