Unit 1: Matrix Methods and Linear Systems

1. Mathematical Foundations of Matrices and Vector Spaces

1.1 Matrices as Linear Transformations

In engineering analysis, a matrix is not merely a tabular rectangular array of numbers, but the algebraic representation of a linear operator $T: \mathbb{R}^n \to \mathbb{R}^m$ acting on vector spaces. For any vector $x \in \mathbb{R}^n$, the mapping is governed by matrix-vector multiplication:
$y = T(x) = A x$
where $A$ is an $m \times n$ matrix with real or complex coefficients $a_{ij}$. Linear transformations preserve vector addition and scalar scaling:
$T(c_1 u + c_2 v) = c_1 T(u) + c_2 T(v)$
Matrices govern coordinate rotations, shear strains, scaling transformations, electrical network mesh loops, structural stiffness formulations, and discrete dynamical transitions.

1.2 Fundamental Matrix Algebra and Special Matrix Classes

1. Transpose and Conjugate Transpose:
   • Transpose: $A^T$ where $(A^T)_{ij} = a_{ji}$. Notice $(AB)^T = B^T A^T$.
   • Conjugate Transpose (Hermitian Adjoint): $A^H = (\overline{A})^T$.
2. Symmetric and Skew-Symmetric Matrices:
   • Symmetric: $A^T = A$ ($a_{ij} = a_{ji}$). All eigenvalues of a real symmetric matrix are guaranteed to be real, and eigenvectors corresponding to distinct eigenvalues are mutually orthogonal.
   • Skew-Symmetric (Antisymmetric): $A^T = -A$ ($a_{ij} = -a_{ji}$). Main diagonal entries must be strictly zero ($a_{ii} = 0$), and eigenvalues are either pure imaginary or zero.
   • Any square matrix $M$ can be uniquely decomposed as the sum of a symmetric and skew-symmetric matrix:
     $M = \frac{M + M^T}{2} + \frac{M - M^T}{2}$
3. Orthogonal Matrices:
   • An $n \times n$ real matrix $Q$ is orthogonal if $Q^T Q = Q Q^T = I$, implying $Q^{-1} = Q^T$.
   • The row and column vectors of an orthogonal matrix form an orthonormal basis.
   • Orthogonal transformations preserve Euclidean vector lengths (norms) and angles: $\|Qx\| = \|x\|$ and $(Qx) \cdot (Qy) = x \cdot y$.
   • The determinant of an orthogonal matrix satisfies $\det(Q) = \pm 1$.
4. Hermitian and Unitary Matrices (Complex Spaces):
   • Hermitian: $A^H = A$.
   • Unitary: $U^H U = I$, implying $U^{-1} = U^H$.

2. Elementary Row Operations and Echelon Reductions

2.1 Elementary Row Operations (EROs)

Matrix reduction relies on three row operations that preserve the solution space of the associated linear system:

1. Row Swapping ($R_i \leftrightarrow R_j$): Interchanging two rows.
2. Scalar Multiplication ($R_i \to c R_i$ where $c \neq 0$): Multiplying every entry of a row by a non-zero scalar.
3. Row Addition ($R_i \to R_i + k R_j$): Adding a scalar multiple of row $j$ to row $i$.

Each elementary row operation corresponds to pre-multiplying $A$ by an elementary matrix $E$. Since elementary matrices are invertible, row operations preserve matrix rank and invertibility.

2.2 Row Echelon Form (REF) vs Reduced Row Echelon Form (RREF)

• Row Echelon Form (REF):
  1. All non-zero rows lie above any rows of all zeros.
  2. The leading entry (pivot) of a non-zero row appears strictly to the right of the leading entry of the row above it.
  3. All entries in a column directly below a pivot are zero.
  • Algorithm: Gauss elimination produces REF, enabling back-substitution.
• Reduced Row Echelon Form (RREF):
  1. Satisfies all REF requirements.
  2. Every leading pivot entry is scaled to exactly $1$.
  3. Every pivot is the only non-zero entry in its entire column (all entries above and below pivots are zero).
  • Algorithm: Gauss-Jordan elimination converts any matrix into its unique RREF canonical form.

3. Matrix Rank and Vector Independence

3.1 Definition of Rank

The rank of an $m \times n$ matrix $A$, denoted $\rho(A)$ or $\text{rank}(A)$, is the fundamental dimension metric of the transformation:

• Echelon Definition: The rank equals the total number of non-zero rows (or equivalently, the number of pivot columns) in its row echelon form.
• Determinant Definition: The rank equals the order $r$ of the largest square submatrix of $A$ having a non-vanishing determinant (non-zero minor), while all minors of order $r+1$ and higher vanish.
• Column / Row Space Definition: The rank equals the maximum number of linearly independent column vectors (or row vectors) in $A$. For any matrix, $\text{column rank} = \text{row rank} = \text{rank}(A) \le \min(m, n)$.

3.2 Linear Dependence, Independence, and Bases

• A set of vectors $\{v_1, v_2, \dots, v_k\}$ is linearly independent if the vector equation:
  $c_1 v_1 + c_2 v_2 + \dots + c_k v_k = 0$
  forces all scalar coefficients to be zero: $c_1 = c_2 = \dots = c_k = 0$.
• If there exists a non-trivial linear combination where at least one $c_i \neq 0$, the vectors are linearly dependent.
• Rank-Nullity Theorem (Fundamental Theorem of Linear Algebra):
  For an $m \times n$ matrix $A$ representing linear operator $T: \mathbb{R}^n \to \mathbb{R}^m$:
  $\text{rank}(A) + \text{nullity}(A) = n$
  where $n$ is the number of columns (dimension of the domain), and $\text{nullity}(A) = \dim(\ker(A))$ represents the dimension of the null space (the solution set to $Ax = 0$).

4. Systems of Linear Algebraic Equations

Consider the general system of $m$ linear algebraic equations in $n$ unknowns:
$A x = b$
where $A$ is the coefficient matrix ($m \times n$), $x$ is the column vector of unknowns ($n \times 1$), and $b$ is the non-homogeneous right-hand-side vector ($m \times 1$). The augmented matrix is denoted $[A \mid b]$.

4.1 Consistency and the Rouché-Capelli Theorem

The Rouché-Capelli theorem provides the exact algebraic condition for the solvability of linear systems:

1. Inconsistent System (No Solution):
   $\text{rank}(A) < \text{rank}([A \mid b])$
   Geometrically, the right-hand vector $b$ lies outside the column space of $A$. In echelon reduction, this manifests as an invalid row of the form $[0 \; 0 \; \dots \; 0 \mid d]$ with $d \neq 0$.
2. Consistent System (At Least One Solution Exists):
   $\text{rank}(A) = \text{rank}([A \mid b]) = r$
   • Case 2A: Unique Solution:
     If $r = n$ (rank equals the number of unknowns), the system has zero free variables, yielding a single unique solution vector $x = A^{-1} b$ (for square non-singular systems).
   • Case 2B: Infinitely Many Solutions:
     If $r < n$, there exist $n - r$ free independent variables (parameters). The solution space forms an affine subspace of dimension $k = n - r$.

4.2 Homogeneous Linear Systems ($A x = 0$)

• In a homogeneous system, $b = 0$. The augmented matrix $[A \mid 0]$ has identical rank to $A$, guaranteeing that every homogeneous system is always consistent.
• The vector $x = [0, 0, \dots, 0]^T$ is the trivial solution.
• Condition for Non-Trivial Solutions:
  Non-trivial (non-zero) solutions exist if and only if:
  $\text{rank}(A) < n$
  For a square $n \times n$ system, non-trivial solutions exist if and only if $\det(A) = 0$. The dimension of the non-trivial solution space (the kernel) is $n - \text{rank}(A)$.

5. Matrix Inversion Techniques

A square $n \times n$ matrix $A$ is invertible (non-singular) if there exists a matrix $A^{-1}$ such that $A A^{-1} = A^{-1} A = I_n$. Invertibility is equivalent to $\det(A) \neq 0$ or $\text{rank}(A) = n$.

5.1 Adjoint Method

The classical formula for the matrix inverse:
$A^{-1} = \frac{1}{\det(A)} \text{adj}(A)$
where $\text{adj}(A) = C^T$, the transpose of the matrix of algebraic cofactors $C_{ij} = (-1)^{i+j} M_{ij}$. While conceptually instructive, the adjoint method incurs $O(n!)$ computational complexity, making it impractical for $n > 3$.

5.2 Gauss-Jordan Inversion Algorithm

Gauss-Jordan elimination computes $A^{-1}$ in $O(n^3)$ operations by forming the augmented partitioned matrix $[A \mid I_n]$ and applying elementary row operations:
$[A \mid I_n] \xrightarrow{\text{Row Operations}} [I_n \mid A^{-1}]$
If reduction encounters a zero pivot that cannot be bypassed, $A$ is singular and has no inverse.

6. Spectral Analysis: Eigenvalues and Eigenvectors

6.1 The Characteristic Equation

For an $n \times n$ square matrix $A$, a non-zero vector $v \in \mathbb{C}^n$ is an eigenvector corresponding to scalar eigenvalue $\lambda \in \mathbb{C}$ if:
$A v = \lambda v \iff (A - \lambda I) v = 0$
Because non-trivial solutions require the operator $(A - \lambda I)$ to be singular, eigenvalues are the roots of the characteristic polynomial:
$P(\lambda) = \det(A - \lambda I) = 0$
Expanding the determinant yields an $n$-th degree algebraic polynomial:
$(-1)^n \lambda^n + (-1)^{n-1} \text{tr}(A) \lambda^{n-1} + \dots + \det(A) = 0$

6.2 Key Spectral Theorems and Algebraic Properties

1. Trace Invariant: The sum of all $n$ eigenvalues equals the matrix trace (sum of main diagonal elements):
   $\sum_{i=1}^n \lambda_i = \text{tr}(A) = \sum_{i=1}^n a_{ii}$
2. Determinant Invariant: The product of all $n$ eigenvalues equals the matrix determinant:
   $\prod_{i=1}^n \lambda_i = \det(A)$
   Consequently, a matrix is invertible if and only if all its eigenvalues are non-zero.
3. Triangular and Diagonal Matrices: The eigenvalues of upper triangular, lower triangular, or diagonal matrices are simply the diagonal entries themselves.
4. Shifted and Scaled Matrices:
   • If $A$ has eigenvalue $\lambda$ with eigenvector $v$, then $A^k$ has eigenvalue $\lambda^k$ with the same eigenvector $v$.
   • The matrix $(A + c I)$ has eigenvalues $(\lambda + c)$.
   • For an invertible matrix $A$, $A^{-1}$ has eigenvalue $\frac{1}{\lambda}$.
5. Multiplicities:
   • Algebraic Multiplicity (AM): The multiplicity of $\lambda$ as a root of $P(\lambda) = 0$.
   • Geometric Multiplicity (GM): The dimension of the eigenspace corresponding to $\lambda$, given by $\text{nullity}(A - \lambda I) = n - \text{rank}(A - \lambda I)$.
   • For every eigenvalue, $1 \le \text{GM} \le \text{AM}$.

7. Cayley-Hamilton Theorem and Matrix Diagonalization

7.1 Cayley-Hamilton Theorem

The fundamental theorem states: Every square matrix satisfies its own characteristic equation.
If the characteristic polynomial of $A$ is $P(\lambda) = \lambda^n + c_{n-1} \lambda^{n-1} + \dots + c_1 \lambda + c_0 = 0$, then substituting matrix $A$ yields the zero matrix:
$P(A) = A^n + c_{n-1} A^{n-1} + \dots + c_1 A + c_0 I = 0$

Major Engineering Applications:

1. Computation of Inverse $A^{-1}$:
   Multiplying the equation by $A^{-1}$ gives:
   $A^{n-1} + c_{n-1} A^{n-2} + \dots + c_1 I + c_0 A^{-1} = 0$
   $A^{-1} = -\frac{1}{c_0} \left( A^{n-1} + c_{n-1} A^{n-2} + \dots + c_1 I \right)$
   (provided $c_0 = (-1)^n \det(A) \neq 0$).
2. Evaluation of High Matrix Powers ($A^k$):
   Any polynomial in $A$ of degree $k \ge n$ can be reduced modulo $P(A)$ to a polynomial of degree at most $n-1$, bypassing expensive iterative multiplications.

7.2 Matrix Diagonalization

An $n \times n$ matrix $A$ is diagonalizable if there exists an invertible modal matrix $P$ and a diagonal matrix $D$ such that:
$P^{-1} A P = D \iff A = P D P^{-1}$

• The columns of modal matrix $P = [v_1 \; v_2 \; \dots \; v_n]$ are the $n$ linearly independent eigenvectors of $A$.
• The diagonal matrix $D = \text{diag}(\lambda_1, \lambda_2, \dots, \lambda_n)$ contains the corresponding eigenvalues.
• Criterion for Diagonalizability: A matrix is diagonalizable if and only if the geometric multiplicity of each eigenvalue equals its algebraic multiplicity ($\text{GM} = \text{AM}$) for all eigenvalues. In particular, any matrix with $n$ distinct eigenvalues is guaranteed to be diagonalizable.
• Power Calculation: $A^k = P D^k P^{-1}$, where $D^k = \text{diag}(\lambda_1^k, \dots, \lambda_n^k)$.