Linear Algebra for ML

What you'll be able to do

• Reason about vectors, subspaces and linear maps geometrically and algebraically
• Solve and analyse linear systems
• Derive least squares and PCA from first principles
• Use eigen-decomposition and SVD to compress, denoise and reduce data

Before you start

Course 0 recommended; Class 12 matrices and determinants.

Syllabus

1. Module 1

   Vectors and geometry

   Vectors in ℝⁿ, linear combinations, span, dot product, norms (L1, L2, L∞), angle and cosine similarity, projection onto a vector.

   In ML: Embeddings, similarity search, nearest neighbours

   4 lessons coming soon

2. Module 2

   Matrices and linear maps

   Matrices as linear transformations; matrix products as composition; transpose and inverse; diagonal, symmetric, orthogonal, idempotent, projection and block matrices.

   In ML: A dense layer Wx + b; batching as a matrix product

   5 lessons coming soon

3. Module 3

   Linear systems and determinants

   Systems Ax = b, Gaussian elimination, row echelon forms, LU decomposition, determinants as volume scaling, cofactor expansion, invertibility conditions.

   In ML: Solving for parameters; numerical stability

   4 lessons coming soon

4. Module 4

   Vector spaces and rank

   Vector spaces and subspaces, linear independence, basis and dimension, rank and nullity, the rank–nullity theorem, the four fundamental subspaces.

   In ML: Redundant features, multicollinearity, degrees of freedom

   4 lessons coming soon

5. Module 5

   Orthogonality and least squares

   Orthonormal bases, orthogonal projection, projection matrices, Gram–Schmidt, QR decomposition, least squares and the normal equations, the pseudo-inverse.

   In ML: Linear regression derived geometrically

   5 lessons coming soon

6. Module 6

   Eigenvalues and eigenvectors

   Characteristic polynomial, eigen-decomposition, diagonalisation, the spectral theorem, positive definite matrices, quadratic forms, Markov chains and power iteration.

   In ML: Covariance matrices, curvature (Hessians), PageRank

   5 lessons coming soon

7. Module 7

   SVD and low-rank approximation

   SVD as rotate–stretch–rotate, its link to eigen-decomposition, Frobenius and spectral norms, Eckart–Young low-rank approximation, condition number, pseudo-inverse via SVD.

   In ML: Compression, recommender systems, latent semantic analysis

   4 lessons coming soon

8. Module 8

   PCA and dimensionality reduction

   Covariance matrix, PCA via eigen-decomposition and via SVD, the variance-maximisation derivation, explained variance and choosing k, Fisher's LDA (introduction).

   In ML: Visualisation, preprocessing, eigenfaces

   3 lessons coming soon

9. Module 9

   Advanced matrix theory

   Optional

   Jordan canonical form, minimal polynomial, polar decomposition.

   In ML: Rarely needed in ML practice; included for rigor

   2 lessons coming soon