Calculus & Optimization for ML
Derivatives, Taylor series, integrals and multivariable calculus, through backpropagation and the optimisers that train every model.
37 lessons · about 42 hours
What you'll be able to do
- Differentiate and approximate functions of one and many variables
- Derive gradients of common ML losses
- Implement backpropagation from scratch
- Choose and tune optimisers
- Solve constrained problems with Lagrange multipliers and KKT conditions
Before you start
Course 1 Modules 1–2 (for the multivariable modules); Class 12 calculus.
Syllabus
Module 1
Functions, limits and continuity
Functions and graphs, limits, continuity, differentiability and where it fails (|x|, the ReLU kink), the intermediate value theorem.
In ML: Activation functions
4 lessons coming soon
Module 2
Derivatives
The derivative as slope and as best linear approximation; sum, product, quotient and chain rules; derivatives of exp, log, sigmoid, tanh and softplus; finite differences.
In ML: Activation derivatives; gradient checking
4 lessons coming soon
Module 3
Taylor series and approximation
Taylor and Maclaurin series, the remainder term, linear and quadratic approximation, log-sum-exp and other numerical tricks.
In ML: Why first-order (gradient descent) and second-order (Newton) methods work
3 lessons coming soon
Module 4
Single-variable optimisation
Critical points, first and second derivative tests, local vs global extrema, convexity in 1D, optimisation word problems.
In ML: Learning-rate intuition
4 lessons coming soon
Module 5
Integration essentials
The definite integral as area, the fundamental theorem of calculus, substitution, integration by parts, improper integrals, double integrals (introduction).
In ML: Probabilities as areas and expected values
4 lessons coming soon
Module 6
Multivariable calculus
Partial derivatives, gradient, directional derivatives, Jacobian, Hessian, the multivariable chain rule, matrix-calculus identities, multivariable Taylor expansion.
In ML: Gradients of MSE and cross-entropy losses
5 lessons coming soon
Module 7
Backpropagation and automatic differentiation
Computational graphs, forward vs reverse mode, backprop for an MLP in matrix form, vanishing and exploding gradients.
In ML: How every deep-learning framework computes gradients
4 lessons coming soon
Module 8
Unconstrained optimisation
Gradient descent and step size, convergence intuition, steepest descent, Newton's method, SGD and mini-batches, momentum, RMSProp, Adam.
In ML: Training any model
4 lessons coming soon
Module 9
Convexity and constrained optimisation
Convex sets and functions, Jensen's inequality, Lagrange multipliers, KKT conditions, duality (introduction), linear programming basics, penalty methods.
In ML: L1 vs L2 regularisation geometry; SVM preview
5 lessons coming soon