Matrices as transformations

A matrix moves the whole plane

In the last lesson, a vector was a point or an arrow. A matrix is a machine that moves points: put a vector in, get a new vector out. Do that to every point of the plane at once and the whole plane turns, stretches, slants or flattens.

Definition (Matrix)

An $m \times n$ matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$ is a grid of real numbers with $m$ rows and $n$ columns. The entry in row $i$ and column $j$ is $a_{ij}$ . A $2 \times 2$ matrix is written

\mathbf{A} = \begin{pmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{pmatrix}

Press the presets and watch the grid, the arrows $\hat{\imath}$ and $\hat{\jmath}$ , and the letter F. Each preset is a different $2 \times 2$ matrix acting on the plane.

Notice what never happens: grid lines never bend, and the origin never moves. That's the mark of the transformations matrices make.

Step 2 of 8

Multiplying a matrix by a vector

Definition (Matrix–vector product)

For $\mathbf{A} \in \mathbb{R}^{m \times n}$ and $\mathbf{x} \in \mathbb{R}^n$ , the product $\mathbf{A}\mathbf{x}$ is the vector in $\mathbb{R}^m$ whose $i$ -th entry is the dot product of row $i$ of $\mathbf{A}$ with $\mathbf{x}$ :

\begin{aligned} (\mathbf{A}\mathbf{x})_i &= \sum_{j=1}^{n} a_{ij} x_j \\ &= a_{i1} x_1 + \cdots + a_{in} x_n \end{aligned}

The vector needs as many entries as $\mathbf{A}$ has columns, and the answer has as many entries as $\mathbf{A}$ has rows.

Example (By hand)

Take $\mathbf{A} = \begin{pmatrix} 2 & -1 \\ 1 & 1 \end{pmatrix}$ and $\mathbf{x} = (3, 2)$ . Row 1 gives $2 \cdot 3 + (-1) \cdot 2 = 4$ , and row 2 gives $1 \cdot 3 + 1 \cdot 2 = 5$ , so

\mathbf{A}\mathbf{x} = \begin{pmatrix} 2 & -1 \\ 1 & 1 \end{pmatrix} \begin{pmatrix} 3 \\ 2 \end{pmatrix} = \begin{pmatrix} 4 \\ 5 \end{pmatrix}

The matrix moves the point $(3, 2)$ to $(4, 5)$ .

Exercise, level: Core

Step 3 of 8

The columns are where the axes land

The standard basis vectors of $\mathbb{R}^2$ are $\hat{\imath} = \mathbf{e}_1 = (1, 0)$ and $\hat{\jmath} = \mathbf{e}_2 = (0, 1)$ . There's a second way to read a matrix–vector product, and it explains the whole picture.

Theorem (Column picture)

Let $\mathbf{a}_1$ and $\mathbf{a}_2$ be the columns of a $2 \times 2$ matrix $\mathbf{A}$ . Then $\mathbf{A}\mathbf{e}_1 = \mathbf{a}_1$ , $\mathbf{A}\mathbf{e}_2 = \mathbf{a}_2$ , and for every $\mathbf{x} = (x_1, x_2)$ ,

\mathbf{A}\mathbf{x} = x_1 \mathbf{a}_1 + x_2 \mathbf{a}_2

Proof

By the definition, $\mathbf{A}\mathbf{x} = \begin{pmatrix} a_{11} x_1 + a_{12} x_2 \\ a_{21} x_1 + a_{22} x_2 \end{pmatrix}$ . Split it by which $x_j$ each term contains:

\begin{aligned} \mathbf{A}\mathbf{x} &= x_1 \begin{pmatrix} a_{11} \\ a_{21} \end{pmatrix} + x_2 \begin{pmatrix} a_{12} \\ a_{22} \end{pmatrix} \\ &= x_1 \mathbf{a}_1 + x_2 \mathbf{a}_2 \end{aligned}

With $\mathbf{x} = \mathbf{e}_1$ (so $x_1 = 1$ , $x_2 = 0$ ) this gives $\mathbf{A}\mathbf{e}_1 = \mathbf{a}_1$ , and with $\mathbf{x} = \mathbf{e}_2$ it gives $\mathbf{A}\mathbf{e}_2 = \mathbf{a}_2$ . The same argument works for any number of columns.

End of proof.

So the first column is where $\hat{\imath}$ lands and the second column is where $\hat{\jmath}$ lands, and every other point follows. The point $(3, 2)$ is "3 steps along $\hat{\imath}$ , then 2 steps along $\hat{\jmath}$ ", so it lands 3 steps along the new $\hat{\imath}$ and 2 along the new $\hat{\jmath}$ . For the matrix above:

\mathbf{A}\mathbf{x} = 3 \begin{pmatrix} 2 \\ 1 \end{pmatrix} + 2 \begin{pmatrix} -1 \\ 1 \end{pmatrix} = \begin{pmatrix} 4 \\ 5 \end{pmatrix}

That's the same answer as the rows gave. Here is that matrix. Drag the tips of $\hat{\imath}$ and $\hat{\jmath}$ , or type new entries, and watch the columns change with them.

Exercise, level: Core

Step 4 of 8

Linear maps

Why do grid lines never bend? Because of two rules that every matrix obeys.

Definition (Linear map)

A function $T: \mathbb{R}^n \to \mathbb{R}^m$ is linear if, for all vectors $\mathbf{u}, \mathbf{v}$ and every number $c$ ,

\begin{aligned} T(\mathbf{u} + \mathbf{v}) &= T(\mathbf{u}) + T(\mathbf{v}) \\ T(c\,\mathbf{u}) &= c\,T(\mathbf{u}) \end{aligned}

Theorem (Matrices and linear maps are the same thing)

For every matrix $\mathbf{A}$ , the map $\mathbf{x} \mapsto \mathbf{A}\mathbf{x}$ is linear.
Every linear map $T: \mathbb{R}^2 \to \mathbb{R}^2$ is $\mathbf{x} \mapsto \mathbf{A}\mathbf{x}$ for exactly one matrix $\mathbf{A}$ : the one whose columns are $T(\mathbf{e}_1)$ and $T(\mathbf{e}_2)$ .

Proof

Part 1. Each entry of $\mathbf{A}\mathbf{x}$ is a dot product of a row with $\mathbf{x}$ , and the dot product spreads over addition and lets numbers pull out (lesson 1). So each entry of $\mathbf{A}(\mathbf{u} + \mathbf{v})$ is the matching entry of $\mathbf{A}\mathbf{u}$ plus that of $\mathbf{A}\mathbf{v}$ , and each entry of $\mathbf{A}(c\,\mathbf{u})$ is $c$ times that of $\mathbf{A}\mathbf{u}$ .

Part 2. Every $\mathbf{x} = (x_1, x_2)$ can be written $\mathbf{x} = x_1 \mathbf{e}_1 + x_2 \mathbf{e}_2$ . Using the two rules,

\begin{aligned} T(\mathbf{x}) &= T(x_1 \mathbf{e}_1) + T(x_2 \mathbf{e}_2) \\ &= x_1 T(\mathbf{e}_1) + x_2 T(\mathbf{e}_2) \end{aligned}

By the column picture, that's $\mathbf{A}\mathbf{x}$ for the matrix $\mathbf{A}$ with columns $T(\mathbf{e}_1)$ and $T(\mathbf{e}_2)$ . Any matrix that does the same must send $\mathbf{e}_1$ to $T(\mathbf{e}_1)$ and $\mathbf{e}_2$ to $T(\mathbf{e}_2)$ , so it has the same columns: it's the same matrix.

End of proof.

Two consequences you saw in the widget: a linear map keeps the origin fixed, because $T(\mathbf{0}) = T(0\,\mathbf{0}) = 0\,T(\mathbf{0}) = \mathbf{0}$ ; and it keeps lines straight, because the points $\mathbf{p} + t\,\mathbf{d}$ of a line land on the points $T(\mathbf{p}) + t\,T(\mathbf{d})$ , another line (or a single point when $T(\mathbf{d}) = \mathbf{0}$ ).

The presets are these matrices. Read each column as where $\hat{\imath}$ or $\hat{\jmath}$ goes:

Preset	Matrix	What it does
Rotate	$\begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$	turns the plane $90^\circ$ counterclockwise
Scale	$\begin{pmatrix} 2 & 0 \\ 0 & 1.5 \end{pmatrix}$	stretches $x$ by 2 and $y$ by 1.5
Shear	$\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$	slides each point sideways by its height
Reflect	$\begin{pmatrix} -1 & 0 \\ 0 & 1 \end{pmatrix}$	mirrors the plane in the $y$ -axis
Project	$\begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}$	flattens the plane onto the $x$ -axis

The widget also shows $\det \mathbf{A} = a_{11} a_{22} - a_{12} a_{21}$ , the determinant. Its absolute value is the factor by which the matrix scales areas, and it's negative when the matrix flips the plane over. We state that here without proof; determinants get a full module in Linear Algebra for ML.

Exercise, level: Derivation

Step 5 of 8

One transformation after another

Apply $\mathbf{B}$ first, then $\mathbf{A}$ : the point $\mathbf{x}$ goes to $\mathbf{B}\mathbf{x}$ , then to $\mathbf{A}(\mathbf{B}\mathbf{x})$ . The result is again a linear map, so by the theorem it's a single matrix.

Definition (Matrix product)

The product $\mathbf{A}\mathbf{B}$ is the matrix whose $j$ -th column is $\mathbf{A}$ times the $j$ -th column of $\mathbf{B}$ . It's defined when $\mathbf{A}$ has as many columns as $\mathbf{B}$ has rows, and then $(\mathbf{A}\mathbf{B})\mathbf{x} = \mathbf{A}(\mathbf{B}\mathbf{x})$ for every $\mathbf{x}$ .

Why that column rule: the combined map sends $\mathbf{e}_j$ to $\mathbf{A}(\mathbf{B}\mathbf{e}_j)$ , and $\mathbf{B}\mathbf{e}_j$ is the $j$ -th column of $\mathbf{B}$ . Its matrix has those images as columns. Note the order: in $\mathbf{A}\mathbf{B}$ , the matrix on the right acts first.

Example (Order matters)

Let $\mathbf{R} = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$ (turn $90^\circ$ ) and $\mathbf{S} = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ (shear). Turning twice is $\mathbf{R}\mathbf{R} = \begin{pmatrix} -1 & 0 \\ 0 & -1 \end{pmatrix}$ , a half turn: every point goes to its opposite. But the two orders of $\mathbf{R}$ and $\mathbf{S}$ give different matrices:

$\begin{aligned} \mathbf{R}\mathbf{S} &= \begin{pmatrix} 0 & -1 \\ 1 & 1 \end{pmatrix} \\ \mathbf{S}\mathbf{R} &= \begin{pmatrix} 1 & -1 \\ 1 & 0 \end{pmatrix} \end{aligned}$

Shearing then turning isn't the same as turning then shearing. Matrix products usually depend on the order.

Exercise, level: Derivation

Step 6 of 8

By hand: one layer of a neural network

A layer of a neural network takes a vector $\mathbf{x}$ and computes

\mathbf{y} = \operatorname{ReLU}(\mathbf{W}\mathbf{x} + \mathbf{b})

The weights $\mathbf{W}$ are a matrix, so the first part is a linear map: it turns, stretches and shears space. The bias $\mathbf{b}$ shifts everything over. Then $\operatorname{ReLU}(t) = \max(0, t)$ , applied to each entry, replaces negative numbers with $0$ : it folds the parts of space with negative coordinates flat onto the axes.

Example (One layer)

Take $\mathbf{W} = \begin{pmatrix} 1 & -1 \\ 2 & 1 \end{pmatrix}$ , $\mathbf{b} = (0, -1)$ and the input $\mathbf{x} = (1, 2)$ .

\begin{aligned} \mathbf{W}\mathbf{x} &= \begin{pmatrix} 1 \cdot 1 + (-1) \cdot 2 \\ 2 \cdot 1 + 1 \cdot 2 \end{pmatrix} \\ &= \begin{pmatrix} -1 \\ 4 \end{pmatrix} \\ \mathbf{W}\mathbf{x} + \mathbf{b} &= \begin{pmatrix} -1 \\ 3 \end{pmatrix} \\ \mathbf{y} &= \begin{pmatrix} 0 \\ 3 \end{pmatrix} \end{aligned}

The ReLU matters more than it looks. Without it, two layers in a row would be $\mathbf{W}_2(\mathbf{W}_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2 = (\mathbf{W}_2\mathbf{W}_1)\mathbf{x} + (\mathbf{W}_2\mathbf{b}_1 + \mathbf{b}_2)$ : a single matrix and a single shift, no more powerful than one layer. The fold between layers is what lets a deep network bend space into shapes no single matrix can make.

Step 7 of 8

In code

NumPy writes the matrix–vector product with @, just like the dot product. A[:, 0] is the first column:

Python

import numpy as np

A = np.array([[2.0, -1.0],
              [1.0,  1.0]])
x = np.array([3.0, 2.0])

print(A @ x)                       # rows dotted with x
print(3 * A[:, 0] + 2 * A[:, 1])   # 3 of column 1 plus 2 of column 2

Python loads when you run code

Ctrl+Enter runs · Esc then Tab leaves the code

Both lines print [4. 5.]. Put many points in the columns of one array and a single @ moves them all. Here are 200 points on the unit circle, moved by the same matrix:

Python

import numpy as np

A = np.array([[2.0, -1.0], [1.0, 1.0]])
t = np.linspace(0, 2 * np.pi, 200)
circle = np.stack([np.cos(t), np.sin(t)])  # 2 rows, 200 columns: one point per column
moved = A @ circle
show(moved[0], moved[1], kind="scatter", title="The unit circle after A")

Python loads when you run code

Ctrl+Enter runs · Esc then Tab leaves the code

The circle comes out as a tilted ellipse: a linear map keeps lines straight, but it can stretch a circle. Finally, the layer from the last step:

Python

import numpy as np

W = np.array([[1.0, -1.0], [2.0, 1.0]])
b = np.array([0.0, -1.0])
x = np.array([1.0, 2.0])

print(np.maximum(0, W @ x + b))  # ReLU(Wx + b)

Python loads when you run code

Ctrl+Enter runs · Esc then Tab leaves the code

It prints [0. 3.], as we found by hand. Change x and run it again.

Step 8 of 8

Where it's used in ML

Tip

Every dense layer of a neural network is a matrix. A layer that reads a $28 \times 28$ image as a vector of $784$ pixels and outputs $128$ numbers has a $128 \times 784$ weight matrix and a bias with $128$ entries. Training (lesson 4) adjusts all of those numbers. Language models are built from the same pieces, with matrices of many millions of entries.

Exercise, level: Core

Exercise, level: Exam

Want to go further, with the four fundamental subspaces, determinants and more? Matrices are the second module of Linear Algebra for ML.

A matrix moves the whole plane

Multiplying a matrix by a vector

Exercise, level: CoreExercise◆ Core

The columns are where the axes land

Exercise, level: CoreExercise◆ Core

Linear maps

Exercise, level: DerivationExercise◆◆ Derivation

One transformation after another

Exercise, level: DerivationExercise◆◆ Derivation

By hand: one layer of a neural network

In code

Where it's used in ML

Exercise, level: CoreExercise◆ Core

Exercise, level: ExamExercise◆◆◆ Exam

Exercise, level: Core

Exercise, level: Core

Exercise, level: Derivation

Exercise, level: Derivation

Exercise, level: Core

Exercise, level: Exam