Vectors and similarity

Movies as arrows

Suppose you score every movie from 0 to 5 on two things: how much action it has and how much comedy. A car-chase thriller might score $(4, 2)$ ; a romantic comedy $(1, 3)$ . Each movie is now a pair of numbers, which you can draw as an arrow from the origin.

A search works the same way. Ask for "mostly action, a bit funny" and your request becomes an arrow too, say $\mathbf{q} = (2, 1)$ . The movies to show you are the ones whose arrows point in nearly the same direction as $\mathbf{q}$ .

Definition (Vector)

A vector $\mathbf{x} \in \mathbb{R}^n$ is an ordered list of $n$ real numbers $x_1, x_2, \dots, x_n$ , its components. Vectors are columns; to save space we often write them sideways, as in $\mathbf{x} = (x_1, x_2)$ .

Real systems use the same idea with hundreds of components instead of two. A list of numbers that describes a word, a picture or a movie is called an embedding.

Drag the tips of the movies $\mathbf{b}$ and $\mathbf{c}$ . Which one points closer to the direction of $\mathbf{q}$ ?

Step 2 of 8

How long is a vector?

In two dimensions the arrow $\mathbf{x} = (x_1, x_2)$ is the long side of a right triangle with legs $x_1$ and $x_2$ , so by Pythagoras its length is $\sqrt{x_1^2 + x_2^2}$ . In three dimensions, applying Pythagoras twice gives $\sqrt{x_1^2 + x_2^2 + x_3^2}$ . Beyond three dimensions there's no picture to measure, so the same formula becomes the definition.

Definition (Length (L2 norm))

The length, or L2 norm, of $\mathbf{x} \in \mathbb{R}^n$ is

\lVert \mathbf{x} \rVert_2 = \sqrt{x_1^2 + x_2^2 + \cdots + x_n^2}

A vector of length $1$ is a unit vector.

For example, the query $\mathbf{q} = (2, 1)$ has length $\lVert \mathbf{q} \rVert_2 = \sqrt{2^2 + 1^2} = \sqrt{5} \approx 2.24$ .

Exercise, level: Core

Step 3 of 8

The dot product

Definition (Dot product)

The dot product of $\mathbf{a}, \mathbf{b} \in \mathbb{R}^n$ multiplies matching components and adds the results:

\begin{aligned} \mathbf{a} \cdot \mathbf{b} &= \sum_{i=1}^{n} a_i b_i \\ &= a_1 b_1 + a_2 b_2 + \cdots + a_n b_n \end{aligned}

As a matrix product it's $\mathbf{a}^\top \mathbf{b}$ : a row times a column.

The answer is a single number, not a vector. Each of these four rules takes one line to check from the definition (the last one is the definition of length):

Rule	In symbols
Order doesn't matter	$\mathbf{a} \cdot \mathbf{b} = \mathbf{b} \cdot \mathbf{a}$
It spreads over addition	$\mathbf{a} \cdot (\mathbf{b} + \mathbf{c}) = \mathbf{a} \cdot \mathbf{b} + \mathbf{a} \cdot \mathbf{c}$
Numbers pull out	$\mathbf{a} \cdot (k \mathbf{b}) = k \, (\mathbf{a} \cdot \mathbf{b})$
A vector with itself gives its length squared	$\mathbf{a} \cdot \mathbf{a} = \lVert \mathbf{a} \rVert_2^2$

Example (By hand)

For the query $\mathbf{q} = (2, 1)$ and the comedy $\mathbf{b} = (1, 3)$ :

\mathbf{q} \cdot \mathbf{b} = 2 \cdot 1 + 1 \cdot 3 = 5

Step 4 of 8

What the dot product says about the angle

The dot product is built from components, yet it measures something geometric: the angle between two arrows.

Theorem (Geometric form of the dot product)

Let $\mathbf{a}$ and $\mathbf{b}$ be nonzero vectors in $\mathbb{R}^2$ or $\mathbb{R}^3$ , and let $\theta \in [0, \pi]$ be the angle between them. Then

\mathbf{a} \cdot \mathbf{b} = \lVert \mathbf{a} \rVert_2 \, \lVert \mathbf{b} \rVert_2 \cos\theta

Proof (by the law of cosines)

The arrows $\mathbf{a}$ and $\mathbf{b}$ are two sides of a triangle whose third side is $\mathbf{a} - \mathbf{b}$ . The law of cosines gives

\begin{aligned} \lVert \mathbf{a} - \mathbf{b} \rVert_2^2 = {} & \lVert \mathbf{a} \rVert_2^2 + \lVert \mathbf{b} \rVert_2^2 \\ & - 2 \lVert \mathbf{a} \rVert_2 \lVert \mathbf{b} \rVert_2 \cos\theta \end{aligned}

By the rules in the last step, the left side also expands as

\begin{aligned} & (\mathbf{a} - \mathbf{b}) \cdot (\mathbf{a} - \mathbf{b}) \\ & \quad = \lVert \mathbf{a} \rVert_2^2 - 2 \, \mathbf{a} \cdot \mathbf{b} + \lVert \mathbf{b} \rVert_2^2 \end{aligned}

Setting the two right-hand sides equal and cancelling $\lVert \mathbf{a} \rVert_2^2 + \lVert \mathbf{b} \rVert_2^2$ leaves $-2 \, \mathbf{a} \cdot \mathbf{b} = -2 \lVert \mathbf{a} \rVert_2 \lVert \mathbf{b} \rVert_2 \cos\theta$ . Divide by $-2$ .

When the arrows point the same way or opposite ways ( $\theta = 0$ or $\pi$ ) the triangle is flat, but the law of cosines still holds, so the proof covers those cases too.

End of proof.

Lengths are positive, so the sign of the dot product is the sign of $\cos\theta$ :

$\mathbf{a} \cdot \mathbf{b}$	Angle	The arrows…
positive	less than $90^\circ$	lean the same way
$0$	exactly $90^\circ$	are perpendicular (orthogonal)
negative	more than $90^\circ$	lean opposite ways

In more than three dimensions you can't draw the angle, so this formula becomes its definition: $\cos\theta = \dfrac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert_2 \lVert \mathbf{b} \rVert_2}$ . That's only allowed because the right side always lies between $-1$ and $1$ , which is the Cauchy–Schwarz inequality $\lvert \mathbf{a} \cdot \mathbf{b} \rvert \le \lVert \mathbf{a} \rVert_2 \lVert \mathbf{b} \rVert_2$ . We state it here without proof; you'll prove it in Linear Algebra for ML.

Exercise, level: Core

Exercise, level: Derivation

Step 5 of 8

Cosine similarity

Divide the dot product by both lengths and only the direction is left.

Definition (Cosine similarity)

The cosine similarity of nonzero vectors $\mathbf{a}, \mathbf{b} \in \mathbb{R}^n$ is

\cos\theta = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert_2 \, \lVert \mathbf{b} \rVert_2}

It lies between $-1$ (opposite directions) and $1$ (the same direction), and it's $0$ for orthogonal vectors.

Proposition (Length doesn't matter)

For any number $k > 0$ , $\mathbf{a}$ and $k\mathbf{b}$ have the same cosine similarity as $\mathbf{a}$ and $\mathbf{b}$ .

Proof

The numbers pull out of the dot product and out of the length ( $\lVert k\mathbf{b} \rVert_2 = k \lVert \mathbf{b} \rVert_2$ for $k > 0$ ), so they cancel:

\begin{aligned} \frac{\mathbf{a} \cdot k\mathbf{b}}{\lVert \mathbf{a} \rVert_2 \, \lVert k\mathbf{b} \rVert_2} &= \frac{k \, (\mathbf{a} \cdot \mathbf{b})}{k \, \lVert \mathbf{a} \rVert_2 \, \lVert \mathbf{b} \rVert_2} \\ &= \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert_2 \, \lVert \mathbf{b} \rVert_2} \end{aligned}

End of proof.

Example (By hand, continued)

For $\mathbf{a} = (2, 1)$ and $\mathbf{b} = (1, 3)$ : $\mathbf{a} \cdot \mathbf{b} = 5$ , $\lVert \mathbf{a} \rVert_2 = \sqrt{5}$ and $\lVert \mathbf{b} \rVert_2 = \sqrt{10}$ , so

\begin{aligned} \cos\theta &= \frac{5}{\sqrt{5} \, \sqrt{10}} = \frac{5}{5\sqrt{2}} \\ &= \frac{1}{\sqrt{2}} \approx 0.707 \end{aligned}

That's $\theta = 45^\circ$ .

Drag the tips. Can you make the dot product $0$ ? How close to $-1$ can you get the cosine similarity? And what happens to it when you make $\mathbf{b}$ longer without turning it?

Key idea

The dot product measures how much two vectors agree. Dividing by their lengths removes size and leaves only direction: that's cosine similarity.

Exercise, level: Core

Step 6 of 8

By hand: a tiny movie search

Back to the search for "mostly action, a bit funny", $\mathbf{q} = (2, 1)$ . Here are three movies, scored (action, comedy):

Example (Rank the movies)

Movie A is $(4, 2)$ , movie B is $(1, 3)$ and movie C is $(5, 5)$ . The query has length $\lVert \mathbf{q} \rVert_2 = \sqrt{5}$ .

A: $\mathbf{q} \cdot \mathbf{a} = 2 \cdot 4 + 1 \cdot 2 = 10$ and $\lVert \mathbf{a} \rVert_2 = \sqrt{20} = 2\sqrt{5}$ , so $\cos\theta = \dfrac{10}{\sqrt{5} \cdot 2\sqrt{5}} = 1$ . A is exactly $2\mathbf{q}$ : the same direction.
B: $\mathbf{q} \cdot \mathbf{b} = 5$ and $\cos\theta = \dfrac{1}{\sqrt{2}} \approx 0.707$ , as before.
C: $\mathbf{q} \cdot \mathbf{c} = 2 \cdot 5 + 1 \cdot 5 = 15$ and $\lVert \mathbf{c} \rVert_2 = \sqrt{50} = 5\sqrt{2}$ , so $\cos\theta = \dfrac{15}{\sqrt{5} \cdot 5\sqrt{2}} = \dfrac{3}{\sqrt{10}} \approx 0.949$ .

Movie	Dot product with $\mathbf{q}$	Cosine similarity
A $(4, 2)$	$10$	$1$
B $(1, 3)$	$5$	$0.707$
C $(5, 5)$	$15$	$0.949$

Cosine similarity ranks A, C, B: A matches the query's taste exactly. The dot product ranks C first, only because C scores high on everything and so has the longest arrow. That's why search uses cosine similarity: it compares what a movie is like, not how much of everything it has.

Step 7 of 8

In code

NumPy writes the dot product as @ and the length as np.linalg.norm. Put the movies in the rows of an array, and movies @ q computes every dot product at once. Press Run:

Python

import numpy as np

q = np.array([2.0, 1.0])          # mostly action, a bit funny
movies = np.array([[4.0, 2.0],    # A
                   [1.0, 3.0],    # B
                   [5.0, 5.0]])   # C

dots = movies @ q
cosines = dots / (np.linalg.norm(movies, axis=1) * np.linalg.norm(q))
print("dot products:", dots)
print("cosine similarities:", cosines.round(3))
print("best match:", "ABC"[np.argmax(cosines)])

Python loads when you run code

Ctrl+Enter runs · Esc then Tab leaves the code

It prints the numbers from the table: dot products $10$ , $5$ and $15$ , cosine similarities $1$ , $0.707$ and $0.949$ , and best match A. Add a fourth movie as another row and run it again.

Search engines with millions of items save work by scaling every embedding to length 1 once, in advance. After that, a plain dot product is the cosine similarity:

Python

import numpy as np

q = np.array([2.0, 1.0])
movies = np.array([[4.0, 2.0], [1.0, 3.0], [5.0, 5.0]])

unit_movies = movies / np.linalg.norm(movies, axis=1, keepdims=True)
unit_q = q / np.linalg.norm(q)
print((unit_movies @ unit_q).round(3))  # the same cosine similarities

Python loads when you run code

Ctrl+Enter runs · Esc then Tab leaves the code

show(x, y) draws a chart. This one shows how cosine similarity falls as the angle between two vectors grows: from $1$ at $0^\circ$ , through $0$ at $90^\circ$ , to $-1$ at $180^\circ$ .

Python

import numpy as np

angles = np.linspace(0, 180, 50)  # in degrees
show(angles, np.cos(np.radians(angles)), title="Cosine similarity by angle (degrees)")

Python loads when you run code

Ctrl+Enter runs · Esc then Tab leaves the code

Step 8 of 8

Where it's used in ML

Tip

Semantic search works exactly like the movie search. A language model turns your question and every document into embeddings with hundreds of components, and the search returns the documents with the highest cosine similarity to your question. Chatbots that look things up before they answer, and "you might also like" recommendations, use the same step.

Exercise, level: Derivation

Exercise, level: Exam

Want the full story, with more proofs and problems? Vectors are the first module of Linear Algebra for ML.

Movies as arrows

How long is a vector?

Exercise, level: CoreExercise◆ Core

The dot product

What the dot product says about the angle

Exercise, level: CoreExercise◆ Core

Exercise, level: DerivationExercise◆◆ Derivation

Cosine similarity

Exercise, level: CoreExercise◆ Core

By hand: a tiny movie search

In code

Where it's used in ML

Exercise, level: DerivationExercise◆◆ Derivation

Exercise, level: ExamExercise◆◆◆ Exam

Exercise, level: Core

Exercise, level: Core

Exercise, level: Derivation

Exercise, level: Core

Exercise, level: Derivation

Exercise, level: Exam