11.1  Inner Products and Norms

Nine chapters, and we have not measured anything. We can say that a list is independent, that a subspace has dimension four, that an operator is diagonalizable. We cannot yet say that a vector has length $3$, or that two vectors meet at a right angle. Geometry has appeared only in pictures, never in the theory. This section puts it into the theory, by abstracting the one operation in ${\mathbb{R}}^n$ that already computes lengths and angles: the dot product.

11.1.1. The field for this chapter

Throughout Chapter 11 the field $F$ is $\mathbb{R}$ or $\mathbb{C}$. This is not laziness; two features of these fields are used on every page, and no other field has both.

The first is positivity. The point of the whole construction is that $⟨v,v⟩$ should be the square of a length, so it must be a number we can compare with $0$. Over ${\mathbb{F}}_p$ there is no ordering compatible with the arithmetic at all: in ${\mathbb{F}}_5$, adding $1$ to itself five times returns to $0$, so no nonzero element can be called “positive” in any useful sense. Over $\mathbb{Q}$ there is an ordering, but lengths would not exist, since $\sqrt{2}\notin \mathbb{Q}$. Over $\mathbb{R}$ every sum of squares is $\ge 0$ and every nonnegative number has a square root, and that is exactly what we need.

The second is conjugation. Over $\mathbb{C}$ the naive sum of squares fails immediately: $1^2+i^2=0$, so “$z_1^2+z_2^2$” is not a length for $(1,i)\ne \mathbf{0}$. The repair is to use $z\bar{z}=|z{|}^2$ instead of $z^2$ (Theorem 0.6.2 (c)), and this needs the complex conjugate. We write $\bar{a}$ for the conjugate of $a\in F$, with the convention that $\bar{a}=a$ when $F=\mathbb{R}$; every formula below then reads correctly over both fields at once.

11.1.2. From the dot product to the axioms

In ${\mathbb{R}}^n$ the dot product is $$\mathbf{x}\cdot \mathbf{y}=x_1{y}_1+\cdots +x_n{y}_n.$$ It computes lengths, because $\mathbf{x}\cdot \mathbf{x}=x_1^2+\cdots +x_n^2$ is the square of the Euclidean length by Pythagoras. It computes angles too. For nonzero $\mathbf{x},\mathbf{y}\in {\mathbb{R}}^2$ at an angle $\theta$, the law of cosines applied to the triangle with sides $\mathbf{x}$, $\mathbf{y}$, and $\mathbf{x}-\mathbf{y}$ says $$\Vert \mathbf{x}-\mathbf{y}{\Vert }^2=\Vert \mathbf{x}{\Vert }^2+\Vert \mathbf{y}{\Vert }^2-2\Vert \mathbf{x}\Vert \Vert \mathbf{y}\Vert \cos \theta ,$$ while expanding the left side entry by entry gives $\Vert \mathbf{x}{\Vert }^2+\Vert \mathbf{y}{\Vert }^2-2(\mathbf{x}\cdot \mathbf{y})$. Comparing the two, $\mathbf{x}\cdot \mathbf{y}=\Vert \mathbf{x}\Vert \Vert \mathbf{y}\Vert \cos \theta$. So one bilinear expression carries both length and angle, and perpendicularity is the single equation $\mathbf{x}\cdot \mathbf{y}=0$.

Which properties of the dot product did that argument use? Only three: it is linear in each slot separately, it is symmetric, and $\mathbf{x}\cdot \mathbf{x}>0$ for $\mathbf{x}\ne \mathbf{0}$. Those three we can demand of an abstract space. Over $\mathbb{C}$, as we saw, symmetry has to be weakened to conjugate symmetry, and the price is small: a conjugate bar in the second slot.

An inner product is a way of multiplying two vectors into a scalar so that a vector paired with itself has positive “length squared”.

In words, clause by clause. (IP1) says that if we freeze the second slot, what is left is a linear functional on $V$: the inner product distributes over sums and lets scalars out of the first argument untouched. (IP2) says that swapping the two arguments conjugates the answer; over $\mathbb{R}$ it is plain symmetry, $⟨v,u⟩=⟨u,v⟩$. (IP3) says that a nonzero vector pairs with itself to give a strictly positive number. For the inequality in (IP3) to mean anything, $⟨v,v⟩$ must be a real number; it is, and the reason is (IP2), as the next result shows. Note also what (IP3) does not say: it says nothing about $⟨u,v⟩$ for $u\ne v$, which may be negative, or not even real.

The first consequences are small, and we will use all of them constantly.

Part (d) is the first appearance of the move that runs through the whole chapter: to show a vector is zero, pair it with itself. There is exactly one vector of length zero, so an identity of the form $⟨z,z⟩=0$ is a proof that $z=0$.

Part (b) is where the conjugate becomes visible in practice. An inner product over $\mathbb{C}$ is not bilinear: scalars leave the second slot wearing a bar. A map with properties (IP1) and (b) is called sesquilinear, from the Latin for “one and a half times linear”.

11.1.3. Examples

The dot product on ${\mathbb{R}}^n$. Define $⟨\mathbf{x},\mathbf{y}⟩=x_1{y}_1+\cdots +x_n{y}_n$. (IP1) holds because each term is linear in $x_i$; (IP2) is the commutativity of multiplication in $\mathbb{R}$, and conjugation is the identity; (IP3) holds because $⟨\mathbf{x},\mathbf{x}⟩=x_1^2+\cdots +x_n^2$ is a sum of squares of real numbers, which is $0$ only when every $x_i=0$. This is the standard inner product on ${\mathbb{R}}^n$, and it is the model for everything else.

The standard inner product on ${\mathbb{C}}^n$. Define $$⟨\mathbf{x},\mathbf{y}⟩=x_1\overline{y_1}+\cdots +x_n\overline{y_n}.$$ (IP1) is clear. For (IP2), $\overline{⟨\mathbf{x},\mathbf{y}⟩}=\sum \overline{x_i}{y}_i=⟨\mathbf{y},\mathbf{x}⟩$ by Theorem 0.6.2 (a), (b). For (IP3), $⟨\mathbf{x},\mathbf{x}⟩=\sum x_i\overline{x_i}=\sum |x_i{|}^2$ by Theorem 0.6.2 (c), a sum of nonnegative real numbers, zero only when every $x_i=0$. The conjugates are doing exactly one job: making the diagonal values real and nonnegative.

Weighted inner products. Fix real numbers $w_1,\dots ,w_n>0$ and put $⟨\mathbf{x},\mathbf{y}⟩=\sum _i{w}_i{x}_i\overline{y_i}$ on $F^n$. The same three checks go through, with $⟨\mathbf{x},\mathbf{x}⟩=\sum w_i|x_i{|}^2$; the weights must be strictly positive, or (IP3) fails for a standard basis vector. In the degenerate case $n=1$ and $w_1=1$, this is $⟨x,y⟩=x\bar{y}$ on $F$ itself, the smallest nonzero inner product space there is. It matters because it is where the axioms are just the arithmetic of $|\cdot |$.

Continuous functions. Let $V=C[0,1]$ be the space of continuous functions $f:[0,1]\to F$, and set $$⟨f,g⟩={\int }_0^{1}f(t)\overline{g(t)}\mathrm{d}t.$$ (IP1) is linearity of the integral, and (IP2) follows because conjugation commutes with integration. For (IP3), $⟨f,f⟩={\int }_0^1|f(t){|}^2\mathrm{d}t\ge 0$, and we must rule out equality for $f\ne 0$. Here is the one fact from analysis we assume: if $f$ is continuous and $f(c)\ne 0$ for some $c$, then $|f|\ge |f(c)|/2$ on some interval of positive length $\delta$ around $c$, so ${\int }_0^1|f{|}^2\ge \delta |f(c){|}^2/4>0$. Continuity is not decoration; the nonexample below shows what happens without it.

Matrices: the Frobenius inner product. First a piece of notation we will need all chapter.

Over $\mathbb{R}$, ${\mathbf{A}}^{\ast }={\mathbf{A}}^{\mathrm{\top }}$ and Hermitian means symmetric (Definition 0.9.11). Now on $V=M_{m\times n}(F)$ define $$⟨\mathbf{A},\mathbf{B}⟩=\mathrm{tr}({\mathbf{B}}^{\ast }\mathbf{A}).$$ Writing out the trace, $\mathrm{tr}({\mathbf{B}}^{\ast }\mathbf{A})=\sum _j\sum _i\overline{b_{ij}}{a}_{ij}$, so this is the standard inner product of $F^{mn}$ after listing the entries in a column. In particular (IP1)–(IP3) hold, with $⟨\mathbf{A},\mathbf{A}⟩=\sum _{i,j}|a_{ij}{|}^2$. This is the Frobenius inner product; for $n=1$ it is the standard inner product on $F^m$.

Polynomials sampled at nodes. Fix $n+1$ distinct scalars $c_0,\dots ,c_n\in F$ and define, on $V=F[x{]}_{\le n}$, $$⟨p,q⟩=\sum _{i=0}^{n}p(c_i)\overline{q(c_i)}.$$ (IP1) and (IP2) hold because evaluation at $c_i$ is linear (Theorem 0.8.5) and the standard inner product on $F^{n+1}$ has them; indeed $⟨p,q⟩$ is the standard inner product of the sample vectors $(p(c_0),\dots ,p(c_n))$ and $(q(c_0),\dots ,q(c_n))$. Positivity is the interesting clause. If $⟨p,p⟩=\sum |p(c_i){|}^2=0$, then $p$ vanishes at all $n+1$ distinct nodes. A nonzero polynomial of degree at most $n$ has at most $n$ distinct roots (Corollary 6.1.4), so $p=0$. The number of nodes is what makes this work, and the nonexample below removes one.

11.1.4. Nonexamples, by minimal change

Change a sign: the hyperbolic form on ${\mathbb{R}}^2$. Let $\beta (\mathbf{x},\mathbf{y})=x_1{y}_1-x_2{y}_2$. Bilinearity survives, since each term is still linear in each slot, and so does symmetry. What fails is (IP3), and the witness is ${\mathbf{e}}_2$: $\beta ({\mathbf{e}}_2,{\mathbf{e}}_2)=-1<0$. The vector $(1,1)$ is worse still: it is nonzero, yet $\beta ((1,1),(1,1))=0$, so it would be a nonzero vector of length $0$. Forms like $\beta$ are useful (they are the subject of the chapter on bilinear forms), but they are not inner products.

Drop the conjugate on ${\mathbb{C}}^2$. Let $⟨\mathbf{u},\mathbf{v}⟩={\mathbf{u}}^{\mathrm{\top }}\mathbf{v}=u_1{v}_1+u_2{v}_2$ on ${\mathbb{C}}^2$. This map is linear in both slots and symmetric, so it satisfies (IP1) and looks tidier than the standard product. Take $\mathbf{u}=(1,i)$. Then $$⟨\mathbf{u},\mathbf{u}⟩=1^2+i^2=1-1=0,$$ with $\mathbf{u}\ne \mathbf{0}$. Clause (IP3) fails. This one example is the whole reason for the conjugate: over $\mathbb{C}$, sums of squares vanish on nonzero vectors, sums of squared moduli do not.

Enlarge the space: drop continuity. Keep the formula $⟨f,g⟩={\int }_0^{1}fg$, but let $V$ be all Riemann-integrable functions $[0,1]\to \mathbb{R}$. Linearity and symmetry are untouched. Let $f$ be $1$ at $t=\frac{1}{2}$ and $0$ elsewhere. It is integrable, nonzero as a function, and ${\int }_0^1{f}^2=0$. Positivity fails, so $⟨\cdot ,\cdot ⟩$ is only positive semidefinite here. There are two standard repairs: restrict to continuous functions, as above, or regard two functions as equal when they differ on a negligible set. We take the first, and every function space in this chapter will be a space of continuous functions.

Remove a node. On $\mathbb{R}[x{]}_{\le 2}$, put $⟨p,q⟩=p(0)q(0)+p(1)q(1)$: the node example with one node too few. Bilinearity and symmetry hold. But $p=x^2-x$ is a nonzero polynomial of degree $2$ with $p(0)=p(1)=0$, so $⟨p,p⟩=0$. The clause that fails is (IP3), and it fails because two nodes cannot pin down a quadratic.

Why this definition. Each clause earns its place. Without (IP3) there can be nonzero vectors $v$ with $⟨v,v⟩=0$, and the proof of Theorem 11.1.1 (d) breaks down, because pairing a vector with itself no longer detects zero. (The form $x_1{y}_1-x_2{y}_2$ below shows that the conclusion of (d) may survive anyway; what is lost is the argument for it, and with it every later proof that runs through $⟨v,v⟩=0\Rightarrow v=0$.) Without (IP2) the diagonal values need not be real, and “$⟨v,v⟩>0$” is meaningless. Linearity in the first slot rather than the second is a convention, but once chosen it must be kept: with (IP2) in place, linearity in one slot forces conjugate-linearity in the other, so a map cannot be linear in both unless $F=\mathbb{R}$.

11.1.5. Norms

By Theorem 11.1.1 (a), the number $⟨v,v⟩$ is real and $\ge 0$, so it has a unique nonnegative real square root. That square root is the length we were after.

On ${\mathbb{R}}^n$ with the dot product this is the Euclidean length $\sqrt{x_1^2+\cdots +x_n^2}$, and on ${\mathbb{C}}^n$ it is $\sqrt{\sum |x_i{|}^2}$. On $C[0,1]$ it is $({\int }_0^1|f{|}^2{)}^{1/2}$, and on matrices it is the Frobenius norm $(\sum |a_{ij}{|}^2{)}^{1/2}$. By definition ${\Vert v\Vert }^2=⟨v,v⟩$ always, and this identity, read from right to left, is the most-used step in the chapter: work with the norm squared, then expand.

Part (c) is the workhorse. Over $\mathbb{R}$ the middle term is just $2⟨u,v⟩$, and the identity is the familiar $(a+b{)}^2=a^2+2ab+b^2$ for vectors. Replacing $v$ by $-v$ gives ${\Vert u-v\Vert }^2={\Vert u\Vert }^2-2\mathrm{Re}⟨u,v⟩+{\Vert v\Vert }^2$, and adding the two versions gives the parallelogram law $$\begin{array}{}\text{(}\ast \text{)}& {\Vert u+v\Vert }^2+{\Vert u-v\Vert }^2=2{\Vert u\Vert }^2+2{\Vert v\Vert }^2.\end{array}$$ We return to $(\ast )$ in the last warning of this section.

11.1.6. Orthogonality and Pythagoras

With a norm in hand, the special relation $⟨u,v⟩=0$ deserves its name.

The relation is symmetric: $⟨u,v⟩=0$ if and only if $⟨v,u⟩=\overline{⟨u,v⟩}=0$, so “$u\perp v$” needs no order. The zero vector is orthogonal to everything (Theorem 11.1.1 (c)), and by part (a) of the same result it is the only vector orthogonal to itself. In ${\mathbb{R}}^n$ this is perpendicularity in the usual sense, by the law of cosines computation above: $\cos \theta =0$ exactly when $\mathbf{x}\cdot \mathbf{y}=0$. In ${\mathbb{C}}^2$, the vectors $(1,i)$ and $(1,-i)$ are orthogonal, since $1\cdot \bar{1}+i\cdot \overline{(-i)}=1+i\cdot i=0$.

So orthogonality is exactly the condition under which squared lengths add. This is why the relation is worth naming, and why the rest of the chapter works so hard to produce orthogonal vectors.

The hypothesis “nonzero” cannot be dropped: the list $({\mathbf{e}}_1,\mathbf{0})$ is orthogonal and dependent. With it, the result is a free supply of independent lists, and this is what makes the next section possible.

11.1.7. The Cauchy–Schwarz inequality

Angles came from the formula $\cos \theta =(\mathbf{x}\cdot \mathbf{y})/(\Vert \mathbf{x}\Vert \Vert \mathbf{y}\Vert )$. To define an angle by that formula in an abstract space, we must know that the right-hand side lies in $[-1,1]$. That is exactly the following inequality, the most used inequality in the subject.

The proof is worth rereading for its shape: the vector $z$ was found from a picture and then checked from the axioms. That is the standard way to work in an abstract inner product space.

Together, Theorem 11.1.2 (a), (b), and Corollary 11.1.6 say that $\Vert \cdot \Vert$ is a norm in the general sense studied in Chapter 16. Chapter 16 also shows that not every norm arises from an inner product; the last warning of this section explains how to tell.

The definition needs Theorem 11.1.5 twice over. Cauchy–Schwarz says the quotient has absolute value at most $1$, so it lies in $[-1,1]$; and $\cos$ is a strictly decreasing bijection from $[0,\pi ]$ onto $[-1,1]$, so exactly one $\theta$ qualifies. Without the inequality the formula could ask for the angle whose cosine is $7$. Note also that $\theta =\pi /2$ if and only if $⟨u,v⟩=0$, so the definition is consistent with Definition 11.1.4.

Over $\mathbb{C}$ there is no angle, because $⟨u,v⟩$ is not real. The quantity $|⟨u,v⟩|/(\Vert u\Vert \Vert v\Vert )\in [0,1]$ still makes sense, and it still measures how far the two vectors are from being orthogonal.

11.1.8. Exercises

A. Check your understanding

B. Practice