11.9 Bessel, Parseval, and Fourier Series

Section 3 measured a vector against a finite-dimensional subspace and produced the nearest point in it. Read that answer coefficient by coefficient and it becomes an inequality: the numbers $\langle v, e_i \rangle$ collected against an orthonormal list can never carry more squared length than $v$ itself has. That is Bessel’s inequality. It costs nothing in dimension, it holds in spaces far too large for Chapter 1’s machinery, and its equality case is Parseval’s identity. This section proves both, applies them to the sines and cosines, and then stops at the exact place where linear algebra stops and analysis begins.

Throughout, $V$ is an inner product space over $F = \mathbb{R}$ or $F = \mathbb{C}$ , with inner product $\langle \cdot, \cdot \rangle$ (Definition 11.1.1) and induced norm $\left\| \cdot \right\|$ (Definition 11.1.3). We do not assume that $V$ is finite-dimensional. What we assume each time is that the list we compute against is finite.

11.9.1. Coefficients against an orthonormal list

In Theorem 11.2.1 the list was an orthonormal basis, and the coefficients $\langle v, e_i \rangle$ recovered $v$ exactly. Suppose now the list is only a list: too short to be a basis, or sitting inside a space that has no finite basis at all, like the continuous functions on an interval. The numbers $\langle v, e_i \rangle$ are still there. What can be said about them when they no longer determine $v$ ?

Against a shorter list you can only lose length, never gain it.

Definition 11.9.1 (Fourier Coefficients With Respect to an Orthonormal List)

Let $(e_1, \dots, e_k)$ be an orthonormal list in an inner product space $V$ and let $v \in V$ . The scalars $\langle v, e_1 \rangle,\ \dots,\ \langle v, e_k \rangle \in F$ are the Fourier coefficients of $v$ with respect to the list.

The name is borrowed from the classical case below, where the $e_i$ are sines and cosines; nothing in the definition mentions trigonometry. When the list happens to be a basis, these are exactly the coordinates of $v$ , by Theorem 11.2.1 (a). When it is not, they are the coordinates of a shadow of $v$ , and the next theorem measures that shadow.

Theorem 11.9.1 (Bessel’s Inequality)

Let $V$ be an inner product space over $F$ , not assumed finite-dimensional, and let $(e_1, \dots, e_k)$ be an orthonormal list in $V$ . Then for every $v \in V$ , $\sum_{i=1}^{k} \lvert \langle v, e_i \rangle \rvert^2 \;\le\; \left\| v \right\|^2 .$

Idea. The left-hand side is not a quantity out of nowhere. Let $U$ be the span of the list. By the projection formula, $P_Uv$ is exactly $\sum_i \langle v, e_i \ranglee_i$ , and its summands are pairwise orthogonal, so Pythagoras turns its squared length into precisely that sum. The inequality then says no more than “a shadow is no longer than what casts it”, which we already proved. The one point needing care is that $V$ may be infinite-dimensional; it does not matter, because Section 3 only ever asked the subspace to be finite-dimensional.

Proof. Put $U = \operatorname{span}(e_1, \dots, e_k)$ . The vectors $e_i$ are orthogonal and nonzero, so the list is linearly independent by Theorem 11.1.4; being a spanning list of $U$ as well, it is an orthonormal basis of $U$ , and $\dim U = k$ is finite. Hence Theorem 11.3.3 applies to $U$ even though $V$ need not be finite-dimensional, and part (a) of it gives $P_Uv = \sum_{i=1}^{k} \langle v, e_i \rangle\,e_i .$ The summands $\langle v, e_i \ranglee_i$ are pairwise orthogonal, since $e_i \perp e_j$ for $i \ne j$ and scalars come out of both slots. So Theorem 11.1.3 and Theorem 11.1.2 (b) give $\left\| P_Uv \right\|^2 = \sum_{i=1}^{k} \lvert \langle v, e_i \rangle \rvert^2 \left\| e_i \right\|^2 = \sum_{i=1}^{k} \lvert \langle v, e_i \rangle \rvert^2 ,$ using $\left\| e_i \right\| = 1$ . By Theorem 11.3.3 (c), $\left\| P_Uv \right\| \le \left\| v \right\|$ . Both sides are nonnegative, so squaring preserves the inequality, and this proves the theorem.

The proof gives more than the inequality, and the surplus is worth recording. The vector $v - P_Uv$ lies in $U^{\perp}$ by Theorem 11.3.2, so it is orthogonal to $P_Uv \in U$ , and Theorem 11.1.3 turns $v = P_Uv + (v - P_Uv)$ into $\left\| v \right\|^2 = \left\| P_Uv \right\|^2 + \left\| v - P_Uv \right\|^2$ . Rearranging, and substituting $\left\| P_Uv \right\|^2 = \sum_{i=1}^{k}\lvert\langle v, e_i \rangle\rvert^2$ from the proof above, $\left\| v - P_Uv \right\|^2 = \left\| v \right\|^2 - \sum_{i=1}^{k} \lvert \langle v, e_i \rangle \rvert^2 . \tag{$\ast$}$ In words: the deficit in Bessel’s inequality is the squared distance from $v$ to the span of the list (Definition 11.3.3). An inequality with a known defect is much more useful than an inequality, and we will read $(\ast)$ from both sides.

Remark. For an infinite orthonormal list $(e_1, e_2, e_3, \dots)$ , apply the theorem to the first $k$ terms for each $k$ . The partial sums $\sum_{i \le k}\lvert\langle v, e_i \rangle\rvert^2$ increase with $k$ and are all bounded above by $\left\| v \right\|^2$ , so they converge, and $\sum_{i=1}^{\infty}\lvert\langle v, e_i \rangle\rvert^2 \le \left\| v \right\|^2$ . This is the only fact from analysis used in the present paragraph: a bounded increasing sequence of real numbers converges.

Now the equality case, which is where the word basis returns.

Theorem 11.9.2 (Parseval’s Identity)

Let $(e_1, \dots, e_k)$ be an orthonormal list in an inner product space $V$ and let $v \in V$ . Then $\sum_{i=1}^{k} \lvert \langle v, e_i \rangle \rvert^2 = \left\| v \right\|^2 \quad\text{if and only if}\quad v \in \operatorname{span}(e_1, \dots, e_k) .$ In particular, if $(e_1, \dots, e_k)$ is an orthonormal basis of $V$ , the identity holds for every $v \in V$ .

Proof. Keep the notation of the proof of Theorem 11.9.1, where we showed $\sum_i \lvert\langle v, e_i \rangle\rvert^2 = \left\| P_Uv \right\|^2$ with $U = \operatorname{span}(e_1, \dots, e_k)$ . So the stated equality says $\left\| P_Uv \right\| = \left\| v \right\|$ , and by Theorem 11.3.3 (c) this holds if and only if $v \in U$ . If the list is a basis of $V$ , then $U = V$ and every $v \in V$ lies in $U$ .

For an orthonormal basis of a finite-dimensional space this is Theorem 11.2.1 (c) again; what is new is the equality case, which identifies exactly the vectors for which no length is lost. Keep the two statements apart:

Bessel is about an orthonormal list, in any inner product space, and it is an inequality.
Parseval is the equality, and it holds exactly for the vectors the list already spans.

The whole difference is whether the list reaches $v$ .

Warning. An orthonormal list is not an orthonormal basis, and the deficit in Bessel can be everything. In $C[-\pi, \pi]$ below, the list $(\cos x, \sin x)$ is orthonormal, and for the constant function $f = 1$ both coefficients are $0$ , so the left-hand side of Bessel is $0$ while $\left\| f \right\|^2 = 2$ . Nothing at all is captured. A short orthonormal list is not a bad basis; it is not a basis.

Quick check. In $\mathbb{R}^4$ with the dot product, let $\mathbf{e}_1 = \tfrac12(1, 1, 1, 1)$ and $\mathbf{e}_2 = \tfrac12(1, 1, -1, -1)$ , and let $\mathbf{v} = (1, 2, 3, 4)$ . Check that $(\mathbf{e}_1, \mathbf{e}_2)$ is orthonormal, compute both sides of Bessel’s inequality, and say what the deficit equals.

Solution. Each has norm $\tfrac12\sqrt{4} = 1$ , and $\langle \mathbf{e}_1, \mathbf{e}_2 \rangle = \tfrac14(1 + 1 - 1 - 1) = 0$ , so the list is orthonormal. The coefficients are $\langle \mathbf{v}, \mathbf{e}_1 \rangle = \tfrac12(1 + 2 + 3 + 4) = 5$ and $\langle \mathbf{v}, \mathbf{e}_2 \rangle = \tfrac12(1 + 2 - 3 - 4) = -2$ , so the left-hand side is $25 + 4 = 29$ , while $\left\| \mathbf{v} \right\|^2 = 1 + 4 + 9 + 16 = 30$ . The inequality is strict, and by $(\ast)$ the deficit $1$ is the squared distance from $\mathbf{v}$ to $\operatorname{span}(\mathbf{e}_1, \mathbf{e}_2)$ . Indeed $P_U\mathbf{v} = 5\mathbf{e}_1 - 2\mathbf{e}_2 = \big(\tfrac32, \tfrac32, \tfrac72, \tfrac72\big)$ and $\mathbf{v} - P_U\mathbf{v} = \tfrac12(-1, 1, -1, 1)$ , whose squared norm is $4 \cdot \tfrac14 = 1$ .

11.9.2. The trigonometric system

The classical home of these two statements is a space of functions, and the first thing to fix is the inner product, constant included.

Let $V = C[-\pi, \pi]$ be the space of continuous functions $f \colon [-\pi, \pi] \to \mathbb{R}$ , a real vector space, and define $\langle f, g \rangle = \frac{1}{\pi}\int_{-\pi}^{\pi} f(t)\,g(t)\,\mathrm{d} t .$ This is the continuous-function inner product of Definition 11.1.1 on a different interval and rescaled by the positive constant $1/\pi$ ; rescaling by a positive constant preserves all three axioms, and positive definiteness is the argument given there. The constant is chosen for one reason only: it makes the sines and cosines unit vectors. With $1/(2\pi)$ in front they would have norm $1/\sqrt2$ , and with no constant at all, norm $\sqrt{\pi}$ .

Warning. Always state the normalizing constant before computing. Formulas for Fourier coefficients differ between books by exactly such a factor, and a misplaced $\pi$ or $2$ changes every number below. In this section $\langle f, g \rangle = \frac1\pi\int_{-\pi}^{\pi} fg$ , and every coefficient, norm, and inequality is written for that choice.

Here are the facts we assume. From calculus: the integral of a finite sum is the sum of the integrals; $\int_{-\pi}^{\pi} \mathrm{d} t = 2\pi$ ; $\int_{-\pi}^{\pi}\sin(mt)\,\mathrm{d} t = 0$ for every integer $m$ and $\int_{-\pi}^{\pi}\cos(mt)\,\mathrm{d} t = 0$ for every nonzero integer $m$ (for $m \neq 0$ evaluate the antiderivative $-\cos(mt)/m$ , respectively $\sin(mt)/m$ , each of which takes the same value at $t = -\pi$ as at $t = \pi$ ; for $m = 0$ the sine integrand is $0$ ); integration by parts, and the substitution $t \mapsto -t$ , both of which we use later in the section when computing particular Fourier coefficients. From trigonometry: the product-to-sum identities. No theorem about convergence is used anywhere in this section; every integral we compute is the integral of a finite sum of sines and cosines.

Example (The Trigonometric System Is Orthonormal)

Fix $n \ge 1$ . Show that the list of $2n + 1$ functions $\frac{1}{\sqrt2},\ \cos x,\ \sin x,\ \cos 2x,\ \sin 2x,\ \dots,\ \cos nx,\ \sin nx$ is orthonormal in $V = C[-\pi, \pi]$ with $\langle f, g \rangle = \frac1\pi\int_{-\pi}^{\pi} fg$ .

Solution. There are five kinds of pair. Write $j, k$ for indices in $\{1, \dots, n\}$ .

The constant with itself. $\left\| 1/\sqrt2 \right\|^2 = \frac1\pi\int_{-\pi}^{\pi}\tfrac12\,\mathrm{d} t = \frac1\pi \cdot \pi = 1$ .

Constant against $\cos kx$ or $\sin kx$ . These are $\frac{1}{\pi\sqrt2}\int_{-\pi}^{\pi}\cos(kt)\,\mathrm{d} t = 0$ and $\frac{1}{\pi\sqrt2}\int_{-\pi}^{\pi}\sin(kt)\,\mathrm{d} t = 0$ , since $k \ne 0$ .

Two cosines. By $\cos A\cos B = \tfrac12[\cos(A - B) + \cos(A + B)]$ , $\langle \cos jx, \cos kx \rangle = \frac{1}{2\pi}\int_{-\pi}^{\pi}\!\big[\cos((j - k)t) + \cos((j + k)t)\big]\mathrm{d} t .$ If $j \ne k$ , both $j - k$ and $j + k$ are nonzero integers, so both integrals vanish and the value is $0$ . If $j = k$ , the first integrand is the constant $1$ and the second has nonzero frequency $2k$ , so the value is $\frac{1}{2\pi}(2\pi + 0) = 1$ .

Two sines. By $\sin A\sin B = \tfrac12[\cos(A - B) - \cos(A + B)]$ , the same two integrals appear with a minus sign, giving $0$ for $j \ne k$ and $\frac{1}{2\pi}(2\pi - 0) = 1$ for $j = k$ .

A sine against a cosine. By $\sin A\cos B = \tfrac12[\sin(A + B) + \sin(A - B)]$ , $\langle \sin jx, \cos kx \rangle = \frac{1}{2\pi}\int_{-\pi}^{\pi}\!\big[\sin((j + k)t) + \sin((j - k)t)\big]\mathrm{d} t = 0 ,$ because every integral of a sine over $[-\pi, \pi]$ vanishes, including the case $j = k$ where the second integrand is $\sin 0 = 0$ .

So each of the $2n + 1$ functions is a unit vector and distinct ones are orthogonal, which is Definition 11.2.1.

Write $W_n = \operatorname{span}(1, \cos x, \sin x, \dots, \cos nx, \sin nx)$ , the space of trigonometric polynomials of degree at most $n$ . The list above is an orthonormal list, hence independent (Theorem 11.1.4), and it spans $W_n$ , since $1 = \sqrt2 \cdot \tfrac{1}{\sqrt2}$ . So it is an orthonormal basis of $W_n$ and $\dim W_n = 2n + 1$ .

For $f \in V$ the classical Fourier coefficients of $f$ are $a_k = \frac1\pi\int_{-\pi}^{\pi} f(t)\cos(kt)\,\mathrm{d} t, \qquad b_k = \frac1\pi\int_{-\pi}^{\pi} f(t)\sin(kt)\,\mathrm{d} t ,$ for $k \ge 0$ and $k \ge 1$ respectively. Comparing with the inner product, $a_k = \langle f, \cos kx \rangle$ and $b_k = \langle f, \sin kx \rangle$ , while the coefficient against the constant of the orthonormal list is $\langle f, 1/\sqrt2 \rangle = a_0/\sqrt2$ . These are exactly the Fourier coefficients of Definition 11.9.1 for this list.

Theorem 11.9.3 (Fourier Partial Sums Are Best Approximations)

Let $f \in C[-\pi, \pi]$ , let $n \ge 1$ , and let $a_k, b_k$ be the Fourier coefficients of $f$ . Put $S_nf = \frac{a_0}{2} + \sum_{k=1}^{n}\big(a_k\cos kx + b_k\sin kx\big) .$ Then, with $\langle f, g \rangle = \frac1\pi\int_{-\pi}^{\pi} fg$ :

(a) $S_nf = P_{W_n}f$ ;
(b) $\left\| f - S_nf \right\| \le \left\| f - g \right\|$ for every $g \in W_n$ , with equality only for $g = S_nf$ ;
(c) $\displaystyle \frac{a_0^2}{2} + \sum_{k=1}^{n}\big(a_k^2 + b_k^2\big) \le \frac1\pi\int_{-\pi}^{\pi} f(t)^2\,\mathrm{d} t$ .

Idea. Nothing new happens here. The subspace $W_n$ is finite-dimensional and we have an orthonormal basis of it, so Section 3 answers the approximation question and Theorem 11.9.1 answers the coefficient question; the work is only to match the classical notation to ours. The one thing to watch is the factor $\tfrac12$ in front of $a_0$ : it is not a convention pulled from the air, it is what $\langle f, 1/\sqrt2 \rangle \cdot \tfrac{1}{\sqrt2}$ equals.

Proof.

(a) By Example (The Trigonometric System Is Orthonormal) the list $\big(\tfrac{1}{\sqrt2}, \cos x, \sin x, \dots, \cos nx, \sin nx\big)$ is an orthonormal basis of $W_n$ . By Theorem 11.3.3 (a), $P_{W_n}f = \Big\langle f, \tfrac{1}{\sqrt2} \Big\rangle \tfrac{1}{\sqrt2} + \sum_{k=1}^{n}\big(\langle f, \cos kx \rangle\cos kx + \langle f, \sin kx \rangle\sin kx\big).$ The first term is $\frac{a_0}{\sqrt2}\cdot\frac{1}{\sqrt2} = \frac{a_0}{2}$ , and the remaining terms are $a_k\cos kx + b_k\sin kx$ . Hence $P_{W_n}f = S_nf$ .
(b) This is Theorem 11.3.5 applied to the subspace $U = W_n$ , which is legitimate because $\dim W_n = 2n + 1$ is finite, together with (a).
(c) This is Theorem 11.9.1 for the same orthonormal list, since $\lvert\langle f, 1/\sqrt2 \rangle\rvert^2 = a_0^2/2$ and $\left\| f \right\|^2 = \frac1\pi\int_{-\pi}^{\pi} f^2$ .

So the Fourier partial sum is not a guess at $f$ that happens to work: it is the orthogonal projection of $f$ onto the trigonometric polynomials of degree at most $n$ , and by Theorem 11.3.5 it is the unique trigonometric polynomial of that degree closest to $f$ in this norm. The mysterious $a_0/2$ of the classical formula is an artifact of normalizing the constant function.

11.9.3. A worked computation

Example (The Fourier Coefficients of

f(x) = x

)

In $C[-\pi, \pi]$ with $\langle f, g \rangle = \frac1\pi\int_{-\pi}^{\pi} fg$ , let $f(x) = x$ . Find all $a_k$ and $b_k$ , write down $S_nf$ , and say what Theorem 11.9.3 (c) gives.

Solution. We use one further calculus fact: a continuous odd function integrates to $0$ over $[-\pi, \pi]$ , and a continuous even one integrates to twice its integral over $[0, \pi]$ . Both come from the substitution $t \mapsto -t$ .

The cosine coefficients. For every $k \ge 0$ the function $t\cos(kt)$ is odd, so $a_k = 0$ . In particular $a_0 = 0$ .

The sine coefficients. The function $t\sin(kt)$ is even, so for $k \ge 1$ , $b_k = \frac1\pi\int_{-\pi}^{\pi} t\sin(kt)\,\mathrm{d} t = \frac2\pi\int_{0}^{\pi} t\sin(kt)\,\mathrm{d} t .$ Integrating by parts, with $\mathrm{d}(-\cos(kt)/k) = \sin(kt)\,\mathrm{d} t$ , $\int_{0}^{\pi} t\sin(kt)\,\mathrm{d} t = \Big[\frac{-t\cos(kt)}{k}\Big]_0^{\pi} + \frac1k\int_0^{\pi}\cos(kt)\,\mathrm{d} t = \frac{-\pi\cos(k\pi)}{k} ,$ since the last integral is $[\sin(kt)/k^2]_0^\pi = 0$ . As $\cos(k\pi) = (-1)^k$ , this equals $\pi(-1)^{k+1}/k$ , and therefore $b_k = \frac{2(-1)^{k+1}}{k} .$

The partial sums. Hence $S_nf = \sum_{k=1}^{n}\frac{2(-1)^{k+1}}{k}\sin kx = 2\sin x - \sin 2x + \tfrac23\sin 3x - \dots ,$ and by Theorem 11.9.3 (b) this is the closest trigonometric polynomial of degree at most $n$ to the function $x$ in this norm.

What Bessel’s inequality (Theorem 11.9.1) gives. First $\left\| f \right\|^2 = \frac1\pi\int_{-\pi}^{\pi} t^2\,\mathrm{d} t = \frac1\pi\cdot\frac{2\pi^3}{3} = \frac{2\pi^2}{3}$ . Since $a_k = 0$ and $b_k^2 = 4/k^2$ , part (c) of Theorem 11.9.3 reads $\sum_{k=1}^{n}\frac{4}{k^2} \le \frac{2\pi^2}{3}, \qquad\text{that is}\qquad \sum_{k=1}^{n}\frac{1}{k^2} \le \frac{\pi^2}{6}$ for every $n$ . The partial sums on the left increase and are bounded above, so they converge, and $\sum_{k=1}^{\infty} 1/k^2 \le \pi^2/6$ .

That bound is sharp: the true value of the sum is $\pi^2/6$ . But sharpness is not something Bessel can deliver. By Theorem 11.9.2, equality for a fixed finite list holds only when $f$ lies in the span of that list, and the function $x$ lies in no $W_n$ — it is not a trigonometric polynomial, since every element of $W_n$ takes the same value at $-\pi$ and at $\pi$ . Getting the equality requires a statement about the infinite system, and that statement is an input from analysis; Exercise (C1: Parseval for the full trigonometric system) carries it out and flags it.

Quick check. Let $g(x) = \cos^2 x$ in $C[-\pi, \pi]$ . Using $\cos^2 x = \tfrac12 + \tfrac12\cos 2x$ , write down all Fourier coefficients of $g$ , find $S_1g$ , and compute the deficit in Bessel’s inequality for $n = 1$ .

Solution. The right-hand side $\tfrac12 + \tfrac12\cos 2x$ already lies in $W_2$ , and coefficients against an orthonormal basis are unique (Theorem 11.2.1 (a)), so, applying that result inside $W_2$ and reading the coefficients off: $a_0 = 1$ , $a_2 = \tfrac12$ , and all other $a_k$ and all $b_k$ are $0$ . Hence $S_1g = a_0/2 = \tfrac12$ , a constant. For the deficit, $\left\| g \right\|^2 = \frac1\pi\int_{-\pi}^{\pi}\cos^4 t\,\mathrm{d} t = \frac1\pi\cdot\frac{3\pi}{4} = \tfrac34$ , while the Bessel sum for $n = 1$ is $a_0^2/2 = \tfrac12$ . The deficit is $\tfrac14$ , and by $(\ast)$ it should be $\left\| g - S_1g \right\|^2 = \big\lVert\tfrac12\cos 2x\big\rVert^2 = \tfrac14\cdot 1 = \tfrac14$ . It is.

11.9.4. What best approximation does and does not say

It is worth being exact about the boundary here, because it is easy to read more into Theorem 11.9.3 than it contains.

Identity $(\ast)$ applied to $U = W_n$ says $\left\| f - S_nf \right\|^2 = \left\| f \right\|^2 - \frac{a_0^2}{2} - \sum_{k=1}^{n}\big(a_k^2 + b_k^2\big) .$ The subtracted sum never decreases as $n$ grows, so the numbers $\left\| f - S_nf \right\|^2$ never increase, and they are bounded below by $0$ . Hence they converge to some limit $L \ge 0$ . Everything up to this sentence is linear algebra plus one fact about monotone bounded sequences.

Whether $L = 0$ is a different kind of question. It asks whether the trigonometric polynomials come arbitrarily close, in this norm, to every continuous function, and no argument in this book decides it. The statement is true, and it is the analytic content of the theory of Fourier series; it is what makes the infinite trigonometric system behave like an orthonormal basis even though it is not a basis in the sense of Chapter 1.

And even $L = 0$ would not be a statement about individual points. Our norm averages the error over the whole interval, so it tolerates a large error on a short interval. The function $f(x) = x$ shows this at once: $S_nf(\pi) = \sum_{k=1}^{n}\frac{2(-1)^{k+1}}{k}\sin(k\pi) = 0$ for every $n$ , while $f(\pi) = \pi$ . The error at the single point $x = \pi$ never shrinks at all.

Warning. Mean-square best approximation is not pointwise approximation. Theorem 11.9.3 says that $S_nf$ minimizes $\int_{-\pi}^{\pi}(f - g)^2$ over $g \in W_n$ . It does not say that $S_nf(x)$ is close to $f(x)$ for a given $x$ , and for $f(x) = x$ at $x = \pi$ it is not. Whether a Fourier series converges at a point is a question in analysis with a delicate answer; it is not decided by anything in this chapter, and a projection theorem cannot decide it.

11.9.5. Legendre and Chebyshev

Two families of orthogonal polynomials show the same two theorems at work with a different weight.

Legendre. Example (Gram–Schmidt on $1, x, x^2$ ) ran Gram–Schmidt on $(1, x, x^2)$ in $\mathbb{R}[x]_{\le 2}$ with $\langle p, q \rangle = \int_{-1}^{1} pq$ , producing the orthogonal list $1,\ x,\ x^2 - \tfrac13$ and, after normalizing, the orthonormal basis $e_1 = \tfrac{1}{\sqrt2}, \qquad e_2 = \sqrt{3/2}\,x, \qquad e_3 = \tfrac{\sqrt{10}}{4}\big(3x^2 - 1\big).$ Inside $\mathbb{R}[x]_{\le 2}$ this is a basis, so Theorem 11.9.2 gives $\left\| p \right\|^2 = \sum_{i=1}^{3}\lvert\langle p, e_i \rangle\rvert^2$ for every quadratic $p$ . Viewed inside the larger space $C[-1, 1]$ of continuous functions, the same three vectors are only a list, and for an $f$ outside $\mathbb{R}[x]_{\le 2}$ Theorem 11.9.1 is strict, the deficit being the squared distance from $f$ to $\mathbb{R}[x]_{\le 2}$ by $(\ast)$ . That deficit is what Example (The best quadratic approximation to a cubic) computed for $f = x^3$ . One list, two readings, and which reading applies depends on the ambient space, not on the list. In practice one projects with the orthogonal list $1,\ x,\ x^2 - \tfrac13$ through Corollary 11.3.4 and avoids the square roots entirely.

Chebyshev. Change the weight. On $\mathbb{R}[x]$ define $\langle p, q \rangle_w = \int_{-1}^{1}\frac{p(t)\,q(t)}{\sqrt{1 - t^2}}\,\mathrm{d} t .$ The integrand is unbounded at $t = \pm 1$ , so this is an improper integral; it converges for polynomials, because $\lvert pq \rvert$ is bounded on $[-1, 1]$ and $\int_{-1}^{1}\mathrm{d} t/\sqrt{1 - t^2} = \pi$ . The three axioms are checked as in Definition 11.1.1, positivity because $1/\sqrt{1 - t^2} > 0$ on $(-1, 1)$ .

The Chebyshev polynomials are defined by the requirement $T_n(\cos\theta) = \cos n\theta \qquad \text{for all } \theta,$ which does determine a polynomial of degree $n$ , and determines it uniquely, since $\cos\theta$ runs over the whole of $[-1, 1]$ and two polynomials agreeing on an infinite set are equal (Corollary 6.1.4). The first few are $T_0 = 1$ , $T_1 = x$ , $T_2 = 2x^2 - 1$ (from $\cos 2\theta = 2\cos^2\theta - 1$ ), and Exercise (C2: Chebyshev orthogonality by substitution) handles the general case. They are orthogonal for $\langle \cdot, \cdot \rangle_w$ : $\langle T_m, T_n \rangle_w = 0 \quad (m \ne n), \qquad \langle T_n, T_n \rangle_w = \begin{cases} \pi, & n = 0, \\ \pi/2, & n \ge 1. \end{cases}$ For $m, n \le 2$ this can be checked by hand, using $\int_{-1}^{1}\mathrm{d} t/\sqrt{1 - t^2} = \pi$ and $\int_{-1}^{1} t^2\,\mathrm{d} t/\sqrt{1 - t^2} = \pi/2$ (both come from $t = \cos\theta$ ). The pairs $(T_0, T_1)$ and $(T_1, T_2)$ have odd integrands on a symmetric interval, so those integrals are $0$ by the substitution $t \mapsto -t$ . For the remaining pair, $\langle T_0, T_2 \rangle_w = \int_{-1}^{1}\frac{2t^2 - 1}{\sqrt{1 - t^2}}\,\mathrm{d} t = 2\cdot\frac{\pi}{2} - \pi = 0 .$

Why should such a weight produce orthogonality? Because it is the trigonometric system in disguise. Substituting $t = \cos\theta$ turns $\langle p, q \rangle_w$ into $\int_0^{\pi} p(\cos\theta)q(\cos\theta)\,\mathrm{d}\theta$ , and on the Chebyshev polynomials that integrand is $\cos m\theta\cos n\theta$ — the very product whose integral we computed in Example (The Trigonometric System Is Orthonormal). Exercise (C2: Chebyshev orthogonality by substitution) asks for this in full.

Since the $T_n$ are orthogonal but not normalized, projections onto their span are computed with Corollary 11.3.4 rather than Theorem 11.3.3. Section 10 of this chapter takes up orthogonal polynomials as a subject in their own right, including the recurrence that generates the $T_n$ and the quadrature rules they produce.

11.9.6. Exercises

A. Check your understanding

Exercise (A1)

(a) State Bessel’s inequality, saying exactly what is assumed about the space and what is assumed about the list.
(b) State Parseval’s identity, including the condition under which equality holds for a given vector.
(c) True or false: Bessel’s inequality requires the ambient space to be finite-dimensional. Justify your answer.
(d) Write down the inner product used on $C[-\pi, \pi]$ in this section, including its normalizing constant, and say what the constant is for.
(e) State what Theorem 11.9.3 gives about $S_nf$ , and name one thing it does not give.
(f) True or false: if $(e_1, \dots, e_k)$ is an orthonormal list and $\sum_i\lvert\langle v, e_i \rangle\rvert^2 = \left\| v \right\|^2$ for one particular $v \ne 0$ , then the list is an orthonormal basis of $V$ . Justify your answer.

Solution.

(a) If $(e_1, \dots, e_k)$ is an orthonormal list in an inner product space $V$ over $\mathbb{R}$ or $\mathbb{C}$ , then $\sum_{i=1}^{k}\lvert\langle v, e_i \rangle\rvert^2 \le \left\| v \right\|^2$ for every $v \in V$ (Theorem 11.9.1). The list is assumed finite and orthonormal; the space is assumed only to be an inner product space, with no dimension hypothesis.
(b) Equality $\sum_i\lvert\langle v, e_i \rangle\rvert^2 = \left\| v \right\|^2$ holds if and only if $v \in \operatorname{span}(e_1, \dots, e_k)$ ; in particular it holds for every $v$ when the list is an orthonormal basis of $V$ (Theorem 11.9.2).
(c) False. Only the span of the list has to be finite-dimensional, and it automatically is, being spanned by $k$ vectors. The proof uses Theorem 11.3.3, whose hypothesis is that the subspace projected onto is finite-dimensional. The application to $C[-\pi, \pi]$ is exactly a case where $V$ is not finite-dimensional.
(d) $\langle f, g \rangle = \frac1\pi\int_{-\pi}^{\pi} f(t)g(t)\,\mathrm{d} t$ . The constant $1/\pi$ makes $\cos kx$ and $\sin kx$ unit vectors for every $k \ge 1$ ; without it their norms would be $\sqrt{\pi}$ .
(e) It gives that $S_nf$ is the orthogonal projection of $f$ onto $W_n$ , hence the unique element of $W_n$ minimizing $\left\| f - g \right\|$ , and it gives Bessel’s inequality for the coefficients. It does not give that $S_nf(x) \to f(x)$ at any particular $x$ , nor even that $\left\| f - S_nf \right\| \to 0$ .
(f) False. Equality for one $v$ says only that $v \in \operatorname{span}(e_1, \dots, e_k)$ (Theorem 11.9.2). In $\mathbb{R}^3$ take the list $(\mathbf{e}_1)$ and $\mathbf{v} = \mathbf{e}_1$ : equality holds, and the list is not a basis.

B. Practice

Exercise (B1: Fourier coefficients)

Work in $C[-\pi, \pi]$ with $\langle f, g \rangle = \frac1\pi\int_{-\pi}^{\pi} fg$ . Find all Fourier coefficients $a_k$ and $b_k$ of each of the following, and in each case write down $S_2f$ .

(a) $f(x) = \lvert x \rvert$ .
(b) $f(x) = x^2$ .
(c) $f(x) = 3 - 2\sin x + \sin 3x$ .

Solution.

(a) $\lvert t \rvert \sin(kt)$ is odd, so $b_k = 0$ for all $k$ . For the cosines, $\lvert t \rvert\cos(kt)$ is even, so $a_k = \frac2\pi\int_0^{\pi} t\cos(kt)\,\mathrm{d} t$ . For $k = 0$ this is $\frac2\pi\cdot\frac{\pi^2}{2} = \pi$ . For $k \ge 1$ , integrating by parts, $\int_0^{\pi} t\cos(kt)\,\mathrm{d} t = \Big[\frac{t\sin(kt)}{k}\Big]_0^{\pi} - \frac1k\int_0^{\pi}\sin(kt)\,\mathrm{d} t = \frac{(-1)^k - 1}{k^2},$ since the bracket vanishes and $\int_0^\pi \sin(kt)\,\mathrm{d} t = (1 - (-1)^k)/k$ . Hence $a_k = \frac{2\big((-1)^k - 1\big)}{\pi k^2} = \begin{cases} -\dfrac{4}{\pi k^2}, & k \text{ odd},\\[2pt] 0, & k \text{ even},\end{cases}$ and $S_2f = \dfrac{\pi}{2} - \dfrac{4}{\pi}\cos x$ .
(b) $t^2\sin(kt)$ is odd, so $b_k = 0$ . Also $a_0 = \frac2\pi\int_0^\pi t^2\,\mathrm{d} t = \frac{2\pi^2}{3}$ . For $k \ge 1$ , integrating by parts twice gives $\int_0^{\pi} t^2\cos(kt)\,\mathrm{d} t = \Big[\frac{t^2\sin(kt)}{k}\Big]_0^\pi - \frac2k\int_0^\pi t\sin(kt)\,\mathrm{d} t = \frac{2\pi(-1)^{k}}{k^2},$ using $\int_0^\pi t\sin(kt)\,\mathrm{d} t = \pi(-1)^{k+1}/k$ from Example (The Fourier Coefficients of $f(x) = x$ ). Hence $a_k = \frac2\pi\cdot\frac{2\pi(-1)^k}{k^2} = \frac{4(-1)^k}{k^2}$ , so $a_1 = -4$ , $a_2 = 1$ , and $S_2f = \frac{\pi^2}{3} - 4\cos x + \cos 2x .$
(c) The function already lies in $W_3$ . Coefficients against an orthonormal basis are unique (Theorem 11.2.1 (a)), so, applying that result inside $W_3$ , we may read them off: $a_0/2 = 3$ , that is $a_0 = 6$ , together with $b_1 = -2$ and $b_3 = 1$ ; every other coefficient is $0$ . Hence $S_2f = 3 - 2\sin x$ , which is not $f$ : the term $\sin 3x$ lies outside $W_2$ and is discarded by the projection. (From $n = 3$ on, $S_nf = f$ , as Theorem 11.3.3 (a) predicts, since $f \in W_3$ and a projection fixes its own subspace.)

Exercise (B2: Verifying Bessel for a truncation)

(a) In $\mathbb{R}^4$ with the dot product let $\mathbf{e}_1 = \tfrac12(1, 1, 1, 1)$ , $\mathbf{e}_2 = \tfrac12(1, -1, 1, -1)$ and $\mathbf{v} = (2, 0, 0, 2)$ . Verify Theorem 11.9.1 for $\mathbf{v}$ and this list, compute the deficit, and check against $(\ast)$ by computing $\left\| \mathbf{v} - P_U\mathbf{v} \right\|$ directly, where $U = \operatorname{span}(\mathbf{e}_1, \mathbf{e}_2)$ .
(b) For $f(x) = \lvert x \rvert$ and $n = 1$ , write out both sides of Theorem 11.9.3 (c) using Exercise (B1: Fourier coefficients) (a), and show that the resulting inequality is equivalent to $96 \le \pi^4$ .

Solution.

(a) Both vectors have norm $\tfrac12\sqrt4 = 1$ and $\langle \mathbf{e}_1, \mathbf{e}_2 \rangle = \tfrac14(1 - 1 + 1 - 1) = 0$ , so the list is orthonormal. The coefficients are $\langle \mathbf{v}, \mathbf{e}_1 \rangle = \tfrac12(2 + 0 + 0 + 2) = 2$ and $\langle \mathbf{v}, \mathbf{e}_2 \rangle = \tfrac12(2 - 0 + 0 - 2) = 0$ , so the left-hand side of Bessel is $4$ , while $\left\| \mathbf{v} \right\|^2 = 4 + 4 = 8$ . The inequality $4 \le 8$ holds, with deficit $4$ . By Theorem 11.3.3 (a), $P_U\mathbf{v} = 2\mathbf{e}_1 = (1, 1, 1, 1)$ , so $\mathbf{v} - P_U\mathbf{v} = (1, -1, -1, 1)$ and $\left\| \mathbf{v} - P_U\mathbf{v} \right\|^2 = 4$ , matching $(\ast)$ .
(b) With $a_0 = \pi$ and $a_1 = -4/\pi$ , and $b_1 = 0$ , the left-hand side of (c) for $n = 1$ is $\frac{a_0^2}{2} + a_1^2 = \frac{\pi^2}{2} + \frac{16}{\pi^2} .$ The right-hand side is $\left\| f \right\|^2 = \frac1\pi\int_{-\pi}^{\pi} t^2\,\mathrm{d} t = \frac{2\pi^2}{3}$ . So the inequality reads $\frac{\pi^2}{2} + \frac{16}{\pi^2} \le \frac{2\pi^2}{3} ,$ that is $16/\pi^2 \le 2\pi^2/3 - \pi^2/2 = \pi^2/6$ , that is $96 \le \pi^4$ . Since $\pi > 3.14$ and $3.14^4 > 97$ , the inequality is true, and it is tight: the deficit is $\pi^2/6 - 16/\pi^2 \approx 0.024$ , so the single cosine $\cos x$ already captures almost all of $\lvert x \rvert$ .

Exercise (B3: The best approximation by

\operatorname{span}(1, \cos x)

)

In $C[-\pi, \pi]$ with $\langle f, g \rangle = \frac1\pi\int_{-\pi}^{\pi} fg$ , let $U = \operatorname{span}(1, \cos x)$ and $f(x) = x^2$ . Find $P_Uf$ and $d(f, U)^2$ . Hint: the list $(1, \cos x)$ is orthogonal but not orthonormal.

Solution. The two functions are orthogonal, since $\langle 1, \cos x \rangle = \frac1\pi\int_{-\pi}^{\pi}\cos t\,\mathrm{d} t = 0$ , and both are nonzero, with $\langle 1, 1 \rangle = 2$ and $\langle \cos x, \cos x \rangle = 1$ by Example (The Trigonometric System Is Orthonormal). So $(1, \cos x)$ is an orthogonal basis of $U$ , and Corollary 11.3.4 gives $P_Uf = \frac{\langle f, 1 \rangle}{2}\cdot 1 + \frac{\langle f, \cos x \rangle}{1}\cdot\cos x .$ By Exercise (B1: Fourier coefficients) (b), $\langle f, 1 \rangle = a_0 = 2\pi^2/3$ and $\langle f, \cos x \rangle = a_1 = -4$ . Hence $P_Uf = \frac{\pi^2}{3} - 4\cos x .$ For the distance, normalize: $(1/\sqrt2,\ \cos x)$ is an orthonormal basis of the same $U$ , with coefficients $\langle f, 1/\sqrt2 \rangle = a_0/\sqrt2$ and $\langle f, \cos x \rangle = a_1$ . Now $\left\| f \right\|^2 = \frac1\pi\int_{-\pi}^{\pi} t^4\,\mathrm{d} t = \frac1\pi\cdot\frac{2\pi^5}{5} = \frac{2\pi^4}{5}$ and the Bessel sum is $a_0^2/2 + a_1^2 = 2\pi^4/9 + 16$ , so $(\ast)$ gives $d(f, U)^2 = \frac{2\pi^4}{5} - \frac{2\pi^4}{9} - 16 = \frac{8\pi^4}{45} - 16 \approx 1.32 .$ It is positive, as it must be, since $x^2 \notin U$ .

C. Going deeper

Exercise (C1: Parseval for the full trigonometric system)

This exercise uses an input from analysis that this book does not prove: the trigonometric system is complete in $C[-\pi, \pi]$ , meaning that for every $f \in C[-\pi, \pi]$ , $\frac{a_0^2}{2} + \sum_{k=1}^{\infty}\big(a_k^2 + b_k^2\big) = \frac1\pi\int_{-\pi}^{\pi} f(t)^2\,\mathrm{d} t .$

(a) Apply this to $f(x) = x$ and deduce that $\sum_{k=1}^{\infty} 1/k^2 = \pi^2/6$ .
(b) Hence deduce that $\sum_{k=1}^{\infty} 1/(2k - 1)^2 = \pi^2/8$ .
(c) Say precisely which step used the analysis input, and explain why no result of this section can supply it.

Solution.

(a) By Example (The Fourier Coefficients of $f(x) = x$ ), $a_k = 0$ for all $k$ and $b_k = 2(-1)^{k+1}/k$ , so $b_k^2 = 4/k^2$ ; and $\left\| f \right\|^2 = 2\pi^2/3$ . The stated identity becomes $\sum_{k=1}^{\infty}\frac{4}{k^2} = \frac{2\pi^2}{3},$ so $\sum_{k=1}^{\infty} 1/k^2 = \pi^2/6$ .
(b) All terms are positive and the series converges, so it may be split into its even-index and odd-index parts. The even part is $\sum_{j=1}^{\infty}\frac{1}{(2j)^2} = \frac14\sum_{j=1}^{\infty}\frac{1}{j^2} = \frac{\pi^2}{24}.$ Hence the odd part is $\pi^2/6 - \pi^2/24 = (4\pi^2 - \pi^2)/24 = \pi^2/8$ , which is the claim.
(c) The analysis input is used exactly once, at the equality sign in (a). This section proves only the inequality $\sum_{k\le n} 4/k^2 \le 2\pi^2/3$ for each $n$ (Theorem 11.9.1), and Theorem 11.9.2 turns that inequality into an equality only when $f$ lies in the span of the finitely many functions used. The function $x$ lies in no $W_n$ , since every element of $W_n$ takes equal values at $-\pi$ and $\pi$ while $f(-\pi) = -\pi \ne \pi = f(\pi)$ . Passing to the infinite system is a statement about a limit of functions, and nothing in this book proves such a statement.

Remark. The classical route to $\sum 1/(2k-1)^2 = \pi^2/8$ applies the same identity to the square wave, the function equal to $-1$ on $[-\pi, 0)$ and $1$ on $[0, \pi]$ , whose coefficients are $b_k = 4/(\pi k)$ for odd $k$ and $0$ otherwise. That function is not continuous, so it does not live in the space $C[-\pi, \pi]$ of this section, and enlarging the space to all integrable functions destroys positive definiteness, as the third nonexample of Section 1 shows. Working with $f(x) = x$ keeps everything inside a genuine inner product space.

Exercise (C2: Chebyshev orthogonality by substitution)

Let $\langle p, q \rangle_w = \int_{-1}^{1} p(t)q(t)(1 - t^2)^{-1/2}\,\mathrm{d} t$ on $\mathbb{R}[x]$ , the Chebyshev inner product of the text.

(a) Prove that for every $n \ge 0$ there is a polynomial $T_n$ of degree $n$ with $T_n(\cos\theta) = \cos n\theta$ for all $\theta$ , and find $T_0, T_1, T_2, T_3$ .
(b) Prove that $\langle p, q \rangle_w = \int_0^{\pi} p(\cos\theta)\,q(\cos\theta)\,\mathrm{d}\theta$ for all polynomials $p, q$ .
(c) Deduce that $\langle T_m, T_n \rangle_w = 0$ for $m \ne n$ , and compute $\langle T_n, T_n \rangle_w$ .

Hint for (a): the identity $\cos((n+1)\theta) + \cos((n-1)\theta) = 2\cos\theta\cos n\theta$ .

Solution.

(a) Induction on $n$ . Take $T_0 = 1$ and $T_1 = x$ , which satisfy $T_0(\cos\theta) = 1 = \cos 0\theta$ and $T_1(\cos\theta) = \cos\theta$ , of degrees $0$ and $1$ . Suppose $T_{n-1}$ and $T_n$ exist, of degrees $n - 1$ and $n$ . Put $T_{n+1} = 2xT_n - T_{n-1}$ . Then for every $\theta$ , $T_{n+1}(\cos\theta) = 2\cos\theta\cos n\theta - \cos((n-1)\theta) = \cos((n+1)\theta)$ by the stated identity. Its degree is $n + 1$ , since the leading term of $2xT_n$ has degree $n + 1$ and $T_{n-1}$ has degree $n - 1$ . This completes the induction, and the recurrence gives $T_2 = 2x^2 - 1$ and $T_3 = 2x(2x^2 - 1) - x = 4x^3 - 3x$ .
(b) The integral is improper at both endpoints, so it means the limit as $\delta \to 0^{+}$ of the integrals over $[-1 + \delta,\ 1 - \delta]$ . On each such interval the integrand is continuous, and $t = \cos\theta$ is a continuously differentiable bijection from the corresponding $\theta$ -interval, so the change-of-variables rule applies there. With $t = \cos\theta$ , $\theta$ running from $\pi$ down to $0$ , we have $\mathrm{d} t = -\sin\theta\,\mathrm{d}\theta$ and $\sqrt{1 - t^2} = \sin\theta$ , which is $\ge 0$ on $[0, \pi]$ . The two minus signs cancel when the limits are reversed, and letting $\delta \to 0^{+}$ , $\int_{-1}^{1}\frac{p(t)q(t)}{\sqrt{1 - t^2}}\,\mathrm{d} t = \int_{0}^{\pi} p(\cos\theta)\,q(\cos\theta)\,\mathrm{d}\theta .$ The right-hand side is a proper integral of a continuous function, so both sides converge, and the limits agree.
(c) By (a) and (b), $\langle T_m, T_n \rangle_w = \int_0^{\pi}\cos m\theta\cos n\theta\,\mathrm{d}\theta$ . Using $\cos A\cos B = \tfrac12[\cos(A-B) + \cos(A+B)]$ and $\int_0^{\pi}\cos(p\theta)\,\mathrm{d}\theta = [\sin(p\theta)/p]_0^{\pi} = 0$ for every nonzero integer $p$ :

if $m \ne n$ , then $m - n \ne 0$ , and $m + n \ge 1$ because the two indices differ, so both integrals vanish and the value is $0$ ;

if $m = n \ge 1$ , the value is $\tfrac12\big(\int_0^\pi 1\,\mathrm{d}\theta + 0\big) = \pi/2$ ;

if $m = n = 0$ , the integrand is $1$ and the value is $\pi$ .

So the $T_n$ are orthogonal for $\langle \cdot, \cdot \rangle_w$ , with $\langle T_0, T_0 \rangle_w = \pi$ and $\langle T_n, T_n \rangle_w = \pi/2$ for $n \ge 1$ . Consequently $\big(T_0/\sqrt{\pi},\ T_1\sqrt{2/\pi},\ \dots,\ T_n\sqrt{2/\pi}\big)$ is an orthonormal list, and Theorem 11.9.1 applies to it verbatim.