13.5 The Loewner Order

Every theorem so far in this chapter has compared a self-adjoint matrix with $0$ . Comparing it with $0$ is only the first case of comparing it with something, and the analogy with real numbers suggests the general one: say that $\mathbf{A}$ is at least $\mathbf{B}$ when the difference is nonnegative. That is a genuine order, and a surprising amount of ordinary arithmetic survives it. A little does not, and what fails is the point of the section: squaring both sides of an inequality between matrices is not allowed.

Throughout, $F = \mathbb{R}$ or $F = \mathbb{C}$ , all matrices are in $M_n(F)$ , and $\lambda_1(\mathbf{A}) \ge \dots \ge \lambda_n(\mathbf{A})$ are the eigenvalues of a Hermitian $\mathbf{A}$ in decreasing order.

13.5.1. Comparing two self-adjoint matrices

The relation $\mathbf{A} \succeq \mathbf{O}$ already behaves like “ $\ge 0$ ” for numbers: it is preserved by sums, by nonnegative scalars, and by congruence (Proposition 13.1.2). For real numbers, $a \ge b$ means nothing other than $a - b \ge 0$ . Copying that sentence is the whole definition.

One self-adjoint matrix dominates another when their difference is positive semidefinite.

Definition 13.5.1 (The Loewner order)

Let $\mathbf{A}, \mathbf{B} \in M_n(F)$ be Hermitian. Write $\mathbf{A} \succeq \mathbf{B} \quad \text{to mean} \quad \mathbf{A} - \mathbf{B} \succeq \mathbf{O} ,$ and $\mathbf{A} \succ \mathbf{B}$ to mean $\mathbf{A} - \mathbf{B} \succ \mathbf{O}$ . This relation on the set of Hermitian matrices is the Loewner order.

In words: $\mathbf{A} \succeq \mathbf{B}$ says that $\langle \mathbf{A}\mathbf{x}, \mathbf{x} \rangle \ge \langle \mathbf{B}\mathbf{x}, \mathbf{x} \rangle$ for every $\mathbf{x} \in F^n$ , since $\langle (\mathbf{A} - \mathbf{B})\mathbf{x}, \mathbf{x} \rangle = \langle \mathbf{A}\mathbf{x}, \mathbf{x} \rangle - \langle \mathbf{B}\mathbf{x}, \mathbf{x} \rangle$ . So the order compares the two quadratic forms pointwise, and it is exactly as strong as that comparison. Note also that the difference of two Hermitian matrices is Hermitian, so clause (P1) of Definition 13.1.1 is automatic here and only (P2) has to be checked. Taking $\mathbf{B} = \mathbf{O}$ recovers $\mathbf{A} \succeq \mathbf{O}$ , so the notation is consistent.

Three examples, each of which will be used later.

Diagonal matrices. $\operatorname{diag}(a_1, \dots, a_n) \succeq \operatorname{diag}(b_1, \dots, b_n)$ if and only if $a_i \ge b_i$ for every $i$ , since the difference is diagonal and a diagonal Hermitian matrix is positive semidefinite exactly when its entries, which are its eigenvalues, are $\ge 0$ (Theorem 13.1.4).
Eigenvalue bounds. For Hermitian $\mathbf{A}$ , $\lambda_1(\mathbf{A})\mathbf{I} \succeq \mathbf{A} \succeq \lambda_n(\mathbf{A})\mathbf{I}$ . Indeed $\lambda_1(\mathbf{A})\mathbf{I} - \mathbf{A}$ is Hermitian with eigenvalues $\lambda_1(\mathbf{A}) - \lambda_i(\mathbf{A}) \ge 0$ .
Projections. If $\mathbf{P} \in M_n(F)$ is the matrix of an orthogonal projection onto a subspace of $F^n$ , then $\mathbf{I} \succeq \mathbf{P} \succeq \mathbf{O}$ : it is Hermitian, and its eigenvalues are $0$ and $1$ .

A nonexample by minimal change: replace $\operatorname{diag}(1, 1)$ by $\operatorname{diag}(1, -1)$ in the first item. Then $\operatorname{diag}(1, -1) \succeq \operatorname{diag}(0, 0)$ fails, and it fails only in the second coordinate — the test is coordinatewise for diagonal matrices, and one bad coordinate is enough.

Proposition 13.5.1 (The Loewner order is a partial order)

On the set of Hermitian matrices in $M_n(F)$ , the relation $\succeq$ is reflexive, transitive, and antisymmetric: $\mathbf{A} \succeq \mathbf{A}$ ; if $\mathbf{A} \succeq \mathbf{B}$ and $\mathbf{B} \succeq \mathbf{C}$ then $\mathbf{A} \succeq \mathbf{C}$ ; and if $\mathbf{A} \succeq \mathbf{B}$ and $\mathbf{B} \succeq \mathbf{A}$ then $\mathbf{A} = \mathbf{B}$ .

Proof. Reflexivity: $\mathbf{A} - \mathbf{A} = \mathbf{O}$ , and $\langle \mathbf{O}\mathbf{x}, \mathbf{x} \rangle = 0 \ge 0$ .

Transitivity: $\mathbf{A} - \mathbf{C} = (\mathbf{A} - \mathbf{B}) + (\mathbf{B} - \mathbf{C})$ , and for every $\mathbf{x}$ , $\langle (\mathbf{A} - \mathbf{C})\mathbf{x}, \mathbf{x} \rangle = \langle (\mathbf{A} - \mathbf{B})\mathbf{x}, \mathbf{x} \rangle + \langle (\mathbf{B} - \mathbf{C})\mathbf{x}, \mathbf{x} \rangle \ \ge\ 0 ,$ both summands being $\ge 0$ by hypothesis.

Antisymmetry: put $\mathbf{S} = \mathbf{A} - \mathbf{B}$ , which is Hermitian. From $\mathbf{A} \succeq \mathbf{B}$ we get $\langle \mathbf{S}\mathbf{x}, \mathbf{x} \rangle \ge 0$ for every $\mathbf{x}$ , and from $\mathbf{B} \succeq \mathbf{A}$ we get $\langle -\mathbf{S}\mathbf{x}, \mathbf{x} \rangle \ge 0$ , that is $\langle \mathbf{S}\mathbf{x}, \mathbf{x} \rangle \le 0$ . Hence $\langle \mathbf{S}\mathbf{x}, \mathbf{x} \rangle = 0$ for every $\mathbf{x} \in F^n$ , and $\mathbf{S}$ is self-adjoint, so $\mathbf{S} = \mathbf{O}$ by Theorem 12.2.5. Thus $\mathbf{A} = \mathbf{B}$ .

Antisymmetry is the clause that needed an argument, and the argument is the one the chapter keeps reusing: a self-adjoint matrix is determined by its quadratic form. Over $\mathbb{R}$ this is genuinely a theorem — the quarter turn of Definition 13.1.1’s warning has vanishing quadratic form and is not $\mathbf{O}$ — and it is self-adjointness that rules that out.

Warning. The Loewner order is partial, and it is not the entrywise order. Two things go wrong if one forgets this. First, most pairs are incomparable: $\mathbf{A} = \operatorname{diag}(1, 0)$ and $\mathbf{B} = \operatorname{diag}(0, 1)$ satisfy neither $\mathbf{A} \succeq \mathbf{B}$ nor $\mathbf{B} \succeq \mathbf{A}$ , so an argument that splits into “either $\mathbf{A} \succeq \mathbf{B}$ or $\mathbf{B} \succeq \mathbf{A}$ ” is not an argument. Second, $\mathbf{A} \succeq \mathbf{B}$ says nothing entrywise, in either direction: $\begin{pmatrix} 2 & 0 \\ 0 & 2 \end{pmatrix} \succeq \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}, \qquad\text{yet the off-diagonal entries went down;}$ the difference is $\begin{psmallmatrix} 1 & -1 \\ -1 & 1\end{psmallmatrix}$ , with eigenvalues $0$ and $2$ . Conversely $\begin{psmallmatrix} 1 & 2 \\ 2 & 1\end{psmallmatrix}$ has every entry $\ge 0$ and is not $\succeq \mathbf{O}$ , its eigenvalues being $3$ and $-1$ .

Quick check. Is the set of Hermitian $\mathbf{A}$ with $\mathbf{I} \succeq \mathbf{A} \succeq \mathbf{O}$ closed under multiplication?

Solution. No. Take $\mathbf{A} = \begin{psmallmatrix} 1 & 0 \\ 0 & 0\end{psmallmatrix}$ and $\mathbf{B} = \tfrac12\begin{psmallmatrix} 1 & 1 \\ 1 & 1\end{psmallmatrix}$ , both orthogonal projections, so both lie in the set. Then $\mathbf{A}\mathbf{B} = \tfrac12\begin{psmallmatrix} 1 & 1 \\ 0 & 0\end{psmallmatrix}$ , which is not even Hermitian, so it is not in the set and the order does not apply to it. The order lives on Hermitian matrices, and Hermitian matrices are not closed under multiplication (Exercise (C2: Sums and products)).

13.5.2. What survives

One small fact is needed first, and it is worth stating on its own because it is the elementary half of a theme Chapter 17 takes up in full: the largest and smallest eigenvalues of a Hermitian matrix are the largest and smallest values of its quadratic form on the unit sphere.

Lemma 13.5.2 (The extreme eigenvalues as extreme values)

Let $\mathbf{A} \in M_n(F)$ be Hermitian. Then for every $\mathbf{x} \in F^n$ with $\left\| \mathbf{x} \right\| = 1$ , $\lambda_n(\mathbf{A}) \ \le\ \langle \mathbf{A}\mathbf{x}, \mathbf{x} \rangle \ \le\ \lambda_1(\mathbf{A}) ,$ and both bounds are attained, at unit eigenvectors for $\lambda_n(\mathbf{A})$ and $\lambda_1(\mathbf{A})$ respectively.

Proof. By Corollary 12.4.3 (over $\mathbb{C}$ ) or Corollary 12.5.2 (over $\mathbb{R}$ ), $F^n$ has an orthonormal basis $(\mathbf{q}_1, \dots, \mathbf{q}_n)$ with $\mathbf{A}\mathbf{q}_i = \lambda_i(\mathbf{A})\mathbf{q}_i$ . Write $\mathbf{x} = \sum_i c_i\mathbf{q}_i$ ; then $\sum_i\lvert c_i\rvert^2 = \left\| \mathbf{x} \right\|^2 = 1$ by Theorem 11.2.1, and $\langle \mathbf{A}\mathbf{x}, \mathbf{x} \rangle = \Big\langle \sum_i \lambda_i(\mathbf{A})c_i\mathbf{q}_i, \sum_j c_j\mathbf{q}_j\Big\rangle = \sum_{i} \lambda_i(\mathbf{A})\lvert c_i\rvert^2 ,$ the cross terms vanishing by orthonormality. The right-hand side is a weighted average of the $\lambda_i(\mathbf{A})$ with nonnegative weights summing to $1$ , hence lies between the smallest and the largest. Taking $\mathbf{x} = \mathbf{q}_1$ gives $\lambda_1(\mathbf{A})$ and $\mathbf{x} = \mathbf{q}_n$ gives $\lambda_n(\mathbf{A})$ .

Theorem 13.5.3 (Basic Properties of the Loewner Order)

Let $\mathbf{A}, \mathbf{B} \in M_n(F)$ be Hermitian with $\mathbf{A} \succeq \mathbf{B}$ . Then:

(a) $\mathbf{S}^{*}\mathbf{A}\mathbf{S} \succeq \mathbf{S}^{*}\mathbf{B}\mathbf{S}$ for every $\mathbf{S} \in M_{n \times m}(F)$ ;
(b) $\lambda_1(\mathbf{A}) \ge \lambda_1(\mathbf{B})$ and $\lambda_n(\mathbf{A}) \ge \lambda_n(\mathbf{B})$ ;
(c) if moreover $\mathbf{B} \succeq \mathbf{O}$ , then $\operatorname{tr}\mathbf{A} \ge \operatorname{tr}\mathbf{B}$ and $\det\mathbf{A} \ge \det\mathbf{B}$ ;
(d) if moreover $\mathbf{B} \succ \mathbf{O}$ , then $\mathbf{A} \succ \mathbf{O}$ and $\mathbf{B}^{-1} \succeq \mathbf{A}^{-1}$ .

Idea. Each clause has a one-line mechanism. (a) is the substitution $\mathbf{x} = \mathbf{S}\mathbf{y}$ . (b) evaluates the quadratic form at the vector that is extremal for the other matrix. (c) splits: the trace is a sum of eigenvalues, and for the determinant we whiten by $\mathbf{B}^{-1/2}$ so that the comparison becomes “ $\succeq \mathbf{I}$ ”, where the determinant is visibly $\ge 1$ . (d) is the same whitening, plus the observation that inverting reverses the order for numbers $\ge 1$ .

Proof.

(a) This is Proposition 13.1.2 (a) applied to the Hermitian matrix $\mathbf{A} - \mathbf{B} \succeq \mathbf{O}$ , together with $\mathbf{S}^{*}\mathbf{A}\mathbf{S} - \mathbf{S}^{*}\mathbf{B}\mathbf{S} = \mathbf{S}^{*}(\mathbf{A} - \mathbf{B})\mathbf{S}$ .
(b) Let $\mathbf{x}$ be a unit eigenvector of $\mathbf{B}$ for $\lambda_1(\mathbf{B})$ . By Lemma 13.5.2 applied twice, and $\mathbf{A} \succeq \mathbf{B}$ in between, $\lambda_1(\mathbf{B}) = \langle \mathbf{B}\mathbf{x}, \mathbf{x} \rangle \le \langle \mathbf{A}\mathbf{x}, \mathbf{x} \rangle \le \lambda_1(\mathbf{A}) .$ Now let $\mathbf{y}$ be a unit eigenvector of $\mathbf{A}$ for $\lambda_n(\mathbf{A})$ . Then $\lambda_n(\mathbf{A}) = \langle \mathbf{A}\mathbf{y}, \mathbf{y} \rangle \ge \langle \mathbf{B}\mathbf{y}, \mathbf{y} \rangle \ge \lambda_n(\mathbf{B}) .$
(c) The matrix $\mathbf{A} - \mathbf{B}$ is Hermitian and $\succeq \mathbf{O}$ , so all its eigenvalues are $\ge 0$ by Theorem 13.1.4, and $\operatorname{tr}(\mathbf{A} - \mathbf{B})$ , being their sum (Theorem 9.3.3), is $\ge 0$ . Hence $\operatorname{tr}\mathbf{A} \ge \operatorname{tr}\mathbf{B}$ .

For the determinant, first note $\mathbf{A} \succeq \mathbf{B} \succeq \mathbf{O}$ gives $\mathbf{A} \succeq \mathbf{O}$ by Proposition 13.5.1, so $\det\mathbf{A} \ge 0$ and $\det\mathbf{B} \ge 0$ , both being products of nonnegative eigenvalues. If $\mathbf{B}$ is singular then $\det\mathbf{B} = 0 \le \det\mathbf{A}$ and we are done. Otherwise $\mathbf{B} \succeq \mathbf{O}$ with no zero eigenvalue means $\mathbf{B} \succ \mathbf{O}$ (Theorem 13.1.5), so $\mathbf{B}^{1/2} \succ \mathbf{O}$ is invertible and we may set $\mathbf{C} = \mathbf{B}^{-1/2}\mathbf{A}\mathbf{B}^{-1/2}$ , where $\mathbf{B}^{-1/2} = (\mathbf{B}^{1/2})^{-1}$ is Hermitian. By (a) with $\mathbf{S} = \mathbf{B}^{-1/2}$ , $\mathbf{C} = \mathbf{B}^{-1/2}\mathbf{A}\mathbf{B}^{-1/2} \ \succeq\ \mathbf{B}^{-1/2}\mathbf{B}\mathbf{B}^{-1/2} = \mathbf{I}_n ,$ so $\mathbf{C} - \mathbf{I} \succeq \mathbf{O}$ and every eigenvalue of $\mathbf{C}$ is $\ge 1$ . Hence $\det\mathbf{C} \ge 1$ . Since $\mathbf{A} = \mathbf{B}^{1/2}\mathbf{C}\mathbf{B}^{1/2}$ , Theorem 7.5.4 gives $\det\mathbf{A} = \det(\mathbf{B}^{1/2})^2\det\mathbf{C} = \det\mathbf{B}\cdot\det\mathbf{C} \ \ge\ \det\mathbf{B} .$
(d) Suppose $\mathbf{B} \succ \mathbf{O}$ . For $\mathbf{x} \ne \mathbf{0}$ , $\langle \mathbf{A}\mathbf{x}, \mathbf{x} \rangle \ge \langle \mathbf{B}\mathbf{x}, \mathbf{x} \rangle > 0$ , so $\mathbf{A} \succ \mathbf{O}$ and in particular $\mathbf{A}$ is invertible (Theorem 13.1.5). Let $\mathbf{C} = \mathbf{B}^{-1/2}\mathbf{A}\mathbf{B}^{-1/2} \succeq \mathbf{I}$ as in (c). Then $\mathbf{C} \succ \mathbf{O}$ , so $\mathbf{C}$ is invertible, and $\mathbf{C}^{-1}$ is Hermitian with eigenvalues the reciprocals of those of $\mathbf{C}$ , all lying in $(0, 1]$ . Therefore $\mathbf{I} - \mathbf{C}^{-1}$ is Hermitian with nonnegative eigenvalues, that is $\mathbf{I} \succeq \mathbf{C}^{-1}$ (Theorem 13.1.4). Applying (a) with $\mathbf{S} = \mathbf{B}^{-1/2}$ to this inequality, $\mathbf{B}^{-1} = \mathbf{B}^{-1/2}\mathbf{I}\mathbf{B}^{-1/2} \ \succeq\ \mathbf{B}^{-1/2}\mathbf{C}^{-1}\mathbf{B}^{-1/2} = \mathbf{B}^{-1/2}\mathbf{B}^{1/2}\mathbf{A}^{-1}\mathbf{B}^{1/2}\mathbf{B}^{-1/2} = \mathbf{A}^{-1} ,$ using $\mathbf{C}^{-1} = (\mathbf{B}^{-1/2}\mathbf{A}\mathbf{B}^{-1/2})^{-1} = \mathbf{B}^{1/2}\mathbf{A}^{-1}\mathbf{B}^{1/2}$ . This proves the theorem.

Clause (d) is the memorable one: the order reverses under inversion, exactly as $a \ge b > 0$ gives $1/b \ge 1/a$ for numbers. Clause (b) is the beginning of a longer story. The full statement is that $\lambda_i(\mathbf{A}) \ge \lambda_i(\mathbf{B})$ for every $i$ , not only for $i = 1$ and $i = n$ ; but the intermediate eigenvalues have no description as a plain maximum over the whole unit sphere, and the min–max description that does the job is the Courant–Fischer theorem of Chapter 17. Nothing in this chapter uses the intermediate cases, so we prove the two extreme ones and leave the rest there.

Remark. Clause (c) does not extend to $\mathbf{A} \succeq \mathbf{B}$ without $\mathbf{B} \succeq \mathbf{O}$ . Take $\mathbf{A} = \operatorname{diag}(1, -1)$ and $\mathbf{B} = \operatorname{diag}(0, -2)$ : then $\mathbf{A} - \mathbf{B} = \operatorname{diag}(1, 1) \succeq \mathbf{O}$ , so $\mathbf{A} \succeq \mathbf{B}$ , yet $\det\mathbf{A} = -1 < 0 = \det\mathbf{B}$ . The hypothesis $\mathbf{B} \succeq \mathbf{O}$ is what lets “bigger” be read off the eigenvalues one at a time, and so multiplied up into a determinant.

13.5.3. What fails: squaring

For positive numbers, $a \ge b \ge 0$ implies $a^2 \ge b^2$ . The matrix version is false, and the smallest counterexample has integer entries.

Example (Squaring does not preserve the Loewner order)

Let $\mathbf{A} = \begin{pmatrix} 2 & 1 \\ 1 & 1 \end{pmatrix}, \qquad \mathbf{B} = \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix} .$ Show that $\mathbf{A} \succeq \mathbf{B} \succeq \mathbf{O}$ and that $\mathbf{A}^2 \succeq \mathbf{B}^2$ is false.

Solution. The hypotheses. Both matrices are real symmetric. $\mathbf{B} \succeq \mathbf{O}$ : its leading principal minors are $1$ and $0$ , and directly $\mathbf{x}^{\top}\mathbf{B}\mathbf{x} = (x_1 + x_2)^2 \ge 0$ . Its eigenvalues are $0$ and $2$ . Next, $\mathbf{A} - \mathbf{B} = \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix} \succeq \mathbf{O} ,$ so $\mathbf{A} \succeq \mathbf{B}$ . (In fact $\mathbf{A} \succ \mathbf{O}$ : $\det\mathbf{A} = 1 > 0$ and $a_{11} = 2 > 0$ , so Sylvester’s criterion applies, Theorem 13.1.5.)

The squares. $\mathbf{A}^2 = \begin{pmatrix} 5 & 3 \\ 3 & 2 \end{pmatrix}, \qquad \mathbf{B}^2 = \begin{pmatrix} 2 & 2 \\ 2 & 2 \end{pmatrix}, \qquad \mathbf{A}^2 - \mathbf{B}^2 = \begin{pmatrix} 3 & 1 \\ 1 & 0 \end{pmatrix} .$ Its determinant is $0 - 1 = -1 < 0$ , so its two eigenvalues have opposite signs and it is not positive semidefinite (Theorem 13.1.4). A witness is available without computing eigenvalues: with $\mathbf{x} = (1, -2)$ , $\mathbf{x}^{\top}(\mathbf{A}^2 - \mathbf{B}^2)\mathbf{x} = 3 - 4 + 0 = -1 < 0 .$ So $\mathbf{A}^2 \not\succeq \mathbf{B}^2$ .

Where does the argument for numbers break? It uses $a^2 - b^2 = (a - b)(a + b)$ , a product of two nonnegative numbers. For matrices, $(\mathbf{A} - \mathbf{B})(\mathbf{A} + \mathbf{B}) = \mathbf{A}^2 + \mathbf{A}\mathbf{B} - \mathbf{B}\mathbf{A} - \mathbf{B}^2$ , which is $\mathbf{A}^2 - \mathbf{B}^2$ only when $\mathbf{A}$ and $\mathbf{B}$ commute — and in the example $\mathbf{A}\mathbf{B} = \begin{psmallmatrix} 3 & 3 \\ 2 & 2 \end{psmallmatrix}$ while $\mathbf{B}\mathbf{A} = \begin{psmallmatrix} 3 & 2 \\ 3 & 2\end{psmallmatrix}$ . Noncommutativity is the whole obstruction, and when it is removed the number argument goes through; that is exercise C1.

What is true is the same statement for the square root, and it is true with no commuting hypothesis at all.

Proposition 13.5.4 (The square root is order preserving)

Let $\mathbf{A}, \mathbf{B} \in M_n(F)$ with $\mathbf{A} \succeq \mathbf{B} \succeq \mathbf{O}$ . Then $\mathbf{A}^{1/2} \succeq \mathbf{B}^{1/2}$ .

Idea. Suppose not. Then $\mathbf{X} - \mathbf{Y}$ , with $\mathbf{X} = \mathbf{A}^{1/2}$ and $\mathbf{Y} = \mathbf{B}^{1/2}$ , has a negative eigenvalue $\mu$ , with unit eigenvector $\mathbf{u}$ . Two different consequences of $\mathbf{X}\mathbf{u} = \mathbf{Y}\mathbf{u} + \mu\mathbf{u}$ then collide. Pairing with $\mathbf{u}$ says $\langle \mathbf{Y}\mathbf{u}, \mathbf{u} \rangle \ge \lvert\mu\rvert$ , because $\langle \mathbf{X}\mathbf{u}, \mathbf{u} \rangle \ge 0$ . Taking norms and using $\left\| \mathbf{X}\mathbf{u} \right\|^2 \ge \left\| \mathbf{Y}\mathbf{u} \right\|^2$ , which is exactly the hypothesis $\mathbf{A} \succeq \mathbf{B}$ , says $\lvert\mu\rvert \ge 2\langle \mathbf{Y}\mathbf{u}, \mathbf{u} \rangle$ . Together: $\lvert\mu\rvert \ge 2\lvert\mu\rvert$ .

Proof. Write $\mathbf{X} = \mathbf{A}^{1/2}$ and $\mathbf{Y} = \mathbf{B}^{1/2}$ , both Hermitian and $\succeq \mathbf{O}$ with $\mathbf{X}^2 = \mathbf{A}$ and $\mathbf{Y}^2 = \mathbf{B}$ (Theorem 13.2.1). Then $\mathbf{X} - \mathbf{Y}$ is Hermitian. Suppose, for a contradiction, that $\mathbf{X} - \mathbf{Y} \not\succeq \mathbf{O}$ . By Theorem 13.1.4 it has a negative eigenvalue $\mu < 0$ ; choose a unit eigenvector $\mathbf{u}$ , so that $\mathbf{X}\mathbf{u} = \mathbf{Y}\mathbf{u} + \mu\mathbf{u} .$

Pairing this with $\mathbf{u}$ and using $\left\| \mathbf{u} \right\| = 1$ , $\langle \mathbf{X}\mathbf{u}, \mathbf{u} \rangle = \langle \mathbf{Y}\mathbf{u}, \mathbf{u} \rangle + \mu .$ All three numbers are real (Proposition 11.6.7), and $\langle \mathbf{X}\mathbf{u}, \mathbf{u} \rangle \ge 0$ because $\mathbf{X} \succeq \mathbf{O}$ . Therefore $\langle \mathbf{Y}\mathbf{u}, \mathbf{u} \rangle \ \ge\ -\mu = \lvert\mu\rvert . \tag{$\ast$}$

Now take squared norms. Since $\mathbf{X}$ is Hermitian, $\left\| \mathbf{X}\mathbf{u} \right\|^2 = \langle \mathbf{X}^2\mathbf{u}, \mathbf{u} \rangle = \langle \mathbf{A}\mathbf{u}, \mathbf{u} \rangle$ , and likewise $\left\| \mathbf{Y}\mathbf{u} \right\|^2 = \langle \mathbf{B}\mathbf{u}, \mathbf{u} \rangle$ ; so $\mathbf{A} \succeq \mathbf{B}$ gives $\left\| \mathbf{X}\mathbf{u} \right\|^2 \ge \left\| \mathbf{Y}\mathbf{u} \right\|^2$ . On the other hand, expanding $\mathbf{X}\mathbf{u} = \mathbf{Y}\mathbf{u} + \mu\mathbf{u}$ , $\left\| \mathbf{X}\mathbf{u} \right\|^2 = \left\| \mathbf{Y}\mathbf{u} \right\|^2 + 2\mu\langle \mathbf{Y}\mathbf{u}, \mathbf{u} \rangle + \mu^2 ,$ where the cross term is $2\mu\langle \mathbf{Y}\mathbf{u}, \mathbf{u} \rangle$ because $\mu$ and $\langle \mathbf{Y}\mathbf{u}, \mathbf{u} \rangle$ are real. Comparing the two, $2\mu\langle \mathbf{Y}\mathbf{u}, \mathbf{u} \rangle + \mu^2 \ \ge\ 0 .$ Substituting $\mu = -\lvert\mu\rvert$ turns this into $\lvert\mu\rvert^2 \ge 2\lvert\mu\rvert\langle \mathbf{Y}\mathbf{u}, \mathbf{u} \rangle$ , and dividing by $\lvert\mu\rvert > 0$ gives $\lvert\mu\rvert \ \ge\ 2\langle \mathbf{Y}\mathbf{u}, \mathbf{u} \rangle . \tag{$\ast\ast$}$ Combining $(\ast)$ and $(\ast\ast)$ , $\lvert\mu\rvert \ge 2\lvert\mu\rvert$ , so $\lvert\mu\rvert \le 0$ and $\mu = 0$ , contradicting $\mu < 0$ . Hence $\mathbf{X} - \mathbf{Y} \succeq \mathbf{O}$ , which is the assertion.

So among the powers $t \mapsto t^{s}$ , the exponent $s = 1/2$ preserves the Loewner order while $s = 2$ does not. The dividing line is at $s = 1$ : the functions preserving the order are called operator monotone, and $t \mapsto t^{s}$ is operator monotone on $[0, \infty)$ exactly for $0 \le s \le 1$ . Chapter 21 proves this, along with the companion fact that $t \mapsto \log t$ is operator monotone while $t \mapsto e^{t}$ is not.

13.5.4. Exercises

A. Check your understanding

Exercise (A1)

(a) Define $\mathbf{A} \succeq \mathbf{B}$ , and say on which set of matrices the relation is defined.
(b) Determine whether the following statement is correct, and justify your answer: for Hermitian $\mathbf{A}, \mathbf{B}$ , either $\mathbf{A} \succeq \mathbf{B}$ or $\mathbf{B} \succeq \mathbf{A}$ .
(c) Determine whether the following statement is correct, and justify your answer: $\mathbf{A} \succeq \mathbf{B}$ implies $a_{ij} \ge b_{ij}$ for all $i, j$ .
(d) State what fails when both sides of $\mathbf{A} \succeq \mathbf{B} \succeq \mathbf{O}$ are squared, and what happens instead when both sides have their square roots taken.
(e) Which clause of Theorem 13.5.3 reverses an inequality, and what is the analogous fact about real numbers?

Solution.

(a) For Hermitian $\mathbf{A}, \mathbf{B} \in M_n(F)$ , $\mathbf{A} \succeq \mathbf{B}$ means $\mathbf{A} - \mathbf{B} \succeq \mathbf{O}$ , equivalently $\langle \mathbf{A}\mathbf{x}, \mathbf{x} \rangle \ge \langle \mathbf{B}\mathbf{x}, \mathbf{x} \rangle$ for every $\mathbf{x} \in F^n$ (Definition 13.5.1). It is defined on the Hermitian matrices of a fixed size.
(b) Incorrect: the order is partial. $\operatorname{diag}(1,0)$ and $\operatorname{diag}(0,1)$ are incomparable, since $\operatorname{diag}(1,-1)$ and $\operatorname{diag}(-1,1)$ are both indefinite.
(c) Incorrect. $\operatorname{diag}(2,2) \succeq \begin{psmallmatrix}1&1\\mathbf{1}&1\end{psmallmatrix}$ , because the difference $\begin{psmallmatrix}1&-1\\-1&1\end{psmallmatrix}$ has eigenvalues $0$ and $2$ ; yet the $(1,2)$ -entries are $0$ and $1$ .
(d) Squaring fails: $\mathbf{A} \succeq \mathbf{B} \succeq \mathbf{O}$ does not give $\mathbf{A}^2 \succeq \mathbf{B}^2$ (Example (Squaring does not preserve the Loewner order)). Taking square roots is legitimate: $\mathbf{A}^{1/2} \succeq \mathbf{B}^{1/2}$ (Proposition 13.5.4).
(e) Clause (d): $\mathbf{A} \succeq \mathbf{B} \succ \mathbf{O}$ gives $\mathbf{B}^{-1} \succeq \mathbf{A}^{-1}$ . For real numbers, $a \ge b > 0$ gives $1/b \ge 1/a$ .

B. Practice

Exercise (B1: Decide the relation)

For each pair of real symmetric matrices, determine whether $\mathbf{A} \succeq \mathbf{B}$ , whether $\mathbf{B} \succeq \mathbf{A}$ , or neither. Justify your answer.

(i) $\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 1 & 3\end{pmatrix}$ , $\mathbf{B} = \mathbf{I}_2$ .
(ii) $\mathbf{A} = \begin{pmatrix} 2 & 0 \\ 0 & 1\end{pmatrix}$ , $\mathbf{B} = \begin{pmatrix} 1 & 1 \\ 1 & 1\end{pmatrix}$ .
(iii) $\mathbf{A} = \begin{pmatrix} 1 & 0 \\ 0 & 3\end{pmatrix}$ , $\mathbf{B} = \begin{pmatrix} 2 & 0 \\ 0 & 1\end{pmatrix}$ .

Solution.

(i) $\mathbf{A} - \mathbf{B} = \begin{psmallmatrix} 2 & 1 \\ 1 & 2\end{psmallmatrix}$ , with leading principal minors $2$ and $3$ , both $> 0$ , so $\mathbf{A} - \mathbf{B} \succ \mathbf{O}$ by Theorem 13.1.5 and $\mathbf{A} \succ \mathbf{B}$ . Then $\mathbf{B} \succeq \mathbf{A}$ would force $\mathbf{A} = \mathbf{B}$ by Proposition 13.5.1, which is false, so $\mathbf{B} \not\succeq \mathbf{A}$ .
(ii) $\mathbf{A} - \mathbf{B} = \begin{psmallmatrix} 1 & -1 \\ -1 & 0\end{psmallmatrix}$ , with determinant $-1 < 0$ , so it is indefinite and $\mathbf{A} \not\succeq \mathbf{B}$ . Also $\mathbf{B} - \mathbf{A}$ has determinant $-1$ , so $\mathbf{B} \not\succeq \mathbf{A}$ . Neither.
(iii) $\mathbf{A} - \mathbf{B} = \operatorname{diag}(-1, 2)$ , indefinite; $\mathbf{B} - \mathbf{A} = \operatorname{diag}(1, -2)$ , indefinite. Neither. (For diagonal matrices the comparison is entrywise, and the two entries disagree.)

Exercise (B2: The order reverses)

Let $\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 1 & 3\end{pmatrix}$ and $\mathbf{B} = \begin{pmatrix} 2 & 1 \\ 1 & 2\end{pmatrix}$ . Verify that $\mathbf{A} \succeq \mathbf{B} \succ \mathbf{O}$ , compute $\mathbf{A}^{-1}$ and $\mathbf{B}^{-1}$ , and check Theorem 13.5.3 (d) directly. Check the determinant clause as well.

Solution. $\mathbf{A} - \mathbf{B} = \mathbf{I}_2 \succ \mathbf{O}$ , so $\mathbf{A} \succ \mathbf{B}$ . And $\mathbf{B} \succ \mathbf{O}$ , its leading principal minors being $2$ and $3$ . Then $\det\mathbf{A} = 8$ , $\det\mathbf{B} = 3$ , so $\mathbf{A}^{-1} = \frac{1}{8}\begin{pmatrix} 3 & -1 \\ -1 & 3 \end{pmatrix}, \qquad \mathbf{B}^{-1} = \frac{1}{3}\begin{pmatrix} 2 & -1 \\ -1 & 2 \end{pmatrix},$ and $\mathbf{B}^{-1} - \mathbf{A}^{-1} = \frac{1}{24}\begin{pmatrix} 16 - 9 & -8 + 3 \\ -8 + 3 & 16 - 9\end{pmatrix} = \frac{1}{24}\begin{pmatrix} 7 & -5 \\ -5 & 7\end{pmatrix} .$ Its leading principal minors are $7/24 > 0$ and $(49 - 25)/576 = 1/24 > 0$ , so $\mathbf{B}^{-1} \succ \mathbf{A}^{-1}$ , as Theorem 13.5.3 (d) predicts. For the determinant clause, $\det\mathbf{A} = 8 \ge 3 = \det\mathbf{B}$ ; and for the trace, $6 \ge 4$ .

Exercise (B3: Another failure of squaring)

Let $\mathbf{A} = \begin{pmatrix} 3 & 1 \\ 1 & 1\end{pmatrix}$ and $\mathbf{B} = \begin{pmatrix} 1 & 1 \\ 1 & 1\end{pmatrix}$ . Verify that $\mathbf{A} \succeq \mathbf{B} \succeq \mathbf{O}$ , and exhibit a vector $\mathbf{x}$ with $\mathbf{x}^{\top}(\mathbf{A}^2 - \mathbf{B}^2)\mathbf{x} < 0$ .

Solution. $\mathbf{A} - \mathbf{B} = \operatorname{diag}(2, 0) \succeq \mathbf{O}$ , so $\mathbf{A} \succeq \mathbf{B}$ ; and $\mathbf{x}^{\top}\mathbf{B}\mathbf{x} = (x_1 + x_2)^2 \ge 0$ , so $\mathbf{B} \succeq \mathbf{O}$ . Squaring, $\mathbf{A}^2 = \begin{pmatrix} 10 & 4 \\ 4 & 2 \end{pmatrix}, \qquad \mathbf{B}^2 = \begin{pmatrix} 2 & 2 \\ 2 & 2 \end{pmatrix}, \qquad \mathbf{A}^2 - \mathbf{B}^2 = \begin{pmatrix} 8 & 2 \\ 2 & 0 \end{pmatrix},$ whose determinant is $-4 < 0$ . With $\mathbf{x} = (1, -3)$ , $\mathbf{x}^{\top}(\mathbf{A}^2 - \mathbf{B}^2)\mathbf{x} = 8 - 12 + 0 = -4 < 0 .$ So $\mathbf{A}^2 \not\succeq \mathbf{B}^2$ . (Any $\mathbf{x} = (1, t)$ with $t < -2$ works, since the value is $8 + 4t$ .)

C. Going deeper

Exercise (C1: When squaring does work)

Let $\mathbf{A}, \mathbf{B} \in M_n(F)$ with $\mathbf{A} \succeq \mathbf{B} \succeq \mathbf{O}$ and $\mathbf{A}\mathbf{B} = \mathbf{B}\mathbf{A}$ .

(a) Prove that if $\mathbf{P}, \mathbf{Q} \succeq \mathbf{O}$ and $\mathbf{P}\mathbf{Q} = \mathbf{Q}\mathbf{P}$ , then $\mathbf{P}\mathbf{Q} \succeq \mathbf{O}$ .
(b) Deduce that $\mathbf{A}^2 \succeq \mathbf{B}^2$ .
(c) Deduce that $\mathbf{A}^k \succeq \mathbf{B}^k$ for every integer $k \ge 1$ .

Hint for (a): $\mathbf{P}^{1/2}$ is a polynomial in $\mathbf{P}$ .

Solution.

(a) By Theorem 13.2.1, $\mathbf{P}^{1/2}$ is a polynomial in $\mathbf{P}$ , so it commutes with every matrix that commutes with $\mathbf{P}$ ; in particular $\mathbf{P}^{1/2}\mathbf{Q} = \mathbf{Q}\mathbf{P}^{1/2}$ . Hence $\mathbf{P}\mathbf{Q} = \mathbf{P}^{1/2}\mathbf{P}^{1/2}\mathbf{Q} = \mathbf{P}^{1/2}\mathbf{Q}\mathbf{P}^{1/2} = (\mathbf{P}^{1/2})^{*}\mathbf{Q}\mathbf{P}^{1/2} ,$ which is $\succeq \mathbf{O}$ by Proposition 13.1.2 (a), since $\mathbf{Q} \succeq \mathbf{O}$ and $\mathbf{P}^{1/2}$ is Hermitian.
(b) Put $\mathbf{P} = \mathbf{A} - \mathbf{B}$ and $\mathbf{Q} = \mathbf{A} + \mathbf{B}$ . Both are $\succeq \mathbf{O}$ : the first by hypothesis, the second as a sum of two matrices $\succeq \mathbf{O}$ , using $\mathbf{A} \succeq \mathbf{B} \succeq \mathbf{O}$ and Proposition 13.5.1. They commute, because $\mathbf{A}\mathbf{B} = \mathbf{B}\mathbf{A}$ makes $\mathbf{P}\mathbf{Q} = \mathbf{A}^2 + \mathbf{A}\mathbf{B} - \mathbf{B}\mathbf{A} - \mathbf{B}^2 = \mathbf{A}^2 - \mathbf{B}^2$ and $\mathbf{Q}\mathbf{P} = \mathbf{A}^2 - \mathbf{A}\mathbf{B} + \mathbf{B}\mathbf{A} - \mathbf{B}^2 = \mathbf{A}^2 - \mathbf{B}^2$ as well. By (a), $\mathbf{A}^2 - \mathbf{B}^2 = \mathbf{P}\mathbf{Q} \succeq \mathbf{O}$ .
(c) Induction on $k$ . The case $k = 1$ is the hypothesis. Assume $\mathbf{A}^{k-1} \succeq \mathbf{B}^{k-1}$ . Then $\mathbf{A}^k - \mathbf{B}^k = \mathbf{A}^{k-1}(\mathbf{A} - \mathbf{B}) + (\mathbf{A}^{k-1} - \mathbf{B}^{k-1})\mathbf{B} ,$ and each term is $\succeq \mathbf{O}$ by (a). Indeed $\mathbf{A} \succeq \mathbf{O}$ is Hermitian with eigenvalues $\ge 0$ , so $\mathbf{A}^{k-1}$ is Hermitian with eigenvalues $\ge 0$ and hence $\mathbf{A}^{k-1} \succeq \mathbf{O}$ (Theorem 13.1.4); it commutes with $\mathbf{A} - \mathbf{B}$ , since $\mathbf{A}\mathbf{B} = \mathbf{B}\mathbf{A}$ . Likewise $\mathbf{A}^{k-1} - \mathbf{B}^{k-1} \succeq \mathbf{O}$ by the induction hypothesis and it commutes with $\mathbf{B} \succeq \mathbf{O}$ . Finally a sum of two matrices $\succeq \mathbf{O}$ is $\succeq \mathbf{O}$ , directly from Definition 13.1.1.

Exercise (C2: Kernels and ranks)

Let $\mathbf{A}, \mathbf{B} \in M_n(F)$ with $\mathbf{A} \succeq \mathbf{B} \succeq \mathbf{O}$ .

(a) Prove that $\operatorname{nul}(\mathbf{A}) \subseteq \operatorname{nul}(\mathbf{B})$ .
(b) Deduce that $\operatorname{rank}\mathbf{A} \ge \operatorname{rank}\mathbf{B}$ .
(c) Give an example with $\mathbf{A} \succeq \mathbf{B} \succeq \mathbf{O}$ and $\operatorname{rank}\mathbf{A} = \operatorname{rank}\mathbf{B}$ but $\mathbf{A} \ne \mathbf{B}$ .

Hint for (a): pair $\mathbf{A}\mathbf{x} = \mathbf{0}$ with $\mathbf{x}$ , then use $\mathbf{B}^{1/2}$ .

Solution.

(a) Let $\mathbf{x} \in \operatorname{nul}(\mathbf{A})$ . Then $\langle \mathbf{A}\mathbf{x}, \mathbf{x} \rangle = 0$ , and $0 \le \langle \mathbf{B}\mathbf{x}, \mathbf{x} \rangle \le \langle \mathbf{A}\mathbf{x}, \mathbf{x} \rangle = 0$ , so $\langle \mathbf{B}\mathbf{x}, \mathbf{x} \rangle = 0$ . Writing $\mathbf{B} = \mathbf{B}^{1/2}\mathbf{B}^{1/2}$ with $\mathbf{B}^{1/2}$ Hermitian (Theorem 13.2.1), $0 = \langle \mathbf{B}\mathbf{x}, \mathbf{x} \rangle = \langle \mathbf{B}^{1/2}\mathbf{x}, \mathbf{B}^{1/2}\mathbf{x} \rangle = \left\| \mathbf{B}^{1/2}\mathbf{x} \right\|^2 ,$ so $\mathbf{B}^{1/2}\mathbf{x} = \mathbf{0}$ and hence $\mathbf{B}\mathbf{x} = \mathbf{B}^{1/2}(\mathbf{B}^{1/2}\mathbf{x}) = \mathbf{0}$ .
(b) By (a), $\operatorname{nullity}\mathbf{A} \le \operatorname{nullity}\mathbf{B}$ , and by Rank–Nullity (Theorem 2.3.1) applied to both matrices on $F^n$ , $\operatorname{rank}\mathbf{A} = n - \operatorname{nullity}\mathbf{A} \ge n - \operatorname{nullity}\mathbf{B} = \operatorname{rank}\mathbf{B}$ .
(c) $\mathbf{A} = \operatorname{diag}(2, 2)$ and $\mathbf{B} = \operatorname{diag}(1, 1)$ : both have rank $2$ , $\mathbf{A} - \mathbf{B} = \mathbf{I}_2 \succeq \mathbf{O}$ , and $\mathbf{A} \ne \mathbf{B}$ .

Exercise (C3: Largest eigenvalue of a sum)

Let $\mathbf{A}, \mathbf{B} \in M_n(F)$ be Hermitian.

(a) Prove that $\lambda_1(\mathbf{A})\mathbf{I} \succeq \mathbf{A}$ , and deduce $\lambda_1(\mathbf{A} + \mathbf{B}) \le \lambda_1(\mathbf{A}) + \lambda_1(\mathbf{B})$ .
(b) Deduce that if $\mathbf{A} \succeq \mathbf{B}$ , then $0 \le \lambda_1(\mathbf{A}) - \lambda_1(\mathbf{B}) \le \lambda_1(\mathbf{A} - \mathbf{B})$ .

Solution.

(a) The matrix $\lambda_1(\mathbf{A})\mathbf{I} - \mathbf{A}$ is Hermitian with eigenvalues $\lambda_1(\mathbf{A}) - \lambda_i(\mathbf{A}) \ge 0$ , so it is $\succeq \mathbf{O}$ by Theorem 13.1.4; that is $\lambda_1(\mathbf{A})\mathbf{I} \succeq \mathbf{A}$ . Adding the same inequality for $\mathbf{B}$ , and using that a sum of two matrices $\succeq \mathbf{O}$ is again $\succeq \mathbf{O}$ (Exercise (C2: The cone of positive matrices) (a)), $\big(\lambda_1(\mathbf{A}) + \lambda_1(\mathbf{B})\big)\mathbf{I} \ \succeq\ \mathbf{A} + \mathbf{B} .$ By Theorem 13.5.3 (b) applied to this pair, $\lambda_1(\mathbf{A}) + \lambda_1(\mathbf{B}) = \lambda_1\big((\lambda_1(\mathbf{A})+\lambda_1(\mathbf{B}))\mathbf{I}\big) \ge \lambda_1(\mathbf{A} + \mathbf{B})$ .
(b) Write $\mathbf{A} = \mathbf{B} + (\mathbf{A} - \mathbf{B})$ . Applying (a) to the pair $\mathbf{B}$ and $\mathbf{A} - \mathbf{B}$ , $\lambda_1(\mathbf{A}) \le \lambda_1(\mathbf{B}) + \lambda_1(\mathbf{A} - \mathbf{B}) ,$ so $\lambda_1(\mathbf{A}) - \lambda_1(\mathbf{B}) \le \lambda_1(\mathbf{A} - \mathbf{B})$ . For the lower bound, $\mathbf{A} \succeq \mathbf{B}$ gives $\lambda_1(\mathbf{A}) \ge \lambda_1(\mathbf{B})$ by Theorem 13.5.3 (b). Note that (a) itself used no order hypothesis at all; only the lower bound needs $\mathbf{A} \succeq \mathbf{B}$ .