Course 4 Book 4A: Function Spaces and Sobolev SpacesChapter 4

Hilbert Spaces and Lax–Milgram

Projection, Riesz representation, orthonormal bases, and the existence theorem for weak solutions.

24 min read · Updated Oct 2, 2026

Read with Brezis, chapter 5, "Hilbert Spaces" (projection onto closed convex sets, the dual of a Hilbert space, Stampacchia and Lax–Milgram, Hilbert sums and orthonormal bases), or Kreyszig chapter 3 (§3.1–3.10), which is gentler and has more examples, including Legendre polynomials. Pick one, not both.

In this chapter · 8 sections

4.1Least squares and the orbit of Ceres
4.2Inner product spaces
4.3Projection
4.4Riesz representation
4.5Orthonormal bases
4.6Lax–Milgram
4.7History
4.8Exercises

A Hilbert space is a Banach space whose norm comes from an inner product, so that angles, orthogonality and projections make sense. That one extra structure makes Hilbert spaces the most tractable infinite-dimensional spaces by far. Every closed convex set has a nearest point to any given point. Every bounded linear functional is an inner product with a fixed vector (Riesz representation), so the space is its own dual. Every separable Hilbert space has an orthonormal basis and is, after a choice of basis, the sequence space $\ell^2$ . The Fourier series of 2B.7 Fourier Series and the First Heat Equation and 3A.7 Lᵖ Spaces and Jensen’s Inequality are one instance.

The chapter's last theorem, Lax–Milgram, is where Hilbert spaces meet PDE. It says that a "coercive" bilinear form represents every functional, and that is precisely the statement that weak solutions of elliptic equations exist (6A.5 Weak Solutions and Elliptic Regularity). The finite element method, the main tool of computational engineering, is Lax–Milgram restricted to a finite-dimensional subspace, and its error estimate is a projection theorem.

By the end of this chapter you will be able to:

use the parallelogram law to recognise inner product norms, and prove the projection theorem onto closed convex sets;
decompose a Hilbert space into a closed subspace and its orthogonal complement, and solve least-squares problems as projections;
prove the Riesz representation theorem;
work with orthonormal bases: Bessel, Parseval, Gram–Schmidt and Legendre polynomials;
prove the Lax–Milgram theorem and write a two-point boundary value problem in weak form.

Least squares and the orbit of Ceres

In the world Data The asteroid that was lost and found

On 1 January 1801 Giuseppe Piazzi discovered a faint moving object, later named Ceres, and followed it for about six weeks, through an arc of only a few degrees of its orbit, before it was lost in the glare of the Sun. To find it again, its orbit had to be computed from a handful of imprecise observations. Carl Friedrich Gauss, then aged 24, devised methods that did this, and using his predicted positions Franz Xaver von Zach found Ceres again on 31 December 1801, with Heinrich Olbers confirming it the next night. Gauss published his methods of orbit determination in Theoria motus corporum coelestium (1809), including the method of least squares, which he said he had used since 1795; Adrien-Marie Legendre had published it first, in 1805. Historians have debated how much least squares in the modern sense entered the 1801 computation itself.

The method is a projection. To fit a model with parameters $x \in \mathbb{R}^n$ to $m > n$ measurements $b \in \mathbb{R}^m$ , where the model predicts $Ax$ , choose $x$ to minimise $\|Ax - b\|^2$ , the sum of squared residuals. The minimising $Ax$ is the orthogonal projection of $b$ onto the column space of $A$ : the residual $b - Ax$ is perpendicular to every column, which is the system of normal equations $A^{\mathsf T}Ax = A^{\mathsf T}b$ (Figure 4.1). Fitting a straight line to measurements, the commonest calculation in experimental science, is the case $n = 2$ . Gauss also showed that, when the measurement errors are independent and normally distributed, the least-squares estimate is the most probable one, which is why the bell curve is often called Gaussian.

Figure 4.1. Left: least squares as projection. The best approximation to

b

from a subspace is its orthogonal projection

Pb

, and the residual

b - Pb

is perpendicular to the subspace. Right: a least-squares line through illustrative data, with the residuals whose squares it minimises.

Inner product spaces

Definition 4.1 Inner product, Hilbert space

An inner product on a real (or complex) vector space $H$ is a map $\langle\cdot,\cdot\rangle : H \times H \to \mathbb{R}$ (or $\mathbb{C}$ ) that is linear in the first slot, symmetric (conjugate-symmetric), and positive: $\langle x, x\rangle > 0$ for $x \neq 0$ . It defines a norm $\|x\| = \langle x, x\rangle^{1/2}$ . A Hilbert space is an inner product space that is complete in this norm.

The examples: $\mathbb{R}^n$ and $\mathbb{C}^n$ ; $\ell^2$ with $\langle x, y\rangle = \sum x_k\bar y_k$ ; $L^2(X, \mu)$ with $\langle f, g\rangle = \int f\bar g\,d\mu$ (3A.7 Lᵖ Spaces and Jensen’s Inequality); and, from 4A.9 Sobolev Spaces, the Sobolev space $H^1$ with $\langle u, v\rangle = \int(uv + \nabla u\cdot\nabla v)$ . The Cauchy–Schwarz inequality $|\langle x, y\rangle| \leq \|x\|\,\|y\|$ holds, with the proof of 2B.1 Metric Spaces, and gives the triangle inequality.

Which norms come from inner products? Exactly those satisfying the parallelogram law

\|x + y\|^2 + \|x - y\|^2 = 2\|x\|^2 + 2\|y\|^2,

the sum of the squares of the diagonals of a parallelogram equals the sum of the squares of its sides (Jordan and von Neumann, 1935). The inner product is then recovered by polarisation, $\langle x, y\rangle = \tfrac14(\|x + y\|^2 - \|x - y\|^2)$ in the real case. The $L^p$ norms satisfy the law only for $p = 2$ (Exercise 4.7), which is why $L^2$ is special among them.

Projection

The parallelogram law is what makes nearest points exist.

Theorem 4.2 Projection onto a closed convex set

Let $K$ be a non-empty closed convex subset of a Hilbert space $H$ and $x \in H$ . There is a unique $p \in K$ with $\|x - p\| = \inf_{y\in K}\|x - y\|$ . It is characterised by

\langle x - p, y - p\rangle \leq 0 \quad \text{for all } y \in K.

Proof. Existence. Let $d$ be the infimum and $y_n \in K$ with $\|x - y_n\| \to d$ . The parallelogram law applied to $x - y_n$ and $x - y_m$ gives

\|y_n - y_m\|^2 = 2\|x - y_n\|^2 + 2\|x - y_m\|^2 - 4\Big\|x - \frac{y_n + y_m}{2}\Big\|^2 \leq 2\|x - y_n\|^2 + 2\|x - y_m\|^2 - 4d^2,

since $\frac{y_n + y_m}2 \in K$ by convexity. The right side tends to $0$ , so $(y_n)$ is Cauchy. By completeness it converges, to some $p \in K$ ( $K$ is closed), and $\|x - p\| = d$ .

Uniqueness is the same computation with two minimisers in place of $y_n$ and $y_m$ .

Characterisation. For $y \in K$ and $0 < t \leq 1$ , $p + t(y - p) \in K$ , so $\|x - p - t(y - p)\|^2 \geq \|x - p\|^2$ ; expanding and letting $t \to 0$ gives $\langle x - p, y - p\rangle \leq 0$ . Conversely, the inequality gives $\|x - y\|^2 = \|x - p\|^2 - 2\langle x - p, y - p\rangle + \|y - p\|^2 \geq \|x - p\|^2$ .

Compare 2B.3 Compactness: there, a nearest point existed because closed bounded sets in $\mathbb{R}^n$ are compact. Here there is no compactness, and the parallelogram law, through completeness, does the job instead. In a general Banach space nearest points need not exist (Exercise 4.8).

For a closed subspace $M$ , the characterisation becomes $\langle x - p, m\rangle = 0$ for all $m \in M$ : the residual is orthogonal to $M$ . The map $x \mapsto p = P_Mx$ is linear, of norm $1$ (if $M \neq 0$ ), and every $x$ splits uniquely as

x = P_Mx + (x - P_Mx) \in M \oplus M^\perp,

where $M^\perp = \{z : \langle z, m\rangle = 0 \text{ for all } m \in M\}$ is the orthogonal complement. Least squares is the case $M$ = the column space of $A$ .

Riesz representation

Theorem 4.3 Riesz representation theorem

For every bounded linear functional $\phi$ on a Hilbert space $H$ there is a unique $y \in H$ with $\phi(x) = \langle x, y\rangle$ for all $x$ , and $\|\phi\| = \|y\|$ .

Proof. If $\phi = 0$ take $y = 0$ . Otherwise the kernel $M = \ker\phi$ is a closed proper subspace, so $M^\perp$ contains a unit vector $z$ . For any $x$ , the vector $\phi(x)z - \phi(z)x$ lies in $M$ , hence is orthogonal to $z$ : $\phi(x) - \phi(z)\langle x, z\rangle = 0$ . So $\phi(x) = \langle x, \overline{\phi(z)}z\rangle$ , and $y = \overline{\phi(z)}z$ works. Uniqueness: if $\langle x, y - y'\rangle = 0$ for all $x$ , take $x = y - y'$ . The norm identity is Cauchy–Schwarz with equality at $x = y$ .

So a Hilbert space is its own dual, and every Hilbert space is reflexive (4A.3 Hahn–Banach and Duality). The theorem turns abstract existence questions into concrete ones: to find a vector, it is enough to exhibit a bounded functional it should represent.

Orthonormal bases

An orthonormal set $\{e_\alpha\}$ consists of unit vectors that are pairwise orthogonal. For finitely many $e_1, \ldots, e_n$ , the projection onto their span is $\sum\langle x, e_k\rangle e_k$ , and Pythagoras gives Bessel's inequality $\sum_k|\langle x, e_k\rangle|^2 \leq \|x\|^2$ , for any orthonormal sequence.

Theorem 4.4 Orthonormal bases

For an orthonormal sequence $(e_k)$ in a Hilbert space $H$ , the following are equivalent:

the span of the $e_k$ is dense in $H$ ;
$x = \sum_k\langle x, e_k\rangle e_k$ for every $x$ , the series converging in norm;
(Parseval) $\|x\|^2 = \sum_k|\langle x, e_k\rangle|^2$ for every $x$ ;
if $\langle x, e_k\rangle = 0$ for every $k$ , then $x = 0$ .

Such a sequence is an orthonormal basis.

The proof is 2B.7 Fourier Series and the First Heat Equation's argument for Fourier series, done abstractly (Exercise 4.10). Every separable Hilbert space has an orthonormal basis (apply Gram–Schmidt to a dense sequence), and the map $x \mapsto (\langle x, e_k\rangle)_k$ is an isometry onto $\ell^2$ . So all infinite-dimensional separable Hilbert spaces are, abstractly, the same space, $\ell^2$ ; what differs is the concrete basis, and choosing a good one is half of applied analysis.

Gram–Schmidt turns linearly independent vectors $v_1, v_2, \ldots$ into an orthonormal sequence with the same nested spans: subtract from $v_n$ its projection onto the span of $e_1, \ldots, e_{n-1}$ , then normalise. Applied to $1, x, x^2, \ldots$ in $L^2(-1, 1)$ , it produces (up to scaling) the Legendre polynomials $1$ , $x$ , $\frac12(3x^2 - 1)$ , $\frac12(5x^3 - 3x)$ , … (Exercise 4.11). Expanding in them instead of in monomials cures the ill-conditioning of 4A.1 Banach Spaces and Bounded Operators's Hilbert matrix, because orthonormal coordinates are perfectly conditioned.

In the world In use Denoising by projection

A noisy measured signal is often modelled as a smooth signal plus rapidly fluctuating noise. Projecting onto the first $N$ Fourier modes, $P_Nf = \sum_{|k|\leq N}\hat f(k)e_k$ , keeps the smooth part (whose energy is in low frequencies, 2B.7 Fourier Series and the First Heat Equation) and discards most of the noise (whose energy is spread over all frequencies). Too few modes blur the signal; too many let the noise back in (Figure 4.2). Because $P_N$ is an orthogonal projection, the energy removed is exactly $\sum_{|k|>N}|\hat f(k)|^2$ , and the trade-off can be chosen by looking at how the coefficients decay. JPEG's discarding of high-frequency coefficients (2B.7 Fourier Series and the First Heat Equation) is the same idea with a different orthonormal basis.

Figure 4.2. A noisy signal (grey; the noise is computer-generated) and its orthogonal projections onto

5

20

and

100

Fourier modes. Five modes oversmooth; twenty capture the signal; a hundred bring the noise back.

In the world Model Quantum states are unit vectors

In quantum mechanics the state of a system is a unit vector $\psi$ in a Hilbert space, for a single particle in space $L^2(\mathbb{R}^3)$ . A measurement of a quantity with a discrete set of outcomes corresponds to an orthonormal basis $(e_k)$ , one vector per outcome, and the probability of outcome $k$ is $|\langle\psi, e_k\rangle|^2$ (the Born rule). Parseval's identity, $\sum_k|\langle\psi, e_k\rangle|^2 = \|\psi\|^2 = 1$ , is the statement that the probabilities add up to $1$ . Projection onto the subspace spanned by $e_k$ is what textbook quantum mechanics calls the collapse of the state after the measurement.

Lax–Milgram

Riesz represents a functional through the inner product. Lax–Milgram represents it through any bilinear form that behaves enough like one.

Theorem 4.5 Lax–Milgram

Let $H$ be a real Hilbert space and $a : H \times H \to \mathbb{R}$ a bilinear form that is

bounded: $|a(u, v)| \leq M\|u\|\,\|v\|$ , and
coercive: $a(u, u) \geq \alpha\|u\|^2$ for some $\alpha > 0$ .

Then for every $\phi \in H^*$ there is a unique $u \in H$ with $a(u, v) = \phi(v)$ for all $v \in H$ , and $\|u\| \leq \|\phi\|/\alpha$ .

Proof. For fixed $u$ , $v \mapsto a(u, v)$ is a bounded functional, so by Riesz there is $Au \in H$ with $a(u, v) = \langle Au, v\rangle$ ; $A$ is linear with $\|A\| \leq M$ . Similarly $\phi(v) = \langle f, v\rangle$ for some $f$ . We must solve $Au = f$ . Coercivity gives $\alpha\|u\|^2 \leq \langle Au, u\rangle \leq \|Au\|\,\|u\|$ , so $\|Au\| \geq \alpha\|u\|$ : $A$ is injective with closed range. Its range is also dense: if $z \perp \operatorname{ran}A$ , then $0 = \langle Az, z\rangle \geq \alpha\|z\|^2$ , so $z = 0$ . A closed dense subspace is everything, so $A$ is onto, and $\|u\| \leq \|f\|/\alpha$ .

When $a$ is also symmetric, $a$ is itself an inner product, equivalent to the original by (1) and (2), and Lax–Milgram is just Riesz representation in that inner product. In that case the solution $u$ is also the unique minimiser of the energy

E(v) = \tfrac12a(v, v) - \phi(v),

since $E(u + w) = E(u) + \tfrac12a(w, w) \geq E(u)$ . This is the Dirichlet principle: solving a symmetric elliptic equation is the same as minimising an energy (4A.6 Weak Convergence and the Direct Method, 6A.9 Calculus of Variations and Gradient Flows).

Example 4.6 A boundary value problem in weak form

Consider $-u'' + u = f$ on $(0, 1)$ with $u(0) = u(1) = 0$ . Multiply by a function $v$ vanishing at the ends and integrate by parts (2A.11 The Riemann Integral): a classical solution satisfies

\int_0^1(u'v' + uv)\,dx = \int_0^1fv\,dx \qquad\text{for all such } v.

Take $H = H^1_0(0, 1)$ , the completion of smooth functions vanishing near $0$ and $1$ in the norm $\|u\|^2 = \int(u'^2 + u^2)$ (made precise in 4A.9 Sobolev Spaces). The left side is $a(u, v) = \langle u, v\rangle_H$ , bounded and coercive with $M = \alpha = 1$ ; the right side is a bounded functional for $f \in L^2$ . Lax–Milgram gives a unique weak solution $u \in H$ , with $\|u\|_H \leq \|f\|_{L^2}$ , for every $f \in L^2$ , including discontinuous $f$ for which no classical solution exists. Showing that $u$ is in fact smooth when $f$ is, is a separate question, regularity (6A.5 Weak Solutions and Elliptic Regularity).

Where this goes From Lax–Milgram to finite elements and Perelman

Weak solutions (6A.5 Weak Solutions and Elliptic Regularity): every second-order elliptic equation $-\sum\partial_i(a_{ij}\partial_ju) + cu = f$ with uniformly elliptic coefficients (2B.3 Compactness's rehearsal) and suitable $c$ is solved this way, in $H^1_0$ . Finite elements (4A.9 Sobolev Spaces): replacing $H$ by a finite-dimensional subspace $V_h$ of piecewise polynomials gives a finite linear system; the discrete solution is the $a$ -orthogonal projection of $u$ onto $V_h$ , and Céa's lemma bounds the error by the best approximation from $V_h$ (Exercise 4.13). Perelman (12A.2 Ricci Flow as a Gradient Flow): the functional $\mathcal{F}$ , rewritten as $\int(4|\nabla w|^2 + Rw^2)\,dV$ (3A.4 Measures, Probability and Weights), is a quadratic form on the Hilbert space $H^1$ of the manifold, and its infimum over unit vectors is the bottom of the spectrum of $-4\Delta + R$ (4A.7 Compact Operators and Spectra).

History

David Hilbert's work on integral equations (1904–1910) introduced the space of square-summable sequences; Erhard Schmidt (1908) gave it its geometric language of orthogonality and projection, and the orthogonalisation process carries his name with Jørgen Gram's (1883). Frigyes Riesz and Maurice Fréchet proved the representation theorem independently in 1907, for $L^2$ . John von Neumann gave the abstract definition of a Hilbert space in 1929, for quantum mechanics, and with Pascual Jordan characterised inner product norms by the parallelogram law in 1935. Peter Lax and Arthur Milgram published their theorem in 1954. Legendre's method of least squares appeared in 1805, Gauss's in 1809.

Recall Where we stand

A Hilbert space is complete with an inner product; the parallelogram law characterises its norms. Every closed convex set has a unique nearest point, characterised by an obtuse-angle condition; closed subspaces have orthogonal complements and orthogonal projections, and least squares is a projection. Riesz identifies $H^*$ with $H$ . Orthonormal bases give Bessel, Parseval and the isometry with $\ell^2$ ; Gram–Schmidt builds them, and gives the Legendre polynomials. Lax–Milgram solves $a(u, v) = \phi(v)$ for bounded coercive forms, which is the existence theorem for weak solutions of elliptic equations. 4A.5 The Fourier Transform turns to the Fourier transform on $\mathbb{R}^n$ , a unitary map of $L^2$ that diagonalises derivatives.

Exercises

Exercise 4.7 Only

p = 2

Show that the parallelogram law fails for the $\ell^1$ and $\ell^\infty$ norms on $\mathbb{R}^2$ (try $x = (1, 0)$ , $y = (0, 1)$ ), and for the $L^p$ norm on $[0, 1]$ with $p \neq 2$ (try $x = 1_{[0,1/2]}$ , $y = 1_{[1/2,1]}$ ).

Solution

For $\ell^1$ : $\|x \pm y\|^2 = 4$ , so the left side is $8$ , the right side $4$ . For $\ell^\infty$ : left $2$ , right $4$ . For $L^p$ : $\|x \pm y\|_p = 1$ and $\|x\|_p = \|y\|_p = 2^{-1/p}$ , so $2 = 4\cdot2^{-2/p}$ , which holds only for $p = 2$ .

Exercise 4.8 No nearest point

In $C([0, 1])$ with the sup norm, let $K = \{f : \int_0^{1/2}f - \int_{1/2}^1f = 1\}$ , a closed affine subspace (hence convex). Show that $\inf_{f\in K}\|f\| = 1$ but no $f \in K$ has $\|f\| = 1$ . So the projection theorem needs the Hilbert structure.

Exercise 4.9 Least squares

Fit a line $y = c_0 + c_1t$ to the points $(0, 1), (1, 2), (2, 2), (3, 4)$ by solving the normal equations, and check that the residual vector is orthogonal to $(1, 1, 1, 1)$ and to $(0, 1, 2, 3)$ .

Solution

$A^{\mathsf T}A = \begin{pmatrix}4 & 6\\6 & 14\end{pmatrix}$ , $A^{\mathsf T}b = (9, 18)$ , so $c_0 = 0.9$ , $c_1 = 0.9$ . Residuals $(0.1, 0.2, -0.7, 0.4)$ sum to $0$ and $0\cdot0.1 + 1\cdot0.2 + 2\cdot(-0.7) + 3\cdot0.4 = 0$ .

Exercise 4.10 Orthonormal bases

Prove Theorem 4.4. ((1) ⇒ (2): the partial sums are the best approximations from the first $N$ vectors' span, as in 2B.7 Fourier Series and the First Heat Equation. (2) ⇒ (3): continuity of the norm. (3) ⇒ (4): immediate. (4) ⇒ (1): if the closure $M$ of the span were not everything, a non-zero vector of $M^\perp$ would contradict (4).)

Exercise 4.11 Legendre polynomials

Apply Gram–Schmidt to $1, x, x^2, x^3$ in $L^2(-1, 1)$ , and show the results are proportional to $1$ , $x$ , $3x^2 - 1$ , $5x^3 - 3x$ .

Exercise 4.12 Weak solutions

For $-(p(x)u')' + q(x)u = f$ on $(0, 1)$ , $u(0) = u(1) = 0$ , with $p \geq p_0 > 0$ and $q \geq 0$ bounded, write the weak formulation and show the form is bounded and coercive on $H^1_0(0, 1)$ , assuming the Poincaré inequality $\int u^2 \leq \frac{1}{\pi^2}\int u'^2$ for $u \in H^1_0(0, 1)$ (proved in 4A.9 Sobolev Spaces).

Solution

$a(u, v) = \int(pu'v' + quv)$ . Bounded: $|a(u, v)| \leq \max(\sup p, \sup q)\|u\|_H\|v\|_H$ . Coercive: $a(u, u) \geq p_0\int u'^2 \geq \frac{p_0}{1 + 1/\pi^2}\int(u'^2 + u^2)$ , using Poincaré to absorb $\int u^2$ .

Exercise 4.13 Rehearsal: Céa's lemma

In the setting of Lax–Milgram, let $V_h \subseteq H$ be a closed subspace (for instance, finite-dimensional) and $u_h \in V_h$ the solution of $a(u_h, v) = \phi(v)$ for all $v \in V_h$ . Show Galerkin orthogonality $a(u - u_h, v) = 0$ for $v \in V_h$ , and deduce

\|u - u_h\| \leq \frac{M}{\alpha}\inf_{v\in V_h}\|u - v\|.

The finite element method's accuracy is therefore governed by how well piecewise polynomials can approximate the true solution in the energy norm, which is an approximation question in Sobolev spaces (4A.9 Sobolev Spaces).

Solution

Subtract the two equations for $v \in V_h$ . Then for any $v \in V_h$ : $\alpha\|u - u_h\|^2 \leq a(u - u_h, u - u_h) = a(u - u_h, u - v) \leq M\|u - u_h\|\,\|u - v\|$ .

© 2026 NeckPinch (www.neckpinch.com). All content in the guidebook (text, mathematics, figures and exercises) is protected by copyright. All rights reserved. No part may be copied, republished or redistributed without written permission.

4.1Least squares and the orbit of Ceres

4.2Inner product spaces

4.3Projection

4.4Riesz representation

4.5Orthonormal bases

4.6Lax–Milgram

4.7History

4.8Exercises

Least squares and the orbit of Ceres

Inner product spaces

Projection

Riesz representation

Orthonormal bases

Lax–Milgram

History

Exercises