Entropy, Information and Diffusion

Perelman's 2002 preprint introduced an entropy for the Ricci flow, and explained that its monotonicity is "in a sense" the same as the monotonicity of entropy in statistical physics. This chapter is about the original: entropy along the heat equation. A density that diffuses becomes more spread out, and its Boltzmann–Shannon entropy $H = -\int u\log u$ increases, at a rate equal to its Fisher information $\int\frac{|\nabla u|^2}{u}$. How large the entropy can be for a given amount of Fisher information is the content of the logarithmic Sobolev inequality, and with the right scale parameter that inequality is literally the statement that Perelman's $\mathcal W$-entropy of flat space is non-negative. The chapter also proves the sharpest pointwise statement about positive solutions of the heat equation, the Li–Yau inequality, whose equality case is the heat kernel.

All three results share one pattern, which is the thread of the second half of this guide: a quantity that is monotone along a flow, with equality exactly for a special self-similar solution. Entropy and the Gaussian here; Huisken's density and self-shrinkers in 6A.8 Curve Shortening and the First Geometric Flows; and Perelman's $\mathcal W$ and reduced volume with shrinking solitons in 12A.3 The 𝓦-Entropy and 12A.5 Reduced Distance and Reduced Volume.

By the end of this chapter you will be able to:

• compute the entropy and Fisher information of a density, and prove that entropy increases along the heat equation at the rate of the Fisher information;
• state Gross's Gaussian log-Sobolev inequality, derive its Euclidean form with a scale parameter $\tau$, and recognise it as $\mathcal W \geq 0$ on flat space;
• explain why log-Sobolev is "Sobolev in infinite dimensions";
• state and check the Li–Yau differential Harnack inequality, and integrate it to compare values at different points and times;
• describe the "monotone quantity plus equality case" template.

10.1Milk in coffee

10.2Entropy and Fisher information

Let $u > 0$ be a probability density on $\mathbb{R}^n$ ($\int u = 1$) with enough decay for the integrals below. Its entropy and Fisher information are

For the heat kernel $\Phi(\cdot, t)$, a Gaussian of variance $2t$ in each coordinate (6A.3 The Heat Equation on ℝⁿ),

(Exercise 10.5). Among all densities with a given covariance, the Gaussian has the largest entropy (3A.7 Lᵖ Spaces and Jensen’s Inequality, by Jensen's inequality). Entropy can be any real number; it increases by $n\log\lambda$ when $u$ is spread out by a factor $\lambda$, because $H$ measures a logarithm of volume.

For the heat kernel this reads $\frac{d}{dt}\frac n2\log(4\pi et) = \frac{n}{2t}$, as it must. The Fisher information itself is also monotone: it decreases, since

(Exercise 10.6). So $H$ is increasing and concave in $t$. The computation behind this, a Bochner-type identity for $\log u$, is the one that, on a manifold, produces a Ricci curvature term; with $\operatorname{Ric} \geq 0$ the sign survives. This is the first sign of how curvature enters entropy, which is the heart of 12A.3 The 𝓦-Entropy.

10.3The logarithmic Sobolev inequality

Entropy production says $\frac{dH}{dt} = I$. To turn this into a statement about rates of mixing, one needs an inequality comparing entropy with Fisher information. It is cleanest relative to a Gaussian.

Two features make this inequality special. The constant $2$ does not depend on the dimension $n$, unlike the constant in the Sobolev inequality (4A.10 Sobolev Embeddings and Critical Exponents), which degenerates as $n \to \infty$; Gross's motivation was quantum field theory, where the dimension is infinite. And it controls $g^2\log g^2$, a quantity only "logarithmically" better than $g^2$, which is the best one can hope for in infinite dimensions. It is equivalent to hypercontractivity of the heat flow for the Gaussian measure (Edward Nelson, 1973): the Ornstein–Uhlenbeck semigroup maps $L^2(\gamma)$ into $L^p(\gamma)$ for some $p > 2$ after a positive time, with norm $1$.

Now rewrite it in the language of Perelman. Let $\tau > 0$, and write a probability density on $\mathbb{R}^n$ as

The reference density is the Gaussian of variance $2\tau$, which corresponds to $f = \frac{|x|^2}{4\tau}$: the heat kernel at time $\tau$, or the backward Gaussian of 6A.3 The Heat Equation on ℝⁿ's rehearsal.

10.4Nash's use of entropy

The log-Sobolev inequality and entropy production together control how fast a density spreads, and John Nash used this kind of argument in 1958 for a striking purpose: to prove that solutions of parabolic equations $u_t = \partial_j(a^{ij}\partial_iu)$ with merely bounded measurable coefficients are Hölder continuous (6A.6 Parabolic Regularity). Among his tools was the entropy of the fundamental solution: he showed that it increases at least like $\frac n2\log t$, just as for the heat kernel, using only the ellipticity bounds, and deduced that the fundamental solution spreads out at the parabolic rate and cannot concentrate. Entropy was a tool for regularity decades before Perelman used it for geometry, and the quantity $H(u) - \frac n2\log(4\pi et)$, entropy measured relative to the heat kernel, is now called the Nash entropy. It reappears in the work of Hein and Naber and of Bamler on Ricci flows (12C.5 After Perelman).

10.5The Li–Yau inequality

The Harnack inequality of 6A.2 Harmonic Functions compared a positive harmonic function at nearby points. For positive solutions of the heat equation, Peter Li and Shing-Tung Yau found in 1986 a sharp differential inequality, which integrates to a Harnack inequality across space and time.

The two forms agree because $\Delta\log u = \frac{\Delta u}{u} - \frac{|\nabla u|^2}{u^2} = \frac{u_t}{u} - \frac{|\nabla u|^2}{u^2}$. For the heat kernel, $\log\Phi = -\frac n2\log(4\pi t) - \frac{|x|^2}{4t}$, so $\Delta\log\Phi = -\frac{n}{2t}$ exactly. The proof applies the maximum principle to $F = t\big(|\nabla\log u|^2 - \partial_t\log u\big)$, using the Bochner identity of 6A.2 Harmonic Functions for $\log u$; on a manifold the Bochner formula contributes $\operatorname{Ric}(\nabla\log u, \nabla\log u)$, which is why $\operatorname{Ric} \geq 0$ is needed (9B.7 The Heat Equation on a Manifold).

Integrating it. Along any path $\gamma(s)$ from $(x_1, t_1)$ to $(x_2, t_2)$ with $t_1 < t_2$, the Li–Yau inequality bounds the change of $\log u$, and optimising over straight paths gives

(Exercise 10.8). This is the parabolic Harnack inequality in sharp form: a positive solution now is bounded below by its value earlier and nearby, with the Gaussian factor that the heat kernel saturates (Figure 10.2).

10.6The template

monotone quantity	flow	equality case	where
entropy $H$, Nash entropy	heat equation	Gaussian (heat kernel)	this chapter
Li–Yau quantity $\Delta\log u + \frac{n}{2t}$	heat equation	heat kernel	this chapter
Huisken's Gaussian density $\Theta$	mean curvature flow	self-shrinkers	6A.8 Curve Shortening and the First Geometric Flows
Perelman's $\mathcal W$ and $\mu$	Ricci flow + conjugate heat equation	shrinking solitons	12A.3 The 𝓦-Entropy
Perelman's reduced volume	Ricci flow	shrinking solitons (Gaussian on flat space)	12A.5 Reduced Distance and Reduced Volume

The pattern is the one first met as "monotone and bounded sequences converge" (2A.6 Sequences), with an extra ingredient: the equality case is a self-similar solution. So a solution along which the quantity becomes constant, such as a limit of rescalings near a singularity, must be self-similar. That is how Perelman shows that singularity models are shrinking solitons, and it is why entropy is at the centre of the proof.

10.7History

Ludwig Boltzmann's entropy dates from the 1870s; Claude Shannon's information entropy and the entropy power inequality from 1948. Ronald Fisher introduced his information in statistics in the 1920s. Aart Stam's proof of the entropy power inequality, using de Bruijn's identity, appeared in 1959. John Nash's paper on continuity of solutions of parabolic and elliptic equations, with its entropy argument, appeared in 1958. Edward Nelson proved the hypercontractivity of the Ornstein–Uhlenbeck semigroup in 1973, and Leonard Gross his logarithmic Sobolev inequality, and its equivalence with hypercontractivity, in 1975. Peter Li and Shing-Tung Yau's inequality appeared in 1986, and Hamilton's matrix Harnack inequality for the Ricci flow in 1993. Perelman's $\mathcal W$-entropy appeared in his 2002 preprint.

10.8Exercises

Solution

On the heat kernel, $-2\int u|\nabla^2v|^2 = -2\cdot\frac{n}{4t^2}\int u = -\frac{n}{2t^2}$, which matches. The general computation: $\frac{d}{dt}\int u|\nabla v|^2 = \int\Delta u\,|\nabla v|^2 + 2\int u\,\nabla v\cdot\nabla(\Delta v + |\nabla v|^2)$; integrate the first term by parts twice to $\int u\,\Delta|\nabla v|^2$, and use $\nabla u = u\nabla v$ to combine; the Bochner identity turns the result into $-2\int u|\nabla^2v|^2$.

Solution

(a) $g = \sqrt{u/\gamma}$, so $\nabla g = \frac12g\nabla\log\frac u\gamma$ and $|\nabla g|^2\gamma = \frac14\frac u\gamma\gamma|\nabla\log\frac u\gamma|^2$. (b) Under $x = \sqrt{2\tau}y$, densities transform with the Jacobian, ratios $u/\gamma_\tau$ are unchanged, the left side is invariant and $|\nabla_y\cdot|^2 = 2\tau|\nabla_x\cdot|^2$, which turns the factor $\frac12$ into $\tau$.

Solution

This is Exercise 10.5 with $2t = \sigma^2$: $H = \frac n2\log(4\pi e\cdot\frac{\sigma^2}{2}) = \frac n2\log(2\pi e\sigma^2)$, so $e^{2H/n} = 2\pi e\sigma^2$ and $N = \sigma^2$. $X + Y$ is Gaussian with covariance $(\sigma^2 + \rho^2)I$.

Solution

$\tau|\nabla f|^2 = \frac{|x|^2}{4\tau} = f$, so the integrand is $(2f - n)u$, and $\int2fu = \frac{2\cdot2n\tau}{4\tau} = n$. For $f_\lambda$, $u$ is the Gaussian of variance $2\lambda\tau$, with $\int|x|^2u = 2n\lambda\tau$. Then $\tau|\nabla f_\lambda|^2 = \frac{|x|^2}{4\lambda^2\tau}$, integrating to $\frac{n}{2\lambda}$; $\int f_\lambda u = \frac n2 + \frac n2\log\lambda$. Total: $\frac{n}{2\lambda} + \frac n2 + \frac n2\log\lambda - n = \frac n2\big(\frac1\lambda - 1 + \log\lambda\big)$, which is $\geq 0$ because $\log\lambda \geq 1 - \frac1\lambda$, with equality only at $\lambda = 1$.