The Inverse and Implicit Function Theorems

When can a nonlinear equation be solved? For a linear system $Ax = y$ the answer is clean: if $A$ is invertible, there is exactly one solution, $x = A^{-1}y$, and it depends continuously on $y$. For a nonlinear system $f(x) = y$ there is no general answer. But there is a local one, and it is the most important theorem of this book: if the linearisation is invertible, the nonlinear problem is solvable nearby. Precisely, if $f(x_0) = y_0$ and the derivative $Df(x_0)$ is an invertible matrix, then for every $y$ close to $y_0$ the equation $f(x) = y$ has exactly one solution $x$ close to $x_0$, and that solution depends smoothly on $y$. This is the inverse function theorem.

Its proof combines the two engines built so far: the derivative (2B.8 Calculus in Several Variables) says $f$ is nearly linear near $x_0$, and the contraction mapping principle (2B.2 Completeness and Contraction) turns "nearly linear and invertible" into "invertible". The implicit function theorem follows, and with it the geometric fact that a level set $\{F = c\}$ is a smooth surface wherever $\nabla F \neq 0$, and the method of Lagrange multipliers. The pattern of the proof, linearise, invert, contract, is thread L of the guidebook, and it is exactly how the existence of Ricci flow is proved (11A.3 Short-Time Existence and Uniqueness).

By the end of this chapter you will be able to:

• state the inverse function theorem and prove it from the contraction mapping principle;
• explain why the theorem is local, with examples of maps that are locally but not globally invertible;
• state and use the implicit function theorem, including the formula for the derivative of an implicitly defined function;
• show that a level set is locally a smooth graph where the gradient is non-zero, and find where it fails;
• derive and use Lagrange multipliers.

9.1When a robot arm gets stuck

9.2The inverse function theorem

The formula for $D(f^{-1})$ is forced by the chain rule: differentiating $f(f^{-1}(y)) = y$ gives $Df(x)\,D(f^{-1})(y) = I$. The content of the theorem is that the inverse exists and is differentiable.

9.2.1The proof

9.2.2Local, not global

The theorem is local, and it has to be. The map $f(x, y) = (e^x\cos y, e^x\sin y)$ has $\det Df = e^{2x} \neq 0$ everywhere, so it is locally invertible at every point. But it is not injective: $f(x, y + 2\pi) = f(x, y)$. (It is the complex exponential $z \mapsto e^z$ written in real coordinates, and its local inverses are branches of the logarithm.) An invertible derivative tells you that nearby points have distinct images, not that distant ones do.

Nor can the hypothesis be dropped. $f(x) = x^3$ is a bijection of $\mathbb{R}$, but $f'(0) = 0$, and the inverse $y^{1/3}$ is not differentiable at $0$. Where the derivative is singular, the inverse, if it exists at all, is not differentiable.

9.3The implicit function theorem

Often the equation to solve has more unknowns than equations. A single equation $F(x, y) = 0$ in two unknowns usually defines a curve, and the question is whether that curve can be written as a graph $y = g(x)$, and whether $g$ is differentiable.

Write points of $\mathbb{R}^{n + m}$ as $(x, y)$ with $x \in \mathbb{R}^n$, $y \in \mathbb{R}^m$. For $F : \mathbb{R}^{n+m} \to \mathbb{R}^m$, the derivative $DF(x, y)$ is an $m \times (n + m)$ matrix, which splits into an $m \times n$ block $D_xF$ (derivatives in the $x$-variables) and an $m \times m$ block $D_yF$.

9.3.1Level sets are smooth where the gradient isn't zero

The case $m = 1$ is the geometric heart of the theorem. Let $F : \mathbb{R}^{n+1} \to \mathbb{R}$ be $C^1$ and $p$ a point of the level set $\{F = c\}$ where $\nabla F(p) \neq 0$. Some partial derivative is non-zero at $p$; renumber the coordinates so it is the last one. Then the implicit function theorem says that near $p$, the level set is the graph of a $C^1$ function of the other $n$ coordinates: a smooth hypersurface, with $\nabla F(p)$ as its normal (2B.8 Calculus in Several Variables). A value $c$ such that $\nabla F \neq 0$ at every point of $\{F = c\}$ is called a regular value, and then the whole level set is a smooth hypersurface. This is the regular value theorem, the main way manifolds are produced in practice (8A.4 Submanifolds).

Where the gradient vanishes, the level set can do anything. For the lemniscate of Bernoulli, $F(x, y) = (x^2 + y^2)^2 - 2(x^2 - y^2) = 0$, the gradient vanishes at the origin, and there the curve crosses itself: near the origin it is not the graph of any function of either variable (Figure 9.2). The crossing contour through the mountain pass in 2B.8 Calculus in Several Variables is the same phenomenon: a pass is a critical point of the elevation, and the contour through it is singular.

9.4Lagrange multipliers

To find the maximum of $f$ on a constraint set $\{g = 0\}$, we can't simply set $\nabla f = 0$: the maximum on the surface need not be a critical point in the whole space.

In words: at a constrained extremum, the level set of $f$ is tangent to the constraint surface, so their normals are parallel. With several constraints $g_1 = \cdots = g_k = 0$ whose gradients are linearly independent at $x_0$, the same proof gives $\nabla f = \sum_i\lambda_i\nabla g_i$.

9.5History

Lagrange introduced his multipliers in mechanics, in the Méchanique analitique of 1788, to handle constrained motion. Cauchy proved an implicit function theorem for analytic functions in the 1830s, using power series. Ulisse Dini proved the real-variable implicit function theorem, with continuously differentiable functions, in his Pisa lectures of 1877–78. The proof through a contraction is from the 20th century, once Banach's principle (2B.2 Completeness and Contraction) was available, and it is the version that extends to infinite-dimensional spaces, where most of its modern uses lie.

9.6Exercises

Solution

(a) $Df = \begin{pmatrix}-\ell_1s_1 - \ell_2s_{12} & -\ell_2s_{12}\\ \ell_1c_1 + \ell_2c_{12} & \ell_2c_{12}\end{pmatrix}$ with $s_1 = \sin\theta_1$, $s_{12} = \sin(\theta_1 + \theta_2)$, etc. Its determinant is $\ell_1\ell_2(c_1s_{12} - s_1c_{12}) = \ell_1\ell_2\sin\theta_2$. (b) By the law of cosines, $r^2 = \ell_1^2 + \ell_2^2 + 2\ell_1\ell_2\cos\theta_2$, which ranges over $[(\ell_1 - \ell_2)^2, (\ell_1 + \ell_2)^2]$. (c) For $r$ strictly inside, $\cos\theta_2$ is determined, giving two values $\pm\theta_2$, and then $\theta_1$ is determined by the direction. On the boundary $\theta_2 \in \{0, \pi\}$ is unique. The two branches meet at the singular configurations, where the Jacobian is singular, as the inverse function theorem requires.

Solution

(b) $\arctan$, or $e^x$. No: by the intermediate value theorem for derivatives (Darboux) or, for $C^1$ maps, the intermediate value theorem for $f'$, $f'$ has constant sign, so $f$ is strictly monotone. In one dimension, local invertibility everywhere implies global injectivity; in two dimensions it doesn't.

Solution

(a) $\partial_yF = 3y^2 - 3x = 0$ with $F = 0$: $x = y^2$ and $y^6 + y^3 - 3y^3 = 0$, so $y = 0$ (the origin) or $y^3 = 2$, giving $(2^{2/3}, 2^{1/3})$. Symmetrically $\partial_xF = 0$ at the origin and $(2^{1/3}, 2^{2/3})$. At the origin the curve crosses itself. (b) $dy/dx = -\frac{3x^2 - 3y}{3y^2 - 3x} = -\frac{9/4 - 3/2}{9/4 - 3/2} = -1$.

Solution

$\nabla f = 2Ax$ and $\nabla g = 2x$ for $g = |x|^2 - 1$, so at an extremum $Ax = \lambda x$: an eigenvector, with $f(x) = x^{\mathsf T}\lambda x = \lambda$. The maximum of $f$ is therefore the largest eigenvalue, and the minimum the smallest.

Solution

(a) $\sum_iu_iu_i^{\mathsf T} = \tfrac32I$ for three unit vectors $120°$ apart. (b) $H^{\mathsf T}H = \mathrm{diag}\big(2\cos^2 75°,\ 1 + 2\sin^2 75°\big) \approx \mathrm{diag}(0.134, 2.866)$, so the DOP is $\sqrt{1/0.134 + 1/2.866} \approx 2.80$. The horizontal direction, across the line of sight, is poorly determined: all three beacons are nearly overhead, so moving sideways barely changes any range.

Solution

$D_xF(x_0, 0) = \nabla^2f(x_0)$ is invertible, so near $(x_0, 0)$ the zeros of $F$ are a smooth curve $x(\varepsilon)$, with $x'(0) = -D_xF^{-1}\partial_\varepsilon F = -\nabla^2f(x_0)^{-1}\nabla h(x_0)$. The Hessian $\nabla^2f(x(\varepsilon)) + \varepsilon\nabla^2h(x(\varepsilon))$ depends continuously on $\varepsilon$ and is invertible at $\varepsilon = 0$; its eigenvalues depend continuously on it, and none can cross $0$ while it stays invertible (by continuity of the determinant, for small $\varepsilon$), so the number of negative ones is constant.