Stochastic Control and the HJB Equation

In 6D.6 Black–Scholes the investor's task was to hedge: hold whatever position replicates a contract. Here the task is to choose: how much to invest, how fast to steer, how much to consume, to make some expected reward as large as possible. When the system is a diffusion and the choice can be revised continuously, the answer is governed by a nonlinear second-order PDE, the Hamilton–Jacobi–Bellman equation

the stochastic version of the first-order HJB equation of 6C.9 Hamilton–Jacobi Equations and Viscosity Solutions. The supremum over controls of a family of linear operators makes it a fully nonlinear equation of the Bellman type of 6C.10 Fully Nonlinear Equations and Monge–Ampère.

The strategy is always the same. Guess or compute a solution of the HJB equation; read off the optimal control as the maximiser in the supremum, a feedback rule depending on the current state; and confirm, by a verification theorem proved with Itô's formula, that the candidate really is optimal. We carry this out for Robert Merton's portfolio problem, whose answer, a constant fraction of wealth in the risky asset, is one of the classical results of financial economics, and for the linear-quadratic regulator of engineering.

By the end of this chapter you will be able to:

• set up a stochastic control problem, state the dynamic programming principle, and derive the HJB equation from it with Itô's formula;
• prove the verification theorem;
• solve Merton's problem for power utility, and derive the Kelly criterion as the logarithmic case;
• solve the linear-quadratic regulator through a Riccati equation;
• explain why value functions are in general viscosity solutions, and connect the Li–Yau Harnack inequality to an optimal control problem.

8.1How much to bet

8.2Controlled diffusions and dynamic programming

Let $A$ be a set of control values (a subset of some $\mathbb{R}^k$). A control is an adapted process $\alpha_s \in A$, and the controlled state follows

with $b$, $\sigma$ Lipschitz in $x$ uniformly in $a$, so that the SDE has a unique solution for each admissible control (6D.3 Stochastic Differential Equations and Feynman–Kac). The controller collects a running reward $f(X_s, \alpha_s)$ and a terminal reward $g(X_T)$, and the value function is

(Minimising a cost is the same with $f, g$ replaced by $-f, -g$.)

The dynamic programming principle (statement; Fleming and Soner, chapter IV, and Pham, section 3.3): for $t \leq t + h \leq T$,

It is Bellman's principle of optimality, and its proof is the same as in the deterministic case (6C.9 Hamilton–Jacobi Equations and Viscosity Solutions) except for a measurable selection of nearly optimal controls from each intermediate state, which is technical.

Formal derivation of HJB. Suppose $u$ is $C^{1,2}$. Use the constant control $a$ on $[t, t + h]$ in the principle, and Itô's formula (6D.2 Martingales and the Itô Integral) for $u(s, X_s)$:

with $\mathcal L^a = b(x, a)\cdot\nabla + \frac12\operatorname{tr}\big(\sigma\sigma^T(x, a)D^2\big)$ the generator for the fixed control $a$ (the stochastic integral has mean zero). Divide by $h$ and let $h \to 0$: $0 \geq \partial_tu + \mathcal L^au + f(x, a)$ for every $a$. For an optimal (or nearly optimal) control the inequality becomes an equality in the limit. So

The equation is degenerate elliptic in the sense of 6C.9 Hamilton–Jacobi Equations and Viscosity Solutions, and parabolic backwards in time.

8.3The verification theorem

The theorem needs a smooth solution of (HJB). When one exists (as in the examples below), it settles the problem completely and produces the optimal control in feedback form, which is how controls are implemented: measure the state, apply $a^*(t, x)$. When the value function is not smooth, the dynamic programming principle still shows that it is a viscosity solution of (HJB), and comparison principles of the Crandall–Ishii–Lions type (6C.9 Hamilton–Jacobi Equations and Viscosity Solutions) show it is the unique one (statement; Fleming and Soner, chapters IV–V). Pierre-Louis Lions established this framework for stochastic control in 1983. It is the reason value functions are well defined and computable (by monotone finite difference schemes, which converge to viscosity solutions by a theorem of Guy Barles and Panagiotis Souganidis, 1991) even when they have kinks.

8.4Merton's portfolio problem

An investor has wealth $W_t$, holds the amount $\pi_t$ in a stock with $dS = \mu S\,dt + \sigma S\,dB$ and the rest in a bank account at rate $r$, with $\mu > r$. Since $\pi_t$ in stock earns $\pi_t\frac{dS}{S}$ and $W_t - \pi_t$ in the bank earns $r(W_t - \pi_t)\,dt$,

The investor maximises expected utility of terminal wealth, $\mathbb{E}\,U(W_T)$, with the power utility

Concavity of $U$ expresses risk aversion: a certain amount is preferred to a gamble with the same mean. The parameter $\gamma = -\frac{wU''(w)}{U'(w)}$ is the relative risk aversion, constant for this family. Write $\theta = \frac{\mu - r}{\sigma}$ for the market price of risk (6D.6 Black–Scholes).

The HJB equation for $u(t, w)$ is

If $\partial_{ww}u < 0$ (concavity in $w$), the supremum of the quadratic in $\pi$ is attained at

Three features of the answer deserve comment. The fraction is constant: it does not depend on wealth or on the horizon, a consequence of the scale invariance of power utility and of the constant coefficients (Paul Samuelson proved the discrete-time analogue in 1969). It is myopic: the investor behaves as if the horizon were one instant away. And it is the mean–variance trade-off: the instantaneous excess return of the portfolio is $c(\mu - r)$ and its variance $c^2\sigma^2$, and $c^*$ maximises $c(\mu - r) - \frac\gamma2c^2\sigma^2$. With $\mu = 0.07$, $r = 0.02$, $\sigma = 0.2$ and $\gamma = 2$, the fraction is $\frac{0.05}{2\times0.04} = 0.625$.

8.5The Kelly criterion

For $\gamma = 1$, the investor maximises $\mathbb{E}\log W_T$. For a constant fraction $c$, by Itô's formula,

so $\frac1T\log\frac{W_T}{W_0} \to g(c) = r + c(\mu - r) - \frac12c^2\sigma^2$ almost surely as $T \to \infty$ (the Brownian term grows like $\sqrt T$). The long-run growth rate $g(c)$ is maximised at the Kelly fraction $c_K = \frac{\mu - r}{\sigma^2}$, with $g(c_K) = r + \frac{\theta^2}{2}$. Over long horizons, the Kelly strategy's wealth eventually exceeds that of any other constant-fraction strategy, almost surely (Leo Breiman proved a general version in 1961).

The parabola $g$ shows the danger of overbetting: $g(2c_K) = r$ (Exercise 8.4), so betting twice the Kelly fraction gives no more growth than the bank account while taking large risks, and beyond $2c_K$ the growth rate is below $r$ and eventually negative. The Itô correction $-\frac12c^2\sigma^2$ is the volatility drag of 6D.2 Martingales and the Itô Integral (geometric Brownian motion there): it is why the median outcome of a leveraged position can lose money even when its mean grows.

8.6The linear-quadratic regulator

In engineering, the standard control problem steers a linear system near a target with quadratic costs. In one dimension, with $dX = \alpha\,dt + \sigma\,dB$ and the cost

to be minimised ($q, k \geq 0$), the HJB equation is $\partial_tu + \inf_a\{a\,\partial_xu + qx^2 + a^2\} + \frac12\sigma^2\partial_{xx}u = 0$, $u(T) = kx^2$. The infimum is at $a^* = -\frac12\partial_xu$, and the ansatz $u = p(t)x^2 + c(t)$ gives the Riccati equation

with optimal feedback $\alpha^* = -p(t)X$ (Exercise 8.6). In $n$ dimensions with $dX = (AX + B\alpha)\,dt + \Sigma\,dW$, costs $X^TQX + \alpha^TR\alpha$, the same calculation gives the matrix Riccati equation $-P' = A^TP + PA - PBR^{-1}B^TP + Q$ and the feedback $\alpha^* = -R^{-1}B^TPX$. Rudolf Kálmán's work on this problem and on filtering (1960) became standard in aerospace guidance and control engineering; the noise affects the cost but not the optimal feedback, a form of certainty equivalence.

8.7History

Richard Bellman formulated dynamic programming in the 1950s, and Lev Pontryagin's school developed the maximum principle for deterministic control at the same time. Wendell Fleming and others developed stochastic control in the 1960s, and Robert Merton solved the portfolio problem in 1969 (with consumption in 1971). Rudolf Kálmán's linear-quadratic theory dates from 1960, John Kelly's criterion from 1956, and Leo Breiman's optimality theorem from 1961. Pierre-Louis Lions proved in 1983 that value functions of stochastic control problems are viscosity solutions of the HJB equation, and Barles and Souganidis's convergence theorem for numerical schemes dates from 1991.

8.8Exercises

Solution

$\partial_wu = \frac1w$, $\partial_{ww}u = -\frac{1}{w^2}$, so $\pi^* = \frac{(\mu - r)/w}{\sigma^2/w^2} = \frac{\mu - r}{\sigma^2}w$. The equation becomes $\psi' + r + \frac{(\mu - r)^2}{2\sigma^2} = 0$ (since $-\frac{(\mu - r)^2}{2\sigma^2}\frac{w^{-2}}{-w^{-2}} = \frac{\theta^2}{2}$), with $\psi(T) = 0$.

Solution

(a) $c_K = \frac{0.05}{0.04} = 1.25$ (that is, $125\%$ of wealth, borrowing the rest), $\theta = 0.25$, $g(c_K) = 0.02 + 0.03125 = 0.05125$. (b) $g(c) = r + c\theta\sigma - \frac12c^2\sigma^2$; with $c = \lambda c_K = \lambda\frac\theta\sigma$, $g = r + \theta^2(\lambda - \frac{\lambda^2}{2})$. For $\lambda = 2$: $r$. For $\lambda = \frac12$: $r + \frac38\theta^2 = 0.02 + 0.0234 = 0.0434$. Half-Kelly keeps three quarters of the excess growth with half the volatility.

Solution

Fraction $\frac{0.05}{2\times0.04} = 0.625$; $\rho = 0.02 + \frac{0.0625}{4} = 0.0356$. Indeed $U(we^{\rho T}) = \frac{w^{1-\gamma}e^{(1-\gamma)\rho T}}{1 - \gamma} = u(0, w)$.

Solution

(a) With $u = px^2 + c$: $\partial_xu = 2px$, $a^* = -px$, $\inf_a\{\cdots\} = -p^2x^2 + qx^2$, $\frac12\sigma^2\partial_{xx}u = \sigma^2p$. So $p'x^2 + c' - p^2x^2 + qx^2 + \sigma^2p = 0$ for all $x$. (b) $p' = q - q = 0$ with $p(T) = \sqrt q$. (c) $dX = -\sqrt qX\,dt + \sigma\,dB$, with stationary variance $\frac{\sigma^2}{2\sqrt q}$.

Solution

For minimisation, $\partial_tu + \inf_{|a|\leq1}\{a\,\partial_xu\} + \frac12\partial_{xx}u = 0$, i.e. $\partial_tu - |\partial_xu| + \frac12\partial_{xx}u = 0$, $u(T) = x^2$. The infimum of the linear function $a\,\partial_xu$ over $[-1, 1]$ is at $a = -\operatorname{sign}(\partial_xu)$, which is $-\operatorname{sign}(x)$ when $\partial_xu$ has the sign of $x$ (as it does by symmetry and monotonicity). Optimal controls of linear-in-control problems with bounded controls take extreme values: "bang-bang".

Solution

(a) $\frac{d}{dt}\log u(\gamma, t) = \partial_t\log u + \nabla\log u\cdot\dot\gamma \geq |\nabla\log u|^2 + \nabla\log u\cdot\dot\gamma - \frac{n}{2t} \geq -\frac{|\dot\gamma|^2}{4} - \frac{n}{2t}$. (b) Integrating, $\log\frac{u(x_2, t_2)}{u(x_1, t_1)} \geq -\frac14\int_{t_1}^{t_2}|\dot\gamma|^2\,dt - \frac n2\log\frac{t_2}{t_1}$. By Cauchy–Schwarz, $\int|\dot\gamma|^2 \geq \frac{|x_2 - x_1|^2}{t_2 - t_1}$, with equality for the straight line at constant speed, the optimal control. (c) As stated: the minimal cost $\frac{|x_2 - x_1|^2}{4(t_2 - t_1)}$ is the value function of a deterministic control problem, the Hopf–Lax value of 6C.9 Hamilton–Jacobi Equations and Viscosity Solutions with $L(v) = \frac{|v|^2}{4}$.