Characteristics of Functions

Extreme Points

Definition: Extreme points include,

1. Local minima: If $f:U\subset {\mathbb{R}}^n\to \mathbb{R}$, a point $x_0\in U$ is a local minima of $f$ if there exists $B_r(x_0)$ such that for all $x\in B_r(x_0)$, $f(x)\ge f(x_0)$
2. Local maxima: If $f:U\subset {\mathbb{R}}^n\to \mathbb{R}$, a point $x_0\in U$ is a local maxima of $f$ if there exists $B_r(x_0)$ such that for all $x\in B_r(x_0)$, $f(x)\le f(x_0)$
3. A point is called a local extrema if it is either a local minima or a local maxima
4. A point $x_0$ is called a critical point of $f:{\mathbb{R}}^n\to \mathbb{R}$ if $\nabla f(x_0)=0$
5. Saddle point: A critical point which is not a local extrema

Theorem: Say $f:U\subset {\mathbb{R}}^n\to \mathbb{R}$ is differentiable, if $x_0\in U$ is a local extrema then $\nabla f(x_0)=0$ and $x_0$ has to be critical

Example: $f:{\mathbb{R}}^2\to \mathbb{R},f(x,y)=x^2+y^2$
$\nabla f(x,y)=0$
$\nabla f(x,y)=(2x,2y)$
The only critical point is $(0,0)$
It is pretty clear in this case that this is a local minima

However, this is not always clear. The second derivative test can be used to test to better test for local extrema for most functions

Definition: If $f:U\subset {\mathbb{R}}^n\to \mathbb{R}$ has continuous 2nd derivatives, the Hessian of $f$ at $x_0\in {\mathbb{R}}^n$ along $\vec{h}\in {\mathbb{R}}^n$ $Hf(x_0)(h)=\frac{1}{2}\sum _{i,j=1}^n\frac{{\partial }^{2}f}{\partial x_i\partial x_j}(x_0)h_i{h}_j$

This helps write the 2nd order Taylor series out as $f(x_0+h)=f(x_0)+\nabla f(x_0)h+\frac{1}{2}Hf(x_0)(h)+R(x_0,h)$
At a critical point, this simplifies to $f(x_0+h)=f(x_0)+\frac{1}{2}Hf(x_0)(h)+R(x_0,h)$

Definition: $Hf(x_0)$ is called positive definite if $Hf(x_0)(h)>0$ for all $h$, $Hf(x_0)$ is called negative definite if $Hf(x_0)(h)<0$ for all $h$

Theorem: If $f:U\subset {\mathbb{R}}^n$ is twice differentiable at $x_0$ and $x_0$ is a critical point of $f$, then $Hf(x_0)$ is positive definite $⟹$ $x_0$ is a local minima and $Hf(x_0)$ is negative definite $⟹$ $x_0$ is a local maxima

Note that since the second derivatives are constant, the Hessian for a two variable function can be written like
$\frac{1}{2}a{h}_1^2+b{h}_1{h}_2+\frac{1}{2}c{h}_2^2$
$=\frac{1}{2}{h}_2^2\left(a{\left(\frac{h_1}{h_2}\right)}^2+2b\frac{h_1}{h_2}+c\right)$
$=\frac{1}{2}{h}_2^2\left(a{\left(\frac{h_1}{h_2}\right)}^2+2b\frac{h_1}{h_2}+\frac{b^2}{a}+c-\frac{b^2}{a}\right)$
$=\frac{1}{2}{h}_2^2\left(a\left[{\left(\frac{h_1}{h_2}\right)}^2+2\frac{b}{a}\frac{h_1}{h_2}+\frac{b^2}{a^2}\right]+c-\frac{b^2}{a}\right)$
$=\frac{1}{2}{h}_2^2\left(a{\left(\frac{h_1}{h_2}+\frac{b}{a}\right)}^2+c-\frac{b^2}{a}\right)$
$=\frac{1}{2}{h}_2^{2}a{\left(\frac{h_1}{h_2}+\frac{b}{a}\right)}^2+\frac{1}{2}{h}_2^2\left(c-\frac{b^2}{a}\right)$

Everything here is necessarily positive except for $a$ and $c-\frac{b^2}{a}$
If $a>0$ and $ca-b^2>0$ then $(x_0,y_0)$ is a strict local minima
If $a<0$ and $ca-b^2>0$ then $(x_0,y_0)$ is a strict local maxima

$\frac{{\partial }^{2}f}{\partial y^2}\cdot \frac{{\partial }^{2}f}{\partial x^2}-{\left(\frac{{\partial }^{2}f}{\partial x\partial y}\right)}^2>0$
This is also the determinant of the Hessian matrix, $\det \left[\begin{array}{cc}\frac{{\partial }^{2}f}{\partial x^2}& \frac{{\partial }^{2}f}{\partial x\partial y}\\ \frac{{\partial }^{2}f}{\partial x\partial y}& \frac{{\partial }^{2}f}{\partial y^2}\end{array}\right]$
Example: $f(x,y)=x^2-2xy+y^2$
$\frac{\partial f}{\partial x}=2x-2y=0$, $\frac{\partial f}{\partial y}=-2x+4y=0$
$(0,0)$ is the only critical point

$\frac{{\partial }^{2}f}{\partial x^2}=2,\frac{{\partial }^{2}f}{\partial x\partial y}=-2,\frac{{\partial }^{2}f}{\partial y^2}=4$
${\mathbf{H}}_f(0,0)=4$ so it is a strict local minima

Global Extrema and Lagrange Multiplier

Definition: Suppose $f:A\subset {\mathbb{R}}^n\to \mathbb{R}$, a point $x_0\in A$ is said to be a global maxima (or absolute maxima) of $f$ if $f(x)\le f(x_0)$ for all $x\in A$ and a global minima of $f$ if $f(x)\ge f(x_0)$ for all $x\in A$

Definition: A set $D\in {\mathbb{R}}^n$ is bounded if there is a number $n>0$ such that $\Vert x\Vert <M$ for all $x\in D$
Definition: A set $D\in {\mathbb{R}}^n$ is closed if it contains all of its boundary points (this definition was given in an earlier class)

Theorem: Suppose $D\subset {\mathbb{R}}^n$ is closed and bounded and $f:D\to \mathbb{R}$ is continuous, then $f$ must have an absolute minima and an absolute maxima in $D$

To find global minima/maxima of a differentiable function $f:D\subset {\mathbb{R}}^2\to \mathbb{R}$ where $D$ is closed and bounded,

1. Locate all critical points
2. Find all critical points of $f$ as a function on boundary of $D$
3. Compute the value of $f$ along all these critical points (of step 1 and step 2)
4. Compare the different values of step 3 and select the smallest and largest

Example: Find the global extrema of $f$ in $D$
$D=\left\{(x,y):x^2+y^2\le 1\right\}$
$f:D\subset {\mathbb{R}}^2\to \mathbb{R}$, $f(x)=x^2+y^2-x-y+1$

$\frac{\partial f}{\partial x}=2x-1$, $\frac{\partial f}{\partial y}=2y-1$
$\left(\frac{1}{2},\frac{1}{2}\right)$ is in $D$

Write $f$ as a function on the boundary points of $D$ (notated as $\partial D$), $x^2+y^2=1$
Parameterize $f(x,y)$ like so $f(t)={\cos }^{2}t+{\sin }^{2}t-\cos t-\sin t+1=2-\cos t-\sin t$
$\frac{df}{dt}=\sin t-\cos t=0$
Find the solutions for $t$ and then compare the critical points

Alternative approach might work?
$y=\pm \sqrt{1-x^2}$
$f(x)=-x-\sqrt{1-x^2}+2$
$\frac{df}{dx}=-1+\frac{2x}{\sqrt{1-x^2}}$
$\sqrt{1-x^2}=2x$
$1-x^2=4x^2$, $x=\pm \frac{\sqrt{5}}{5}$

In general, the problem of finding maxima/minima according to a constraint is difficult and needs some specialized techniques. The problem is maximizing/minimizing $f(x,y)$ subject to the side condition $g(x,y)=c$.

Theorem: Suppose that $f:U\subset {\mathbb{R}}^n\to \mathbb{R}$ and $g:U\subset {\mathbb{R}}^n\to \mathbb{R}$ are $C^1$ functions. Let $x_0\in U$ and $g(x_0)=c$ and let $S$ be the level set of $g$ with value $c$. Assume $\nabla g(x_0)\ne 0$, then if $f|S$ (denotes $f$ restricted to $S$) has a local maximum or minimum on $S$ at $x_0$ then there is a real number $\lambda$ such that $\nabla f(x_0)=\lambda \nabla g(x_0)$

Essentially, $f(x)$ is parallel to $g(x)$ and perpendicular to the level set $S$

Theorem: If $f$, when constrained to a surface $S$, has a maximum or minimum at $x_0$, then $\nabla f(x_0)$ is perpendicular to $S$ at $x_0$

So the goal is to find a point $x_0$ and a constant $\lambda$ called a Lagrange multiplier such that $\nabla f(x_0)=\lambda \nabla g(x_0)$. This means solving the simultaneous equations $\frac{\partial f}{\partial x_i}(x)=\lambda \frac{\partial g}{\partial x_i}(x)$.

In other words, the Lagrange multiplier theorem says that to find the extreme points of $f|S$ one should examine the critical points of $h(x_1,\dots ,x_n,\lambda )=f(x_1,\dots ,x_n)-\lambda [g(x_1,\dots ,x_n)-c]$

Example: Maximize $f(x,y,z)=x+z$ subject to the constraint $x^2+y^2+z^2=1$
This has a maximum and minimum since $f$ is continuous and $D$ is closed and bounded
$1=2x\lambda ,0=2y\lambda ,1=2z\lambda$, and $x^2+y^2+z^2=1$
This gives $(1/\sqrt{2},0,1/\sqrt{2})$ and $(-1/\sqrt{2},0,-1/\sqrt{2})$

$f|S$ can have a maxima and minima even when $S$ is unbounded, depending on the situation.

If $S$ is bounded then $f$ must have a maximum and a minimum. If only two points satisfy the Lagrange multiplier than one is the maximum and one is the minimum. If there are more than some can be saddle points. If $S$ is not bounded then $f$ need not have any maxima or minima

For multiple conditions, the Lagrange multiplier theorem can be generalized with the equation $\nabla f(x_0)={\lambda }_1\nabla g_1(x_0)+\cdots +{\lambda }_k\nabla g_k(x_0)$

Definition: Let $U$ be an open region in ${\mathbb{R}}^n$ with boundary $\partial U$, $\partial U$ is smooth if $\partial U$ is the level set of a smooth function $g$ whose gradient $\nabla g$ never vanishes ($\nabla g\ne 0$)

Now, to find the absolute maxima and minima,

1. Locate all critical points of $f$ in $U$
2. Use the method of Lagrange multiplier to locate all the critical points of $f|\partial U$
3. Compute the values of $f$ at all these critical points
4. Select the largest and the smallest

Is there an equivalent Hessian matrix technique for problems with constraints?

Theorem: Let $f:U\subset {\mathbb{R}}^2\to \mathbb{R}$ and $g:U\subset {\mathbb{R}}^2\to \mathbb{R}$ (and at least $C^2$). Let $v_0\in U$ and $g(v_0)=c$, and $S$ be the level curve for $g$ with value $c$. From the auxiliary function $h=f-\lambda g$, the bordered Hessian determinant is $|\overline{H}|$=$\left|\begin{array}{ccc}0& \frac{-\partial g}{\partial x}& \frac{-\partial g}{\partial y}\\ \frac{-\partial g}{\partial x}& \frac{{\partial }^{2}h}{\partial x^2}& \frac{{\partial }^{2}h}{\partial x\partial y}\\ \frac{-\partial g}{\partial y}& \frac{{\partial }^{2}h}{\partial x\partial y}& \frac{{\partial }^{2}h}{\partial y^2}\end{array}\right|$
Now, if $|\overline{H}|>0$ then $v_0$ is a local maximum point for $f|S$. If $|\overline{H}|<0$ then $v_0$ is a local minimum point for $f|S$. Otherwise, the test is inconclusive.

In the general case, one tests the $n\times n$ submatrices of the matrix,

$$|\overline{H}|=\left|\begin{array}{ccccc}0& \frac{-\partial g}{\partial x_1}& \frac{-\partial g}{\partial x_2}& \dots & \frac{-\partial g}{\partial x_n}\\ \frac{-\partial g}{\partial x_1}& \frac{{\partial }^{2}h}{\partial x_1^2}& \frac{{\partial }^{2}h}{\partial x_1\partial x_2}& \dots & \frac{{\partial }^{2}h}{\partial x_1\partial x_n}\\ \frac{-\partial g}{\partial x_2}& \frac{{\partial }^{2}h}{\partial x_1\partial x_2}& \frac{{\partial }^{2}h}{\partial x_2^2}& \dots & \frac{{\partial }^{2}h}{\partial x_2\partial x_n}\\ ⋮& ⋮& ⋮& & ⋮\\ \frac{-\partial g}{\partial x_n}& \frac{{\partial }^{2}h}{\partial x_1\partial x_n}& \frac{{\partial }^{2}h}{\partial x_2\partial x_n}& \dots & \frac{{\partial }^{2}h}{\partial x_n^2}\end{array}\right|$$

If these submatrices are all negative then we are a local minimum. If they alternative in sign then we are at a local maximum.