Csdrhrt

Goal

I would like to proof than the Negative Log Likelihood Function of Sample drawn from a Normal Distribution is convex.

Below a Figure showing an example of such function:

Motivation of this question is detailed at the end of the post.

Sketching the demonstration

What I did so far...

First I have written the likelihood function for a single observation:

$$mathcal{L} (mu, sigma mid x) = frac{1}{sqrt{2pisigma^2} } e^{ -frac{(x-mu)^2}{2sigma^2} }$$

For convenience I take the Negative Logarithm of the Likelihood function, I think it does not change the extremum because Logarithm is a monotonic function:

$$f(mu, sigma mid x) = -ln mathcal{L} (mu, sigma mid x) = frac{1}{2}ln(2pi) + ln(sigma) +frac{(x-mu)^2}{2sigma^2}$$

Then I can assess the Jacobian (check: 1 and 2):

$$
mathbf{J}_f = left[
begin{matrix}
-frac{x-mu}{sigma^2}\
-frac{(x-mu)^2 - sigma^2}{sigma^3}
end{matrix}
right]
$$

And the Hessian (check 3, 4 and 5):

$$
mathbf{H} = left[
begin{matrix}
frac{1}{sigma^2} & frac{2(x - mu)}{sigma^3} \
frac{2(x - mu)}{sigma^3} & frac{3(x - mu)^2 - sigma^2}{sigma^4}
end{matrix}
right]
$$

Of the Negative Log Likelihood function.

Now, I compute the Hessian of the Negative Log Likelihood function for $N$ observations:

$$
mathbf{A} = frac{1}{N}sumlimits_{i=1}^{N}mathbf{H} = left[
begin{matrix}
frac{1}{sigma^2} & frac{2(bar{x} - mu)}{sigma^3} \
frac{2(bar{x} - mu)}{sigma^3} & frac{frac{3}{N}sumlimits_{i=1}^{N}(x-mu)^2 - sigma^2}{sigma^4}
end{matrix}
right]
$$

If everything is right at this point:

• Proving the function is convex is equivalent to prove than the Hessian is semi-positive definite;

Additionally I know that:

• A semi-positive definite Matrix must have all its eigenvalues non-negative;


• Because the Hessian Matrix is symmetric, all eigenvalues are real;

So if I prove than all eigenvalues are positive real numbers, then I can claim the function is convex. We can also check, as @LinAlg suggested, that both determinant and trace of Matrix $mathbf{A}$ are positive:

$$
begin{align}
det(mathbf{A}) geq 0 Leftrightarrow & frac{3}{N}sumlimits_{i=1}^{N}(x-mu)^2 - sigma^2 - 4(bar{x} - mu)^2 geq 0 \
operatorname{tr}(mathbf{A}) geq 0 Leftrightarrow & frac{3}{N}sumlimits_{i=1}^{N}(x-mu)^2 geq 0
end{align}
$$

It is obvious that $operatorname{tr}(mathbf{A}) geq 0$.

Inequality

The inequality $det(mathbf{A}) geq 0$ is not obvious at the first glance, it requires a bit of algebra. Expanding all squares, applying sum, simplifying and grouping gives:

$$
3left[frac{1}{N}sumlimits_{i=1}^{N}x^2 - bar{x}^2right] -(bar{x}-mu)^2 -sigma^2 geq 0
$$

Now I can rewrite it using Standard Deviation Estimation:

$$
3left[frac{N-1}{N}s^2_x - sigma^2right] + 2sigma^2 geq (bar{x}-mu)^2
$$

For $N$ sufficiently large it tends to:

$$
3left(s^2_x - sigma^2right) + 2sigma^2 geq (bar{x}-mu)^2
$$

Or:

$$
left|bar{x}-muright| leq sqrt{3s^2_x - sigma^2}
$$

Provided the radicand is non -negative. This last inequality provides a bound for Mean Asbolute Error which must be lower than approximately $sqrt{2}sigma$. Finally, if the estimators converge to expected values it reduces to:

$$
sigma geq 0
$$

Which is trivially true by definition.

My interpretation of this inequality is:

If we have sufficiently large statistics, drawn from a Normal Distribution, and the Mean and Variance Estimation are close enough to their expected value then the Negative Likelihood Function should be convex. If expected values $mu$ and $sigma$ are known, it is possible to assess the convexity.

Questions

• Are my reasoning and the interpretation of the result correct?


• Can we formally prove that $det(mathbf{A}) geq 0$?

Motivation

This question arose from a numerical example I develop. I am sampling from a Normal Distribution $mathcal{N}(mu=2, sigma=3)$ with $N=1000$ and I would like to emphasize all steps of a Maximum Likelihood Estimation. Then when I visualized the function to minimize I wondered: Can we say that this function is convex? Which made me write this post on MSE.