Prelude
Consider a canonical block diagram of a negative feedback loop with gain \$K\$, plant transfer function \$G(s)\$ and feedback transfer function \$H(s)\$: -
Regardless of what functions K, G, and H are, the closed loop transfer function for this system is
$$T_\text{cl}(s) = \frac{KG(s)}{1+KG(s)H(s)} \tag{1}$$
The term \$KG(s)H(s)\$ is called the open loop gain (in some literature just "loop gain") and this is of particular interest. Only signals that can be physically made are of interest so we substitute \$s = j\omega\$ and arrive at
$$T_\text{ol}(j\omega) = KG(j\omega)H(j\omega) = |T_\text{ol}(j\omega)|e^{j\theta_\text{ol}(j\omega)} $$
\$T_\text{ol}(j\omega)\$ is a complex function that varies with frequency.
The frequency at which \$T_\text{ol}(j\omega) = -1\$, i.e. \$|T_\text{ol}(j\omega)| = 1 \$ and \$\theta_\text{ol}(j\omega) = \pm180^\circ \$ is special, because it causes the denominator in \$(1)\$ to become \$0\$ and \$T_\text{cl}(j\omega)\$ becomes undefined.
At this point, all textbooks I've read about this subject define the frequency at which \$|T_\text{ol}(j\omega)| = 1 = 0 \: \text{dB}\$ as the crossover frequency \$\omega_c\$. Incidentally, another parameter used to characterize the stability of a closed loop system is defined:
$$\phi_m = \theta_\text{ol}(j\omega_c) + 180^\circ $$
The importance of \$\phi_m\$ is that it reveals how close to instability the closed loop system will be. So a system with \$\phi_m = 5^\circ\$ will be close to instability, while a system with \$\phi_m = 60^\circ\$ will robust.
This all makes perfect sense. So far so good.
Problem
The question is: what happens when \$\phi_m\$ becomes negative?
The textbooks claim that the closed loop system will be unstable, but with no mathematical argument. A simulation supports this claim but with no satisfactory explanation. Let's take an example. I am only interested in the unity feedback case, i.e. \$H(s) = 1\$:
Consider a system with \$H(s) = 1\$ and \$KG(s) = \frac{10^5}{s^3+60s^2+1100s+6000} \$, whose open loop system has negative phase margin.
$$T_\text{ol}(s) = \frac{10^5}{s^3+60s^2+1100s+6000} \Rightarrow$$$$T_\text{cl}(s) = \frac{10^5}{s^3+60s^2+1100s+106000} $$
But why is the closed loop system unstable? The answer is obviously that \$T_\text{cl}(s)\$ has poles in the right half plane (in this case \$3.56 \pm j39.5\$) but why does a negative phase margin imply that?
Another thing I have seen mentioned in some of TI's documents is that \$\phi_m\$ should be measured with respect to the phase at DC. So if \$\theta_\text{ol}(0) = 180^\circ\$ and \$\theta_\text{ol}(j\omega_c) = 30^\circ \$ then \$\phi_m = 180^\circ - 150^\circ = 30^\circ\$. Where they get this from I don't know.
Question
I am looking for a mathematical explanation of why a negative phase margin in a negative feedback loop implies an unstable closed loop system.
I guess a mathematical description of the question would be:
Let \$T_\text{ol}(s)\$ be a continuous and differentiable function \$T_\text{ol} : \mathbb{C} \to \mathbb{C}\$. Define \$T_\text{cl}(s) = \frac{T_\text{ol}(s)}{1+T_\text{ol}(s)} = \frac{N(s)}{D(s)}, \: s \in \mathbb{C} \$.
Show that if there exists an \$\omega_c \in \mathbb{R}\$ such that \$T_\text{ol}(j\omega_c) =|T_\text{ol}(j\omega_c)|e^{j\theta_\text{ol}(j\omega_c)}\$ where \$|T_\text{ol}(j\omega_c)| \geq 1\$ and \$ \theta_\text{ol}(j\omega_c) = \pm \pi \$ that the polynomial \$D(s)\$ will have at least one root with positive real part.
Sources
Texas instruments claim: https://i.stack.imgur.com/nZhh8.png
Block diagram: Signal processing and linear systems - B.P.Lathi
Edit
I investigated the transfer function \$T_\text{ol}(s) = 10^4 \frac{s^3+3s^2+3s+1}{s^3+60s^2+1100s+6000} \Rightarrow T_\text{cl}(s) = 10^4\frac{s^3+3s^2+3s+1}{10001s^3+30060s^2+31100s+16000}\$.
Have a look at the Bode Plot for \$T_\text{ol}(s)\$ and the impulse response for \$T_\text{cl}(s)\$: -
What do we see? There are in this case two frequencies where the gain is larger than \$1\$ and the phase shift is \$180^\circ\$ so the closed loop system should be unstable, right? But no. Looking at the impulse response for the closed loop system we see that the system is stable.
Investigating \$T_\text{ol}(s) = 10\frac{(s+1)^2}{s^3} \Rightarrow T_\text{cl}(s) = 10\frac{(s+1)^2}{s^3+10s^2+20s+10} \$ shows this equally interesting result: -
Again, the gain is larger than \$1\$ while the phase shift now is \$-180^\circ\$. The closed loop system is still stable.
Edit 2
For completeness sake, consider the system \$T_\text{ol}(s) = 10\frac{(s+1)^2}{(s+0.01)^3} \Rightarrow T_\text{cl}(s) = 10\frac{(s+1)^2}{s^3+10.03s^2+20s+10}\$ to avoid having poles on the imaginary axis. We still arrive at a system that is closed loop stable: -
