In any linear Hamiltonian system, instability associated with eigenvalues $\pm\alpha\pm i\beta$ with $\alpha\ne0$ requires the
Hamiltonian $H$ to be of indefinite sign. This is necessary but not sufficient. The unstable solution must lie on the level set $H=0$ (since it approaches zero in backward time)
and this level set must be unbounded.
In classical mechanics, the simplest example of spiral-type instability for a
Hamiltonian system (both $\alpha$ and $\beta$ nonzero) involves 2 degrees of freedom. Start with
a planar oscillator, with spring constants $a$ and $b$ along the $x$ and $y$ axes
respectively, and impose a rotation of plane with angular velocity $\omega$. (This classical example is mentioned in
this paper,
see the references there for more information concerning gyroscopic stabilization.)
The Lagrangian for this system takes the form
$$
L = \frac12 ((\dot x-\omega y)^2+(\dot y+\omega x)^2)-(a x^2+b y^2))
$$
and the Hamiltonian is
$$
H = \frac12 (p_x^2+p_y)^2+ax^2+by^2)+ \omega(p_xy-p_yx) .
$$
If $0<a<-b<3a$, the system is unstable without rotation, and as the
rotation speed $\omega$ is increased it becomes first stable, but then unstable
with oscillatory instability:
The characteristic equation for solutions of the system that are proportional
to $e^{\lambda t}$ reduces to the equation
$$
0={\rm det}( (\lambda I-\omega J)^2 +A ) =0,
\quad J = \left(\begin{matrix} 0& 1\\ -1&0 \end{matrix}\right),
\quad A = \left(\begin{matrix}a&0\\ 0&b\end{matrix}\right).
$$
One computes that the characteristic roots satisfy
$$
-\lambda^2 = \omega^2+ \frac{a+b}2 \pm
\left(
\left(\omega+\frac{a+b}2\right)^2-(\omega^2-a)(\omega^2-b)\right)^{1/2}
$$
or
$$-\lambda^2 = \omega^2+ \frac{a+b}2 \pm
\frac 12
\left(
\left({a-b}\right)^2 + 8\omega^2(a+b)\right)^{1/2}
$$
The roots are of the form $\lambda =\pm\alpha\pm i\beta$ with nonzero
$\alpha$ and $\beta$ whenever $-\lambda^2$ is not real.
Clearly this is the case and the system has an oscillatory instability
if and only if $a+b<0$ and
$\omega^2$ is large enough so that
$$
\Delta:=8\omega^2(a+b)+(b-a)^2<0.
$$
For the value of $\omega^2$ that makes $\Delta=0$, we have
$$
-8(a+b)\lambda^2 = -(b-a)^2+ 4(a+b)^2.
$$
One finds that the system is stable ($\lambda^2<0$) at the onset of oscillatory instability ($\Delta=0$)
if
$$0<a<-b<3a.$$
In this situation,
as $\omega^2$ increases from 0,
the system goes from i) being unstable with one real root $\lambda>0$,
to ii) stable with four pure imaginary roots, to iii) unstable with
four roots $\lambda =\pm\alpha\pm i\beta$ with nonzero
$\alpha$ and $\beta$.