You start with the standard trick to get from dynamic equations to energy equations, which is multiply first equation by $\dot x$, second by $\dot y$ and sum them:
$$
\ddot x\dot x+\zeta_1\dot x^2+(k_{11}+k_{12}x^2)x\dot x -2\gamma \dot y\dot x=
\frac{d}{dt}\left(\frac12\dot x^2+\frac{k_{11}}2x^2+\frac{k_{12}}4x^4\right)+\zeta_1\dot x^2-2\gamma\dot x\dot y=0,\\
\ddot y\dot y+\zeta_2\dot y^2+(k_{21}+k_{22}y^2)y\dot y +2\gamma \dot x\dot y=
\frac{d}{dt}\left(\frac12\dot y^2+\frac{k_{21}}2y^2+\frac{k_{22}}4y^4\right)+\zeta_2\dot y^2+2\gamma\dot x\dot y=0,\\
\frac{dE}{dt}=-\zeta_1\dot x^2-\zeta_2\dot y^2, \qquad\text{where}\qquad E=\frac14\left(2\dot x^2+2\dot y^2+2k_{11}x^2+2k_{21}y^2+k_{12}x^4+k_{22}y^4\right).
$$
So now we know for sure, that if both $\zeta_1,\zeta_2>0$ (or both $<0$), then energy is strictly decreasing (or increasing) and therefore there are no periodic orbits.
However, if $\zeta_1=\zeta_2=0$ or $\zeta_1\zeta_2<0$, then with some parameters there can be periodic orbits.
For example, if $k_{11}=k_{21}=1$ and everything else is zero, it's just two harmonic oscillator and every orbit is periodic. Or if $\zeta_2=-\zeta_1=1$, all $k=0$ and $\gamma=1$, then for $z=(\dot x, \dot y)$ it is linear equation $\dot z=Az$ with pure imaginary eigenvalues. Again, every orbit is periodic.