$$(x^3-3x+1)\sqrt{x^2-1}+x^4-3x^2+x+1=0$$
$$(x^3-3x+1)\sqrt{x^2-1}=-(x^4-3x^2+x+1)$$
$$(x^3-3x+1)^2(x^2-1)=(x^4-3x^2+x+1)^2$$
If you expand everything, and then subtract the two sides of the equation, you get
$$x^6-4x^4+2x^3+3x^2-4x+2 = 0$$
By the Rational Root Theorem, potential rational roots are $\pm 1$ and $\pm 2$, of which it turns out that $x = 1$ is the only one that works. Divide the polynomial by $x - 1$.
$$x^5+x^4-3x^3-x^2+2x-2=0$$
Unfortunately, there's no Quintic Formula. And no rational roots. But maybe we can find a quadratic factor somehow.
$$(x^3+ax^2+bx+c)(x^2+dx+e)=0$$
$$x^5+(a+d)x^4+(b+ad+e)x^3+(c+bd+ae)x^2+(cd+be)x+ce=0$$
It's a bit tedious, but we can match up the coefficients and solve the system of 5 equations for 5 variables, ultimately getting
$$(x^3+x^2-x+1)(x^2-2)=0$$
The quadratic factor obviously gives the roots $x = \pm \sqrt{2}$.
The cubic, AFAICT, has no simple solution, so I solved it numerically to get $x \approx -1.8392867552141612$. However, this turns out to be an extraneous solution that does not solve the original equation. The other two roots of the cubic are complex numbers, so I assume you're not interested in them.
Therefore, $x \in \lbrace 1, \sqrt{2}, -\sqrt{2} \rbrace$.