My calculus book says that if $x\le-1$ then: $$\sec^{-1}{x}=\pi-\sin^{-1}\frac{\sqrt{x^2-1}}{x}$$
I have limited experience with mathematics, but my calculator disagreed with the above statement. Shouldn't it be:
$$\sec^{-1}{x}=\pi+\sin^{-1}\frac{\sqrt{x^2-1}}{x}$$
For reference: it is stated in exercize 48 of Chapter 3.5 from the book "Calculus, a complete course", 8th edition.
It depends on how the “arcsecant” is defined.
While the values for $x\ge1$ are “naturally” taken in $[0,\pi/2)$ one can choose to take the values for $x\le-1$ in $(\pi/2,\pi]$ or in $[\pi,3\pi/2)$.
Some people make the latter choice, which has an impact on how the derivative of the arcsecant is represented, see Definition of $\operatorname{arcsec}(x)$ (which is a question of yours, by the way).
As explained there, the derivative of the arcsecant, with the latter definition, is $$ \frac{1}{x\sqrt{x^2-1}} $$ and the derivative of $\arcsin\frac{\sqrt{x^2-1}}{x}$ (for $x\le-1$) is $$ \frac{1}{\sqrt{1-\dfrac{x^2-1}{x^2}}} \frac{\dfrac{x^2}{\sqrt{x^2-1}}-\sqrt{x^2-1}}{x^2} =\sqrt{x^2}\frac{1}{x^2\sqrt{x^2-1}} =-\frac{1}{x\sqrt{x^2-1}} $$ This means that, for $x\le-1$, $$ \sec^{-1}x=c-\sin^{-1}\frac{\sqrt{x^2-1}}{x} $$ and it's easy to see that $c=\pi$.
So, apparently, your textbook is using the definition of the arcsecant as taking its values in $[0,\pi/2)\cup[\pi,3\pi/2)$.
You are right, but let's think about why this is:
If $\sec \theta \leq -1$, then $-1 \leq \cos \theta < 0$, which implies Quadrant II or III. Thus, since the range of $\sec^{-1}$ is $[0, \pi]-\{\frac \pi 2\}$ (Quadrant I or II), we must have $\theta \in (\frac \pi 2, \pi]$ (Quadrant II)
Now, if $x \leq -1$, then $-1 < \frac{\sqrt{x^2-1}}{x}=\sin \alpha \leq 0$, which implies Quadrant III or Quadrant IV. Thus, since the range of $\sin^{-1}$ is $[-\frac \pi 2, \frac \pi 2]$ (Quadrant I or IV), we must have $\alpha \in (-\frac \pi 2, 0]$ (Quadrant IV).
Now, let's consider a few values of $x$: $$x=-1 \implies \sin \alpha=0 \wedge \sec \theta=-1 \implies \alpha=0 \wedge \theta=\pi$$ $$x=-\frac{2}{\sqrt 3} \implies \sin \alpha=-\frac 1 2 \wedge \sec \theta=-\frac{2}{\sqrt 3} \implies \alpha=-\frac \pi 6 \wedge \theta=\frac{5\pi}{6}$$
As you see in our two examples, both $\alpha$ and $\theta$ have the same reference angles, but $\alpha$ is in Quadrant IV while $\theta$ is in Quadrant II. Thus, we just need to shift $\alpha$ into Quadrant II by rotating it $\pi$ radians, so we get $\alpha+\pi=\theta$, or: $$\sin^{-1} \frac{\sqrt{x^2-1}}{x}+\pi=\sec^{-1}x$$
Notice that $\dfrac{\sqrt{x^2-1}} x > 0$. And one expects $\sec^{-1}(\cdot) \le \pi$.
Draw the graph of the secant function. It is not one-to-one, so one restricts the domain of the secant function to get a one-to-one function whose inverse is called the inverse secant or arcsecant. The restriction by convention is the interval $[0,\pi]$, or $[0,\pi]\setminus\{\pi/2\}$ if you don't want $\infty$ to be one of the values of the secant function. (This is not $+\infty$ or $-\infty$, but a single $\infty$ that is approached by going in either the positive direction or the negative direction, thus making the secant function continuous on $\mathbb R$ and the restriction of the secant function continuous on $[0,\pi]\setminus\{\pi/2\}$.)
Therefore the range of the arcsecant function, or inverse secant function, is included in the set $[0,\pi]$.
