I'm trying to figure out a (computationally efficient) way to determine whether, given the x and y coordinates of points A, B, and C, both the angle going from A to B to C and the angle from A to C to B are less than 90 degrees.
Basically, I want to determine whether Point A falls within the green area or the red area in this image
.
I could think of some convoluted ways to do this, but I feel like there should be a simpler one. Thanks!
$\angle ABC<90^\circ$ if and only if $\vec{BC}\cdot\vec{BA}>0$.
That is, $(x_c-x_b)(x_a-x_b)+(y_c-y_b)(y_a-y_b)>0$
Observe that the line through $B,C$ has the slope $$m=\frac{y_c-y_b}{x_c-x_b}$$ Therefore, the lines $l_1$ and $l_2$, both perpendicular to $BC$ and through $B$ and $C$ respectively can be defined as follows
\begin{align*}l_1: f(x)&=\frac{x_c-x_b}{y_b-y_c}\cdot x+y_b-\frac{x_c-x_b}{y_b-y_c}\cdot x_b=\frac{x_c-x_b}{y_b-y_c}\cdot (x-x_b)+y_b\\l_2:g(x)&=\ldots=\frac{x_c-x_b}{y_b-y_c}\cdot (x-x_c)+y_c\end{align*} Hence, $A$ will lie between $l_1$ and $l_2$ if
$$f(x_a)<x_a<g(x_a)$$
Observation This method assumes that $x_b<x_c$. If $x_c<x_b$, simply change $B$ for $C$ and vice-versa in the calculations. What happens if $x_c=x_b$?
Define $$a:=\sqrt{(x_c-x_b)^2+(y_c-y_b)^2}\qquad b:=\ldots \qquad c:=\ldots$$
Observe that, in any triangle, $\angle ABC$ is acute, if and only if $$\cos(\angle ABC)=\frac{a^2+c^2-b^2}{2ac}>0$$
By your method:
Shift the origin to $B$. Rotate the axes by the original slope of line $BC$, and check if $y$ coordinate of $A$ is positive and less than $y$ coordinate of $C$ in the new transformed coordinate system.
Vector dot product decides obtuse/acute condition. Tail of arrow of vectors is at $A:$
If sign $(\vec{CA}\cdot\vec{BA})>0,$ then $\angle A$ is acute
If sign $(\vec{CA}\cdot\vec{BA})<0,$ then $\angle A$ is obtuse
If $(\vec{CA}\cdot\vec{BA})=0$ then $\angle A$ is a right angle
Apply the above Rule successively at vertices $B$ and $C$.
Your question is a bit misleading, since it hints to a possible solution of your problem. To solve your problem this way, you would apply, e.g., the answer by CY Aries twice. But your original problem is to determine whether point A is within the green strip (defined by points B and C) or not. To solve it, you can calculate $\vec{BA}\cdot\vec{BC}$ and compare it to $\vec{BC}\cdot\vec{BC}$, i.e., to $\Vert\vec{BC}\Vert^2$. The advantage is that you don't need to repeat this last calculation for any further point to be checked. In other words, if you have $n$ points like A, you only need to calculate $n+1$ dot products instead of $2n$.
Remember that, in 2D (just add terms for $z$ in 3D), $$\vec{BA}\cdot\vec{BC} = (x_a-x_b)(x_c-x_b)+(y_a-y_b)(y_c-y_b)$$ and $$\Vert\vec{BC}\Vert^2 = (x_c-x_b)^2+(y_c-y_b)^2$$ Referring to your image, $$\vec{BA}\cdot\vec{BC} < 0 \implies A \in red_{left} $$ $$\vec{BA}\cdot\vec{BC} = 0 \implies \vec{BA} \perp \vec{BC} \lor A \equiv B$$ $$0 < \vec{BA}\cdot\vec{BC} < \Vert\vec{BC}\Vert^2 \implies A \in green $$ $$\vec{BA}\cdot\vec{BC} = \Vert\vec{BC}\Vert^2 \implies \vec{CA} \perp \vec{CB} \lor A \equiv C$$ $$\vec{BA}\cdot\vec{BC} > \Vert\vec{BC}\Vert^2 \implies A \in red_{right} $$
