From what I understand what you're trying to do is equivalent to: treat numbers that differ by a multiple of $180$ as equivalent, and consider the distance of $x$ and $y$ to be the minimum of $|x' - y'|$ such that $x'$ is equivalent to $x$ and $y'$ is equivalent to $y$.
The modulo operation is designed so that x % 180 returns the unique number $x'$ so that $0 \le x' < 180$ (at least if $x \ge 0$, conventions might differ if $x$ is allowed to be negative). For $0 \le x < 360$, the code in the question is performing the exactly same operation. Applying this operation ensures that the outputs are equal exactly when the inputs are equivalent. This more or less solves the first problem, but complicates the second a little.
The issue, as you've identified, is that for numbers near but on either side of the wraparound threshold, their shifted copies may not be close to each other. The direct distance between $x$ and $y$ is $|x - y|$, while the distance while wrapping around is $180 - |x - y|$ (if you're not convinced of this, see below for a proof). So $$\min (|x - y|, 180 - |x -y|)$$ is a correct formula for the distance you want. Your other formula involving $\cos$ and $\arccos$ is actually equivalent to this, using that $\cos(180 - \theta) = - \cos(\theta)$ and $\cos(\theta) = \cos(|\theta|)$ (where $\theta$ is in degrees).
A proof that the formula is correct
To start with an example, when comparing $179$ with $1$, we want to actually compare $179 \equiv 359 \equiv 539 \equiv -1 \equiv \dots$ with $1 \equiv 181 \equiv 361 \equiv 541 \equiv -179 \equiv \dots$. But because the distance is at most $180$ anyway, it is enough to actually just compare $179$ with $1$, $179$ with $181$ and $359$ with $1$ (and take the minimum distance).
In general, to compare $x$ with $y$ (which are in the range $0 \le x, y < 180$), you want to compare $x$ with $y$, $x + 180$ with $y$ and $x$ with $y + 180$ in absolute value. But we can find a single formula that deals with the last two cases. Using $180 - x \ge 0$ and $180 - y \ge 0$,
$$|(x + 180) - y | = |x + (180 - y)| = x + 180 - y = 180 + (x - y)$$
and
$$|x - ( y + 180)| = |-180 + x - y| = |180 - x + y| = |(180 - x) + y| = 180 - x + y = 180 - (x-y).$$
So these two values are $180 \pm |x - y|$ in some order. Obviously the smaller of these is $180 - |x - y|$, so since we're looking for the minimum, we can replace this pair with the single formula $180 - |x - y|$.
0 <= x < 360and I can check for angle>= 180instead of using>, that was just a typo, but I'm wondering whether there is a general formula for doing this conversion in trigonometry rather than writing my own by just thinking through the various cases. – Hugh_Kelley Feb 27 '23 at 21:14