Everyone is just being a bit vague in their terminology. Taken literally, elementary functions are not necessarily holomorphic, e.g. $|z| = \sqrt{z^2}$ is not differentiable at $0$. For that matter, rational functions are not holomorphic, they're meromorphic, e.g. $1/z$ blows up at $z=0$. These functions don't typically have inverses, e.g. $\cos(z)$ is not invertible even when restricted to real $z$. More seriously, $z^2$ isn't even locally invertible in any neighborhood of $z=0$. You've got to make some choices when picking inverses and try to "patch them together" in some consistent way, e.g. by choosing a branch cut of the complex logarithm.
These issues aren't particularly serious, though. The building blocks of polynomials and exponential functions are holomorphic everywhere. Holomorphic functions are locally invertible at a point (moreover with holomorphic local inverse) iff their derivative is non-zero there. For polynomials you get finitely many derivative-zero points, and in general they'll be isolated. So however you interpret "inverses" of polynomials and exponentials, you're going to come up with things that are locally holomorphic "most of the time". Things get sketchy when you define something like the complex logarithm with a branch cut, which will give you a whole line's worth of discontinuities.
Any reasonable definition of "elementary function" should result in things that are at least locally holomorphic at some point of their domain. $\sqrt{z}$ is indeed, despite having a branch cut and being badly behaved at $z=0$. Complex conjugation is nowhere holomorphic, so there's just not going to be a way to define it as an elementary function.
