The formula $tan^{-1}$ does not denote $1/\tan$. It denotes the inverse tangent function, also known as "arctan" (i.e., the function that takes a number $t$ and tells you what angle (between $-\pi/2$ and $\pi/2$) has $t$ as its tangent. Thus $\arctan(1) = \pi/4$, for instance.
Also: if you use "floor", you get the wrong picture; what's probably wanted is the "fractional part", which is $x - floor(x)$. See this desmos plot:

As you'll notice (perhaps), I left out the "cot"; that gives as a result an actual "curved" saw-tooth. TO get an ordinary sawtooth,
$$
y=\frac{1}{2}-\arctan\left(\cot\left(\pi x\right)\right)
$$
suffices, as this plot shows. Note that no "floor" function is needed.
