What if $f(x)$ is neither increasing nor decreasing.
The answer is NO. Indeed, here is a counterexample.
First, partition the interval $(0,\infty)$ into pairs of two elements.
This can be done for example, by paring $x \in (0, \infty)$ with $\frac{1}{x}$, as long as they are both not integers, and pairing $n$ with $\frac{1}{n+1}$ (the last one to avoid the fact that $x=\frac{1}{x}$ for $x=1$).
This way, we get a family $\{ A_i\}_{i \in I}$ of sets, with the following properties:
- $|A_i|=2$ for each $i$.
- $A_i \cap A_j= \emptyset$ for all $i \neq j$.
- $\bigcup_{i \in I} A_i =(0,\infty)$.
Now, define $f: \mathbb R \to \mathbb R$ the following way: First set $f(0)=0$.
Next, for each $i \in I$, if $A_i=\{ a,b\}$ define
$$f(a)=b \\
f(b)=-a \\
f(-a)=-b \\
f(-b)=-a$$
Then $f \circ f(x)=-x$ for all $x \in \mathbb R$.
By chosing the right $A_i$ you can make $f$ discontinuous, but $f \circ f$ is continuous and decreasing.
P.S. A simpler way to pair $(0, \infty)$ is by pairing the intervals $(2n, 2n+1]$ and $(2n+1, 2n+2]$ via the $x \to x+1$.
Then, if I didn't make a mistake, here is your function
$$f(x)=
\left\{
\begin{array}{lc}
f(0)=0 & \\
f(x)=x+1 & \mbox{ if } \lceil x \rceil \mbox{ is positive and odd } \\
f(x)=-x+1 & \mbox{ if } \lceil x \rceil \mbox{ is positive and even} \\
f(x)=x-1 & \mbox{ if } \lfloor x \rfloor \mbox{ is negative and odd} \\
f(x)=-x-1 & \mbox{ if } \lfloor x \rfloor \mbox{ is negative and odd} \\
\end{array}
\right. $$
Here $\lceil x \rceil$ and $ \lfloor x \rfloor$ are the ceiling and floor functions.
Added Note that actually you can prove something simpler, and probably this is what the question asks:
Lemma If $f \circ f$ is strictly decreasing, then $f$ cannot be continuous on $\mathbb R$.
Proof: Assume by contradiction that $f$ is continuous. Since $f\circ f$ is one to one, $f$ must be a one-to-one function.
Since $f$ is one-to-one and continuous, by a simple application of the Intermediate value Theorem it is monotonic.
But then, by the argument you made $f$ is increasing, contradiction.