Problem
Find functions $f(x)$ and $g(x)$ such that $f(g(x))=x^2-2x-4$ and $g(f(x))=x^2-6x+6$.
Finding some points
Notice that $f(g(f(x)))=f^2(x)-2f(x)-4=f(x^2-6x+6)$ and the equation $x^2-6x+6=x$ has roots $1$ and $6$. This gives $f^2(1)-2f(1)-4=f(1)$ and $f^2(6)-2f(6)-4=f(6)$. The former gives $f(1)=4$ or $f(1)=-1$ while the latter gives $f(6)=4$ or $f(6)=-1$.
Note that $g(f(1))=1$ and $g(f(6))=6$. We have [($f(1)=4$ and $g(4)=1$) or ($f(1)=-1$ and $g(-1)=1$)] and [($f(6)=4$ and $g(4)=6$) or ($f(6)=-1$ and $g(-1)=6$)].
Similarly, notice that $g(f(g(x)))=g^2(x)-6g(x)+6=g(x^2-2x-4)$ and the equation $x^2-2x-4=x$ has roots $-1$ and $4$. This gives $g^2(-1)-6g(-1)+6=g(-1)$ and $g^2(4)-6g(4)+6=g(4)$. The former gives $g(-1)=1$ or $g(-1)=6$ while the latter gives $g(4)=1$ or $g(4)=6$.
Note that $f(g(-1))=-1$ and $f(g(4))=4$. We have [($g(-1)=1$ and $f(1)=-1$) or ($g(-1)=6$ and $f(6)=-1$)] and [($g(4)=1$ and $f(1)=4$) or ($g(4)=6$ and $f(6)=4$)].
Polynomial solutions
Assume that $f(x)$ and $g(x)$ are polynomials. Since $f(g(x))$ and $g(f(x))$ are monic and of degree $2=1\cdot 2$. One of $f(x)$ and $g(x)$ is monic and of degree $2$ and the other is monic and of degree $1$.
Case 1: $f(x)$ is of degree $1$.
Let $f(x)=x+k$. Then $f^{-1}(x)=x-k$. Then $g(x)=f^{-1}(f(g(x)))=x^2-2x-4-k$ and $g(x)=g(f(f^{-1}(x)))=(x-k)^2-6(x-k)+6$. By compare the coefficient of the $x$ term, $k=-2$. Hence, $f(x)=x-2$ and $g(x)=x^2-2x-2$.
Case 2: $g(x)$ is of degree $1$.
Let $g(x)=x+k$. Then $g^{-1}(x)=x-k$. Then $f(x)=g^{-1}(g(f(x)))=x^2-6x+6-k$ and $f(x)=f(g(g^{-1}(x)))=(x-k)^2-2(x-k)-4$. By compare the coefficient of the $x$ term, $k=2$. Hence, $f(x)=x^2-6x+4$ and $g(x)=x+2$.
Question
Notice that the solution in Case 1 matches the point ($f(1)=-1$ and $g(-1)=1$) and the solution in Case 2 matches the points ($f(6)=4$ and $g(4)=6$).
My question is what solution (should be non-polynomial) of $f(x)$ and $g(x)$ matches the remaining points, namely ($f(1)=4$ and $g(4)=1$) and ($f(6)=-1$ and $g(-1)=6$).
