Given graph of $f(x)=1/x$, i.e. $G=\{(x,y)^t:xy=1\}$, I want to show that rotation of this grph by 45° clockwise gives us a hyperbolic graph.
This means I need to come to something of form $\{(x,y)^t: {x^2 \over a^2}-{y^2\over b^2}=1\}$. How do I calculate $a$ and $b$?
I wanted to roteate the original graph for 315° counter-clockwise (positive direction) using rotation matrix $A=\frac{1}{\sqrt{2}}\pmatrix{1 & 1 \\ -1 & 1}$, but don't know how to proceed.
With your rotation you have $$\pmatrix {x' \\ y'}=\frac{1}{\sqrt{2}}\pmatrix{1 & 1 \\ -1 & 1}\pmatrix {x \\ y}$$ Use this to get $x=f(x',y'), y=g(x',y')$ Substitute those into $xy=1$ and you will get an equation in $x',y'$ that should be what you want.
You're on the right track. Notice that you can parametrize the the original curve as $\left(x, 1/x\right)$. Now just apply the transformation to get $$ \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ -1 & 1 \end{pmatrix} \begin{pmatrix} x \\ 1/x \end{pmatrix}\\ = \frac{1}{\sqrt{2}} \begin{pmatrix} \frac{1 + x^2}{x}\\ \frac{1 - x^2}{x} \end{pmatrix}. $$ This is the parametric equation of the new hyperbola. Put this into the hyperbola equation to get $$ \frac{1}{2a^2}\left(\frac{x^4 + 2x^2 + 1}{x^2}\right) - \frac{1}{2b^2}\left(\frac{x^4 - 2x^2 + 1}{x^2}\right) = 1 \\ \implies b^2(x^4 + 2x^2 + 1) - a^2(x^4 - 2x^2 + 1) = 2a^2b^2x^2. $$ Compare the terms of the two polynomials on both sides to find $a$ and $b$.
