What I tried was pretty basic.
$$A=2xy=2x\cos x$$
$$A'=2\cos x-2x\sin x$$
$$A' = 0 \implies 2\cos x-2x\sin x=0$$
$$x=\frac{\cos x}{\sin x}=\cot x$$
But I got stuck here. I didn't know what to do next.
You cannot express the solution of this equation by means of the usual functions, and you need to resort to so-called numerical methods that let you evaluate approximate values.
Consider the sequence of numbers
$$x_{n+1}=\arctan\frac1{x_n},$$ starting from $x_0=1$. If these values get closer and closer to a certain $x$, then you can say that "in the end",
$$x=\arctan\frac1x,$$ which is the same as $x=\cot x$. Have a try with a calculator.
You will later learn under what conditions such a process can converge to the solution of an equation (and why I chose this particular form of recurrence). You will also learn faster methods.
before they told us not to anywaysThat's probably when someone realized that there is no "nice" closed form solution. You can prove that $x = \cot x$ has a unique solution in $(0,\pi / 2)$ and that value gives the maximum area, but that's about as far as you can push it symbolically i.e. without solving the equation numerically. – dxiv Dec 14 '16 at 23:07