It is possible to give a very quick answer if we use this nice property of the ellipse:
the locus of the intersections of perpendicular tangents to an
ellipse is a circle called director circle,
and the square of its radius is the sum of the squares of
the ellipse semi-axes.
Tangents drawn from any point on the director circle, and from its reflection about the center, form then the sides of a circumscribed rectangle. If we choose those points as the intersections between director circle and axes of the ellipse, the rectangle is by symmetry a square.
In your particular case the vertices of the circumscribed square lie then at points $(0,\pm\sqrt{10})$ and $(\pm\sqrt{10},0)$.