Let A$_{3×2}$ and B$_{2×3}$ be matrices such that their product $AB$ is $$AB=\begin{pmatrix} 8&2 & -2\\ 2&5&4 \\ -2&4&5 \\ \end{pmatrix}$$ And $BA$ is nonsingular
Find the determinant of $BA$.
I have no idea , how to solve this type of question. All I could notice is that $|AB| = 0$ and it's a symmetric matrix. I tried assuming a general matrix , but I get simply too many unknowns and very few equations.
A direct calculation is also possible in case one wants to find out the answer easily. If we denote the coefficients of $A$ by $a_i$ and the ones of $B$ by $b_j$, and the given matrix by $C$, then the matrix equation $AB=C$ is equivalent to equations in $a_i,b_j$. We can solve them case by case. The first equation is $a_1b_1 + a_2b_4 =8$. For $a_1=0$ we obtain $a_2\neq 0$ and $$ b_4=\frac{8}{a_2}, b_5= \frac{2}{a_2}, b_6=-\frac{2}{a_2}, a_4= \frac{a_2a_3b_3 - 4a_2}{2}, b_3= -\frac{1}{a_3(a_3b_2 - 9}, b_2= \frac{1}{(4a_3)(a_3b_1 + 18)}, a_6= -\frac{a_2a_5b_1 + 2a_2}{8}, a_5=a_3. $$ Then we obtain $$ BA=\begin{pmatrix}9 & 0 \cr 0 & 9\end{pmatrix}. $$ The other case $a_1\neq 0$ is similar. Note that $\det(AB)\neq \det(BA)$ in general, but $tr(AB)=tr(BA)$ is true in general. So the determinant is $81$.
There is one really simple solution. Key fact (proof below): If $A$ is $m\times n$, $B$ is $n\times m$ and $n \geq m$ then $\chi_{BA}(x) = \lambda^{n-m}\chi_{AB}(x)$ where $\chi_M(x)$ denotes a characteristic polynomial $\chi_M(x) = det(x - M)$. Using this fact your task is solved this way.
Proof:
First, lets show another subfact: for square matrices $A$ and $B$ holds $det(AB - x) = det(BA - x) \Leftrightarrow \chi_{AB}(x) = \chi_{BA}(x)$.
If $det(A) \neq 0$ then it follows from $det(AB - x) = det(A^{-1}A)det(AB - x) = det(A^{-1})det(AB - x)det(A) = det(BA - x)$.
If $det(A) = 0$ there are finite number of such $s \in \mathbb R$ that $\chi_A(s)=0$ because $\chi_A(s)$ is a finite-degree polynomial. Then there are infinite number of such $s$ that $\chi_A(s) \neq 0$. For all such $s$ we know $\chi_{(A-s)B}(x) = \chi_{B(A-s)}(x)$ as a result of a previous case. For every fixed $x$ we see two finite-degree polynomials ($x$ is fixed, $s$ is variable) $\chi_{(A-s)B}(x)$ and $\chi_{B(A-s)}(x)$ which are equal in infinite number of points. Then we conclude they are equal at every $s$. At $s = 0$ we get the result $\chi_{AB}(x) = \chi_{BA}(x)$ at every $x$.
Subfact is done!
Main fact proof now:
Consider $n\times n$ matrices $A' = \left(\dfrac{A}{0}\right)$ and $B' = (B\mid0)$. We just put zero rows and columns to make matrices $n\times n$.
First, $B'A' = BA \Rightarrow x - B'A' = x - BA \Rightarrow \chi_{B'A'}(x) = \chi_{BA}(x)$
Second, $A'$ and $B'$ are square matrices. Then due to the subfact we have $\chi_{B'A'}(x) = \chi_{A'B'}(x)$.
Third, $\chi_{A'B'}(x) = det(x - A'B') = det\begin{pmatrix}x - AB & 0 \\ 0 & \begin{matrix}x & 0 & \ldots & 0 \\ 0 & x & \ldots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \ldots & x \\\end{matrix}\end{pmatrix} = det(x - AB)x^{n - m} = x^{n-m}\chi_{AB}(x)$
So, we see $\chi_{BA}(x) = \chi_{B'A'}(x) = \chi_{A'B'}(x) = x^{n-m}\chi_{AB}(x)$ and we are done!
The characteristic polynomial of $AB$ is $$ p(x)=\left| \begin{array}{rrr} 8-x & 2 & -2 \\ 2 & 5-x & 4 \\ -2 & 4 & 5-x \end{array} \right|=\cdots =-x^3+18x^2-81x=-x(x-9)^2 $$ Cayley-Hamilton Theorem implies that $p(AB)=0$, i.e., $$ -(AB)^3+18(AB)^2-81AB=0 $$ In fact, since $AB$ is symmetric, it is diagonalisable and hence its minimal polynomial is $m(x)=x(x-9)$.
Hence $$ (AB)^2-9AB=0 \quad\Longrightarrow\quad B(AB)^2A-9BABA=0 \quad\Longrightarrow\quad (BA)^3-9(BA)^2=0. \quad\Longrightarrow\quad (BA)^2\big((BA)-9\big)=0. $$ Clearly, rank$(AB)=2$, and also rank$((AB)^2)=2$ (as $(AB)^2$ is also symmetric with eigenvalues $81,81,0$) and hence rank$(BA)=2$, and thus $BA$ is invertible and thus $$ (BA)^2\big((BA)-9\big)=0\quad\Longrightarrow\quad (BA)-9=0, $$ i.e., $$ BA=\left(\begin{array}{cc} 9&0 \\ 0&9\end{array}\right) $$ and hence det$(BA)=81$.
