Consider the following homogeneous equation where $A$ and $X$ are matrices.
$$AX = 0$$
I want to know whether there are non trivial solutions for this equations. Now, if $A^{-1}$ exists, then I can multiply throughout by it and get $X = 0$, so if $A$ is invertible, only the trivial solution exists. However, I do not understand why $A$ being non-invertible would imply that non-trivial solutions exists, shouldn't it just imply that no solutions exist?
I'm going to assume that your $\mathbf A$ is a square matrix (or else $\mathbf A^{-1}$ doesn't really make sense).
An alternative formula for the matrix multiplication between a $n\times n$ matrix and a $n\times 1$ matrix is a sum of the scalar multiples of $n\times 1$ matrices. That is, let $\mathbf A=\begin{bmatrix} \vec a\ \vec b\ \vec c\end{bmatrix}$, where $\vec a, \vec b, \vec c$ are column vectors, and let $\vec x = \begin{bmatrix} x \\ y \\ z\end{bmatrix}$. Then $\mathbf A\vec x = x\vec a + y\vec b + z \vec c$.
But if $\mathbf A$ is not invertible, then the vectors $\vec a, \vec b, \vec c$ are linearly dependent. So then there will be nontrivial solutions to $x\vec a + y\vec b + z \vec c=\vec 0$ and thus to $\mathbf A\vec x=x\vec a + y\vec b + z \vec c=\vec 0$.
Consider the general case $Ax = b$. If $A$ is non-invertible - ie $\det(A) = 0$ - then we have two cases:
(i) There are no solutions non-trivial (ie $x \neq 0$) solutions - this happens when $b$ is not in the image of $A$.
(ii) There are infinitely many solutions given by a subspace.
(i) There may be a contradiction for non-zero $x$ such as the following: $$ A = \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}, b = \begin{pmatrix} 1 \\ 2 \end{pmatrix}$$
$$\Rightarrow \begin{matrix} x_1 + x_2 = 1 \\ x_1 + x_2 = 2 \end{matrix} \Rightarrow 1 = 2$$
which is clearly a contradiction. This is because $b \notin Im(A) = \{Ax | x \in V\}$, where $V$ is your vector space.
(ii) If there exists a solution, then there exists a subspace of solutions. Suppose that $x$ satisfies $Ax = b$ (so $x \neq 0$). Since $\det(A) = 0$, we can find a vector $y \neq 0$ such that $Ay = 0$. As such, $A(x + \lambda y) = Ax + \lambda Ay = b$ for all scalars $\lambda$. This gives a 1D subspace of your vector space that satisfies the equation. Similarly, if you can find a vector $z \neq 0 $ such that $Az = 0$ and $y$ and $z$ are linearly independent (ie we do not have $y = cz$ for a constant $c$, recalling that $y,z \neq 0$), then $A(x + \lambda y + \mu z) = Ax + \lambda Ay + \mu z= b$. Thus any vector $v$ with $v \in \{b + \lambda y + \mu z | \lambda,\mu \text{scalars} \}$ satisfies $Av = b$. This gives a 2D subspace of solutions. We can then continue similarly (if su
I assume you mean $A$ is square, say $n \times n$.
If you already know a good deal about linear mappings, then there is a relatively simple explanation. Let $T : R^n \to R^n$ be the linear mapping associated with $A$. $A$ being invertible is equivalent to $T$ being bijective. On the other hand, the existence of a nonzero $X$ such that $AX = 0$ is equivalent to the kernel of the $T$ being nontrivial, which in turn is equivalent to $T$ being injective. But for a linear mapping $T$ between vector spaces of the same finite dimension, $T$ is injective if and only if it is bijective.
However, if your course has focused mainly on computations with matrices, then it is not that obvious. A proof might be as follows.
Try to apply the inversion algorithm to $A$. Since $A$ isn't invertible, it can't be changed to $I$ using row operations.
Now solve the system $AX = 0$ by row-reducing $\pmatrix{A & 0}$. Since $A$ can't be transformed to the identity matrix, this will result in a system with at least one parameter, hence at least one nontrivial solution $X$.
