You could put it like this (there are many different but "equivalent" ways to express it): the mod(12) of an integer $k$ is the unique number $m$ between $0$ and $11$ that can be the rest of the division of $k$ by $12$. For instance, the mod(12) of any number between $0$ and $11$ is just the number itself, and then it starts again: the mod(12) of $12$ is $0$, the mod(12) of $13$ is $1$,..., the mod(12) of $136$ is $4$ since $136 = 12 \cdot 11 + 4$. For negative numbers, you still take a positive mod. For instance, the mod(12) of $-1$ is $11$, since $-1 = -1 \cdot 12 + 11$.
For your equation, you can compute every step in mod(12), for example $7^2 = 1$ since in $\mathbb{Z}$ the number $7^2$ is $49$ which has $1$ as mod(12).
Another way would be to do the whole computation and then do mod(12) only at the end (but you should explain why this works).
In the case of $1$, your computation is actually almost correct: you never reach the number $12$, so everything is fine and you can compute as usual. You should just write $[1]^2 + [9]\cdot[1]-[10] = [1]+[9]-[10]=[10]-[10]=[0]$, as desired.
Now you just need to check all the other 11 possibilities. For instance, 7 is not a solution, since $[7]^2 + [9]\cdot[7]-[10] = [1]+[3]-[10] = [6] \neq [0]$.