Don't try to convert from signed-magnitude to two's complement. As you said, you can't express $32$ in $6$-bit signed magnitude. Just express it directly.
Your place values in unsigned $n$-bit binary are: $2^{n-1}, 2^{n-2}, \ldots, 2, 1$. But this only allows positive numbers. In two's complement, the sign of the leading bit is reversed. So your place values for 6-bit two's complement are: $-32, 16, 8, 4, 2, 1$. This allows you to express numbers from $-32$ to $31$, and lets you add and subtract numbers with different signs normally. In this case, $-32$ is really easy to express, it's just $100000$.
This also makes the whole $-x == \:\: \sim\! x + 1 == \:\: \sim\!(x - 1)$ trick make more sense.