For whatever reason, the scipy implementation of affine transformation requires an inverse transformation matrix: https://docs.scipy.org/doc/scipy/reference/generated/scipy.ndimage.affine_transform.html
How can I know in advance whether a certain combination of scaling/rotation/shearing will give me a non-invertible matrix, and what would that look like geometrically?
A non-invertible transformation collapses the space along some direction(s). If you know about eigenvalues and eigenvectors, the eigenvectors of $0$ give you the directions in which this collapse happens.
Rotations are isometries, so they never contribute to making a transformation non-invertible. A shear shouldn’t cause any collapse, either. It shifts things parallel to some direction, which keeps areas/volumes constant, so there’s no collapse there, either. That leaves scaling as the likely culprit, but that’s easy to detect: the result is not invertible iff any of the scale factors is zero.
