Apparently my confusion was in assuming that if a diagonal contains a 0, then both its row and column must contain only 0's.
This is not the case. Here are the rules for creating a Hermite canonical matrix
- Diagonal must consist of only ones and zeros.
- Everything below the diagonal must be zeros.
- If a diagonal element is 0, then every other element in that row must also be 0. (Note that in this case, any value may be above the diagonal 0 in the column.)
If a diagonal element is 1, then every other entry in that column (above and below) must be 0.
Here is an example of a matrix in Hermite canonical form:
1 0 2 0
0 1 3 0
0 0 0 0
0 0 0 1
And multiplying this by itself indeed yields the same matrix.
And it is indeed easy to verify that H*H=H, because the 1 diagonal elements obviously produce the same row (as with the identity matrix I) and the 0 diagonal elements will produce a zero row in the product, since the row in that position is the sum of the row vectors with each row multiplied by the value in the row, i.e, all zeros.