Given a matrix A, how to measure its similarity (row and column permutations can be performed on A) to triangular form, here the triangular form is like
\begin{align}
M=\begin{bmatrix}
1 & 1 & 1 & 1 \\
0 & 1 & 1 & 1 \\
0 & 0 & 1 & 1 \\
0 & 0 & 0 & 1
\end{bmatrix}
\end{align}
This triangular form is different from upper triangular matrix. Besides requiring all the entries below the main diagonal are zero, I'd like the entries above the main diagonal are 1 as much as possible.
For example, the similarity between A,B,C and M should satisfy $sim(A,M)>sim(B,M)>sim(C,M)$
A has the highest similarity, which is actually same as M after swapping column 1 and 4.
\begin{align} A = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & 1 & 1 & 0 \\ 1 & 0 & 1 & 0 \\ 1 & 0 & 0 & 0 \end{bmatrix} & B = \begin{bmatrix} 1 & 1 & 1 & 0 \\ 0 & 1 & 1 & 1 \\ 0 & 0 & 1 & 1 \\ 0 & 0 & 0 & 1 \end{bmatrix} & C = \begin{bmatrix} 1 & 1 & 1 & 0 \\ 0 & 1 & 1 & 0 \\ 0 & 0 & 1 & 1 \\ 0 & 0 & 0 & 1 \end{bmatrix} \end{align}