There is a result apparently known as Jacobi's formula for the minors of an inverse matrix. We need to define the following notation:
- For subsets $I,J \subset \{1,\dots,N\}$, $A_{I,J}$ denotes the submatrix of $A$ using only the $i$th rows and $j$th columns of $A$ for $i \in I$ and $j \in J$
- $I^c$ denotes the complement of $I$ (i.e. $I^c =\{i : 1 \leq i \leq n, i \notin I\}$)
- $\sigma(I)$ denotes the sum of all elements of $I$
With that: if $I$ and $J$ are sets of $2$ elements, we have
$$
\det(B_{I,J}) = (-1)^{\sigma(I) + \sigma(J)}\frac{\det A_{J^c,I^c}}{\det(A)}.
$$
For $N = 3$, this gives you the formula you found. Taking $I = \{i,k\}$ and $J = \{j,k\}$ yields
$$
\det(B_{I,J}) = b_{ij}b_{kk} - b_{ik}b_{kj},\\
(-1)^{\sigma(I) + \sigma(J)}\frac{\det A_{J^c,I^c}}{\det(A)} = (-1)^{i + j + 2k} \frac{a_{ij}}{\det(A)} =
(-1)^{i + j} \frac{a_{ij}}{\det(A)}.
$$
For $N = 4$, here is an example. Taking $I = \{1,2\}$ and $J = \{3,4\}$ yields
$$
\det(B_{I,J}) = b_{ip}b_{jq} - b_{iq}b_{jp},\\
(-1)^{\sigma(I) + \sigma(J)} \frac{\det A_{J^c,I^c}}{\det(A)} =
\frac{(-1)^{1+2+3+4}}{\det(A)}(\det A_{J^c,I^c}) =
\frac 1{\det(A)} (a_{13}a_{24} - a_{14}a_{23}).
$$