How big a text file will be for a wordlist composed of 10 HEX characters given certain conditions?

Question

While discussing with another friend the insecurity with some wifi systems against dictionary attacks we couldn't reach a common ground on how big a WPA wordlist dictionary would be for the most common password we work with here.

Not knowing if my calculations are correct or not, I've reached the estimated size of ~14Tb while my friend says that it is over 20Tb. Without any way to be certain of who is doing the math correctly I have decided to ask here for help on this matter.

The question is as the title says; How big a file will be for a wordlist composed of 10 HEX characters (for each word, each character can be from 0 to 9 or A to F) with the following conditions:

• Each character cannot appear more than 3 times in the entire word
• The words are all capitalized, no lower caps

We have tried to calculate these values with Permutations and Combinations (and without them), but with no way to confirm the results we can't tell if we used the correct methods for the intended goal.

Answer 1

The trick is enumerating all of the ways that the repeat groups can be split up.

No repeating characters is easy: $KLMNOPQRST$.

Characters not repeating or repeating once is also easy:

$$KKLMNOPQRS, KKLLMNOPQR, KKLLMMNOPQ, KKLLMMNNOP, KKLLMMNNOO.$$

Characters not repeating, repeating once, or repeating twice gives the most.

If one character repeats twice:

$$KKKLMNOPQR, KKKLLMNOPQ, KKKLLMMNOP, KKKLLMMNNO.$$

If two characters repeat twice:

$$KKKLLLMNOP, KKKLLLMMNO, KKKLLLMMNN.$$

If three characters repeat twice:

$$KKKLLLMMMN.$$

And that's it for the groupings.

Now for each grouping, you have to choose which hex characters, and then choose their order.

You specify that a "hex character" is a hexadecimal digit between $0$ and $F$ ($0$ to $15$ decimal).

Let's do the case $KKKLLMMNNO$.

First choose which character is repeated twice ($_{16}C_1$), then choose the places where it goes in the word ($_{10}C_3$).

Next, choose the three characters from what's left that are repeated once ($_{15}C_3$). Assign the smallest character to $L$, the next largest to $M$, and the largest to $N$. (This prevents counting duplicates.) Then, choose the two locations for $B$ from what's left ($_7C_2$), then $C$ ($_5C_2$), and then $D$ ($_3C_2$).

There's no choice where to put $O$ now, but you still choose what it is ($_{12}C_1$).

So, the number of possible words with one character repeated twice, three repeated once, and one not repeated is:

$$N(KKKLLMMNNO) = 16 \cdot \frac{10 \cdot 9 \cdot 8}{3 \cdot 2} \cdot \frac{15 \cdot 14 \cdot 13}{3 \cdot 2} \cdot \frac{7 \cdot 6}{2} \cdot \frac{5 \cdot 4}{2} \cdot \frac{3 \cdot 2}{2} \cdot 12.$$

That's one of fourteen cases.

Then, after you've calculated them all, add them up and multiply the sum by $8 \cdot 10$ to get the number of bits in the file.