If $N$ represents the total number of sheep that were originally in the flock, then the brothers were paid a total of $N^2$ rubles. From the description of how the proceeds were divided, we know that if $N^2$ is divided by 20, the remainder will be between 10 and 19 (inclusive). That is, there are nonnegative integers $q$ and $r$ such that
$$ N^2 = 20q + r
\qquad\text{and}\qquad
10 \le r \le 19. \tag{$\ast$}$$
While there are likely cleverer ways of approaching the problem at this point (the term "quadratic residues" comes to mind...), we can just brute force this problem: listing the remainders when square integers are divided by 20, we obtain
\begin{align}
1^2 &\equiv 1 \pmod{20} \\
2^2 &\equiv 4 \pmod{20} \\
3^2 &\equiv 9 \pmod{20} \\
4^2 &\equiv 16 \pmod{20} \\
5^2 &\equiv 5 \pmod{20} \\
6^2 &\equiv 16 \pmod{20} \\
7^2 &\equiv 9 \pmod{20} \\
8^2 &\equiv 4 \pmod{20} \\
9^2 &\equiv 1 \pmod{20} \\
10^2 &\equiv 0 \pmod{20} \\
11^2 &\equiv 1 \pmod{20} \\
12^2 &\equiv 4 \pmod{20} \\
13^2 &\equiv 9 \pmod{20} \\
14^2 &\equiv 16 \pmod{20} \\
15^2 &\equiv 5 \pmod{20} \\
16^2 &\equiv 16 \pmod{20} \\
17^2 &\equiv 9 \pmod{20} \\
18^2 &\equiv 4 \pmod{20} \\
19^2 &\equiv 1 \pmod{20} \\
20^2 &\equiv 0 \pmod{20},
\end{align}
at which point the pattern will repeat. Note that there is exactly one term that satisfies our requirements: 16. That is, if ($\ast$) holds, then $r=16$ is the only possibility. Therefore there were 6 rubles left over when the process finished.
Hence in the last division, the younger brother took 6 rubles. The older brother had just taken 10 rubles, which means that the older brother was up by 4. To make things even, the older brother would have to give the younger brother 2 rubles (subtracting two from his own total, and adding two to the total of the younger brother, netting each brother 8 rubles in the last division), therefore the penknife must be worth two rubles.
