I've got some partial results:
By definition: $\boxed{n = m_1 + ... + m_k}$ ; $\boxed{m_k \geq 1}$ ; $\boxed{n \geq k}$
$$f(m_1,...,m_k) = \binom{m_1}{2}\binom{m_2}{2}...\binom{m_k}{2}$$
$$\binom{m_i}{2} = \frac{[m_i-1]\,m_i}{2}$$
$$f(m_1,...,m_k) = \frac{[m_1\,m_2\,...\,m_k][m_1-1][m_2-1]...[m_k-1]}{2^k}$$
Let's first solve for the small cases:
If $k \leq n < 2\,k$, there is at least one $m_i=1$, so $f(m_1,...,m_k)=0$
If $n = 2\,k$, $m_1=m_2=...=m_k=2$ would yield the maximal value: $f(m_1,...,m_k)=1$
Now, let's assume $\boxed{m_k\geq 2}$, so that $f(m_1,...,m_k)$ is strictly positive.
If we would write this as an continuous maximization problem, we would get the following derivatives.
$\nabla f = \frac{1}{2^k}\begin{bmatrix}
[m_2\,m_3\,...\,m_k][m_2-1][m_3-1]...[m_k-1]\,[2\,m_1-1]\\
[m_1\,m_3\,...\,m_k][m_1-1][m_3-1]...[m_k-1]\,[2\,m_2-1]\\
\vdots\\
[m_1\,m_2\,...\,m_{k-1}][m_1-1][m_2-1]...[m_{k-1}-1]\,[2\,m_k-1]\\
\end{bmatrix}$
$g(m_1,...,m_k) = m_1+...+m_k - n = 0$
$\nabla g = \begin{bmatrix}
1\\
1\\
\vdots\\
1
\end{bmatrix}$
From these derivatives, it can be shown that $m_1=m_2=...=m_k=\frac{n}{k}$ is a local maximum that can be achieved in the discrete original problem whenever $\boxed{n = d\,k}$, with $d\in\{1,2,...\}$.
For this particular cases, the local maximal value of $f$ would be:
$$\boxed{\frac{n^k\,[n-k]^k}{2^k\,k^{2k}}}$$
If we can now show that this is a global maximum for the relaxed continuous problem, then it will be an upper bound for the discrete original function!
Note that, if $m_1=m_2=...=m_k=\frac{n}{k}$ is the global solution of the optimization problem:
$$\text{maximize} \;\; m_1\,m_2\,...\,m_k$$
$$\text{subject to} \;\; m_1+m_2+...+m_k=n$$
Then, $m_1-1 = m_2-1 = ... = m_k-1 = \frac{n-k}{k}$ will solve:
$$\text{maximize} \;\; [m_1-1][m_2-1]...[m_k-1]$$
$$\text{subject to} \;\; [m_1-1] + [m_2-1] + ... + [m_k-1] = n-k$$
Since our function of interest is the product of these two (divided by $2^k$), this would mean that the presented local optimum is global!
So now the problem is reduced to prove that the product of $k$ positive values with fixed sum is maximal when all the factors are equal.
Note that if $k>n$, there are no $m_i$ which satisfy your conditions, since $m_1+\ldots + m_k\geqslant 1+\ldots + 1 =k>n$.
– Aug 24 '23 at 12:54