Proof.
By your work, $n=3$ $$3\le \sum_{k=1}^{9}\{\sqrt{k}\}\le 4.$$
The proof for RHS.$$\sum_{k=1}^{n^2}\{\sqrt{k}\}\le \frac{n^2-1}{2}.\tag {*}$$
Assuming that $n=m\ge 3.$ It gives$$\sum_{k=1}^{m^2}\{\sqrt{k}\}\le \frac{m^2-1}{2}.$$
We'll prove $(*)$ is true for $n=m+1.$
Notice that$$\sum_{k=1}^{(m+1)^2}\{\sqrt{k}\}=\sum_{k=1}^{m^2}\{\sqrt{k}\}+\sum_{k=m^2+1}^{(m+1)^2}\{\sqrt{k}\}\le \frac{m^2-1}{2}+\sum_{k=m^2+1}^{(m+1)^2}\{\sqrt{k}\}.$$
Thus, it suffices to prove$$\frac{m^2-1}{2}+\sum_{k=m^2+1}^{(m+1)^2}\{\sqrt{k}\}\le \frac{(m+1)^2-1}{2}.$$Or$$\sum_{k=m^2+1}^{(m+1)^2}\{\sqrt{k}\}\le \frac{2m+1}{2}.$$
Since $\{\sqrt{(m+1)^2}\}=\{m+1\}=0,$ it's$$\{\sqrt{m^2+1}\}+\{\sqrt{m^2+2}\}+...+\{\sqrt{m^2+2m}\}\le\frac{2m+1}{2} .$$
Also, it's obvious that$$m<\sqrt{m^2+1}<\sqrt{m^2+2}<...<\sqrt{m^2+2m}<m+1.$$
Thus, $[\sqrt{m^2+t}]=m,\forall 1\le t\le 2m.$ It means$$\{\sqrt{m^2+t}\}=\sqrt{m^2+t}-m,\forall 1\le t\le 2m.$$
Also, we may use$$\sqrt{m^2+t}-m=\sqrt{m^2+2\cdot\frac{t}{2m}\cdot m+\frac{t^2}{4m^2}-\frac{t^2}{4m^2}}-m< \frac{t}{2m},\forall 1\le t\le 2m.$$
Hence, $$\{\sqrt{m^2+1}\}+\{\sqrt{m^2+2}\}+...+\{\sqrt{m^2+2m}\}\le \frac{\sum_{t=1}^{2m}t}{2m}=\frac{2m+1}{2} .$$
The proof for LHS.$$\sum_{k=1}^{n^2}\{\sqrt{k}\}\ge \frac{n(n-1)}{2}.\tag {**}$$
Assuming that $n=m\ge 3.$ It gives$$\sum_{k=1}^{m^2}\{\sqrt{k}\}\le \frac{m(m-1)}{2}.$$
We'll prove $(*)$ is true for $n=m+1.$
Notice that$$\sum_{k=1}^{(m+1)^2}\{\sqrt{k}\}=\sum_{k=1}^{m^2}\{\sqrt{k}\}+\sum_{k=m^2+1}^{(m+1)^2}\{\sqrt{k}\}\ge \frac{m(m-1)}{2}+\sum_{k=m^2+1}^{(m+1)^2}\{\sqrt{k}\}.$$
Thus, it suffices to prove$$\frac{m(m-1)}{2}+\sum_{k=m^2+1}^{(m+1)^2}\{\sqrt{k}\}\ge \frac{m(m+1)}{2}.$$Or$$\{\sqrt{m^2+1}\}+\{\sqrt{m^2+2}\}+...+\{\sqrt{m^2+2m}\}\ge m .$$
Also, it's obvious that$$m<\sqrt{m^2+1}<\sqrt{m^2+2}<...<\sqrt{m^2+2m}<m+1.$$
Thus, $[\sqrt{m^2+t}]=m,\forall 1\le t\le 2m.$ It means$$\{\sqrt{m^2+t}\}=\sqrt{m^2+t}-m=\frac{t}{\sqrt{m^2+t}+m}\ge \frac{t}{\sqrt{m^2+2m+1}+m}=\frac{t}{1+2m} ,\forall 1\le t\le 2m.$$
Hence, $$\{\sqrt{m^2+1}\}+\{\sqrt{m^2+2}\}+...+\{\sqrt{m^2+2m}\}\ge \frac{\sum_{t=1}^{2m}t}{1+2m}= m .$$
We're done!