sum=0
for i=1 to n do
for j=1 to i^2 do
if(j (mod i) = 0) then
for k=1 to j do
sum++
This is the same as
$$\begin{align}
%
S(n) &= \sum_{i = 1}^n \sum_{\begin{array} {l} j = 1 \\ j \equiv 0 \pmod i \end{array}}^{i^2} \sum_{k = 1}^j 1
%
\\ \\ & \text{Let } m = j/i
%
\\ \\ &= \sum_{i = 1}^n \sum_{m = 1}^{i} \sum_{k = 1}^{m^2} 1
%
\end{align}$$
You don't have to evaluate the sum exactly, all you need to be able to determine is that $\sum_{i = 1}^n \sum_{m = 1}^{i} \sum_{k = 1}^{m^2} 1$ is a 4th degree polynomial.
$$\begin{align}
%
S(n) &= \sum_{i = 1}^n \sum_{m = 1}^{i} m^2
%
\\ \\ &= \sum_{i = 1}^n A~i^3 + \dots
%
\\ \\ &= A~n^4 + \dots
%
\end{align}$$
Since an increment takes a logarithmic amount of time to compute, your runtime is
$$\begin{align}
%
R(n) & \in O(S(n)\log(S(n))
%
\\ \\ &= O(~ (A~n^4 + \dots) \log( An^4 + \dots)~)
%
\\ \\ &= O(n^4\log(n))
%
\end{align}$$
The exact computation, which isn't necessary but might help understand, is:
$$\begin{align}
%
S(n) &= \sum_{i = 1}^n \sum_{m = 1}^{i} m^2
%
\\ \\ &= \sum_{i = 1}^n \frac{1}{6}\bigg(2i^3 + 3i^2 + i\bigg)
%
\\ \\ &= \frac{1}{12} \bigg(n^4 + 4n^3 + 5n^2 + 2n\bigg)
%
\end{align}$$
for k=1 to ksupposed to mean? Typo I assume – BrianO Feb 05 '16 at 01:55j (mod i) = 0) are the values $i, 2i, 3i, 4i, \ldots, i^2$. If you make $j$ just iterate over those values without taking any of the other values between $1$ and $i^2$, theifstatement will always be true and you can get rid of it. – David K Feb 05 '16 at 07:36sumand what the values ofi,j, andkwere each time. Then maybe you'll see a pattern. – David K Feb 05 '16 at 07:42