Let us do the things exactly:
$$
\begin{aligned}
I &=
\int_{0}^{1}dx\int_{0}^{1} \left\{\frac xy\right\}\; dy
\\
&\qquad\text{(Substitution: $y=x/z$)}
\\
&=
\int_0^1dx \int_x^\infty \left\{ z\right\}\;\frac x{z^2}\; dz
\\
&=
\int_0^1x\; dx
\left(\
\int_x^1 \left\{ z\right\}\;\frac {dz}{z^2}
+
\int_1^\infty \left\{ z\right\}\;\frac {dz}{z^2}
\
\right)
\\
&=
\int_0^1x\; dx
\underbrace{\int_x^1 z\;\frac {dz}{z^2}}_{-\log x}
\qquad +\qquad
\underbrace{\int_0^1x\; dx}_{=1/2}
\sum_{n\ge 1}\int_n^{n+1} \{z\}\;\frac {dz}{z^2}
\\
&=
\frac 14
+
\frac 12\sum_{n\ge 1}
\int_0^1 t\;\frac {dt}{(n+t)^2}
\\
&=
\frac 14
+
\frac 12\sum_{n\ge 1}
\left(\log\frac{n+1}n-\frac 1{n+1}\right)
\\
&=
\frac 14
+
\frac 12\lim_{N\to\infty}
\left(\log(N+1)-\frac 12-\frac 13-\dots-\frac1{N+1}\right)
\\
&=
\frac 14
+
\frac 12\left(1-\gamma\right)
\\
&=
\frac 14(3-2\gamma)
\\
&\approx
0.46139216754923\dots
\end{aligned}
$$
Computer checks:
Sage gives the numerical value for $(3-2\gamma)/4$:
sage: (3 - 2*euler_gamma.n())/4
0.461392167549234
Pari/gp has a better numerical integration, so i tried...
? \p 300
realprecision = 308 significant digits (300 digits displayed)
? intnum(x=0, 1, intnum(y=0.00001, 1, frac(x/y)))
%20 = 0.4632302758838776079638791660602840407334814843635587641353594901119...
(Last result was manually truncated.)
? (3 - 2*Euler)/4
%22 = 0.4613921675492335696967439549587987844789203320300382005971163825575...