I have been given a hint that based off of
that $(\angle AIB')^r+(\angle BIC')^r+(\angle CIA')^r=\pi$ and that each of these three angles is less than $\Pi(r)$. Using this information I must then apply the Bolyai-Lobachevsky formula to find x such that $\Pi(x)=\pi/3$. I feel like the solution has something to do with the fact that the vertices of the triangle are all poles of chords of the circle but I don't know how to quantitatively relate it.
As recommended by MvG in a comment let's examine triangles of infinite sides. Below is depicted in the Klein model a doubly infinite hyperbolic triangle $ABC$.
Drop a perpendicular to $AC$ from $B$. Now, we have two infinite right triangles $ABD$ and $BCD$. Since these triangles are congruent: $\alpha=\beta$. $BD$ is the angle bisector of the angle $ABC$. The center of the inscribed circle of $ABC$ will be on $BD$ even if we extend $AB$ as far as we please.
Now, consider the following (triply) infinite triangle depicted in the Klein model. (Since all the triply infinite triangles are congruent, we can take the one whose situation in the Klein model is special. The center of the model halves the base and the triangle itself looks isosceles.)
The inscribed circle is centered at the point where the red angle bisector and the white angle bisector intersect. It is clear that the red and the white lines are angle bisectors because they are perpendicular to the facing sides. (see the tangent lines).
The two angle bisectors meet at the point whose Euclidean distance from the origin is $\frac12$. Now, anyone can calculate the corresponding hyperbolic distance. The length of the diameter of the inscribed circle being the largest such circle is $\ln(3)$.
Note:I think there is a typo in the OP, the correct value is $\frac{1}{2}\log 3$. Not a big deal.
Recall the formula for the radius $\rho$ of the inscribed circle in a triangle of sides $a$, $b$, $c$ in the spherical geometry (sphere of radius $R$)
$$\tan\frac{\rho}{R}= \sqrt{\frac{\sin\frac{s-a}{R}\cdot \sin\frac{s-b}{R}\cdot\sin\frac{s-c}{R}}{\sin\frac{s}{R} } }$$ (see Todhunter --Spherical Trigonometry)
To get the radius of the inscribed circle in the hyperbolic geometry ( curvature $= R^2 =-1$), put $R=i$ in the above formula. We get
$$\tanh \rho = \sqrt{\frac{\sinh(s-a)\cdot\sinh(s-b)\cdot\sinh(s-c)}{\sinh s}}$$
Now, the function $t \mapsto \log \sinh t$ is concave on $(0, \infty)$. Therefore, the right hand side in the above equality is $\le \sqrt{\frac{\sinh^3(\frac{s}{3})}{\sinh s}}$. This is a strictly increasing function of $s$ on $(0, \infty)$ with limit $\frac{1}{2}$ at infinity. Therefore we have $$\rho \le \tanh^{-1} \frac{1}{2}= \frac{1}{2}\log 3$$
We do have equality for the ideal triangles (sides $=\infty$).
