Edit : Here are two solutions. The first one uses analytic geometry. The second one, simpler, elaborated later, uses only trigonometry on triangles.
First solution : Taking "natural" coordinates representation wrt inner rectangle (red axes on figure below), let us consider the extreme points of the brace have coordinates :
$$A(b-x,0), B(b,x), C(0,a-x),D(x,a).$$

Straight lines $AC$ and $BD$ (brace borders) have resp. equations :
$$\begin{cases}x_1(a-x)+y_1(b-x)-(a-x)(b-x)&=&0\\x_1(a-x)+y_1(b-x)+(x^2-ab)&=&0\end{cases}$$
(where $x_1,y_1$ are the generic coordinates).
The shortest distances from the origin $O$ to lines $AC$ and $BD$ are resp. :
$$d_1=\frac{(b-x)(a-x)}{\sqrt{(a-x)^2+(b-x)^2}}, \ d_2=\frac{ab-x^2}{\sqrt{(a-x)^2+(b-x)^2}}$$
(see formula here).
Therefore the distance between the two lines $AC$ and $BD$ is the difference $d_2-d_1$, which must be equal to $c$ :
$$\frac{(a+b)x-2x^2}{\sqrt{(a-x)^2+(b-x)^2}}=c$$
Squaring it, we get the following 4th degree equation :
$$\underbrace{((a+b)x-2x^2)^2-c^2((a-x)^2+(b-x)^2)}_{f(x)}=0,\tag{1}$$
$x$ being one of its roots.
Examples :
- If we take $a=3,b=2$ and $c=0.3$ (see graphical representation of $f$ below), we get the roots :
$$x \approx -0.216, \ 0.217, \ 2.457, \ 2.542$$

The second root (and this one only) is the right one.
- If we take $a=b=3$ and $c=0.5$, we get the roots :
$$x = -\frac{\sqrt{2}}{4}, \ \frac{\sqrt{2}}{4}, \ 3 \ \text{(double root)}$$
Here as well, the second root is the right one.
Second solution : Consider the following figure featuring the north-eastern part of the internal rectangle :

Angle $\theta$ is found in 3 places, allowing us to write two equations :
$$\begin{cases}x \cos(\theta)+x \sin(\theta)&=&c\\ \tan(\theta)&=&\frac{b-x}{a-x}\end{cases}.$$
Out of these two equations, one can choose
$$b \cos(\theta)-a \sin(\theta)=c \frac{\cos(\theta)-\sin(\theta)}{\cos(\theta)+\sin(\theta)}$$
that can be solved by taking half-angle formulas
$$\cos(\theta)=\frac{1-t^2}{1+t^2}, \ \sin(\theta)=\frac{2t}{1+t^2} \ \ \text{where} \ \ t:=\tan(\theta/2)$$
giving rise... to a fourth degree equation.