
Tried attempting by using altitude and similarity of triangles, but the problem is that variables are not getting elliminated.

Tried attempting by using altitude and similarity of triangles, but the problem is that variables are not getting elliminated.
By Cavalieri's principle we can make the outer triangle isosceles without changing the rectangles' area. By hyperbolic transformations we can make this triangle have base 6 and height 4, again keeping the rectangle areas intact.
Let $p=P_1S_1$ and $q=P_2S_2$. The triangle area outside the rectangles, which we want to minimise, is $$p\cdot\frac34p+q\cdot\frac34q+(4-p-q)\cdot\frac34(4-p-q)$$ (The $\frac34$ factors are there because we can rearrange the unoccupied space into three 4:3 rectangles whose longer sides are $p,q,4-p-q$.) $$=\frac34\left(p^2+q^2+(4-(p+q))^2\right)$$ $$=\frac32(p^2+pq+q^2-4p-4q)+12$$ The gradient of the expression in brackets is $(2p+q-4,2q+p-4)$ and this is zero when $p=q=\frac43$. Thus these are the rectangle heights maximising the area occupied, which is $$\frac32\left(8\cdot\frac43-3\cdot\frac43\cdot\frac43\right)=8$$
Let's call $x$ the height from $A$ onto $S_2R_2$, $y=|S_2P_2|$, and $z=|S_1P_1|$. The area of rectangles is then $$A_r=z|S_1R_1|+y|S_2R_2|$$ We can write these segments in terms of $|BC|$ by looking at similar triangles: $$|S_2R_2|=|BC|\frac{x}{x+y+z}\\|S_1R_1|=|BC|\frac{x+y}{x+y+z}$$ Therefore: $$A_r=|BC|\frac{xy+yz+zx}{x+y+z}$$ Since $|BC|$ and $x+y+z$ are constant for a given triangle, we want to maximize $xy+yz+zx$ with the constraint that $x+y+z=k$, where $k$ is some constant. Using Lagrange multiplier method: $$\partial_x (xy+yz+zx-\lambda(x+y+z-k))=0\\\partial_y (xy+yz+zx-\lambda(x+y+z-k))=0\\\partial_z (xy+yz+zx-\lambda(x+y+z-k))=0$$ you get: $$y+z-\lambda=0\\x+z-\lambda=0\\x+y-\lambda=0$$ Using $x+y+z=k$, when adding these equations you get$$2k-3\lambda=0$$ or $\lambda=\frac{2}{3}k$. The solution is $$x=y=z=\frac{k}{3}$$ Then $$\max(A_r)=\frac{3(k/3)^2}{k}|BC|=\frac{k|BC|}{3}$$ Notice that $k|BC|$ is twice the area of the triangle, so $$\max(A_r)=8$$
The two apiled rectangles leave an area associated to three similar triangles. The rectangles will have maximum area when the three similar triangles are equal
or ressuming
$$ \frac 12 b\cdot h = 12 \Rightarrow b\cdot h = 24\\ 3\frac 12 \left(\frac b3\cdot\frac h3\right) = 4 $$
so the maximum area is $12-4 = 8$
NOTE
The problem is the same depending on the number of apiled squares. In the case of $n$ we have the maximum squares area is given by $12 - n\left(\frac{1}{2}\frac{b}{n}\frac{h}{n}\right) = 12-\frac{12}{n}$ for $n = 1, 2,\cdots$
To show the necessity of the equality between the small excess triangles think on the problem depicted in the included picture.
Drop a perpendicular from $A$ to the base, and consider the "right half" $S$ of the triangle, whereby we may assume $A=(0,1)$, $C=(1,0)$, $Q_1=(x,0)$, $\>0\leq x\leq1$. If there were just one rectangle we would have to maximize $x(1-x)$, leading to $x={1\over2}$. For two rectangles we therefore have to maximize $$f(x):=x(1-x)+\left({x\over2}\right)^2=x-{3\over4}x^2\ .$$ The maximum is at $x={2\over3}$ with $f\left({2\over3}\right)={1\over3}$. It follows that $${{\rm area}_\max(R_1\cup R_2)\over{\rm area}(S)}={f\bigl({2\over3}\bigr)\over{1\over2}}={{1\over3}\over{1\over2}}={2\over3}\ ,$$ so that the answer to the question is ${2\over3}\cdot12=8$.