I think you're going to have difficulty trying to establish the boundedness of $S$ using this argument. The reason why is that the Cartesian coordinates are not well-suited to describe $S$. Below I have given another method to establish the boundedness of $S$.
Note that $x^2 + 3xy + 3y^2 = {\bf x}^TA{\bf x}$ where
$$A = \begin{bmatrix} 1 & \frac{3}{2}\\ \frac{3}{2} & 3\end{bmatrix}.$$
As $A$ is symmetric, it can be diagonalised. That is, there is an orthogonal matrix $P$ such that $P^TAP$ is a diagonal matrix $\operatorname{diag}(\lambda_1, \lambda_2)$ where $\lambda_1$, $\lambda_2$ are the eigenvalues of $A$. If we introduce new variables ${\bf \hat{x}} = (\hat{x}, \hat{y})$ given by ${\bf\hat{x}} = P^{-1}{\bf x}$, then as ${\bf x} = P{\bf\hat{x}}$ we have
$${\bf x}^TA{\bf x} = {\bf \hat{x}}^TP^TAP{\bf\hat{x}} = {\bf\hat{x}}^T\operatorname{diag}(\lambda_1, \lambda_2){\bf\hat{x}} = \lambda_1\hat{x}^2 + \lambda_2\hat{y}^2.$$
In these new coordinates, we see that $S$ is simply a conic.
- If $\lambda_1, \lambda_2 > 0$, then $S$ is an ellipse.
- If $\lambda_1 > 0$, $\lambda_2 < 0$ (or vice versa), then $S$ is a hyperbola.
- If $\lambda_1 = 0$, $\lambda_2 > 0$ (or vice versa), then $S$ is the union of two lines.
- If $\lambda_1, \lambda_2 \leq 0$, $S$ is empty.
As we explicitly have the matrix $A$, we can calculate $\lambda_1$ and $\lambda_2$. However, we just need to know the signs of the eigenvalues, which is a bit easier to determine. As $\det A = \lambda_1\lambda_2 = \frac{3}{4} > 0$ and $\operatorname{tr} A = \lambda_1 + \lambda_2 = 4 > 0$, both $\lambda_1$ and $\lambda_2$ are positive. Therefore, $S$ is an ellipse, and hence bounded.
Note, one can ensure that $P$ has determinant $1$, and hence $P$ is a rotation matrix. The change of variables ${\bf\hat{x}} = P^{-1}{\bf x}$ rotate the $x$ and $y$ axes in such a way that the expression $x^2 + 3xy + 3y^2$ is put in standard form. These coordinates are the ones which we can consider to be well-suited to describe $S$. This is the standard procedure for putting a quadric into standard form.