The statement $$S=\{x: f(x)\ge\vert S\vert\}$$ is a meaningful statement about a set $S$. However, it doesn't constitute a definition: you can't use the thing you're trying to define, to define itself.
Now, that said, there are contexts in which self-referential "definitions" work! (A keyword here is "impredicative.") For example, defining a function by recursion can be viewed as such a definition: if we want to build a function $f$ satisfying "$f(n)$ is the number of $k<n$ such that $f(k)$ is even," we can define $f$ as $$f(n)=I(f\upharpoonright n)$$ (where $f\upharpoonright n$ is the function $f$ restricted to numbers $<n$, and $I$ applied to a function with finite domain outputs the number of times that function spits out an even number). This may seem bonkers at first, but it turns out to make perfect sense:
There are no $k<0$, so $f(0)=0$.
There is one $k<1$ - namely $0$ - and $f$ spits out an even number on that input (since $f(0)=0$, as we concluded above) - so $f(1)=1$.
There's still only one $k<2$ with $f(k)$ even - so $f(2)=1$.
And it's not hard to see that indeed $f(n)=1$ for all $n>0$.
See e.g. this paper for a serious treatment of definition by recursion as self reference.
But we have to be careful. Even if we allow $S$ to appear in its own definition, we still don't let anything go: in order to not be nonsense, a self-referential definition has to have two properties:
That is, something satisfying the definition has to exist, and has to be unique. Neither of these is obviously true of the statement you've written down.
Indeed, the second one is false in the case you outline: in the case $f(x)=x^2$, there is no $S$ with this property! Such an $S$ would have to be finite (since we never have $f(x)$ infinite), but then taking $n>\vert S\vert$ we'd have $n\in S$ (since $n^2>n>\vert S\vert$). But there are infinitely many $n>\vert S\vert$ if $S$ is finite, so then $S$ would be infinite; contradiction.
(Note that all we used was that $x^2$ is not bounded. However, boundedness isn't enough to guarantee the existence of such an $S$: consider the function $f(0)=2$, $f(n)=1$ for $n>0$. Can there be any set $S$ such that $S=\{x: f(x)\ge\vert S\vert\}$?
It's actually a good exercise to show that there is any function $f$ for which some $S$ exists! HINT: see what happens if you make $f$ take on its maximum value infinitely many times . . .)