Russell's Paradox involves a test whether a set x is an element of itself.
But from a naive set theory perspective, sets have a hiearchy, per definitionem a set cannot be an element of itself. This seems to be a formalisation problem.
What axioms are meant to preclude this?
The real problem is not with $R\notin R$ itself. In most set theories this is the usual case, and the intention is that $R\notin R$ for all $R$. Instead, the real problem is with a universal set, which is a set $V$ that contains all sets.
Cantor's theorem shows that for any set $S$, there is another set, $$\mathscr P(S)= \{x \mid x\subseteq S\},$$ which is strictly larger than $S$. The proof is: Let $f$ be any map from $S$ to $ \mathscr P(S)$. Construct the set $R$ of all $x$ for which $x\notin f(x)$. Then there is no $r$ for which $f(r) = R$, therefore there are elements of $\mathscr P(S)$ which are not in the range of $f$. Since this is true for any $f$, $ \mathscr P(S)$ is bigger than $S$.
If there were a universal set $V$, this argument would lead us to Cantor's paradox: According to Cantor's theorem, $\mathscr P(V)$ is strictly larger than $V$ itself. But $V$ is the set of all sets, and $\mathscr P(V)$ is a set of some sets, so $V$ must be at least as large as $\mathscr P(V)$.
($V$ has the property that $x\in V$ whenever $x\subseteq V$, so applying the proof of Cantor's theorem to $V$ produces exactly the Russell set $\{R\in V\mid R\notin R\}$.)
So universal sets are impossible. Different theories have different ways of ruling out universal sets.
Most (not all) systems of set theory have your notion of a hierarchy built in. The most common, Zermelo-Fraenkel theory (ZF), was explicitly created with this idea in mind, although its encoding in the the axioms is not explicit. The closest thing ZF has is the so-called axiom of regularity, also called the axiom of foundation.
For technical reasons the axiom is expressed in a rather opaque way. It says that every nonempty set $S$ must contain an element $t$ for which $$t\cap S = \emptyset.$$
It can be shown that this implies that there is no set $S$ for which $S\in S$, and more generally there is no infinite descending chain $$\dots\in S_3 \in S_2\in S_1.$$
The regularity axiom, though, is not what prevents Russell's paradox from arising in ZF. The paradox arises from the idea of a universal set, not from irregularity. Regularity just tells us that $S\notin S$ for all $S$. So if Russell's set $R$ were to exist, it would simply contain all sets—it would be a universal set. But Cantor's paradox shows there can't be any such thing.
Instead, what fixes Russell's paradox in ZF is called restricted comprehension. In ZF you are not allowed to perform unrestricted comprehension, which means you may not construct the set of all $x$ having property $\Psi$, which would be written $$\{x\mid \Psi(x)\}.\tag1$$ Rather, you are only allowed to construct the subset of some other set $S$ that consists of the elements of $S$ that have property $\Psi$: $$\{x\in S\mid \Psi(x)\}\tag2$$ where $S$ is some set. This is called “restricted comprehension” because the construction is restricted to the elements of $S$.
(Note that if there is a universal set $V$, then restricting comprehension accomplishes nothing because $\{x\in V\mid \Psi(x)\}$ is exactly unrestricted comprehension.)
We can still try to proceed with the Russell argument in ZF. Suppose $S$ is some set that we have already. We may construct $$R(S) =\{x\in S\mid x\notin x\},$$ the set of all elements of $S$ for which $x\notin x$ is true. But because the axiom of regularity allows us to prove $x\notin x$ for all $x$, we simply get $$R(S)=S$$ for all $S$.
Morse-Kelley (MK) set theory allows unrestricted comprehension like $(1)$, but interprets it differently. MK distinguishes between “sets” and “classes”. Any property $\Psi$ defines a class, but not every class is a set. The notation $\{x \mid \Psi(x)\}$ is understood to mean “the class of all sets with property $\Psi$”.
In MK set theory, we can construct $R = \{x \mid x\notin x\}$ but it means the class of all sets that satisfy $x\notin x$. Russell's argument simply shows that $R$ is not itself a set, and so that $R\notin R$. Morse-Kelley theory does have a universal class $V$ of all sets but instead of leading to a paradox the Cantor-Russell argument merely shows that $V$ is not a set.
von Neumann-Bernays-Gödel set theory (NBG) is very similar to MK. There are “good” sets and “bad” sets, and only “good” sets are allowed to be members of other sets. For a bad set $b$, $b\in x$ is always false. Russell's $R$ then becomes the set of all good sets that satisfy $x\notin x$, and Russell's argument simply shows that $R$ is a bad set.
Russell himself developed a theory that tried to solve the problem, called the theory of types. In this theory sets belong to different types, and a set may only contain elements whose types are all the same. Then, at a particular type we can construct $R=\{x\mid x\notin x\}$ but it is only the set of all things of a certain type that satisfy $x\notin x$. Here, Russell's argument just shows that $R$ has some other type, and again $R\notin R$.
Russell's theory does have universal sets, but for each type $T$ there is a different universal set $V_T$ that contains everything of type $T$. The paradoxical argument only shows that $V_T$ is not a thing of type $T$.
Russell's theory had some technical problems which led to it falling out of use. Most simply, there are many different empty sets, one for each type.
Quine's New Foundations (NF) and systems based on it are an attempt to rehabilitate the “type” idea behind Russell's theory. The types themselves are abandoned, and the language replaces unlimited comprehension $(1)$ with stratified comprehension. This says that you can construct a set $\{x\mid \Psi(x)\}$ only when $\Psi$ is a “stratified” formula. A formula $\Psi$ is stratified if one can associate each variables in $\Psi$ with a number, and the numbers have to match up in a particular way: if $\Psi$ contains the subformula “$x\in y$” then the number associated with $x$ must be one less than the number associated with $y$. In particular, this means that $x\in x$ and $x\notin x$ are not stratified, and cannot appear in the $\Psi$ that used used in a comprehension.
Most set theories adopt some sort of the hierarchy you suggested, and eliminate any set $x$ for which $x\in x$. Peter Aczel developed a “non-wellfounded” set theory that was intended to permit sets for which $x\in x$ is true, and similarly sets $x$ and $y$ where both $x\in y$ and $y\in x$. I don't remember how Aczel deals with Russell's paradox. It may be that it doesn't come up because there is still no universal set.
Finally, Randall Holmes has done work on set theories that do have a universal set $V$ and that get around Cantor's paradox in different ways.
