It depends on how you define relative denseness for subsets/subspaces of $X.$ For example, we might say:
Given two subsets $A,B$ of a topological space $X,$ we say that $A$ is dense in $B$ if $A\subseteq B$ and the closure of $A$ in $B$ (relative to the subspace topology on $B$) is equal to $B.$
In that case, your relation $R$ is trivially antisymmetric by double-inclusion. The above is the typical definition, as Daniel Wainfleet points out in the comments.
On the other hand, you're considering that we might instead say:
Given two subsets $A,B$ of a topological space $X,$ we say that $A$ is dense in $B$ if the closure of $A\cap B$ in $B$ (relative to the subspace topology on $B$) is equal to $B.$
In that case, we won't be able to prove antisymmetry. Consider for example $X=\Bbb R$ in the usual topology, $A=X\setminus\{0\},$ and $B=X\setminus\{1\}.$
Moreover, we won't be able to prove transitivity, either! Consider for example $A=\Bbb Q,$ $B=\Bbb R,$ and $C=\Bbb R\setminus\Bbb Q$ as subsets of $\Bbb R$ in the usual topology. Then $A$ is dense in $B$ and $B$ is dense in $C,$ but $A$ is not dense in $C.$
Consequently, it seems that the first definition is the one you want, which makes $R$ a partial order, and in fact, $R$ is a sub-relation of $[\subseteq]_{\mathcal{P}(X)}.$ You are correct that, given two sets $A$ and $B$ we don't necessarily have either of them as a subset of the other, which only means that $R$ won't be a total order (unless $X$ is empty).