Here is my interpretation of how Rudin argued the first point in your question (to contradict the case that $y^{n} < x$).
The idea is to find a $y^*$ that would lie in the "gap" of numbers created if $y$ were indeed less than the "true $\sqrt[n]{x}$". (Here $y$ refers to $y=\sup E$ where $E=\{t:t^n < x, t\in\mathbb{R^+}\}$ as defined by Rudin.)
In other words, we are trying to construct such a $y^*$ that has the two properties $(y^*)^n < x$ and $y^* > y$. The first property says that $y^* \in E$, the set which $y$ bounds above. The second property contradicts the fact that $y$ is an upper bound of $E$, since $y^* \not\le y$ and $y^* \in E$.
To construct $y^*$ we need to find a suitable $h$ such that $y^*=y+h$.
Intuitively, one would like to let $h$ be a positive quantity less than the difference $\sqrt[n]{x} - y$. However, we have not yet shown the existence of $\sqrt[n]{x}$ so this complicates our approach.
One alternative is to look at the difference $x - y^n$, which is a valid expression at this point in the proof. Graphically, this approach can be thought of as picking the position of $y+h$ on the horizontal axis based on the function value $(y+h)^n$ on the vertical axis.
In other words, instead of specifying $h$ directly we will try to specify $h$ in terms of what $y+h$ maps to under exponentiation by $n$.
Hence, we look for an $h$ such that $(y+h)^n - y^n < x - y^n$.
(Again, think of comparing function values of the curve $f(t) = t^n$ on a graph. The left-hand side $(y+h)^n - y^n$ is some positive quantity smaller than the height of the vertical "gap" assumed to exist between $y^n$ and $x$ on the graph.)
(Since $f(t) = t^n$ is a strictly increasing function (for positive $t$), inequalities are preserved, so the hypothetical horizontal "gap" (in which we were originally interested) will correspond with a vertical "gap" after this transformation. Intuitively, (pretending $\sqrt[n]{x}$ is defined) we could write,
$$
\sqrt[n]{x} - y > 0 \iff \sqrt[n]{x} > y \iff f(\sqrt[n]{x}) > f(y) \iff \sqrt[n]{x}^n > y^n \iff x > y^n \iff x - y^n > 0
$$
with the following justifications for each of the double arrows (in order from left to right): (1) addition/subtraction by $y$, (2) $f$ is strictly increasing, (3) definition of $f$, (4) definition of $\sqrt[n]{x}$, (5) addition/subtraction by $y^n$.)
Using the observation Rudin made$^1$ we can inject another expression into the inequality to get:
$(y+h)^n - y^n < hn(y+h)^n < x - y^n$
(This "injection" is okay for now because we have technically not defined $h$ yet, but instead are still working backwards to specify $h$.)
The trouble with this inequality is that the middle expression contains $h$ inside a binomial term raised to the power of $n$, which makes it hard to algebraically isolate. I believe Rudin makes the convenient assumption at this point that $h$ is small — in other words $0 < h < 1$ — in order to complete his definition of $h$.
We can now inject another expression into the inequality to yield:
$(y+h)^n - y^n < hn(y+h)^n < hn(y+1)^n < x - y^n$
Isolating just the rightmost two expressions and rearranging gives the final inequality:
$h < \frac{x - y^n}{n(y+1)^n}$
End Note
- Rudin's observation:
$b^n - a^n = (b-a)(b^{n-1} + ab^{n-2} + ... + a^{n-2}b + a^{n-1})$
so
$b^n - a^n < (b-a)n(b^{n-1})$
in the case $0 < a < b$.
(Rudin let $b = y+h$ and $a=y$ in the proof.)