First of all you should review the definition of a "rational function."
It is a function that can be written as a ratio of two polynomials.
The integrand $\frac{1}{x(\sqrt{x} + \sqrt[5]{x^2})}$ is not a rational function of $x.$
There is no way to make it equal to $\frac{p(x)}{q(x)}$ where $p$ and $q$ are polynomial functions.
It is also not true that you "must" substitute $t = \sqrt[10]x.$
It is entirely possible that there is a completely different way to solve this integral.
But you book is showing you a way to solve this integral, and that way involves a $u$-substitution that leaves you with a rational
function in the integrand.
The trick to this particular substitution is to find something that lets you rewrite all the powers of $x$ as integer powers of your substituted variable,
so that when you add and multiply as shown you'll have a polynomial in your new variable.
(It's also necessary that whatever happens in the numerator due to the substitution, that also has to end up as a polynomial in your new variable. We don't know this will happen until we try it.)
The choice $t = \sqrt[{10}]{x}$ is by no means the only way to do this.
But it does allow you to write each of the terms $x$, $\sqrt x$,
and $\sqrt[10]x$ as an integer power of $t.$
So, if you did not have the answer key in front of you, how might you come to the decision to try $t = \sqrt[{10}]{x}$?
You could first convert everything into integer or fractional powers of $x$:
\begin{align}
\sqrt x &= x^{1/2}, \\
\sqrt[5]{x^2} &= x^{2/5}, \\
\end{align}
so now we have
$$ \frac{1}{x(x^{1/2} + x^{2/5})} .$$
Now for $x^{1/2}$ and $x^{2/5}$ both to be integer powers of the same number, it's sufficient for that number to be a power of $x$ with an exponent that divides both $\frac12$ and $\frac25$ evenly.
How to find a number that divides two fractions evenly?
Try putting them in terms of a common denominator,
$$ \frac12 = \frac{5}{10}, \quad \frac25 = \frac{4}{10}. $$
that is, $\frac12 = 5 \times \frac1{10},$ so $x^{1/2} = (x^{1/10})^5.$
And $\frac25 = 4 \times \frac1{10},$ so $x^{2/5} = (x^{1/10})^4.$
What is inside the parentheses in both cases? It's $x^{1/10}.$
So if we suppose that $t = x^{1/10},$ then
$$ x^{1/2} + x^{2/5} = t^4 + t^5 . $$
Remember, this is not the only way to write these terms as integer powers of something. It just happens to be a convenient way to do it.
Again, there is often more than one way to solve an integral. A solution shows you how you can solve it, not how you must solve it.
Especially with $u$-substitutions, unless you've seen the exact same problem before (possibly in a more general form), you're really just guessing that a particular substitution is going to help.
You find out that it was a good one when you actually do it and solve the integral.