If $X$ and $Y$ are irreducible, then rational maps from $X$ to $Y$ correspond to injections of fields $K(Y) \hookrightarrow K(X)$. Now $K(X) = K(X_{\mathbb Q}) = \mathbb Q(t)$, and $f$ induces an automorphism of $\mathbb Q(t)$ by the usual linear fractional transformation. This will extend to a (regular, not just rational) automorphism of $\mathbb P^1_{\mathbb Q}$ (i.e. $X_{\mathbb Q}$), but it may not extend regularly over all of $X$, and the problem is to determine whether or not it does.
Also, the process of converting a map $K(Y) \hookrightarrow K(X)$ to a rational map $X \to Y$ is essentially one of clearing denominators, as you say. But computing the maximal domain of definition of the rational map can be more subtle,
since clearing denominators in different ways may lead to extensions over different open pieces of $X$. (E.g. for rational maps between smooth projective varieties, the complement of the domain of definition is always codimension $2$; and although you are not quite in this context, your scheme $X$ is regular of dimension two, and it is fairly analogous; so one thing you'll need to work out is whether in your context the complement of the maximal domain of definition of your rational map has codimension one or two.)
