Let $x_i, y_i \in \mathbb{R}^n$, and $F,G: \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}^n$.
Consider sequences of vectors $$x_{i+1} = F(x_i, y_i)$$ $$y_{i+1} = G(y_i, x_i)$$
I have two related questions.
What properties should functions $F$ and $G$ have so that sequences $x_1, x_2, \dots$, $y_1, y_2, \dots$ converges?
And given some functions $F$ and $G$, how to determine whether the sequences converge or not.
Any thoughts or links to literature are welcome!
If the functions $F$ and $G$ are continuous they should behave in a way that the function $H = (F,G)$ has a fixed point. This is because if a limit $(x,y)$ exists then by taking limits on the equations we arrive to the functional equation $$(x,y) = (F(x,y), G(x,y)) = H(x,y)$$. To see if there is a fixed point you can use Banach fixed point theorem which guarantees the existence of a fixed point if $H$ is a contraction. You can check also Brower's fixed point theorem.
There certainly is no simple answer in the generality that you ask. This is a central question in a whole field called discrete dynamics. To exaggerate a little bit, that would almost be like asking how to solve a differential equation in general.
Like Bunder pointed out in the comments, if $F$ and $G$ are continuous (which is almost always the case) then the limit point $(x_0,y_0)$ must verify $F(x_0,y_0)=x_0$ and $G(x_0,y_0)=y_0$. If further the maps $F$ and $G$ are differentiable, then the matrix $DH$ (where $H=(F,G)$) must have at least one eigenvalue of modulus less than or equal to one. If $H$ is analytical, a lot more can be said.
Here is another situation (besides the Banach fixed point theorem) where you are guaranteed to converge : assume $(x_0,y_0)$ is an attracting fixed point, i.e. a fixed point where the differential $DH$ as all eigenvalues of modulus strictly less than one. Then if $(x,y)$ is close enough to $(x_0,y_0)$ it will converge to that fixed point. The Banach contraction theorem tells you that if $H$ is a contraction then you can drop the "close enough" part.
