Yes, "pushing $\gamma_r$ across $R_{r+1}$" means using a homotopy; $F|\gamma_r$ is homotopic to $F|\gamma_{r+1}$, since the restriction of $F$ to $R_{r+1}$ provides a homotopy between them via the square lemma (or a slight variation of the square lemma which allows for non-square rectangles). But there's more we can say; the factorization of $[F|\gamma_r]$ is equivalent to the factorization of $[F|\gamma_{r+1}]$.
Each of these two factorizations might contain a product of multiple terms corresponding to the edges around $R_{r+1}$ since, for most of the rectangles $R_i$, you'll hit about three vertices while traveling from one corner of the rectangle to the other (either way around the rectangle). However, all of those terms can be replaced with terms in $\pi_1(A_{ij})$, where $A_{ij}$ is the open set into which $F$ maps $R_{r+1}$; the last bullet point on page 44 says that replacing these terms gives new factorizations which are equivalent to the old ones. Then we can combine all the terms going clockwise around $R_{r+1}$ into a single term (using the second-to-last bullet point on page 44), and we can also combine all the terms going counterclockwise around $R_{r+1}$ into a single term; this gives equivalent factorizations. Now these two terms in $\pi_1(A_{ij})$ are equal, since they are the homotopy classes of two loops in $A_{ij}$ which are homotopic through a homotopy in $A_{ij}$; thus the factorization of $[F|\gamma_r]$ is equivalent to the factorization of $[F|\gamma_{r+1}]$.