$\newcommand{\R}{\mathbf{R}}\renewcommand{\P}{\mathbf{P}}$Note that in the projective plane $\P^{2}$, there's no intrinsic notion of points lying at infinity. Instead, a point of $\P^{2}$ (i.e., a line through the origin of $\R^{3}$) may (or may not) lie at infinity with respect to a choice of "affine coordinate chart" (or "dehomogenization").
As your book says, the set of points at infinity with respect to any particular affine chart is a projective line. The points on this "line at infinity" correspond geometrically to directions in the affine chart. Algebraically, points in the chart $Z \neq 0$ are represented uniquely by dehomogenized triples:
$$
[X : Y : Z] \leftrightarrow [x : y : 1] = (x, y),\quad x = X/Z,\ y = Y/Z.
$$
A point at infinity in this chart is (by definition) a point of $\P^{2}$ not represented by these affine coordinates, namely, a triple with $Z = 0$, a.k.a. a line through the origin of $\R^{3}$ in the $(X, Y)$-plane. Starting at the "origin" $(0, 0)$ of the affine chart and traveling along the line through $(x, y) \neq (0, 0)$ is tantamount to looking at points of $\P^{2}$ represented by
$$
(tx, ty) = [tx : ty : 1] = [x : y : \tfrac{1}{t}];
$$
as $|t| \to \infty$, the point $(tx, ty)$ therefore approaches $[x : y : 0]$, the "point at infinity along the direction $(x, y)$".
Regarding your question about distance and apparent size, it looks to me that you're trying to reconcile two different notions of convergence. Specifically, suppose $[X : Y : Z]$ is a point with $Z \neq 0$ (but free to vary) and $(X, Y) \neq (0, 0)$ is fixed. In the affine chart with $(x, y) = (X/Z, Y/Z)$:
If $Z \to 0$, then $(x, y)$ approaches $[x : y : 0]$ in $\P^{2}$; the same is true of every point on the affine line through $(0, 0)$ and $(x, y)$ (other than $(0, 0)$ itself). This recapitulates the preceding observation that points at infinity in an affine chart correspond to affine directions in that chart.
If $|Z| \to \infty$ (the object moves away from the camera, staying parallel to the viewing axis $[0 : 0 : 1]$), then $(x, y) \to (0, 0)$. Loosely, "distant objects appear to be close to the origin". This recapitulates your video lecture ("...divide by $Z$..."), and explains why parallel railroad tracks "meet at infinite distance". The terminology is potentially confusing: This meeting point at infinite distance along the tracks does not lie in the line at infinity in screen coordinates.
Generally, in the chart $Z \neq 0$, the line $\ell$ in $\R^{3}$ defined parametrically by $[X_{0} + tA : Y_{0} + tB : Z_{0} + tC]$ (with $C \neq 0$, and not passing through the origin of $\R^{3}$) projects to the affine line
$$
\left(\frac{X_{0} + tA}{Z_{0} + tC}, \frac{Y_{0} + tB}{Z_{0} + tC}\right).
$$
As $|t| \to \infty$, this point approaches $(A/C, B/C)$, the common vanishing point for lines with direction vector $(A, B, C)$. Again, this vanishing point lies at "infinite distance" along $\ell$, but represents a (finite) location in the affine $(x, y)$ coordinates.
It may be a helpful exercise to sketch all this in $\R^{3}$, "unwrapping" $\P^{2}$ to the unit sphere.