First, we note that, for any real number $\alpha \neq 0$, we can find an open interval $( \beta, \gamma)$ such that $\alpha \in (\beta, \gamma)$ and such that $0 \not\in (\beta, \gamma)$: in fact we can take
$$
( \beta, \gamma) \colon=
\begin{cases}
\left( \frac{\alpha}{2}, 2 \alpha \right) \ & \mbox{ if } \ \alpha > 0, \\
\left( 2 \alpha, \frac{\alpha}{2} \right) \ & \mbox{ if } \ \alpha < 0.
\end{cases}
\tag{0}
$$
For the box topology:
Let $\mathbf{a} \colon= \left( \alpha_1, \alpha_2, \ldots \right)$ be an arbitrary element of $\mathbb{R}^\omega \setminus \mathbb{R}^\infty$. Then there are infinitely many natural numbers $n$ for which $\alpha_n \neq 0$, and for each one of these $n$ we can find an open interval $\left( \beta_n, \gamma_n \right)$ containing $\alpha_n$ and not containing $0$, as in (0) above.
Let $ \left( n_1, n_2, \ldots \right)$ be the strictly increasing sequence of all those natural numbers $n$ for which $\alpha_n \neq 0$. Let us consider the box topology basis element
$$ B \colon= \prod_{n \in \mathbb{N} } \left( a_n, b_n \right) = \left( a_1, b_1 \right) \times \left( a_2, b_2 \right) \times \cdots, $$
where $$
\left( a_n, b_n \right) \colon=
\begin{cases}
\left( \beta_n, \gamma_n \right) \ & \mbox{ if } \ n = n_k \mbox{ for some } k \in \mathbb{N}, \\
(-1, 1) \ & \mbox{ otherwise}.
\end{cases}
\tag{1}
$$
Then $$\mathbf{a} \in B \subset \mathbb{R}^\omega \setminus \mathbb{R}^\infty, \tag{2}$$
for if $\mathbf{x} \colon= \left( x_1, x_2, \ldots \right) \in B$, then $x_{n_k} \in \left( \beta_{n_k}, \gamma_{n_k} \right)$, which implies that $x_{n_k} \neq 0$ for all $k \in \mathbb{N}$; that is, $x_n \neq 0$ for infinitely many natural numbers $n$, and so $\mathbf{x} \in \mathbb{R}^\omega \setminus \mathbb{R}^\infty$.
Thus we have shown that for each point $\mathbf{a} \in \mathbb{R}^\omega \setminus \mathbb{R}^\infty$, we can find a box topology basis element $B$ such that (2) holds. Therefore $\mathbb{R}^\omega \setminus \mathbb{R}^\infty$ is open, and so $\mathbb{R}^\infty$ is closed in $\mathbb{R}^\omega$ in the box topology, which implies that
$$ \overline{\mathbb{R}^\infty} = \mathbb{R}^\infty. $$
Now for the product topology:
Let $\mathbf{x} \colon= \left( x_1, x_2, \ldots \right)$ be any element of $\mathbb{R}^\omega$, and let $U \colon= \prod_{n \in \mathbb{N} } U_n$ be a product topology basis element containing $\mathbf{x}$. Then $U_n$ is open in $\mathbb{R}$ for each $n \in \mathbb{N}$, and $U_n \neq \mathbb{R}$ for at most finitely many $n \in \mathbb{N}$; let $n_1, \ldots, n_k$ be all the natural numbers $n$ for which $U_n \neq \mathbb{R}$, and let $$N \colon= \max \left\{ \ n_1, \ldots, n_k \ \right\}. $$
Then $U_n = \mathbb{R}$ for every natural number $n > N$. Now let $\mathbf{x}^\prime \colon= \left( x_1^\prime, x_2^\prime, \ldots \right)$ such that, for each $n \in \mathbb{N}$,
$$
x_n^\prime \colon=
\begin{cases}
x_n \ & \mbox{ if } n \leq N, \\
0 \ & \mbox{ if } n > N.
\end{cases}
$$
Then obviously this $\mathbf{x}^\prime \in U \cap \mathbb{R}^\infty$ so that
$U \cap \mathbb{R}^\infty$ is non-empty.
Thus we have shown that every product topology basis element $U$ in $\mathbb{R}^\omega$ containing $\mathbf{x}$ intersects $\mathbb{R}^\infty$. So $\mathbf{x} \in \overline{\mathbb{R}^\infty}$.
But $\mathbf{x}$ was an arbitrary element of $\mathbb{R}^\omega$. Thus we can conclude that every element of $\mathbb{R}^\omega$ is in the closure of $\mathbb{R}^\infty$. That is,
$$ \overline{ \mathbb{R}^\infty } = \mathbb{R}^\omega. $$