$P(i)$ the probability that Markov chain eventually enters state $N$ given that it starts in state $i$. Show $\{P(X_n), n\geq 0\}$ is a martingale

Question

Consider the Markov chain $\{X_n, n \geq 0\}$ with $P_{NN} = 1$. Let $P(i)$ denote the probability that this chain eventually enters state $N$ given that it starts in state $i$. Show that $\{P(X_n), n\geq 0\}$ is a martingale.

This is exercise 6.5 of Sheldon Ross's Stochastic processes. In that chapter, it said (right below (6.1.2) that if we wanted to show that $Z_n$ is a martingale sequence, we can just show $$E[Z_{n+1}|Z_1,...,Z_n, Y] = Z_n$$

for any set of random variables $Y$. Since the Markov chain only has $N$ states, thus, with $Z_n = P(X_n)$, I was thinking of showing $$E[Z_{n+1}|P(1),...,P(N)] = Z_n$$

However, since $Z_{n+1} = P(X_n)$ where $X_n \in [N]$, thus $E[Z_{n+1}|P(1),...,P(N)] = P(X_n) = Z_{n+1}$. However, this is not the result I want.

Answer 1

You weren't using it consistently, but it's still a good idea to define $Z_n = P(X_n)$. Then we want to show $$\mathbb{E}[Z_{n+1} \mid Z_1, \ldots, Z_n, X_n] = Z_n$$

Towards this, note for each $j \in [N]$, $$\begin{align*} \mathbb{E}[\chi[X_{n+1}=j] \mid Z_1, \ldots, Z_n, X_n] &= \mathbb{E}[\mathbb{E}[\chi[X_{n+1}=j] \mid X_1, \ldots, X_n] \mid Z_1, \ldots, Z_n, X_n] \\ &= \mathbb{E}[\mathbb{E}[\chi[X_{n+1}=j] \mid X_n] \mid Z_1, \ldots, Z_n, X_n] \\ &= \mathbb{E}[\chi[X_{n+1}=j] \mid X_n] \\ &= \mathbb{P}[X_{n+1}=j \mid X_n] \\ &= \sum_{i=1}^N P_{ij}\cdot \chi[X_n=i]. \end{align*}$$

Since $Z_{n+1} = \sum_{j=1}^N P(j)\chi[X_{n+1}=j],$ we can apply the above to show $$\begin{align*} \mathbb{E}[Z_{n+1} \mid Z_1, \ldots, Z_n, X_n] &= \sum_{j=1}^NP(j)\mathbb{E}[\chi[X_{n+1}=j] \mid Z_1, \ldots, Z_n, X_n] \\ &= \sum_{j=1}^NP(j)\sum_{i=1}^NP_{ij} \chi[X_n = i] \\ &= \sum_{i=1}^N\left(\sum_{j=1}^NP(j)P_{ij}\right)\chi[X_n=i]. \end{align*}$$

Comparing this to $Z_n = \sum_{i=1}^N P(i)\chi[X_n = i]$, we now see the whole problem comes down to showing that, for each $i\in [N]$, $$P(i) = \sum_{j=1}^NP(j)P_{ij}.$$

---

This last equation is true because $P_{NN}=1$ implies $N$ is recurrent. It's effectively just proposition 4.4.2 on page 186 of the 2nd edition.