2

I have a Markov chain with state space $S = \left\{ 1,2,\dots \right\}$ with transition matrix as follows

$\begin{bmatrix} 0 & 1 & 0 & 0 & 0 & 0 & \dots \\ 0 & 0 & 1 & 0 & 0 & 0 & \dots \\ 0 & 0 & \frac{2}{3} & \frac{1}{3} & 0 & 0 & \dots \\ 0 & 0 & \frac{2}{3} & 0 & \frac{1}{3} & 0 & \dots \\ 0 & 0 & \frac{2}{3} & 0 & 0 & \frac{1}{3} & \dots \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots \end{bmatrix} $

I am trying to find $\lim_{n\to\infty} P^n(4,7)$ and I believe I should be using this theorem:

$\textbf{Theorem 7}$ Let $X_n, n > 0$, be an irreducible positive recurrent Markov chain having stationary distribution $\pi$. If the chain is aperiodic,

$(55) \lim_{n\to\infty} P^n(x,y) \space x,y \in S$

If the chain is periodic with period d, then for each pair $x, y$ of states in $S$ there is an integer $r, 0 \leq r < d$, such that $P^n(x, y) = 0$ unless $n = md + r$ for some nonnegative integer $m$, and

$(56) \lim_{m\to\infty} P^{md+r}(x,y) = d\pi(y)$

The problem is that it does not satisfy the conditions of being irreducible and positive recurrent so I cannot use the theorem. However I see that if I consider only the block not containing states $1,2$ ie

$\begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 & \dots \\ \frac{2}{3} & 0 & \frac{1}{3} & 0 & \dots \\ \frac{2}{3} & 0 & 0 & \frac{1}{3} & \dots \\ \vdots & \vdots & \vdots & \vdots & \ddots \end{bmatrix} $

Then this is irreducible, positive recurrent and would be aperiodic and can use the theorem. Would it be valid to apply the theorem to this block matrix so that I can find

$\lim_{n\to\infty} P^n(4,7)$?

zoli
  • 20,452
lllll
  • 409

1 Answers1

3

Yes it is appropriate to consider the the state transition matrix

$$\begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 & \dots \\ \frac{2}{3} & 0 & \frac{1}{3} & 0 & \dots \\ \frac{2}{3} & 0 & 0 & \frac{1}{3} & \dots \\ \vdots & \vdots & \vdots & \vdots & \ddots \end{bmatrix} $$

since the original chain gets in this chain in two steps. So we can forget about the first two states, 1 and 2, $P_1$ and $P_2$ will be $0$ on the long run because the chain will never get (back) to these states.

In the case of the transition matrix above, it is easy to calculate the stationary probabilities:

So, we have the stationary probabilities in a row vector $\pi=[P_3\ P_4 \ P_5\cdots]$ and we have the equation for the stationary probabilities

$$[P_3\ P_4 \ P_5\cdots]\begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 & \dots \\ \frac{2}{3} & 0 & \frac{1}{3} & 0 & \dots \\ \frac{2}{3} & 0 & 0 & \frac{1}{3} & \dots \\ \vdots & \vdots & \vdots & \vdots & \ddots \end{bmatrix}=[P_3\ P_4 \ P_5\cdots].$$

In the case of the row vector and the firs column we get

$$\frac23(P_3+P_4+\cdots)=\frac23=P_3.$$ Then in the case of the row vector and the second column we have

$$P_3\frac13=\frac29=P_4$$

and so on.

So we have for the row vector of the stationary probabilities

$$\left[0 \ 0\ \frac23\ \ \frac29\ \ \frac2{27}\ \cdots\right].$$

Here $P_1$ and $P_2$ were included.

If I understood the question well then

$$\lim_{n\to\infty} p^n(4,7)=\frac2{3^5}=\pi(7)$$

because

$$\lim_{n\to\infty} \begin{bmatrix} 0&1&0&0&\cdots\\ 0&0&1&0&\cdots\\ \frac{2}{3} & \frac{1}{3} & 0 & 0 & \dots \\ \frac{2}{3} & 0 & \frac{1}{3} & 0 & \dots \\ \frac{2}{3} & 0 & 0 & \frac{1}{3} & \dots \\ \vdots & \vdots & \vdots & \vdots & \ddots \end{bmatrix}^n =\begin{bmatrix} 0&0&\frac23&\frac2{3^2}&\frac2{3^3}&\frac2{3^4}&\frac2{3^5}\cdots\\ 0&0&\frac23&\frac2{3^2}&\frac2{3^3}&\frac2{3^4}&\frac2{3^5}\cdots\\ 0&0&\frac23&\frac2{3^2}&\frac2{3^3}&\frac2{3^4}&\frac2{3^5}\cdots\\ 0&0&\frac23&\frac2{3^2}&\frac2{3^3}&\frac2{3^4}&\boxed{\color{red}{\frac2{3^5}}}\cdots\\ \vdots & \vdots & \vdots & \vdots & \vdots& \vdots & \ddots \end{bmatrix}.$$

zoli
  • 20,452