0

We have the following state-space $S = \{0,1,2,3\} $ with transition matrix:$$ P = \begin{pmatrix}\frac{1}{2} & \frac{1}{2} & 0 & 0 \\ \frac{1}{2}&\frac{1}{2}&0&0 \\ 0 &0 &\frac{1}{3} & \frac{2}{3} \\ 0 &0&0&1 \end{pmatrix}$$

Deduce a general form for $P^n$ in terms of $n$ and use this to calculate $P(X_{20} = 2|X_{10}=2)$.

After plugging in the matrix into R to easily calculate a large number of matrix multiplications to see the trend. I find that all values remain the same besides $X_{3, 3} X_{3, 4}$. We find that as multiplcations increase, then $X_{3,4}$converges closer to one, and $X_{3,3}$ towards zero.

I can find this easily with a programming software but I cannot do the mathematical calculations to derive this. How do I find the general form for $P^n$?

As for the second part:

     [,1] [,2]         [,3] [,4]
[1,]  0.5  0.5 0.000000e+00    0
[2,]  0.5  0.5 0.000000e+00    0
[3,]  0.0  0.0 2.867972e-10    1
[4,]  0.0  0.0 0.000000e+00    1

Is the matrix at the twentieth time-step, and because of floating point arithmetic the software can only compute small numbers, but its largely safe to say it's zero at $P(X_{20}=2|X_{10}=2)$

Peter Strouvelle
  • 923
Working dollar
  • 241

2 Answers2

1

You just need to draw out the transition diagram(or equivalently, observe that only particular transitions are made): $$P(X_t=1|X_{t1} = 1) = \frac{1}{2} = P(X_t=1|X_{t1} = 2) = P(X_t=2|X_{t1} = 1) = P(X_t=2|X_{t1} = 2)$$ Since the states 1 and 2 act as the Gilbert Elliot model(a two-state MDP) independently. The transitions to all other states from both 1 and 0 are clearly 0. The probabilities are half because the one-step transition leads to half-and-half given any state, 1 or 2. Since this is the stationary distribution for the 2 state MDP, it remains that way. If you do not know about stationary distributions, just try multiplying the upper 2x2 matrix twice. You will see it leads to the same matrix.

For the states 3 and 4, I think you have a typo at 3->4 (should be 2/3?). Even they do not transition outside to 1, 2. Additionally, observe that the state 4 acts as a sink. So $P(X_t = 4 | X_{t1} = 4) = 1$ and rest are 0. Finally, for 3 we have $P(X_t = 3 | X_{t1} = 3) = (\frac{1}{3})^{t-t1}$ and $P(X_t = 4 | X_{t1} = 3) = 1-(\frac{1}{3})^{t-t1}$.

cybershiptrooper
  • 753
0

Your matrix has a block structure. Let $$A=\frac12\begin{pmatrix} 1&1\\1&1 \end{pmatrix}$$ and $$B=\begin{pmatrix} \frac13&\frac23\\0&1 \end{pmatrix}.$$

There for $$P^n=\begin{pmatrix} A^n & 0 \\ 0 & B^n \end{pmatrix}$$

Further $A^n=A$, and $B^n$ can be computed easily, since it is triangular (you might use induction or diagonalisation).

PS: can you check the matrix again? The chain would either be absorbed in $\{0,1\}$ or in $2$. The computations with R don't fit to your matrix.

Peter Strouvelle
  • 923