How would this Markov chain problem be solved?

Question

I have the following problem:

Bob is a salesman. Each week, Bob will either make no money, make a small amount of money, or make a large amount of money. And if Bob makes money, he will either make \$100 (small amount) or \$1000 (large amount). For any given week, if Bob doesn't make any money in the previous week, then the probability that he will make a small amount of money is 0.6, and the probability that he will make a large amount of money is 0.3. For a week where he made a small amount of money in the previous week, the probability of making a small amount of money is 0.4 and the probability of making a large amount of money is 0.2. If Bob made a large amount of money in the previous week, then this week he will make a small amount of money with probability 0.3, and no money with probability 0.7. Let this be a Markov chain $(X_n, n \ge 0)$ with state space $\{0, 1, 2 \}$, where making no money is $0$, making a small amount of money is $1$, and making a large amount of money is $2$.

Assume that Bob earned a small amount of money this week. If we then wanted to calculate the expected value of the total number of times he earns money (both small and large amounts) over the next two weeks (after this week), then how would we do so? Is this a so-called 'hitting times' problem, or is it something else? In the field of stochastic processes, these things often go by different names, so I'm not exactly sure what it would be called; I would appreciate a link to some learning material, such as some online lecture notes or something similar.

Answer 1

Let $Y_k$ be the indicator variable for $X_k \in \{1, 2\}$; then, the desired quantity is $$\operatorname{E}(Y_1 + Y_2 \mid X_0 = 1).$$

Since expectation is linear, and the expected value of an indicator for an event is the probability of said event, this boils down to computing

$$ P(X_1 \in \{1, 2\} \mid X_0 = 1) + P(X_2 \in \{1, 2\} \mid X_0 = 1) $$

as pointed out by @Ian in the comments. We can further simplify this to

$$ \sum_{k=1}^{2} \sum_{j=1}^{2} P(X_k = j \mid X_0 = 1) $$

since the events $X_k = 1$ and $X_k = 2$ are clearly disjoint.

From the information given in the question, the probability matrix for this Markov chain is

$$M = \begin{pmatrix} 0.1 & 0.6 & 0.3\\ 0.4 & 0.4 & 0.2\\ 0.7 & 0.3 & 0 \end{pmatrix}.$$

If we index the rows and columns from $0$ to $2$, then the $(i,j)$-entry of $M$ is $P(X_1 = j \mid X_0 = i)$.

Similarly for

$$M^2 = \begin{pmatrix} 0.46 & 0.39 & 0.15\\ 0.34 & 0.46 & 0.2\\ 0.19 & 0.54 & 0.27 \end{pmatrix},$$

the $(i,j)$-entry is $P(X_2 = j \mid X_0 = i)$. If we read the correct entries off of each matrix, we see that the desired sum is

$$ 0.4 + 0.2 + 0.46 + 0.2 = 1.26. $$