Mathematics Stack Exchange
Mathematics Stack Exchange

Questions tagged [markov-chains]

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

A Markov chain is a stochastic process on a discrete (finite or countably infinite) space in which the distribution of the next state depends only on the current state. These objects show up in probability and computer science both in discrete-time and continuous-time models. For Markov processes on continuous spaces please use .

A discrete-time Markov chain is a sequence of random variables $\{X_n\}_{n\geq1}$ with the Markov property, namely that the probability of moving to the next state depends only on the present state and not on the previous states, i.e. $$\mathbb P(X_{n+1}=x\mid X_{1}=x_{1},X_{2}=x_{2},\ldots ,X_{n}=x_{n})=\mathbb P(X_{n+1}=x\mid X_{n}=x_{n}),$$ if both conditional probabilities are well defined, i.e. if $\mathbb P(X_{1}=x_{1},\ldots ,X_{n}=x_{n})>0.$

5938 questions
0
votes
1 answer

Markov chain, being in one state until absorption

Assume that $X_t$ is a continuous time-inhomogeneous Markov process on the states $\{1,2,3\}$ where the state $3$ is absorbing, the other two states are transient. Let the random variable $T$ denote the time until absorbtion in state $3$. I want to…
Jonathan Kiersch
  • 365
0
votes
0 answers

To show that $X$ is a time homogeneous Markov chain if and only if there exists a function $f(x,y)$

Let $X : Ω \to S^{Z_+}$ be a random process for the discrete state space $S$. Show that $X$ is a time homogeneous Markov chain if and only if there exists a function $f(x,y)$ such that for all $x_0, x_1,. . ., x_{n−1}, x,y ∈ S$, we have $$P(X_{n+1}…
Siddhant
  • 473
0
votes
1 answer

Is this sequence Markov chain?

Let $\{X_{n}\}$ be a sequence of iid random variables with $P(X_n = j) = a_j > 0 $ for every $j \geq 0$. and let $\{Y_{n}\}$ be a sequence defined by $Y_{n} = X_{n} + X_{n - 1}$ for each $n\geq 1$ and $Y_0 = 0$. Is $\{Y_{n}\}$ a Markov chain? I…
user709945
0
votes
1 answer

Equivalence of definitions of Markov Chain Number of Visits

Let us have the following definitions for a Markov Chain $(X_n)_{n \ge 0}$ $$ T_i := inf\{n \ge 1 : X_n = i\}\\ T_i^{(r+1)} := inf\{n \ge T_i^{(r)}+1 : X_n = i\}\\ \tilde V_i := inf\{n \ge 1 : T_i^{(n)} = \infty \} \\ V_i :=…
Nikola
  • 1,558
0
votes
1 answer

When does the stationary distribution exist for a Markov chain?

I am trying to find (ideally) an if and only if statement stating the existence of a stationary distribution for a time-discrete, finite state Markov chain. So far I have only found sufficient conditions involving irreducibility of a Markov chain…
Abdul Miah
  • 237
0
votes
1 answer

Finding whether a Markov Chain is recurrent and positive recurrent

I have the following Markov Chain with infinite state space $I=\{0,1,2,3,4,...\}$ and transition matrix $$P = \begin{bmatrix}q_0 & p_0 & 0 & 0 & 0 & 0 & ...\\q_1 & 0 & p_1 & 0 & 0 & 0 & ...\\ q_2 & 0 & 0 & p_2 & 0 & 0 & ... \\ q_3 & 0 & 0 & 0 & p_3…
Nikola
  • 1,558
0
votes
1 answer

calculating infinite step transition probabilities

Consider a Markov chain with transition probability matrix $P$ given by $$\displaystyle{P=\begin{bmatrix} \frac{1}{2} & \frac{1}{2} & 0 \\ 0 & \frac{1}{2} & \frac{1}{2} \\ \frac{1}{3} & \frac{1}{3} & \frac{1}{3} \end{bmatrix}}$$ For any…
am_11235...
  • 2,142
0
votes
1 answer

Probabilty of entering state 4 exactly 4 times-Markov chain

I have Markov chain with states $S=\{1,2,3,4,5\}$ and probability matrix $P= \begin{bmatrix} 0.2 & 0.8 & 0 & 0&0\\ 0 & 0.4 & 0.6&0&0 \\ 0 & 0 & 0.6&0.4&0 \\ 0.2&0&0&0.6&0.2 \\ 0&0&0&0&1 \end{bmatrix} $ If the chain starts from state 1,…
Ivana Bogicevic
  • 3
  • 1
0
votes
1 answer

What does $\delta_n$ refer to in "distribution of $T_k$", when defining mean recurrence time?

What does $\delta_n$ refer to in "distribution of $T_k$", when defining mean recurrence time? Particularly $$T_k:= \inf \{ n \geq 1: f_n=k\}$$ then my notes say: $$\mathbb{P}(f_n=k, f_{n-1} \not=k, ..., f_1 \not=k | f_0=k)= f_{kk}^{(n)}$$ and that…
mavavilj
  • 7,270
0
votes
1 answer

Why can one write $\mathbb{P}(f_{j+i}=m|f_i=l) =\mathbb{P}(f_{j+i}=m|f_i=l, f_0=k)$ for Markov Chain?

Why can one write $\mathbb{P}(f_{j+i}=m|f_i=l) =\mathbb{P}(f_{j+i}=m|f_i=l, f_0=k)$ for Markov Chain? Is this application of Markov property?
mavavilj
  • 7,270
0
votes
1 answer

Why is $\mathbb{P}(f_{j+1}=m, f_i=l | f_0=k) \leq \mathbb{P}(f_{j+1}=m | f_0 =k)$?

Why is $\mathbb{P}(f_{j+1}=m, f_i=l | f_0=k) \leq \mathbb{P}(f_{j+1}=m | f_0 =k)$? Where $i,j \geq 0$ and nothing is assumed by the order of probabilities. And where $(f_i)_i$ is a Markov chain that's homogenous.
mavavilj
  • 7,270
0
votes
1 answer

What does $\lim_{\theta \uparrow 1}$ mean in criticality of Branching process?

What does $\lim_{\theta \uparrow 1}$ mean in criticality of Branching process? Assume that $\lim_{\theta \uparrow 1} F'(\theta)=F'(1) < \infty$. Where $F$ is is prob. generating function: $$F(\theta):= \sum_{l=0}^{\infty} \theta^l p_l$$
mavavilj
  • 7,270
0
votes
1 answer

What does the notation $(p^{(i)}(x_k))_{k=1,...,M}$ mean?

What does the notation $(p^{(i)}(x_k))_{k=1,...,M}$ mean? This is a marginal distribution. But I have troubles understanding whether this is a vector or a product or something? I think the notation refers to sequence notation as in…
mavavilj
  • 7,270
0
votes
2 answers

Markov chain transient state

I have got a Markov Chain as below $\begin{bmatrix}q & p & 0 & 0 &0\\0 & p &q & 0 &0\\0&p&0&q&0\\q&0&0&0&p\\0&0&0&0&1\end{bmatrix}$ I am asked to classify the class and if they are transient. The answer from my lecturer is {0,1,2,3} is a open class…
Thomas
  • 173
0
votes
0 answers

Meaning of the inverse of a Markov transition matrix

Consider a basic Markov process where vector $x$ at time $t$ switches states with a transition matrix $A$, so that: $$ x(t+1) = A x(t). $$ Assume that A is invertible, we can write the above as: $$ A^{-1}x(t+1) = x(t) $$ What is the meaning of…
econ
  • 101
1 2 3
…
30 31