Mathematics Stack Exchange
Mathematics Stack Exchange

Questions tagged [markov-chains]

Stochastic processes (with either discrete or continuous time dependence) on a discrete (finite or countably infinite) state space in which the distribution of the next state depends only on the current state. For Markov processes on continuous state spaces please use (markov-process) instead.

A Markov chain is a stochastic process on a discrete (finite or countably infinite) space in which the distribution of the next state depends only on the current state. These objects show up in probability and computer science both in discrete-time and continuous-time models. For Markov processes on continuous spaces please use .

A discrete-time Markov chain is a sequence of random variables $\{X_n\}_{n\geq1}$ with the Markov property, namely that the probability of moving to the next state depends only on the present state and not on the previous states, i.e. $$\mathbb P(X_{n+1}=x\mid X_{1}=x_{1},X_{2}=x_{2},\ldots ,X_{n}=x_{n})=\mathbb P(X_{n+1}=x\mid X_{n}=x_{n}),$$ if both conditional probabilities are well defined, i.e. if $\mathbb P(X_{1}=x_{1},\ldots ,X_{n}=x_{n})>0.$

5938 questions
0
votes
1 answer

Finding a condition for which it is $P(\exists n\in\mathbb{N}: N_n=0)=0$

Let $X, X_{n,k}$ for $k,n\in\mathbb{N}$ denote independent random variables with values in $\mathbb{N}_0$. Define $N_0:=1$ and for $n\in\mathbb{N}$ set $$ N_n:=\begin{cases}0, & \text{ if }N_{n-1}=0\\X_{n,1}+\cdots+X_{n,N_{n-1}}, & \text{ if…
Salamo
  • 1,094
0
votes
2 answers

Why is P irreducible and aperiodic?

Let $P=(p_{ij})_{i,j\in E}$ denote the transition matrix of a Markov chain with finite state space $E$. Why does the following implication hold: $$ \exists n\in\mathbb{N}: p_{ij}^{(n)}>0~\forall i,j\in E~~\implies P^n\text{ is irreducible for…
user34632
0
votes
2 answers

Does from $P^n$ irreducible for all $n\in\mathbb{N}$ follow that $P$ is aperiodic?

let $P$ be a transition matrix of a Markov chain with state space E, that is finite. Does from $P^n$ irreducible for all $n\in\mathbb{N}$ follow that $P$ is irreducible and aperiodic? The first thing is clear. If $P^n$ is irreducible for all…
user34632
0
votes
1 answer

Is this statement about periodic communicating classes an equivalence statement?

In our reading about Markov chains we had the following theorem (including proof): Let $(X_n)_{n\in\mathbb{N}_0}$ denote a Markov chain with state space $E$. A periodic communicating class $C\subseteq E$ with period $p$ can be decomposed into a…
mathfemi
  • 2,631
-1
votes
1 answer

Discrete Time Markov Chains

Let $\{X_n\}_{n=0}^{\infty}$ be a Discrete Time Markov Chain (DTMC) with finite state space $X$. For any state $y \in X$ we define $Ty := \inf\{n \ge 1 \mid X_n = y\}$. Let $p(n)(x, y)$ denote the $n$-step transition probability of going from state…
Laila
  • 3
  • 2
-1
votes
1 answer

Markov chains and their applications

I'm trying to solve a Markov chain question, I believe this a special kind of those chains, I've solved part a still I'm stuck in part 2. A city is served by two newspapers, the Star and the Times. Each year the Star loses 30% of its subscribers to…
mandez
  • 750
-1
votes
1 answer

Markov chain between between days N-3, N-2, and N-1 and N

Suppose that whether or not a web server is in a fully operational mode at a specific day N depends on previous conditions of the server through the last three days only, i.e., days N-1, N-2 and N-3. Show how the performance of this web server can…
Liky
  • 109
-1
votes
1 answer

Whether the followings are Markov Chain

Suppose that $\{X_n\}$ are i.i.d. random variables with $\mathbb{P}(X_n=0)=\mathbb{P}(X_n=1)=1/2$. Let $Y_n=X_n+X_{n-1}$ and $Z_n=2X_n+X_{n-1}$. Why $\{Z_n\}$ is Markov chain but $\{Y_n\}$ is not?
Yuteng
  • 77
-1
votes
1 answer

How to find a lim$_{n\to\infty} $ P(Xn = z | X0 = y) for a Markov chain?

I am trying to answer some questions related to the following markov chain: matrix.The questions are: Is it irreducible? What's the stationary distribution? And I need to find this and explain why I get this answer. My answers are : Yes it is…
cru3lgenius
  • 19
-1
votes
1 answer

Markov Equations Balance equations and Normalising equations

I am looking at a question involving three equations: $A=0.6667A+0.2222B+0.1667C$ $B= 0.2A+0.3333B+0.5C$ $C=0.1333A+0.4444P+0.3333C$ The solution then goes on to say, that these equations can be re-written…
Rab1
  • 1
-1
votes
1 answer

Repair Chain (Markov Chain Sample Model)

A machine has $3$ critical parts that are subject to failure, but can function as long as two of these parts are working. When two are broken, they are replaced and the machine is back to working order the next day. To formulate a Markov chain model…
user175306
-2
votes
1 answer

Show that $~X_n~:~n\ge 0~$ is a Markov chain.

Consider a communication system which transmits the two digits $~0~$and$~1~$ through several stages. Let $~X_0~$ be the digit transmitted initially (leaving) $~0^{\text{th}}~$ stage and $~X_n~,~n\ge 1~$ be the digit leaving the $~n^{\text{th}}~$…
nmasanta
  • 9,222
-4
votes
1 answer

Markov chain- recurrent and transient?

In this Markov chain, are all the states recurrent since you can return back to every state starting anywhere? or are there any transient states? A Markov chain has transition probability matrix $$P=\begin{pmatrix} 0 & 1 & 0 & 0 \\ 0 & 0 & 1/3…
Rahul Chadha
  • 3
1 2 3
…
30
31