Mathematics Stack Exchange
Mathematics Stack Exchange

Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

A Markov process is a stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. This tag is used for general state space processes both in discrete and continuous time, for countable state spaces use .

2574 questions
0
votes
1 answer

Branching process - probability that the branching process survives forever with 4 individuals

Consider a branching process where the number of offspring of an individual is a binomial random variable with parameters $(2, p),$ with $p \in (0, 1).$ Initially there is one individual. (a) Calculate the probability, as a function of $p,$ that…
underdisplayname
  • 1,240
0
votes
1 answer

Markov process: The population distribution of the system after $n$-transitions

I am familiar with discrete markov processes and idea of their convergence towards a stable distribution after some number of steps. But my question is much simpler--more like a validation of some intuition. Say I am looking at the transition of…
krishnab
  • 2,461
0
votes
0 answers

What's the difference between discounted cost , total expected cost and average expect cost MDP?

What's the difference between discounted cost , total expected cost and average expect cost MDP? Are they just MDP problems with different objective function? When the discounted factor equals 1, then discounted cost mdp becomes total cost? Can…
Cubic
  • 1
0
votes
1 answer

Markov Decision Process for several players

I just read a paper by the authors Kohlberg and Neyman stating that "A single-person stochastic game is known as a Markov Decision Process (MDP)." Does anyone know if the following extension to $n$ players might work? Take, say, 2 players playing a…
Leon
  • 61
  • 2
0
votes
1 answer

Questions about Markov process usage

I've read the answer of George Lowther on this question and I don't understand his proof. I know just a few things on Markov Process. If I define $\left(X_n\right)_{n \in \mathbb{N}}$ with $X_0=1$ and $$ X_{n+1}=\begin{cases} 2X_n,&\text{with…
Atmos
  • 7,369
0
votes
1 answer

What allows us to use Fubini in this passage? A question concerning Liggett's 1985 book.

In the book Interacting Particle system, chapter 1 pg 43 one reads It is often the case that $\Omega$ is an unbounded operator. Considering that this is the case, what allows us to use Fubini's theorem in the…
Conrado Costa
  • 6,071
0
votes
1 answer

How do we verify that these limit objects are martingales? A question concerning a paper from Holley & Stroock, 1976

In page 199 of the paper A Martingale approach to Infinite Systems of Interacting Processes one reads: Bearing in mind that there might be a typo in $|\theta|^2_c$ as mentioned in How to take this derivative? A question concerning a paper from…
Conrado Costa
  • 6,071
0
votes
1 answer

How do we prove that this process is a martingale? A question concerning a paper from Holley Stroock, 1976

In the paper A Martingale approach to Infinite Systems of Interacting Processes one reads: I wasn't able to prove using Theorems 1.1 and 1.2 that (1.3) is a $P$ martingale. context: Here is theorem (1.1) and here is theorem (1.2) (along with some…
Conrado Costa
  • 6,071
0
votes
1 answer

A question concerning a possible typo in Kipnis and Cocozza 1977

In the paper Existence de processus Markoviens pour des systèmes infinis de particules by Cocozza, C. and Kipnis, C. (Ann. lnst. H. Poincaré, Sect. B, 13, 239-257, 1977), one reads (pg 249) While $A^{x_1,\ldots,x_k}_{t_1,\ldots,t_k}$ is defined in…
Conrado Costa
  • 6,071
0
votes
1 answer

Trouble understanding the definitions in Cocozza & Kipnis 1977

In the paper Existence de processus Markoviens pour des systèmes infinis de particules by Cocozza, C. and Kipnis, C. (Ann. lnst. H. Poincaré, Sect. B, 13, 239-257, 1977), one reads If $m^x \in \mathcal{M}^1$ what is the meaning of $m^x]0,k]$ Isn't…
Conrado Costa
  • 6,071
0
votes
0 answers

How do we obtain this inequality? A question concerning an argument in Ethier and Kurtz 1986 (pg 137))

In page 137 of Ethier and Kurtz(1986 - Markov processes, convergence and characterization) one reads: I don't see how that follows, imagine that $\Delta >\delta$ so $q(X(\delta), X(\Delta \wedge \delta)) = 0$ and if (8.27) were true…
Conrado Costa
  • 6,071
0
votes
1 answer

How do we obtain this inequality? A question concerning an argument in Ethier and Kurtz 1986 (pg 135)[answered]

In page 135 of Ethier and Kurtz(1986 - Markov processes, convergence and characterization) one reads: The question is, do we need the factor $a_\beta$ to be squared? Since we have (8.22) it seems that can we use triangle inequality as follows: $$…
Conrado Costa
  • 6,071
0
votes
1 answer

Prove or disprove that the successive maxima of sums of i.i.d. increments are a Markov process

Let $\{\xi_n\}$ be independent, identically distributed, random variables. Define $S_k = \sum\limits_{i=0}^k \xi_i $ and $\eta_k = \max(S_0, ..., S_k)$. How to prove or disprove that $\{ \eta_k\}$ is a Markov process? I have a feeling that…
xolodec
  • 103
0
votes
1 answer

Transition intensities of Markov process

In Markov process, transition intensities from state i to j are defined as derivatives of transition probabilities at zero: $$q_{ij}=p_{ij}'(0)$$ However I can't somehow catch the interpretation of transition intensities. My first thought was that…
Paweł Orliński
  • 626
0
votes
1 answer

Markov processes Hitting times

I'm having trouble understanding what hitting times are in Markov chain processes and how they are calculated. An example follows: A Markov process on $E = \{1, 2, 3\}$ has the following generator matrix $$\begin{pmatrix}−2 & 1 & 1 \\ 2 & −4 & 2…
kullapparos
  • 21
1 2 3
4
5