Mathematics Stack Exchange
Mathematics Stack Exchange

Questions tagged [markov-process]

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space processes.

A Markov process is a stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. This tag is used for general state space processes both in discrete and continuous time, for countable state spaces use .

2574 questions
1
vote
0 answers

How to take this derivative? A question concerning a paper from Holley Stroock 1976

In the page 199 of the paper A Martingale approach to Infinite Systems of Interacting Processes one reads: I suspect that (1.5) is incorrect To simplify notation, let's write $X_\theta^s(t) = \exp(F(\lambda))$ where $$F(\lambda) = \int_s^{s\vee…
Conrado Costa
  • 6,071
1
vote
0 answers

A question concerning pg 124 of Ethier and Kurtz (1986) - Markov Processes

In page 123 of the book Markov Processes (Ethier Kurtz 1986, 2005)- convergence and characterization one reads So far no problem, but when we turn to page 124 a few troubles appear: Question 1 - What is the meaning of $\Gamma_{i/m_l}$ Question 2 -…
Conrado Costa
  • 6,071
1
vote
0 answers

Markov Chains that preserve an ordering of the state space

Suppose $X = (X_k)_{k=0}^\infty$ is a homogeneous Markov chain/process (for example on the state space $E = \lbrace 1, \dots, m\rbrace$). We can interpret the elements of the state space as "values". In the context of optimal stopping for example,…
Haro
  • 78
1
vote
1 answer

Generator - Markov process - Reorganization of the service

Two employees of a brokerage firm receive calls from customers regarding the purchase or sale of mutual funds. When their telephone lines were independent, each was busy a time of exponential expectancy $1/4$ hour with each client, during which time…
user320554
1
vote
0 answers

I have to show that the following stochastic process is a Markov process

I don't understand how to show that some stochastic processes have the Markov property. For example, if I have the following process: $$(\Omega, \mathcal{F}, (X_t)_{t \geq 0}, P^y)$$ where $\Omega = \mathbb{R}$, $\mathcal{F} =…
g.pomegranate
  • 2,215
0
votes
1 answer

Markov process states adding.

I got a question from my exam paper. In its third (c) question the transition lime and orange is combined to a single drink called Li-Ora. How can I add the transition probabilities in this case? Will they change. Below I have attached the question.…
Hiru
  • 151
  • 1
  • 1
  • 4
0
votes
1 answer

Markovian Coefficients Unclear Definitiondtdt

I have come across the following unclear definition: Consider $dS(t) = S(t)[\mu(t)dt + \sigma(t) dW(t)]$ "Assume that the coefficient $\sigma$ is Markovian. That is, (with abuse of notation) $\sigma(t) = \sigma(S(t))$." I don't see what we are…
user30201
  • 729
0
votes
1 answer

inequality for finite state markov chains

Let $X$ be a discrete-time Markov process in $S$ with invariant distribution $\nu$. Show that for any measurable set $B\subset S$ such that $$P_{\nu}\{X_n \in B\, \textrm{i.o.} \}\geq \nu B.$$ I'm honestly also unsure what $P_{\nu}$ means here. Is…
Kashif
  • 1,497
0
votes
1 answer

Asymmetric Markov process

The limiting distribution $\xi(x)$ of a Markov process $$x_0=1\text{ and }x_{i+1}=x_i+\Delta x_i,\tag1$$ where $\Delta x_i=-ax_i$ and $\Delta x_i=a$ occur with equal probability for every $i$, and $a\in(0,1)$ is a fixed parameter, is a good…
Andrey Sokolov
  • 1,536
  • 10
  • 15
0
votes
1 answer

Time Inhomogeneous markov processes to homogeneous one

I started studying RL recently using ashwin rao book "RL for finance".I'm studying Markov processes for now. At some point the author highlights the possibility of transforming by adding the time in the status rendering the process homogeneous. I…
Chg
  • 1
0
votes
1 answer

Why a continuous Markov chain is recurrent iff its embedded jump chain is recurrent?

I read this statement in the proof of this theorem 5 from the website Random Services. I am a little confused and I didn't find proof for it. Thanks for any explanation or clue!
namasikanam
  • 515
0
votes
1 answer

How to determine the number of distinct deterministic policies in a MDP?

I'm trying to tackle this question to understand MDP. Can someone explain how can you determine or calculate the number of distinct deterministic policies in the below MDP? Or resources where I can learn how to do this. I watched various videos and…
Nick
  • 1
0
votes
1 answer

Max. reachability in infinite-state MDP

Following [1], the maximum probability to reach a set of states $B\subseteq S$ from state $s\in S$ in a Markov decision process with finite state space $S$ can be expressed as the unique solution to the following system of equations over variables…
warakawa
  • 103
0
votes
1 answer

Continuous Markov Chain, Find Average of Served Clients per hour?

I'm stuck in a problem on markov jump process: A gas station receives cars at a rate of 20 vehicles per hour, the station has only one gas pump. If the pump is empty, it receives one client, if a new client comes and find that the pump is occupied…
Souames
  • 293
  • 2
  • 7
0
votes
0 answers

Markov decision Processes - Optimal state value function

I want to know how an optimal state value function defined for Markov decision Processes Could anyone be kind enough to define the Optimal State value function for MDP?
Pluviophile
  • 949
1 2
3
4 5