11Cmarkov


 * Markov Chains **


 * Markov chains ** are a __**discrete**__ probability distribution where each outcome is conditional (dependent) __**only**__ on the outcome of the experiment __**immediately**__ before it.

We will only consider two state Markov chains. ** Two state Markov chains ** have two possible outcomes (2 states) at each experiment.

NOTE: In a two state Markov chain, the outcomes are __**not independent**__, so this is __**not**__ a binomial distribution.

To understand a Markov chain, use a tree diagram:


 * Example 1 **

A school has 200 students. Records show that 70% of the students who are absent on a particular day will be absent on the following day as well. Also, 10% of the students who are present will be absent on the following day. On a particular day (day 1), 20 students are absent.

Notice that the probabilities depend only on the outcome of the day before so this is a __**Markov chain**__. Also, notice that there are only two possible outcomes (present or absent) so this is a **__two state Markov chain__**.




 * The __expected number__ of students absent on Day 2 will be the sum of the numbers in red: **

... ... E(Abs day2) ... ... = PA + AA ... ... =18 + 14 ... ... =32 (students absent)

**The __probability__ that a random student is present on Day 2 can be calculated:**

... ... 162 + 6 = 168 students present out of 200 so: ... .... Pr(Pres day2) = 0.84


 * OR **

... ... Pr(Pres day2) ... ... = PP + AP ... ... = 0.9 x 0.9 + 0.1 x 0.3 ... ... = 0.84




 * The __expected number__ of students absent on Day 3 will be the sum of the numbers in red: **

... ... E(Abs day3) ... ... = PPA + PAA + APA + AAA ... ... = 16.2 + 12.6 + 0.6 + 9.8 ... ... = 39.2

Notice that mostly we are only interested in the total number present or absent on a given day.

Just occasionally, you may see a question like this:


 * Find the __probability__ that a random student is present on Day 3 __given that__ they were absent on Day 1: **

... ... Pr(Pres day3 | Abs day1) ... ... = (5.4 + 4.2)/20 ... ... = 0.48


 * OR **

... ... Pr(Pres day3 | Abs day1) ... ... = APP + AAP ... ... = 0.3 x 0.9 + 0.7 x 0.3 ... ... = 0.48

{as you can see, repeating a Markov chain for several cycles makes the tree diagram very big}
 * {Fortunately, there is another way} **

Transition Matrices


 * Markov Chains ** can be solved using matrices provided we are only interested in the final numbers.


 * Example 1 (repeated) **

A school has 200 students. Records show that 70% of students who are absent on a particular day will be absent the following day as well. Also 10% of the students who are present will be absent the following day. On a particular day, 20 students are absent.

__**Solution:**__

This situation can be set out in a ** transition probability table **:

Note: The columns should represent the current state and the rows represent the next state.

Note: Row 1 and Column 1 should refer to the same activity. It doesn't matter which way around you do it provided you are consistent.

The transition probability table can then be represented as a ** transition matrix ** (T) {Keep track of which row/column represents which activity}

math . \qquad T = \left[ \begin{matrix} 0.7 & 0.1 \\ 0.3 & 0.9 \\ \end{matrix} \right] math

The initial state S(0) and the current state S(n) of the system can then be represented by a column matrix.

math . \qquad S \big( 0 \big) = \left[ \begin{matrix} 20 \\ 180 \\ \end{matrix} \right] \qquad \qquad S \big( n \big) = \left[ \begin{matrix} \text{Absent after n days} \\ \text{Present after n days} \\ \end{matrix} \right] math

Note: In **T**, __Absent__ is on the top row, so __Absent__ must be on the top row in **S** as well.

Now, to find S(1) we multiply T × S(0)

math . \qquad S \big( 1 \big) = \left[ \begin{matrix} 0.7 & 0.1 \\ 0.3 & 0.9 \\ \end{matrix} \right] \, \left[ \begin{matrix} 20 \\ 180 \\ \end{matrix} \right] = \left[ \begin{matrix} 32 \\ 168 \\ \end{matrix} \right] math

Similarly, we can find S(2) by multiplying T × S(1)

math . \qquad S \big( 2 \big) = \left[ \begin{matrix} 0.7 & 0.1 \\ 0.3 & 0.9 \\ \end{matrix} \right] \, \left[ \begin{matrix} 32 \\ 168 \\ \end{matrix} \right] = \left[ \begin{matrix} 39.2 \\ 160.8 \\ \end{matrix} \right] math

And we can then find S(3) by multiplying T × S(2)

math . \qquad S \big( 3 \big) = \left[ \begin{matrix} 0.7 & 0.1 \\ 0.3 & 0.9 \\ \end{matrix} \right] \, \left[ \begin{matrix} 39.2 \\ 160.8 \\ \end{matrix} \right] = \left[ \begin{matrix} 43.52 \\ 156.48 \\ \end{matrix} \right] math

OR {This is easy on the calculator}

We can keep referring back to S(0)

math \\ . \qquad S \big( 1 \big) = T \times S \big( 0\big) \\. \\ . \qquad S \big( 2 \big) = T \times T \times S \big( 0 \big) = T^2 \times S \big( 0 \big) \\. \\ . \qquad S \big( 3 \big) = T \times T \times T \times S \big( 0 \big) = T^3 \times S \big( 0 \big) math

math . \qquad S \big( 1 \big) = \left[ \begin{matrix} 0.7 & 0.1 \\ 0.3 & 0.9 \\ \end{matrix} \right] \, \left[ \begin{matrix} 20 \\ 180 \\ \end{matrix} \right] = \left[ \begin{matrix} 32 \\ 168 \\ \end{matrix} \right] math

math . \qquad S \big( 2 \big) = \left[ \begin{matrix} 0.7 & 0.1 \\ 0.3 & 0.9 \\ \end{matrix} \right]^2 \, \left[ \begin{matrix} 20 \\ 180 \\ \end{matrix} \right] = \left[ \begin{matrix} 39.2 \\ 160.8 \\ \end{matrix} \right] math

math . \qquad S \big( 3 \big) = \left[ \begin{matrix} 0.7 & 0.1 \\ 0.3 & 0.9 \\ \end{matrix} \right]^3 \, \left[ \begin{matrix} 20 \\ 180 \\ \end{matrix} \right] = \left[ \begin{matrix} 43.52 \\ 156.48 \\ \end{matrix} \right] math

In general, S(n) = T n × S(0)


 * {This rule is on your formula sheet} **

Steady State of a Markov Chain By collecting the numbers present and absent over a sequence of days, we can see that the values are slowly converging on a specific number of students present/absent each day.

When they reach that number, the values will remain on that number each day without changing. This is called a ** steady state **, or the ** long term behaviour ** of the system.

Note: This is __**not**__ asymptotic behaviour. The values actually adopt the numbers they are converging to and stay there.

Note: For any given system, the __**same**__ steady state values will be achieved regardless of the initial values.

To find the **long term behaviour** (or the ** steady state **) using **__transition matrices__**, choose a large number for n ( n = 50 is usually sufficient) and use your calculator to find S(50)

S(steady) = T 50 × S(0)


 * Example 1 (continued) **

math . \qquad S \big( 50 \big) = \left[ \begin{matrix} 0.7 & 0.1 \\ 0.3 & 0.9 \\ \end{matrix} \right]^{50} \, \left[ \begin{matrix} 20 \\ 180 \\ \end{matrix} \right] = \left[ \begin{matrix} 50 \\ 150 \\ \end{matrix} \right] math

Hence, in the long term, there will be 50 absent and 150 present on each day.

Finding the Steady State without a Calculator
 * {I do this differently from the text} **


 * Example 1 (continued) **

Let P = the number present in the steady state Let A = the number absent in the steady state


 * Total = 200,
 * so P + A = 200
 * so A = 200 – P ** {Equation 1} **

Express the rule for P in terms of P and A {notice that in the steady state, P(next) = P(previous)}

or
 * Present(next day) = 0.9 × Present + 0.3 × Absent
 * P = 0.9P + 0.3A ** {Equation 2} **

{Sub Eqn 1 into Eqn 2}


 * P = 0.9P + 0.3(200 – P)
 * P = 0.9P + 60 – 0.3P
 * P = 0.6P + 60
 * 0.4P = 60
 * ** P = 150 **

Using Eqn 1
 * A = 200 – P
 * ** A = 50 **

ie In the long term in the situation described, on any given day
 * 150 students will be present
 * 50 students will be absent


 * Example 2 **

Commuters travelling into the centre of Trenchtown use either the bus or the train. Research shows that each month, 20% of those using the bus switch to train travel and 30% of those using the train switch to bus travel. At the start of January, 4800 were using the bus and 3600 were using the train.

__**Solution:**__

This situation can be set out in a transition probability table: This gives us a transition matrix:

math . \qquad T = \left[ \begin{matrix} 0.7 & 0.2 \\ 0.3 & 0.8 \\ \end{matrix} \right] math

And the intial state and current state are given by:

math . \qquad S \big( 0 \big) = \left[ \begin{matrix} 3600 \\ 4800 \\ \end{matrix} \right] \qquad \qquad S \big( n \big) = \left[ \begin{matrix} \text{Train after n months} \\ \text{Bus after n months} \\ \end{matrix} \right] math


 * a) Find the number of people using train and bus at the beginning of May **.

**Beginning of May is after 4 months**

math . \qquad S \big( 4 \big) = \left[ \begin{matrix} 0.7 & 0.2 \\ 0.3 & 0.8 \\ \end{matrix} \right]^{4} \, \left[ \begin{matrix} 3600 \\ 4800 \\ \end{matrix} \right] = \left[ \begin{matrix} 3375 \\ 5025 \\ \end{matrix} \right] math

Hence, at the beginning of May,
 * 3375 will use the train
 * 5025 will use the bus.


 * b) Find the number of people using train and bus in the long term **.

math . \qquad S \big( 50 \big) = \left[ \begin{matrix} 0.7 & 0.2 \\ 0.3 & 0.8 \\ \end{matrix} \right]^{50} \, \left[ \begin{matrix} 3600 \\ 4800 \\ \end{matrix} \right] = \left[ \begin{matrix} 3360 \\ 5040 \\ \end{matrix} \right] math

Hence, in the long term,
 * 3360 will use the train
 * 5040 will use the bus.

Use
 * c) Find steady state for this system without a calculator. **
 * T = number on Train
 * B = number of Bus
 * Total = 8400
 * B = 8400 – T .... ** {1} **


 * Express the rule for T in terms of T and B: **
 * T(next) = 0.7T + 0.2B


 * T = 0.7T + 0.2B ..... ** {sub 1} **
 * T = 0.7T + 0.2(8400 – T)
 * T = 0.7T + 1680 – 0.2T
 * T = 0.5T + 1680
 * 0.5T = 1680
 * ** T = 3360 **


 * B = 8400 – T ..... ** {1} **
 * ** B = 5040 **

Hence, in the long term,
 * 3360 would use the train
 * 5040 would use the bus.

NOTE: There is an assumption built into the idea of Markov Chains that the __total__ population stays constant. .