5 - Discrete-time Markov chains

Summary

Up until now, I only have considered processes in which one type of event can occur. In Chapter 3, the event signalled an irreversible change of state; in Chapter 4, it could recur. Except in the general theory of Section 4.1.1, I have made no assumptions about any dependencies among these events over time; the only condition was in the definition of the intensity as the probability of the event happening, given that it had not yet occured.

In this chapter, I shall begin the study of stochastic processes involving several kinds of events. I shall consider one way in which to introduce dependence assumptions, by conditioning on what happened before, here simply on the previous state(s). Thus, the probabilities of the various possible outcomes that can be observed at a given time will depend on the previous history of the process in one specific way. I shall model explicitly the probability of transition between states. In the simplest case, the present state will depend only on that immediately preceding it (Section 1.2.2), a first-order Markov chain. This procedure will yield state dependence models (Section 1.2.3).

I shall assume that each event occurs in a given constant interval of time and that these intervals are small enough so that only one event, however defined, can occur in each. In addition, in contrast to previous chapters, I shall assume that the small. Intervals are all of equal size, so that I am working in discrete time.