Hostname: page-component-78c5997874-t5tsf Total loading time: 0 Render date: 2024-11-10T13:22:38.023Z Has data issue: false hasContentIssue false
  • English
  • Français

Perturbation analysis of inhomogeneous finite Markov chains

Part of: Markov processes Probabilistic methods, simulation and stochastic differential equations

Published online by Cambridge University Press:  24 March 2016

Bernd Heidergott ,
Haralambie Leahu ,
Andreas Löpker  and
Georg Pflug
Bernd Heidergott*
Affiliation:
Vrije Universiteit Amsterdam
Haralambie Leahu*
Affiliation:
University of Amsterdam
Andreas Löpker*
Affiliation:
Helmut Schmidt University
Georg Pflug*
Affiliation:
University of Vienna
*
* Postal address: Korteweg-de-Vries Institute for Mathematics, University of Amsterdam, Science Park 107, Postbus 94248, 1090 GE Amsterdam, The Netherlands. Email address: haralambie@gmail.com
** Postal address: Department of Economics and Social Sciences, Helmut Schmidt University, Hamburg, 22008, Germany. Email address: lopker@hsu-hh.de
*** Postal address: Department of Statistics and Operations Research, University of Vienna, Oskar-Morgenstern Platz 1, Vienna, 1090, Austria. Email address: georg.pflug@univie.ac.at
**** Postal address: Department of Econometrics and Operations Research and Tinbergen Institute, Vrije Universiteit Amsterdam, De Boelelaan 1105, 1081 HV Amsterdam, The Netherlands. Email address: b.f.heidergott@vu.nl
Get access Rights & Permissions [Opens in a new window]

Abstract

In this paper we provide a perturbation analysis of finite time-inhomogeneous Markov processes. We derive closed-form representations for the derivative of the transition probability at time t, with t > 0. Elaborating on this result, we derive simple gradient estimators for transient performance characteristics either taken at some fixed point in time t, or for the integrated performance over a time interval [0 , t]. Bounds for transient performance sensitivities are presented as well. Eventually, we identify a structural property of the derivative of the generator matrix of a Markov chain that leads to a significant simplification of the estimators.

Keywords

Finite Markov processinhomogeneous Markov processsensitivity analysisinfinitesimal generator

MSC classification

Primary: 60J27: Continuous-time Markov processes on discrete state spaces
Secondary: 65C05: Monte Carlo methods
Type
Research Article
Information
Advances in Applied Probability , Volume 48 , Issue 1 , March 2016 , pp. 255 - 273
Copyright
Copyright © Applied Probability Trust 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

[1]Abdalla, N. and Boucherie, R. J. (2002). Blocking probabilities in mobile communications networks with time-varying rates and redialing subscribers. Ann. Operat. Res. 112, 15–34. Google Scholar
[2]Abramov, V. and Liptser, R. (2004). On the existence of limiting distributions for time-nonhomogeneous countable Markov process. Queueing Systems 46, 353–361. Google Scholar
[3]Andreychenko, A., Crouzen, P. and Wolf, V. (2011). On-the-fly uniformization of time-inhomogeneous infinite Markov population models. In Proc. QAPL EPTCS, pp. 1–15. Google Scholar
[4]Artalejo, J. R. and Gómez-Corral, A. (2008). Retrial Queueing Systems. Springer, Berlin. Google Scholar
[5]Banasiak, J. and Arlotti, L. (2006). Perturbations of Positive Semigroups with Applications. Springer, London. Google Scholar
[6]Bellman, R. (1997). Introduction to Matrix Analysis, 2nd edn. SIAM, Philadelphia, PA. Google Scholar
[7]Brémaud, P. (1992). Maximal coupling and rare perturbation sensitivity analysis. Queueing Systems Theory Appl. 11, 307–333. CrossRefGoogle Scholar
[8]Brémaud, P. and Vázquez-Abad, F. J. (1992). On the pathwise computation of derivatives with respect to the rate of a point process: the phantom RPA method. Queueing Systems Theory Appl. 10, 249–269. Google Scholar
[9]Cao, X.-R. (2007). Stochastic Learning and Optimization: A Sensitivity-Based Approach. Springer, New York. Google Scholar
[10]Çinlar, E. (1975). Introduction to Stochastic Processes. Prentice-Hall, Englewood Cliffs, NJ. Google Scholar
[11]Dollard, J. D. and Friedman, C. N. (1978). On strong product integration. J. Funct. Anal. 28, 309–354. CrossRefGoogle Scholar
[12]Falin, G. (1990). A survey of retrial queues. Queueing Systems Theory Appl. 7, 127–167. Google Scholar
[13]Falin, G. I. and Templeton, J. G. C. (1997). Retrial Queues. Chapman & Hall, London. Google Scholar
[14]Feldman, Z., Mandelbaum, A., Massey, W. A. and Whitt, W. (2008). Staffing of time-varying queues to achieve time-stable performance. Manag. Sci. 54, 324–338. Google Scholar
[15]Fu, M. C. (2006). Gradient estimation. In Handbook on Operations Research and Management Science: Simulation, eds S. G. Henderson and B. L. Nelson, Elsevier, Amsterdam, pp. 575–616. Google Scholar
[16]Fu, M. and Hu, J.-Q. (1997). Conditional Monte Carlo: Gradient Estimation and Optimization Applications. Kluwer, Boston, MA. Google Scholar
[17]Gans, N., Koole, G. and Mandelbaum, A. (2003). Telephone call centers: tutorial, review, and research prospects. Manufacturing Service Operat. Manag. 5, 79–141. CrossRefGoogle Scholar
[18]Gill, R. D. and Johansen, S. (1990). A survey of product-integration with a view toward application in survival analysis. Ann. Statist. 18, 1501–1555. CrossRefGoogle Scholar
[19]Glasserman, P. (1991). Gradient Estimation via Perturbation Analysis. Kluwer, Boston, MA. Google Scholar
[20]Glasserman, P. and Gong, W.-B. (1990). Smoothed perturbation analysis for a class of discrete-event systems. IEEE Trans. Automatic Control 35, 1218–1230. Google Scholar
[21]Gong, W.-B. and Ho, Y.-C. (1987). Smoothed (conditional) perturbation analysis of discrete event dynamical systems. IEEE Trans. Automatic Control 32, 858–866. CrossRefGoogle Scholar
[22]Green, L. V., Kolesar, P. J. and Whitt, W. (2007). Coping with time-varying demand when setting staffing requirements for a service system. Production Operat. Manag. 16, 13–39. Google Scholar
[23]Heidergott, B. and Hordijk, A. (2003). Taylor series expansions for stationary Markov chains. Adv. Appl. Prob. 35, 1046–1070. (Correction: 36 (2004), 1300.) Google Scholar
[24]Heidergott, B. and Leahu, H. (2010). Weak differentiability of product measures. Math. Operat. Res. 35, 27–51. Google Scholar
[25]Heidergott, B. and Vázquez-Abad, F. J. (2008). Measure-valued differentiation for Markov chains. J. Optimization Theory Appl. 136, 187–209. CrossRefGoogle Scholar
[26]Ho, Y.-C. and Cao, X.-R. (1991). Perturbation Analysis of Discrete Event Dynamic Systems. Kluwer, Boston, MA. Google Scholar
[27]Johansen, S. (1986). Product integrals and Markov processes. CWI Newslett. 12, 3–13. Google Scholar
[28]Kato, T. (1984). Perturbation Theory for Linear Operators, 2nd edn. Springer, Berlin. Google Scholar
[29]Kulkarni, V. G. and Liang, H. M. (1997). Retrial queues revisited. In Frontiers in Queueing, CRC, Boca Raton, FL, pp. 19–34. Google Scholar
[30]Lewis, P. A. W. and Shedler, G. S. (1979). Simulation of nonhomogeneous Poisson processes by thinning. Naval Res. Logistics Quart. 26, 403–413. CrossRefGoogle Scholar
[31]Mamon, R. S. (2002). A time-varying Markov chain model of term structure. Statist. Prob. Lett. 60, 309–312. CrossRefGoogle Scholar
[32]Massey, W. A. (1985). Asymptotic analysis of the time dependent M/M/1 queue. Math. Operat. Res. 10, 305–327. Google Scholar
[33]Massey, W. A. (2002). The analysis of queues with time-varying rates for telecommunication models. Telecommun. Systems 21, 173–204. Google Scholar
[34]Massey, W. A. and Whitt, W. (1998). Uniform acceleration expansions for Markov chains with time-varying rates. Ann. Appl. Prob. 8, 1130–1155. Google Scholar
[35]Otte, P. (1999). An integral formula for section determinants of semi-groups of linear operators. J. Phys. A 32, 3793–3803. CrossRefGoogle Scholar
[36]Pflug, G. C. (1996). Optimization of Stochastic Models. Kluwer, Boston, MA. Google Scholar
[37]Rubinstein, R. Y. (1992). Sensitivity analysis of discrete event systems by the 'push out' method. Ann. Operat. Res. 39, 229–250. CrossRefGoogle Scholar
[38]Rubinstein, R. Y. and Melamed, B. (1998). Modern Simulation and Modeling. John Wiley, New York. Google Scholar
[39]Rubinstein, R. Y. and Kroese, D. P. (2008). Simulation and the Monte Carlo Method, 2nd edn. John Wiley, Hoboken, NJ. Google Scholar
[40]Van Dijk, N. M. (1992). Uniformization for nonhomogeneous Markov chains. Operat. Res. Lett. 12, 283–291. CrossRefGoogle Scholar
[41]Van Loan, C. (1977). The sensitivity of the matrix exponential. SIAM J. Numer. Anal. 14, 971–981. Google Scholar
[42]Vázquez-Abad, F. J. (1999). Strong points of weak convergence: a study using RPA gradient estimation for automatic learning. Automatica 35, 1255–1274. Google Scholar
[43]Yang, T. and Templeton, J. G. C. (1987). A survey on retrial queues. Queueing Systems 2, 201–233. (Correction: 4 (1989), 94.) Google Scholar
[44]Yin, G. and Zhang, Q. (1989). Continuous Time Markov Chains and Applications. Springer, New York. Google Scholar
[45]Zeifman, A. and Korotysheva, A. (2011). Weak ergodicity of Mt/Mt/N/N+R queue. Pliska Stud. Math. Bulgar. 20, 243–254. Google Scholar