Hostname: page-component-78c5997874-fbnjt Total loading time: 0 Render date: 2024-11-11T10:13:44.566Z Has data issue: false hasContentIssue false
  • English
  • Français

Analysis of a stochastic approximation algorithm for computing quasi-stationary distributions

Part of: Markov processes

Published online by Cambridge University Press:  19 September 2016

J. Blanchet ,
P. Glynn  and
S. Zheng
J. Blanchet*
Affiliation:
Columbia University
P. Glynn*
Affiliation:
Stanford University
S. Zheng*
Affiliation:
Columbia University
*
* Postal address: Department of Industrial Engineering & Operations Research, Columbia University, 500 West 120th Street, New York, NY 10027, USA.
*** Postal address: Department of Management Science & Engineering, Stanford University, Huang Engineering Center, 475 Via Ortega, Stanford, CA 94035-4121, USA. Email address: glynn@stanford.edu
* Postal address: Department of Industrial Engineering & Operations Research, Columbia University, 500 West 120th Street, New York, NY 10027, USA.
Get access Rights & Permissions [Opens in a new window]

Abstract

We study the convergence properties of a Monte Carlo estimator proposed in the physics literature to compute the quasi-stationary distribution on a transient set of a Markov chain (see De Oliveira and Dickman (2005), (2006), and Dickman and Vidigal (2002)). Using the theory of stochastic approximations we verify the consistency of the estimator and obtain an associated central limit theorem. We provide an example showing that convergence might occur very slowly if a certain eigenvalue condition is violated. We alleviate this problem using an easy-to-implement projection step combined with averaging.

Keywords

Quasi-stationary distributionstochastic approximationMarkov chaincentral limit theorem

MSC classification

Primary: 60J22: Computational methods in Markov chains
Secondary: 60J10: Markov chains (discrete-time Markov processes on discrete state spaces)
Type
Research Article
Information
Advances in Applied Probability , Volume 48 , Issue 3 , September 2016 , pp. 792 - 811
Copyright
Copyright © Applied Probability Trust 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

[1] Aldous, D.,Flannery, B. and Palacios, J. L. (1988).Two applications of urn processes to the fringe analysis of search trees and the simulation of quasi-stationary distributions of Markov chains. Prob. Eng. Inf. Sci. 2,293–307.Google Scholar
[2] Athreya, K. B. and Karlin, S. (1968).Embedding of urn schemes into continuous time Markov branching processes and related limit theorems.Ann. Math. Statist. 39,1801–1817.Google Scholar
[3] Benaïm, M. and Cloez, B. (2015).A stochastic approximation approach to quasi-stationary distributions on finite spaces.Electron. Commun. Prob. 20, 14 pp.Google Scholar
[4] Blanchet, J.,Glynn, P. and Zheng, S. (2013).Empirical analysis of a stochastic approximation approach for computing quasi-stationary distributions. In EVOLVE,Springer,Berlin,19–37.Google Scholar
[5] Boyd, S. and Vandenberghe, L. (2004).Convex Optimization.Cambridge University Press.Google Scholar
[6] Burzdy, K.,Hołyst, R. and March, P. (2000).A Fleming‒Viot particle representation of the Dirichlet Laplacian.Commun. Math. Phys. 214,679–703.Google Scholar
[7] Darroch, J. N. and Seneta, E. (1965).On quasi-stationary distributions in absorbing discrete-time finite Markov chains.J. Appl. Prob. 2,88–100.Google Scholar
[8]}Darroch, J. N. and Seneta, E. (1967).On quasi-stationary distributions in absorbing continuous-time finite Markov chains.J. Appl. Prob. 4,192–196.Google Scholar
[9] De Oliveira, M. M. and Dickman, R. (2005).How to simulate the quasistationary state.Phys. Rev. E 71, 016129.Google Scholar
[10] De Oliveira, M. M. and Dickman, R. (2006).Quasi-stationary simulation: the subcritical contact process.Brazilian J. Phys. 36,685–689.Google Scholar
[11] Del Moral, P. and Miclo, L. (2006).Self-interacting Markov chains.Stoch. Anal. Appl. 24,615–660.Google Scholar
[12] Dickman, R. and Vidigal, R. (2002).Quasi-stationary distributions for stochastic processes with an absorbing state.J. Phys. A 35,1147–1166.CrossRefGoogle Scholar
[13] Dimov, I. T.,Karaivanova, A. N. and Yordanova, P. I. (1998).Monte Carlo algorithms for calculating eigenvalues. In Monte Carlo and Quasi-Monte Carlo Methods 1996(Salzburg; Lecture Notes Statist. 127),Springer,New York, pp. 205–220.Google Scholar
[14] Ferrari, P. A. and Marić, N. (2007).Quasistationary distributions and Fleming‒Viot processes in countable spaces.Electron. J. Prob. 12,684–702.Google Scholar
[15] Golub, G. H. and Van Loan, C. F. (1996).Matrix Computations,3rd edn.Johns Hopkins University Press,Baltimore, MD.Google Scholar
[16] Groisman, P. and Jonckheere, M. (2013).Simulation of quasi-stationary distributions on countable spaces.{Markov Process. Relat. Fields 19,521–542.Google Scholar
[17] Karlin, S. and Taylor, H. M. (1975).A First Course in Stochastic Processes,2nd edn.Academic Press,New York.Google Scholar
[18] Krasulina, T. P. (2013).The method of stochastic approximation for the determination of the least eigenvalue of a symmetrical matrix.{USSR Comput. Math. Math. Phys. 9,189–195.CrossRefGoogle Scholar
[19] Krasulina, T. P. (1970).Method of stochastic approximation in the determination of the largest eigenvalue of the mathematical expectation of random matrices.Automat. Rem. Contr. 2,215–221.Google Scholar
[20] Kushner, H. J. and Yin, G. (2003).Stochastic Approximation and Recursive Algorithms and Applications,2nd edn.Springer,New York.Google Scholar
[21] Méléard, S. and Villemonais, D. (2012).Quasi-stationary distributions and population processes.Prob. Surveys 9,340–410.CrossRefGoogle Scholar
[22] Oja, E. and Karhunen, J. (1985).On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix.J. Math. Anal. Appl. 106,69–84.Google Scholar
[23] Permantle, R. (2007).A survey of random processes with reinforcement.Prob. Surveys 4,1–79.Google Scholar
[24] Polyak, B. T. and Juditsky, A. B. (1992).Acceleration of stochastic approximation by averaging.SIAM J. Control Optimization 30,838–855.Google Scholar
[25] Robbins, H. and Monro, S. (1951).A stochastic approximation method.Ann. Math. Statist. 22,400–407.Google Scholar
[26] Zheng, S. (2014).Stochastic approximation algorithms in the estimation of quasi-stationary distribution of finite and general state space Markov chains. Doctoral Thesis, Columbia University.Google Scholar