Hostname: page-component-745bb68f8f-lrblm Total loading time: 0 Render date: 2025-01-15T16:00:24.891Z Has data issue: false hasContentIssue false
  • English
  • Français

ADAPTIVE BAYESIAN ESTIMATION OF CONDITIONAL DENSITIES

Published online by Cambridge University Press:  13 July 2016

Andriy Norets  and
Debdeep Pati
Andriy Norets*
Affiliation:
Brown University
Debdeep Pati
Affiliation:
Florida State University
*
*Address correspondence to Andriy Norets, Associate Professor, Department of Economics, Brown University, Providence, RI 02912; e-mail: andriy_norets@brown.edu.
Get access Rights & Permissions [Opens in a new window]

Abstract

We consider a nonparametric Bayesian model for conditional densities. The model is a finite mixture of normal distributions with covariate dependent multinomial logit mixing probabilities. A prior for the number of mixture components is specified on positive integers. The marginal distribution of covariates is not modeled. We study asymptotic frequentist behavior of the posterior in this model. Specifically, we show that when the true conditional density has a certain smoothness level, then the posterior contraction rate around the truth is equal up to a log factor to the frequentist minimax rate of estimation. An extension to the case when the covariate space is unbounded is also established. As our result holds without a priori knowledge of the smoothness level of the true density, the established posterior contraction rates are adaptive. Moreover, we show that the rate is not affected by inclusion of irrelevant covariates in the model. In Monte Carlo simulations, a version of the model compares favorably to a cross-validated kernel conditional density estimator.

Type
ARTICLES
Information
Econometric Theory , Volume 33 , Issue 4 , August 2017 , pp. 980 - 1012
Copyright
Copyright © Cambridge University Press 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

We thank the editor, the co-editor, and referees for helpful comments. Dr. Pati acknowledges support for this project from the Office of Naval Research (ONR BAA 14-0001) and NSF DMS-1613156.

References

REFERENCES

Barron, A., Schervish, M.J., & Wasserman, L. (1999) The consistency of posterior distributions in nonparametric problems. The Annals of Statistics 27(2), 536–561.Google Scholar
Bhattacharya, A., Pati, D., & Dunson, D. (2014) Anisotropic function estimation using multi-bandwidth Gaussian processes. The Annals of Statistics 42(1), 352–381.Google Scholar
Chung, Y. & Dunson, D.B. (2009) Nonparametric Bayes conditional distribution modeling with variable selection. Journal of the American Statistical Association 104(488), 1646–1660.CrossRefGoogle ScholarPubMed
De Iorio, M., Muller, P., Rosner, G.L., & MacEachern, S.N. (2004) An ANOVA model for dependent random measures. Journal of the American Statistical Association 99(465), 205–215.CrossRefGoogle Scholar
De Jonge, R. & van Zanten, J.H. (2010) Adaptive nonparametric Bayesian inference using location-scale mixture priors. The Annals of Statistics 38(6), 3300–3320.Google Scholar
Dunson, D.B. & Park, J.H. (2008) Kernel stick-breaking processes. Biometrika 95(2), 307–323.Google Scholar
Dunson, D.B., Pillai, N., & Park, J.H. (2007) Bayesian density regression. Journal of the Royal Statistical Society Series B (Statistical Methodology) 69(2), 163–183.Google Scholar
Efromovich, S. (2007) Conditional density estimation in a regression setting. The Annals of Statistics 35(6), 2504–2535.Google Scholar
Geweke, J. & Keane, M. (2007) Smoothly mixing regressions. Journal of Econometrics 138, 252–290.Google Scholar
Ghosal, S., Ghosh, J.K., & Ramamoorthi, R.V. (1999) Posterior consistency of Dirichlet mixtures in density estimation. The Annals of Statistics 27(1), 143–158.Google Scholar
Ghosal, S., Ghosh, J.K., & van der Vaart, A.W. (2000) Convergence rates of posterior distributions. The Annals of Statistics 28(2), 500–531.CrossRefGoogle Scholar
Ghosal, S. & van der Vaart, A.W. (2001) Entropies and rates of convergence for maximum likelihood and Bayes estimation for mixtures of normal densities. The Annals of Statistics 29(5), 1233–1263.Google Scholar
Ghosal, S. & van der Vaart, A.W. (2007) Posterior convergence rates of Dirichlet mixtures at smooth densities. The Annals of Statistics 35(2), 697–723.Google Scholar
Griffin, J.E. & Steel, M.F.J. (2006) Order-based dependent Dirichlet processes. Journal of the American Statistical Association 101(473), 179–194.CrossRefGoogle Scholar
Hall, P., Racine, J., & Li, Q. (2004) Cross-validation and the estimation of conditional probability densities. Journal of the American Statistical Association 99(468), 1015–1026.Google Scholar
Hayfield, T. & Racine, J.S. (2008) Nonparametric econometrics: The np package. Journal of Statistical Software 27(5), 1–32.Google Scholar
Huang, T.M. (2004) Convergence rates for posterior distributions and adaptive estimation. The Annals of Statistics 32(4), 1556–1593.Google Scholar
Jacobs, R.A., Jordan, M.I., Nowlan, S.J., & Hinton, G.E. (1991) Adaptive mixtures of local experts. Neural Computation 3(1), 79–87.Google Scholar
Jordan, M. & Xu, L. (1995) Convergence results for the em approach to mixtures of experts architectures. Neural Networks 8(9), 1409–1431.Google Scholar
Keane, M. & Stavrunova, O. (2011) A smooth mixture of tobits model for healthcare expenditure. Health Economics 20(9), 1126–1153.Google Scholar
Kruijer, W., Rousseau, J., & van der Vaart, A. (2010) Adaptive Bayesian density estimation with location-scale mixtures. Electronic Journal of Statistics 4, 1225–1257.Google Scholar
Li, F., Villani, M., & Kohn, R. (2010) Flexible modeling of conditional distributions using smooth mixtures of asymmetric student t densities. Journal of Statistical Planning and Inference 140(12), 3638–3654.Google Scholar
Li, Q. & Racine, J.S. (2007) Nonparametric Econometrics: Theory and Practice. Princeton University Press.Google Scholar
MacEachern, S.N. (1999) Dependent nonparametric processes. Proceedings of the Section on Bayesian Statistical Science, pp. 50–55. American Statistical Association.Google Scholar
Norets, A. (2010) Approximation of conditional densities by smooth mixtures of regressions. The Annals of Statistics 38(3), 1733–1766.Google Scholar
Norets, A. (2015) Optimal retrospective sampling for a class of variable dimension models. Unpublished manuscript, Brown University.Google Scholar
Norets, A. & Pelenis, J. (2012) Bayesian modeling of joint and conditional distributions. Journal of Econometrics 168, 332–346.CrossRefGoogle Scholar
Norets, A. & Pelenis, J. (2014) Posterior consistency in conditional density estimation by covariate dependent mixtures. Econometric Theory 30, 606–646.Google Scholar
Pati, D., Dunson, D.B., & Tokdar, S.T. (2013) Posterior consistency in conditional distribution estimation. Journal of Multivariate Analysis 116, 456–472.Google Scholar
Peng, F., Jacobs, R.A., & Tanner, M.A. (1996) Bayesian inference in mixtures-of-experts and hierarchical mixtures-of-experts models with an application to speech recognition. Journal of the American Statistical Association 91(435), 953–960.Google Scholar
Rousseau, J. (2010) Rates of convergence for the posterior distributions of mixtures of betas and adaptive nonparametric estimation of the density. The Annals of Statistics 38(1), 146–180.Google Scholar
Scricciolo, C. (2006) Convergence rates for Bayesian density estimation of infinite-dimensional exponential families. Annals of Statatistics 34(6), 2897–2920.Google Scholar
Shen, W. & Ghosal, S. (2016) Adaptive Bayesian density regression for high-dimensional data. Bernoulli 22(1), 396–420.Google Scholar
Shen, W., Tokdar, S.T., & Ghosal, S. (2013) Adaptive Bayesian multivariate density estimation with Dirichlet mixtures. Biometrika 100(3), 623–640.Google Scholar
Tokdar, S., Zhu, Y., & Ghosh, J. (2010) Bayesian density regression with logistic Gaussian process and subspace projection. Bayesian Analysis 5(2), 319–344.Google Scholar
van der Vaart, A.W. & van Zanten, J.H. (2009) Adaptive Bayesian estimation using a Gaussian random field with inverse gamma bandwidth. The Annals of Statistics 37(5B), 2655–2675.Google Scholar
Villani, M., Kohn, R., & Giordani, P. (2009) Regression density estimation using smooth adaptive Gaussian mixtures. Journal of Econometrics 153(2), 155–173.Google Scholar
Villani, M., Kohn, R., & Nott, D.J. (2012) Generalized smooth finite mixtures. Journal of Econometrics 171(2), 121–133.Google Scholar
Wade, S., Dunson, D.B., Petrone, S., & Trippa, L. (2014) Improving prediction from Dirichlet process mixtures via enrichment. The Journal of Machine Learning Research 15(1), 1041–1071.Google Scholar
Wood, S., Jiang, W., & Tanner, M. (2002) Bayesian mixture of splines for spatially adaptive nonparametric regression. Biometrika 89(3), 513–528.Google Scholar
Yang, Y. & Tokdar, S.T. (2015) Minimax-optimal nonparametric regression in high dimensions. The Annals of Statistics 43(2), 652–674.Google Scholar