Hostname: page-component-cd9895bd7-jn8rn Total loading time: 0 Render date: 2025-01-06T03:35:37.399Z Has data issue: false hasContentIssue false
  • English
  • Français

Discrepancy Risk Model Selection Test Theory for Comparing Possibly Misspecified or Nonnested Models

Published online by Cambridge University Press:  01 January 2025

R. M. Golden
R. M. Golden*
Affiliation:
Applied Cognition and Neuroscience Program, University of Texas at Dallas
*
Requests for reprints should be sent to Richard M. Golden, Applied Cognition and Neuroscience Program (GR4.1), University of Texas at Dallas, Box 830688, Richardson, TX 75083-0688. E-Mail: golden@utdallas.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

A new model selection statistical test is proposed for testing the null hypothesis that two probability models equally effectively fit the underlying data generating process (DGP). The new model selection test, called the Discrepancy Risk Model Selection Test (DRMST), extends previous work (see Vuong, 1989) on this problem in four distinct ways. First, generalized goodness-of-fit measures (which include log-likelihood functions) can be used. Second, unlike the classical likelihood ratio test, the models are not required to be fully nested where the nesting concept is defined for generalized goodness-of-fit measures. The DRMST also differs from the likelihood ratio test by not requiring that either competing model provides a completely accurate representation of the DGP. And, fourth, the DRMST may be used to compare competing time-series models using correlated observations as well as data consisting of independent and identically distributed observations.

Keywords

asymptotic statistical theorymodel selectionhypothesis-testingmodel misspecificationtime-seriesm-estimation
Type
Articles
Information
Psychometrika , Volume 68 , Issue 2 , June 2003 , pp. 229 - 249
Copyright
Copyright © 2003 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

This research was supported in part by a University of Texas at Dallas Special Faculty Development Award and the NSF Information Technology Research Initiative through the Research on Learning and Education Program Award 0113369 to the author at the University of Texas at Dallas. This research was also supported by the National Institute on Alcohol Abuse and Alcoholism grant to Martingale Research Corporation (Grant No. R44AA11607). The author thanks Halbert White, Steven Henley, and the UTD A2N2 research group for discussions regarding issues related to this manuscript. The author also thanks three anonymous reviewers for feedback and comments regarding a previous version of this paper.

References

Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In Petrox, B.N., Caski, F. (Eds.), Second International Symposium on Information Theory (pp. 267–267). Budapest: Akademiai Kiado.Google Scholar
Balasubramian, V. (1997). Statistical inference, Occam's razor, and statistical mechanics on the space of probability distributions. Neural Computation, 9, 349–368.CrossRefGoogle Scholar
Barth, M.E., Beaver, W.H., Landsman, W.R. (1998). Relative valuation roles of equity book value and net income as a function of financial health. Journal of Accounting & Economics, 25, 1–34.CrossRefGoogle Scholar
Bozdogan, H. (1987). Model selection and Akaike's information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52, 345–370.CrossRefGoogle Scholar
Chen, K.Y., Plott, C.R. (1998). Nonlinear behavior in sealed bid first price auctions. Game Theory and Economic Behavior, 25, 34–78.CrossRefGoogle Scholar
Clarke, S., Barron, A.R. (1990). Information-theoretic asymptotics of Bayes methods. IEEE Transactions on Information Theory, IEEE-IT, 36, 453–471.CrossRefGoogle Scholar
Davies, R.B. (1980). The distribution of a linear combination ofx 2 random variables. Applied Statistics, 29, 323–333.CrossRefGoogle Scholar
Djuric, P.M. (1998). Asymptotic MAP criteria for model selection. IEEE Transactions on Signal Processing, 46, 2726–2735.CrossRefGoogle Scholar
Draper, N.R., Smith, H. (1981). Applied regression analysis. New York, NY: Wiley.Google Scholar
Efron, B. (1984). Comparing nonnested linear models. Journal of the American Statistical Association, 79, 791–803.CrossRefGoogle Scholar
Franklin, J.N. (1968). Matrix theory. New Jersey, NJ: Prentice-Hall.Google Scholar
Foutz, R.V., Srivastava, R.C. (1977). The performance of the likelihood ratio test when the model is incorrect. Annals of Statistics, 5, 1183–1194.CrossRefGoogle Scholar
Gan, L., Jiang, J. (1999). A test for global maximum. Journal of the American Statistical Association, 94, 847–855.CrossRefGoogle Scholar
Gertham, U.G. (1997). Equity in health care utilization: Further tests based on hurdle models and Swedish micro data. Health Economics, 6, 303–319.3.0.CO;2-P>CrossRefGoogle Scholar
Golden, R.M. (1995). Making correct statistical inferences using a wrong probability model. Journal of Mathematical Psychology, 38, 3–20.CrossRefGoogle Scholar
Golden, R.M. (1996). Mathematical methods for neural network analysis and design. Cambridge, MA: MIT Press.Google Scholar
Golden, R.M. (1998). Knowledge digraph contribution analysis of protocol data. Discourse Processes, 25, 179–210.CrossRefGoogle Scholar
Golden, R.M. (2000). Statistical tests for comparing possibly misspecified and nonnested models. Journal of Mathematical Psychology, 44, 153–170.CrossRefGoogle ScholarPubMed
Gordon, R.A. (1994). The integrals of Lebesgue, Denjoy, Perron, and Henstock. Providence, RI: American Mathematical Society.CrossRefGoogle Scholar
Grootendorst, P.V. (1995). A comparison of alternative models of prescription drug utilization. Health Economics, 4, 183–198.CrossRefGoogle ScholarPubMed
Kass, R.E., Wasserman, L. (1995). A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion. Journal of the American Statistical Association, 90, 928–934.CrossRefGoogle Scholar
Larson, H.J., Shubert, B.O. (1979). Probabilistic models in engineering sciences: Volume 1. New York, NY: Wiley.Google Scholar
Linhart, H. (1988). A test whether two AIC's differ significantly. South African Statistical Journal, 22, 153–161.Google Scholar
Linhart, H., Zucchini, W. (1986). Model selection. New York, NY: Wiley.Google Scholar
Martingale, Research (1997). CCR Data Modeling [Computer software]. Bedford, TX: Author.Google Scholar
Myung, I., Forster, M., Browne, M. (2000). Special issue on model selection. Journal of Mathematical Psychology, 44, 1–2.CrossRefGoogle Scholar
Qian, G., Kunsch, H.R. (1998). Some notes on Rissanen's stochastic complexity. IEEE Transactions on Information Theory, 44, 782–786.CrossRefGoogle Scholar
Rissanen, J.J. (1996). Fisher information and stochastic complexity. IEEE Transactions on Information Theory, 42, 40–47.CrossRefGoogle Scholar
Rivers, D., Vuong, Q. (2002). Model selection tests for nonlinear dynamic models. The Econometrics Journal, 5, 1–39.CrossRefGoogle Scholar
Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464.CrossRefGoogle Scholar
Serfling, R.J. (1980). Approximation theorems of mathematical statistics. New York, NY: Wiley.CrossRefGoogle Scholar
Sheil, J., O'Muircheartaigh, I. (1977). The distribution of nonnegative quadratic forms in normal variables. Applied Statistics, 26, 92–98.CrossRefGoogle Scholar
Shimodaira, H. (1997). Assessing the error probability of the model selection test. Annals of the Institute of Statistical Mathematics, 49, 395–410.CrossRefGoogle Scholar
Sin, C., White, H. (1996). Information criteria for selecting possibly misspecified parametric models. Journal of Econometrics, 71, 207–225.CrossRefGoogle Scholar
Vincent, L. (1999). The information content of funds from operations (FFO) for real estate investment trusts (REITs). Journal of Accounting & Economics, 26, 69–104.CrossRefGoogle Scholar
Vuong, Q.H. (1989). Likelihood ratio tests for model selection and nonnested hypotheses. Econometrica, 57, 307–333.CrossRefGoogle Scholar
Vuong, Q.H., Wang, W. (1993). Minimum chi-square estimation and tests for model selection. Journal of Econometrics, 56, 141–168.CrossRefGoogle Scholar
Wang, Q.B., Halbrendt, C., Johnson, S.R. (1996). A nonnested test of the AIDS vs. the translog demand system. Economics Letters, 51, 139–143.CrossRefGoogle Scholar
White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50, 1–25.CrossRefGoogle Scholar
White, H. (1984). Asymptotic theory for econometricians. San Diego, CA: Academic Press.Google Scholar
White, H. (1989). Some asymptotic results for learning in single hidden-layer feedforward network models. Journal of the American Statistical Association, 84, 1003–1013.CrossRefGoogle Scholar
White, H. (1994). Estimation, inference, and specification analysis. New York, NY: Cambridge University Press.CrossRefGoogle Scholar
Wilks, S.S. (1938). The large sample distribution of the likelihood ratio for testing composite hypotheses. Annals of Mathematical Statistics, 9, 60–62.CrossRefGoogle Scholar