Hostname: page-component-5f745c7db-f9j5r Total loading time: 0 Render date: 2025-01-06T06:47:44.575Z Has data issue: true hasContentIssue false
  • English
  • Français

Data, Model, Conclusion, Doing it Again

Published online by Cambridge University Press:  01 January 2025

Ivo W. Molenaar
Ivo W. Molenaar*
Affiliation:
University of Groningen
*
Requests for reprints should be sent to Ivo W. Molenaar, Rijksuniversiteit te Groningen, Vakgroep Statistiek en Meettheorie, Grote Kruisstraat 2/1, 9712 TS Groningen, THE NETHERLANDS.
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper explores the robustness of conclusions from a statistical model against variations in model choice (rather than variations in random sampling and random assignment to treatments, which are the usual variations covered by inferential statistics). After the problem formulation in section 1, section 2 presents an example from Box and Tiao in which variation in parent distribution is modeled for a one sample location problem. An adaptive Bayesian procedure permits to use a weighted mixture of parent distributions rather than choosing just one, such as a normal or a uniform distribution.

In section 3 similar considerations are applied to an event history model for the influence of education and gender on age at first marriage, but here the conclusions are hardly influenced by the choice of the duration distribution. In section 4 a brief discussion of model choice in factor analysis and structural equation models is followed by a more elaborate treatment of the choice of integer valued slopes for item response functions in the OPLM model which is an extension of the Rasch model. A modest simulation study suggests that Adaptive Bayesian Modeling with a mixture of sets of slopes works better than fixing one set of postulated slopes.

The paper concludes with some remarks on the roles and sources of prior distributions followed by a short epilogue which argues that simultaneous consideration of a class of models for the same data is sometimes superior to exclusively analyzing the data under one specific model chosen from such a class.

Keywords

adaptive Bayesian modelingmodel choiceposterior model weightrobustness under model choiceone parameter logistic model
Type
Original Paper
Information
Psychometrika , Volume 63 , Issue 4 , December 1998 , pp. 315 - 340
Copyright
Copyright © 1998 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

This article is based on the Presidential Address Ivo W. Molenaar gave on June 20, 1998 at the 1998 Annual Meeting of the Psychometric Society held at the University of Illinois in Champaign, Illinois. Thanks is given to John Wiley & Sons and George C. Tiao for granting permission to reprint three figures from the book George C. Tiao wrote with George E. P. Box titled Bayesian Inference in Statistical Analysis.—Editor

Thanks are due to Anne Boomsma, Jeffrey Hoogland, Mark Huisman, Tom Snijders, Marijtje Van Duijn, and Norman Verhelst for suggesting improvements and/or assisting with data analyses.

References

Bernardo, J. M., & Smith, A. F. M. (1994). Bayesian theory, Chichester: John Wiley & Sons.CrossRefGoogle Scholar
Blossfeld, H-P., & Rohwer, G. (1995). Techniques of event history modeling. New approaches to causal analysis, Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Boomsma, A. (1991). BOJA. A program for bootstrap and jackknife analysis (User's manual, PC version), Groningen: iec ProGAMMA.Google Scholar
Box, G. E. P., & Andersen, S. L. (1955). Permutation theory in the derivation of robust criteria and the study of departures from assumptions. Journal of the Royal Statistical Society, Series B, 17.CrossRefGoogle Scholar
Box, G. E. P., & Tiao, G. C. (1973). Bayesian inference in statistical analysis, Menlo Park, CA: Addison-Wesley.Google Scholar
Bozdogan, H. (1987). Model selection and Akaike's information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52, 345–370.CrossRefGoogle Scholar
Browne, M. W. (1984). Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37, 62–83.CrossRefGoogle ScholarPubMed
Camstra, A. (1998). Cross-validation in covariance structure analysis, The Netherlands: University of Groningen.Google Scholar
Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1, 245–276.CrossRefGoogle ScholarPubMed
Cliff, N. (1979). Test theory without true scores?. Psychometrika, 44, 373–393.CrossRefGoogle Scholar
Cox, D. R. (1972). Regression models and life-tables. Journal of the Royal Statistical Society, 34, 187–220.CrossRefGoogle Scholar
Cudeck, R., & Browne, M. W. (1983). Cross-validation for prediction. Multivariate Behavioral Research, 18, 147–167.CrossRefGoogle Scholar
De Gooijer, J. G. (1995). Cross-validation criteria for covariance structures. Communications in Statistics: Simulation and Computation, 24, 1–16.CrossRefGoogle Scholar
De Groot, A. D. (1984). The theory of the science forum: Subject and purport. Methodology and Science, 17, 230–259.Google Scholar
de Leeuw, J. (1988). Models and techniques. Statistica Neerlandica, 42, 91–98.CrossRefGoogle Scholar
Ellis, J. L., & Junker, B. W. (1997). Tail-measurability in monotone latent variable models. Psychometrika, 62, 495–523.CrossRefGoogle Scholar
Fisher, R. A. (1960). The design of experiments 7th ed.,, New York: Hafner.Google ScholarPubMed
Fischer, G. H. (1974). Einführung in die Theorie Psychologischer Tests, Bern: Huber [Introduction to mental test theory.]Google Scholar
Fischer, G. H., & Molenaar, I. W. (1995). Rasch models: Foundations, Recent developments, and applications, New York: Springer.CrossRefGoogle Scholar
Glas, C. A. W. (1988). The derivation of some tests for the Rasch model from the multinomial distribution. Psychometrika, 53, 525–546.CrossRefGoogle Scholar
Hoijtink, H., & Molenaar, I. W. (1997). A multidimensional item response model: Constrained latent class analysis using the Gibbs sampler and posterior predictive checks. Psychometrika, 62, 171–189.CrossRefGoogle Scholar
Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling. Sociological Methods & Research, 26, 329–367.CrossRefGoogle Scholar
Junker, B. W., & Ellis, J. L. (1997). A characterization of monotone unidimensional latent variable models. Annals of Statistics, 25, 1327–1343.CrossRefGoogle Scholar
Kelley, T. L. (1927). Fundamentals of statistics, Cambridge, MA: Harvard University Press.Google Scholar
Luijben, T. C. W. (1989). Statistical guidance for model modification in covariance structure analysis, Amsterdam: Sociometric research Foundation.Google Scholar
Mokken, R. J. (1971). A theory and procedure of scale analysis, The Hague: Mouton/Berlin.CrossRefGoogle Scholar
Molenaar, I. W. (1988). Formal statistics and informal data analysis or why laziness should be discouraged. Statistica Neerlandica, 42, 83–90.CrossRefGoogle Scholar
Molenaar, I. W. (1997). Review of Techniques of Event History Modeling. New Approaches to Causal Analysis by Hans-Peter Blossfeld & Goetz Rohwer. Psychometrika, 62, 461–462.CrossRefGoogle Scholar
Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference, Part I. Biometrika, 20, 175–175.Google Scholar
Novick, M. R., Jackson, P. H., & Thayer, D. T. (1971). Bayesian inference and the classical test theory model: reliability and true scores. Psychometrika, 36, 261–288.CrossRefGoogle Scholar
Novick, M. R., Lewis, C., & Jackson, P. H. (1973). The estimation of proportions in m groups. Psychometrika, 38, 19–46.CrossRefGoogle Scholar
Patel, N., & Mehta, C. R. (1989). StatXact user manual, Cambridge, MA: Cytel Software.Google Scholar
Raftery, A. (1998). How many clusters? Keynote Address presented at the Joint Meeting of Classification Society of North America and Psychometric Society, Champaign IL.Google Scholar
Ramsay, J. O. (1996). A similarity based smoothing approach to nondimensional item analysis. Psychometrika, 60, 323–339.CrossRefGoogle Scholar
Ramsay, J. O., & Abramowicz, M. (1989). Binomial regression with monotone splines: A psychometric application. Journal of the American Statistical Association, 84, 906–915.CrossRefGoogle Scholar
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests, Copenhagen: The Danish Institute of Educational Research.Google Scholar
Saris, W. E., Satorra, A., & Sörbom, D. (1987). The detection and correction of specification errors in structural equation models. In Clogg, C. C. (Eds.), Sociological methodology (pp. 105–129). Washington, DC: American Sociological Association.Google Scholar
Sijtsma, K., & Molenaar, I. W. (1999). A nonparametric item response theory. Submitted for publication.Google Scholar
Smeenk, W. H. (in press). Opportunity and marriage; the impact of individual resources and marriage market structure on the when and whom of first marriage in the Netherlands. Amsterdam: Thesis Publishers.Google Scholar
Sörbom, D. (1989). Model modification. Psychometrika, 54, 371–384.CrossRefGoogle Scholar
Stout, W. F. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589–617.CrossRefGoogle Scholar
Stout, W. F. (1990). A new item response theory modeling approach with applications to unidimensionality assessment and ability estimation. Psychometrika, 55, 293–325.CrossRefGoogle Scholar
Swaminathan, H., & Gifford, J. A. (1985). Bayesian estimation in the two-parameter logistic model. Psychometrika, 50, 349–364.CrossRefGoogle Scholar
Thissen, D., & Steinberg, L. (1984). A response model for multiple choice items. Psychometrika, 49, 501–519.CrossRefGoogle Scholar
Ultee, W. C., & Ganzeboom, H. B. G. (1993). Netherlands Family Survey 1992–1993, Nijmegen, Netherlands: Nijmegen University, Department of Sociology [machine readable data file]Google Scholar
Van Lenthe, J. (1993). ELI: An interactive elicitation technique for subjective probability distributions. Organizational Behavior and Human Decision Processes, 55, 379–413.CrossRefGoogle Scholar
Van Lenthe, J. (1994). Scoring-rule feedforward and the elicitation of subjective probability distributions. Organizational Behavior and Human Decision Processes, 59, 188–209.CrossRefGoogle Scholar
Verhelst, N. D., & Glas, C. A. W. (1995). The one parameter logistic model. In Fischer, G. H., & Molenaar, I. W. (Eds.), Rasch models: Foundations, recent developments and applications (pp. 215–237). New York: Springer.CrossRefGoogle Scholar
Warm, Th. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika, 54, 427–450.CrossRefGoogle Scholar
Wright, G., & Ayton, P. (1994). Subjective probability, New York: John Wiley and Sons.Google Scholar