Hostname: page-component-5f745c7db-rgzdr Total loading time: 0 Render date: 2025-01-06T21:05:34.976Z Has data issue: true hasContentIssue false
  • English
  • Français

Assessing the Size of Model Misfit in Structural Equation Models

Published online by Cambridge University Press:  01 January 2025

Alberto Maydeu-Olivares Open the ORCID record for Alberto Maydeu-Olivares [Opens in a new window]
Alberto Maydeu-Olivares*
Affiliation:
University of Barcelona University of South Carolina
*
Correspondence should be made to Alberto Maydeu-Olivares, Department of Psychology, University of South Carolina, Barnwell College, 1512 Pendleton St., Columbia, SC 29208, USA. Email: amaydeu@sc.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

When a statistically significant mean difference is found, the magnitude of the difference is judged qualitatively using an effect size such as Cohen’s d. In contrast, in a structural equation model (SEM), the result of the statistical test of model fit is often disregarded if significant, and inferences are drawn using “close” models retained based on point estimates of sample statistics (goodness-of-fit indices). However, when a SEM cannot be retained using a test of exact fit, all substantive inferences drawn from it are suspect. It is therefore important to determine the size of the model misfit. Standardized residual covariances and residual correlations provide standardized effect sizes of the misfit of SEM models. They can be summarized using the Standardized Root Mean Squared Residual (SRMSR) and the Correlation Root Mean Squared Residual (CRMSR) which can be used as overall effect sizes of the misfit. Statistical theory is provided that allows the construction of confidence intervals and tests of close fit based on the SRMSR and CRMSR. It is hoped that the use of standardized effect sizes of misfit will help reconcile current practices in SEM and elsewhere in statistics.

Keywords

goodness-of-fitRMSEAeffect size
Type
Original paper
Information
Psychometrika , Volume 82 , Issue 3 , September 2017 , pp. 533 - 558
Copyright
Copyright © 2017 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Presidential Address to the Psychometric Society, delivered at the annual meeting in Madison (WI), July 2014. This research was supported by an ICREA-Academia Award and Grant SGR 2014 1500 from the Catalan Government and Grant PSI2012-33601 from the Spanish Ministry of Education. I am indebted to Peter Bentler, Ke-Hai Yuan, Albert Satorra, Jim Steiger, Haruhiko Ogasawara, and Yves Rosseel for their helpful comments. I am also most thankful to Yves Rosseel for implementing these methods in the Lavaan package in R.

References

Alden, D. L., Steenkamp, J. B. E. M., & Batra, R.. (2006). Consumer attitudes toward marketplace globalization: Structure, antecedents and consequences. International Journal of Research in Marketing, 23(3), 227–239. doi:10.1016/j.ijresmar.2006.01.010.CrossRefGoogle Scholar
Barrett, P.. (2007). Structural equation modelling: Adjudging model fit. Personality and Individual Differences, 42(5), 815–824. doi:10.1016/j.paid.2006.09.018.CrossRefGoogle Scholar
Bentler, P. M.. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. doi:10.1037/0033-2909.107.2.238.CrossRefGoogle ScholarPubMed
Bentler, P. M. (1995). EQS 5 [Computer Program]. Encino, CA: Multivariate Software Inc..Google Scholar
Bollen, K. A., (1989). Structural equations with latent variables. New York: Wiley.CrossRefGoogle Scholar
Bollen, K. A., & Arminger, G.. (1991). Observational residuals in factor analysis and structural equation models. Sociological Methodology, 21, 235. doi:10.2307/270937.CrossRefGoogle Scholar
Bollen, K. A., & Long, J. S., (1993). Testing structural equation models. Newbury Park, CA: Sage.Google Scholar
Browne, M. W.. (2000). Cross-validation methods. Journal of Mathematical Psychology, 44(1), 108–132. doi:10.1006/jmps.1999.1279.CrossRefGoogle ScholarPubMed
Browne, M. W.,Hawkins, D. M.. (1982). Covariance structures. Topics in applied multivariate analysis. (pp. 72–141). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Browne, M. W.. (1984). Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37(1), 62–83. doi:10.1111/j.2044-8317.1984.tb00789.x.CrossRefGoogle ScholarPubMed
Browne, M. W., Arminger, G.Arminger, G.Clogg, C. C., & Sobel, M. E.. (1995). Specification and estimation of mean- and covariance-structure models. Handbook of statistical modeling for the social and behavioral sciences. (pp. 185–249). New York: Plenum.CrossRefGoogle Scholar
Browne, M. W., Cudeck, R.. (1989). Single sample cross-validation indices for covariance structures. Multivariate Behavioral Research, 24(4), 445–455. doi:10.1207/s15327906mbr2404_4.CrossRefGoogle ScholarPubMed
Browne, M. W., Cudeck, R.Bollen, K. A., & Long, J. S.. (1993). Alternative ways of assessing model fit. Testing structural equation models. Newbury Park, CA: Sage 136–162.Google Scholar
Chen, F.Curran, P. J., Bollen, K. A., & Kirby, J.Paxton, P.. (2008). An empirical evaluation of the use of fixed cutoff points in rmsea test statistic in structural equation models. Sociological Methods & Research, 36(4), 462–494. doi:10.1177/0049124108314720.CrossRefGoogle ScholarPubMed
Coffman, D.Millsap, R.. (2006). Evaluating latent growth curve models using individual fit statistics. Structural Equation Modeling, 13(1), 37–41. doi:10.1207/s15328007sem1301_1.CrossRefGoogle Scholar
Cohen, J.Statistical power analysis for the behavioral sciences 1988 2Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
Cohen, J.Cohen, P.West, S. G., & Aiken, L. S., (2003). Applied multiple regression/correlation analysis for behavioral sciences. 3Mahwah, NJ: Erlbaum.Google Scholar
Edwards, J. R., & Lambert, L. S.. (2007). Methods for integrating moderation and mediation: A general analytical framework using moderated path analysis. Psychological Methods, 12(1), 1–22. doi:10.1037/1082-989X.12.1.1.CrossRefGoogle ScholarPubMed
Hildreth, L. (2013). Residual analysis for structural equation modeling. Ph.D. dissertation, Statistics Department, Iowa State University..Google Scholar
Hu, L.Bentler, P. M.. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55. doi:10.1080/10705519909540118.CrossRefGoogle Scholar
Jöreskog, K. G.,Bollen, K. A., & Long, J. S.. (1993). Testing structural equation models. Testing structural equation models. Newbury Park, CA: Sage 294–316.Google Scholar
Jöreskog, K. G., & Sörbom, D. (1981). Analysis of linear structural relationships by maximum likelihood and least squares methods. Research report 81-8, Uppsala, Sweden..Google Scholar
Jöreskog, K. G., & Sörbom, D.LISREL 7. A guide to the program and applications 1988 2Chicago, IL: International Education Services.Google Scholar
Kenny, D. A., & McCoach, D. B.. (2003). Effect of the number of variables on measures of fit in structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 10(3), 333–351. doi:10.1207/S15328007SEM1003_1.CrossRefGoogle Scholar
Lee, S. Y., & Jennrich, R. I.. (1979). A study of algorithms for covariance structure analysis with specific comparisons using factor analysis. Psychometrika, 44(1), 99–113. doi:10.1007/BF02293789.CrossRefGoogle Scholar
MacCallum, R. C., Wegener, D. T., Uchino, B. N., & Fabrigar, L. R.. (1993). The problem of equivalent models in applications of covariance structure analysis. Psychological Bulletin, 114(1), 185–199. doi:10.1037/0033-2909.114.1.185.CrossRefGoogle ScholarPubMed
Marsh, H.Hau, K.Grayson, D.Maydeu-Olivares, A.Mcardle, J. J.. (2005). Goodness of fit in structural equation models. Contemporary psychometrics: A Festschrift for Roderick P. McDonald. Mahwah, NJ: Erlbaum 275–340.Google Scholar
Marsh, H. W., & Hau, K.Wen, Z.. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling: A Multidisciplinary Journal, 11(3), 320–341. doi:10.1207/s15328007sem1103_2.CrossRefGoogle Scholar
Maydeu-Olivares, A.. (2006). Limited information estimation and testing of discretized multivariate normal structural models. Psychometrika, 71(1), 57–77. doi:10.1007/s11336-005-0773-4.CrossRefGoogle Scholar
Maydeu-Olivares, A.Reise, S. P., & Revicki, D. A.. (2015). Evaluating fit in IRT models. Handbook of item response theory modeling: Applications to typical performance assessment. New York: Routledge 111–127.Google Scholar
Maydeu-Olivares, A., & Shi, D. (2017). Effect sizes of model misfit in structural equation models: Standardized residual covariances and residual correlations. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences..Google Scholar
McDonald, R. P., & Ho, M-HR. (2002). Principles and practice in reporting structural equation analyses. Psychological Methods, 7(1), 64–82. doi:10.1037/1082-989X.7.1.64.CrossRefGoogle ScholarPubMed
Muthén, B. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49(1), 115–132. doi:10.1007/BF02294210.CrossRefGoogle Scholar
Ogasawara, H.. (2001). Standard errors of fit indices using residuals in structural equation modeling. Psychometrika, 66(3), 421–436. doi:10.1007/BF02294443.CrossRefGoogle Scholar
Raykov, T.. (2000). On sensitivity of structural equation modeling to latent relation misspecifications. Structural Equation Modeling, 7(4), 596–607. doi:10.1207/S15328007SEM0704_4.CrossRefGoogle Scholar
Reise, S. P., Scheines, R.Widaman, K. F., & Haviland, M. G.. (2012). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 73(1), 5–26. doi:10.1177/0013164412449831.CrossRefGoogle Scholar
Rosseel, Y.. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. doi:10.18637/jss.v048.i02.CrossRefGoogle Scholar
Saris, W. E., & Satorra, A.van der Veld, W. M.. (2009). Testing structural equation models or detection of misspecifications?. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 561–582. doi:10.1080/10705510903203433.CrossRefGoogle Scholar
Satorra, A.. (1989). Alternative test criteria in covariance structure analysis: A unified approach. Psychometrika, 54(1), 131–151. doi:10.1007/BF02294453.CrossRefGoogle Scholar
Satorra, A.. (2015). A comment on a paper by H. Wu and M. W. Browne (2014). Psychometrika, 80(3), 613–618. doi:10.1007/s11336-015-9455-z.CrossRefGoogle Scholar
Satorra, A.Bentler, P. M.,Von Eye, A.Clogg, C. C.. (1994). Corrections to test statistics and standard errors in covariance structure analysis. Latent variable analysis. Applications for developmental research. Thousand Oaks, CA: Sage 399–419.Google Scholar
Savalei, V.. (2012). The relationship between root mean square error of approximation and model misspecification in confirmatory factor analysis models. Educational and Psychological Measurement, 72(6), 910–932. doi:10.1177/0013164412452564.CrossRefGoogle Scholar
Schmidt, F.Hunter, J.Harlow, L. L., & Mulaik, S.Steiger, J. H.. (1997). Eight common but false objections to the discontinuation of significance testing in the analysis of research data. What if there were no significance tests. Mahwah, NJ: Lawrence Erlbaum 37–64.Google Scholar
Schott, J. R., (1997). Matrix analysis for statistics. New York: Wiley.Google Scholar
Steiger, J. H. (1989). EzPATH: A supplementary module for SYSTAT and SYGRAPH [Computer program manual]. Evanston, IL: Systat Inc..Google Scholar
Steiger, J. H.. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25(2), 173–180. doi:10.1207/s15327906mbr2502_4.CrossRefGoogle ScholarPubMed
Steiger, J. H.. (2000). Point estimation, hypothesis testing, and interval estimation using the RMSEA: Some comments and a reply to Hayduck and Glaser. Structural Equation Modeling, 7(2), 149–162. doi:10.1207/S15328007SEM0702_1.CrossRefGoogle Scholar
Steiger, J. H., Fouladi, R.Harlow, L. L., Mulaik, S. A., & Steiger, J. H.. (1997). Noncentrality interval estimation and the evaluation of statistical models. What if there were no significance tests. Mahwah, NJ: Erlbaum 221–257.Google Scholar
Steiger, J. H., & Lind, J. C. (1980). Statistically-based tests for the number of common factors. Paper presented at the Annual Meeting of the Psychometric Society, Iowa..Google Scholar
Tanaka, J. S., & Huba, G. J.. (1989). A general coefficient of determination for covariance structure models under arbitrary GLS estimation. British Journal of Mathematical and Statistical Psychology, 42(2), 233–239. doi:10.1111/j.2044-8317.1989.tb00912.x.CrossRefGoogle Scholar
Wu, H.Browne, M. W.. (2015). Quantifying adventitious error in a covariance structure as a random effect. Psychometrika, 80(3), 571–600. doi:10.1007/s11336-015-9451-3 5439333.CrossRefGoogle Scholar
Yuan, K-HBentler, P. M.. (1997). Mean and covariance structure analysis: Theoretical and practical improvements. Journal of the American Statistical Association, 92, 767–774. doi:10.1080/01621459.1997.10474029.CrossRefGoogle Scholar
Yuan, K-HBentler, P. M.. (1998). Normal theory based test statistics in structural equation modelling. The British Journal of Mathematical and Statistical Psychology, 51(2), 289–309. doi:10.1111/j.2044-8317.1998.tb00682.x.CrossRefGoogle ScholarPubMed
Yuan, K-HBentler, P. M. F\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F$$\end{document} tests for mean and covariance structure analysis Journal of Educational and Behavioral Statistics 1999 24(3), 225–243. doi:10.2307/1165323.CrossRefGoogle Scholar
Yuan, K-HHayashi, K.. (2010). Fitting data to model: Structural equation modeling diagnosis using two scatter plots. Psychological Methods, 15(4), 335–351. doi:10.1037/a0020140.CrossRefGoogle ScholarPubMed
Yuan, K-HMarshall, L. L., & Bentler, P. M.. (2003). Assessing the effect of model misspecifications on parameter estimates in structural equation models. Sociological Methodology, 33(1), 241–265. doi:10.1111/j.0081-1750.2003.00132.x.CrossRefGoogle Scholar