Hostname: page-component-5f745c7db-tvc9f Total loading time: 0 Render date: 2025-01-06T21:28:08.703Z Has data issue: true hasContentIssue false
  • English
  • Français

The Attack of the Psychometricians

Published online by Cambridge University Press:  01 January 2025

Denny Borsboom
Denny Borsboom*
Affiliation:
University of Amsterdam
*
Requests for reprints should be sent to Denny Borsboom, Department of Psychology, Faculty of Social and Behavioral Sciences, University of Amsterdam, Roetersstraat 15, 1018 WB Amsterdam, The Netherlands. E-mail: d.borsboom@uva.nl
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper analyzes the theoretical, pragmatic, and substantive factors that have hampered the integration between psychology and psychometrics. Theoretical factors include the operationalist mode of thinking which is common throughout psychology, the dominance of classical test theory, and the use of “construct validity” as a catch-all category for a range of challenging psychometric problems. Pragmatic factors include the lack of interest in mathematically precise thinking in psychology, inadequate representation of psychometric modeling in major statistics programs, and insufficient mathematical training in the psychological curriculum. Substantive factors relate to the absence of psychological theories that are sufficiently strong to motivate the structure of psychometric models. Following the identification of these problems, a number of promising recent developments are discussed, and suggestions are made to further the integration of psychology and psychometrics.

Keywords

Psychometricsmodern test theoryclassical test theoryconstruct validitypsychological measurement
Type
Original Paper
Information
Psychometrika , Volume 71 , Issue 3 , September 2006 , pp. 425 - 440
Copyright
Copyright © 2006 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

This research was sponsored by NWO Innovational Research grant no. 451-03-068. I would like to thank Don Mellenbergh and Conor Dolan for their comments on an earlier version of this manuscript.

References

AERA, APA, & NCME (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education) Joint Committee on Standards for Educational and Psychological Testing (1999). Standards for educational and psychological testing. Washington, DC: AERA.Google Scholar
Bartholomew, D.J. (2004). Measuring intelligence: Facts and fallacies, Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Blanton, H., Jaccard, J., Gonzales, P.M., Christie, C. (2006). Decoding the implicit association test: Implications for criterion prediction. Journal of Experimental Social Psychology, 42, 192–212.CrossRefGoogle Scholar
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord, F.M., Novick, M.R. (Eds.), Statistical theories of mental test scores, Reading, MA: Addison-Wesley.Google Scholar
Bollen, K.A. (1989). Structural equations with latent variables, Dordrecht: Wiley.CrossRefGoogle Scholar
Bollen, K.A., Lennox, R. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110, 305–314.CrossRefGoogle Scholar
Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics, Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Borsboom, D., Mellenbergh, G.J. (2002). True scores, latent variables, and constructs: A comment on Schmidt and Hunter. Intelligence, 30, 505–514.CrossRefGoogle Scholar
Borsboom, D., Mellenbergh, G.J., Van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110, 203–219.CrossRefGoogle ScholarPubMed
Borsboom, D., Mellenbergh, G.J., Van Heerden, J. (2004). The concept of validity. Psychological Review, 111, 1061–1071.CrossRefGoogle ScholarPubMed
Bouwmeester, S., Sijtsma, K. (2004). Measuring the ability of transitive reasoning, using product and strategy information. Psychometrika, 69, 123–146.CrossRefGoogle Scholar
Bridgman, P.W. (1927). The logic of modern physics, Dordrecht: Macmillan.Google Scholar
Cliff, N. (1992). Abstract measurement theory and the revolution that never happened. Psychological Science, 3, 186–190.CrossRefGoogle Scholar
Coombs, C. (1964). A theory of data, Dordrecht: Wiley.Google Scholar
Cronbach, L.J., Gleser, G.C., Nanda, H., Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles, Dordrecht: Wiley.Google Scholar
Cronbach, L.J., Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.CrossRefGoogle ScholarPubMed
De Boeck, P., Wilson, M. (2004). Explanatory item response models: A generalized linear and nonlinear approach, Dordrecht: Springer.CrossRefGoogle Scholar
Doignon, J.P., Falmagne, J.C. (1999). Knowledge spaces, Dordrecht: Springer-Verlag.CrossRefGoogle Scholar
Dolan, C.V., Jansen, B.R.J., Van der Maas, H.L.J. (2004). Constrained and unconstrained normal finite mixture modeling of multivariate conservation data. Multivariate Behavioral Research, 39, 69–98.CrossRefGoogle Scholar
Dolan, C.V., Roorda, W., Wicherts, J.M. (2004). Two failures of Spearman’s hypothesis: The GATB in Holland and the JAT in South Africa. Intelligence, 32, 155–173.CrossRefGoogle Scholar
Edwards, J.R., Bagozzi, R.P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5, 155–174.CrossRefGoogle ScholarPubMed
Embretson, S.E. (1998). A cognitive design system approach for generating valid tests: Approaches to abstract reasoning. Psychological Methods, 3, 300–396.CrossRefGoogle Scholar
Embretson, S.E. (2004). The second century of ability testing: Some predictions and speculations. Measurement, 2, 1–32.Google Scholar
Embretson, S.E., Hershberger, S.L. (1999). The new rules of measurement: What every psychologist and educator should know, Mahwah, NJ: Erlbaum.CrossRefGoogle Scholar
Embretson, S.E., Reise, S. (2000). Item response theory for psychologists, Mahwah, NJ: Erlbaum.Google Scholar
Falmagne, J.C. (1989). A latent trait theory via stochastic learning theory for a knowledge space. Psychometrika, 54, 283–303.CrossRefGoogle Scholar
Ferrer, E., Nesselroade, J.R. (2003). Modeling affective processes in dyadic relations via dynamic factor analyses. Emotion, 3, 344–360.CrossRefGoogle Scholar
Fraley, R.C., Roberts, B.W. (2005). Patterns of continuity: A dynamic model for conceptualizing the stability of individual differences in psychological constructs across the life course. Psychological Review, 112, 60–74.CrossRefGoogle ScholarPubMed
Frederiksen, N., Mislevy, R.J., Bejar, I.I. (1993). Test theory for a new generation of tests, Hillsdale, NJ: Erlbaum.Google Scholar
Greenwald, A.G., McGhee, D.E., Schwartz, J.L.K. (1998). Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74, 1464–1480.CrossRefGoogle ScholarPubMed
Hagenaars, J.A. (1993). Loglinear models with latent variables, Newbury Park: Sage.CrossRefGoogle Scholar
Hamaker, E.L., Dolan, C.V., Molenaar, P.C.M. (2005). Statistical modeling of the individual: Rationale and application of multivariate time series analysis. Multivariate Behavior Research, 40, 207–233.CrossRefGoogle Scholar
Heinen, T. (1996). Latent class and discrete latent trait models: Similarities and differences, Thousand Oaks: Sage.Google Scholar
Herrnstein, R.J., Murray, C. (1994). The Bell curve, Dordrecht: The Free Press.Google Scholar
Hessen, D.J. (2004). A new class of parametric IRT models for dichotomous item scores. Journal of Applied Measurement, 5, 385–397.Google ScholarPubMed
Hunter, J.E., Schmidt, F.L. (2000). Racial and gender bias in ability and achievement tests. Psychology, Public Policy & Law, 6, 151–158.CrossRefGoogle Scholar
Jansen, B.R.J., Van der Maas, H.L.J. (1997). Statistical tests of the rule assessment methodology by latent class analysis. Developmental Review, 17, 321–357.CrossRefGoogle Scholar
Jansen, B.R.J., Van der Maas, H.L.J. (2002). The development of children’s rule use on the balance scale task. Journal of Experimental Child Psychology, 81, 383–416.CrossRefGoogle ScholarPubMed
Jöreskog, K.G., Sörbom, D. (1996). LISREL 8 User’s reference guide, (2nd ed.). Chicago: Scientific Software International.Google Scholar
Kaplan, D. (2000). Structural equation modeling. Foundations and extensions, Thousand Oaks, CA: Sage.Google Scholar
Krantz, D.H., Luce, R.D., Suppes, P., Tversky, A. (1971). Foundations of measurement, Vol. I, Dordrecht: Academic Press.Google Scholar
Lord, F.M., Novick, M.R. (1968). Statistical theories of mental test scores, Reading, MA: Addison-Wesley.Google Scholar
Lykken, D.T. (1991). What’s wrong with psychology anyway?. In Cicchetti, D., Grove, W.M. (Eds.), Thinking clearly about psychology, Vol. 1 (pp. 3–39). Minneapolis, MN: University of Minnesota Press.Google Scholar
Lynn, R., Vanhanen, T. (2002). IQ and the wealth of nations, Westport, CT: Praeger.CrossRefGoogle Scholar
McCrae, R.R., Costa, P.T. Jr., Ostendorf, F., Angleitner, A., Hrebickova, M., Avia, M.D.et al. (2000). Nature over nurture: Temperament, personality, and life span development. Journal of Personality and Social Psychology, 78, 173–186.CrossRefGoogle ScholarPubMed
McCrae, R.R., Zonderman, A.B., Costa, P.T. Jr., Bond, M.H., Paunonen, (1996). Evaluating replicability of factors in the Revised NEO Personality Inventory: Confirmatory factor analysis versus Procrustes rotation. Journal of Personality and Social Psychology, 70, 552–566.CrossRefGoogle Scholar
Mellenbergh, G.J. (1989). Item bias and item response theory. International Journal of Educational Research, 13, 127–143.CrossRefGoogle Scholar
Mellenbergh, G.J. (1994). Generalized linear item response theory. Psychological Bulletin, 115, 300–307.CrossRefGoogle Scholar
Mellenbergh, G.J. (2001). Outline of a faceted theory of item response data. In Boomsma, A., Van Duijn, M.A.J., Snijders, T.A.B. (Eds.), Essays in item response theory, Dordrecht: Springer-Verlag.Google Scholar
Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58, 525–543.CrossRefGoogle Scholar
Messick, S. (1988). The once and future issues of validity: Assessing the meaning and consequence of measurement. In Wainer, H., Braun, H.I. (Eds.), Test validity (pp. 33–45). Hillsdale, NJ: Erlbaum.Google Scholar
Messick, S. (1989). Validity. In Linn, R.L. (Ed.), Educational measurement (pp. 13–103). Washington, DC: American Council on Education and National Council on Measurement in Education.Google Scholar
Millsap, R.E. (1997). Invariance in measurement and prediction: Their relationship in the single-factor case. Psychological Methods, 2, 248–260.CrossRefGoogle Scholar
Millsap, R.E., Everson, H.T. (1993). Methodology review: Statistical approaches for assessing bias. Applied Psychological Measurement, 17, 297–334.CrossRefGoogle Scholar
Mislevy, R.J., Verhelst, N. (1990). Modeling item responses when different subjects employ different solution strategies. Psychometrika, 55, 195–215.CrossRefGoogle Scholar
Mokken, R.J. (1970). A theory and procedure of scale analysis, The Hague: Mouton.Google Scholar
Molenaar, P.C.M. (2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement, 2, 201–218.Google Scholar
Muthén, L.K., Muthén, B.O. (2001). Mplus user’s guide, (2nd ed.). Los Angeles, CA: Muthén & Muthén.Google Scholar
Neale, M.C., Boker, S.M., Xie, G., & Maes, H.H. (2003). Mx: Statistical modeling (6th ed.). Box 980126 MCV, Richmond, VA 23298, USA.Google Scholar
Popper, K.R. (1959). The logic of scientific discovery, London: Hutchinson Education.Google Scholar
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests, Copenhagen: Paedagogiske Institut.Google Scholar
Scheiblechner, H. (1995). Isotonic ordinal probabilistic models (ISOP). Psychometrika, 60, 281–304.CrossRefGoogle Scholar
Sijtsma, K., Molenaar, I.W. (2002). Introduction to nonparametric item response theory, Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
Society for Industrial Organizational Psychology (2003). Principles for the application and use of personnel selection procedures, Bowling Green, OH: Society for Industrial Organizational Psychology.Google Scholar
Stark, S., Chernyshenko, O.S., Drasgow, F., Williams, B.A. (2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring?. Journal of Applied Psychology, 91, 25–39.CrossRefGoogle ScholarPubMed
Süss, H., Oberauer, K., Wittmann, W.W., Wilhelm, O., Schulze, R. (2002). Working-memory capacity explains reasoning ability—And a little bit more. Intelligence, 30, 261–288.CrossRefGoogle Scholar
Tuerlinckx, F., De Boeck, P. (2005). Two interpretations of the discrimination parameter. Psychometrika, 70, 629–650.CrossRefGoogle Scholar
Van Breukelen, G.J.P. (2005). Psychometric modeling of response speed and accuracy with mixed and conditional regression. Psychometrika, 70, 359–376.CrossRefGoogle Scholar
Venables, W.N., Smith, D.M., and The R Development Core Team (2005). An introduction to R, Version 2.2.0. R-Project, 2005. URL: http://CRAN.R-project.org.Google Scholar
Vermunt, J.K., Magidson, J. (2000). Latent GOLD user’s manual, Boston, MA: Statistical Innovations Inc.Google Scholar