Hostname: page-component-5f745c7db-96s6r Total loading time: 0 Render date: 2025-01-06T07:15:51.281Z Has data issue: true hasContentIssue false
  • English
  • Français

Test Theory without True Scores?

Published online by Cambridge University Press:  01 January 2025

Norman Cliff
Norman Cliff*
Affiliation:
University of Southern California
*
Requests for reprints should be sent to Norman Cliff, Department of Psychology, University of Southern California, University Park, Los Angeles, CA 90007.
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper traces the course of the consequences of viewing test responses as simply providing dichotomous data concerning ordinal relations. It begins by proposing that the score matrix is best considered to be items-plus-persons by items-plus-persons, and recording the wrongs as well as the rights. This shows how an underlying order is defined, and was used to provide the basis for a tailored testing procedure. It also was used to define a number of measures of test consistency. Test items provide person dominance relations, and the relations provided by one item can be in one of three relations with a second one: redundant, contradictory, or unique. Summary statistics concerning the number of relations of each kind are easy to get and provide useful information about the test, information which is related to but different from the usual statistics. These concepts can be extended to form the basis of a test theory which is based on ordinal statistics and frequency counts and which invokes the concept of true scores only in a limited sense.

Keywords

test theoryconsistencytrue scoresordinal measurestailored testing
Type
Original Paper
Information
Psychometrika , Volume 44 , Issue 4 , December 1979 , pp. 373 - 393
Copyright
Copyright © 1979 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

1979 Psychometric Society presidential address.

I want to recognize the contributions which others have made to whatever I have accomplished. First to mention here are my teachers: principally Harold Gulliksen, Ledyard Tucker, and the late Edith Jay. Second, I would like to recognize the importance of my graduate students. Tom Reynolds has been especially important in developing the ideas that I will talk about here today, but at various times, the others have made major contributions in this and other topics. I would like to express also a debt to my family, primarily my wife, Rosemary, who herself has a longterm interest in the psychometric area. Finally, I must acknowledge the financial support of the NIMH some time ago, the Office of Naval Research, until about a year ago, and of the James McKeen Cattell Fund this past year.

References

Reference Notes

McCormick, D. Tailor-APL: An interactive computer program for individual tailored testing, 1978, Los Angeles, CA: University of Southern California, Department of Psychology.Google Scholar
Cliff, N., Cudeck, R., & McCormick, D. Evaluations of implied orders as a basis for tailored testing, 1977, Los Angeles, CA: University of Southern California, Department of Psychology.Google Scholar
Cliff, N. Psychological scaling. In preparation.Google Scholar
Cliff, N., Cudeck, R., & McCormick, D. Implied orders as a basis for tailored testing: Final report, 1978, Los Angeles, CA: University of Southern California, Department of Psychology.Google Scholar
Reynolds, T. J. The analysis of dominance matrices. Extraction of unidimensional orders within a multidimensional context, 1976, Los Angeles, CA: University of Southern California, Department of Psychology.CrossRefGoogle Scholar

References

Airasian, P., & Bart, W. Ordering theory: A new and useful measurement model. Educational Psychology, 1973, 13, 56–60.Google Scholar
Bart, W., & Krus, D. An ordering theoretic method to determine hierarchies among items. Educational and Psychological Measurement, 1973, 33, 291–300.CrossRefGoogle Scholar
Brennan, R. L., & Kane, M. T. Signal/noise ratios for domain referenced tests. Psychometrika, 1977, 42, 609–630.CrossRefGoogle Scholar
Cliff, N. Scaling. Annual Review of Psychology, 1974, 24, 473–506.CrossRefGoogle Scholar
Cliff, N. Complete orders from incomplete data: Interactive ordering and tailored testing. Psychological Bulletin, 1975, 82, 289–302.CrossRefGoogle Scholar
Cliff, N. A theory of consistency of ordering generalizable to tailored testing. Psychometrika, 1977, 42, 375–401.CrossRefGoogle Scholar
Cliff, N. What is and isn't measurement? In Keren, Gideon (Ed.) Statistical and methodological issues in psychological and social science research. New York: Erlbaum, in press.Google Scholar
Coombs, C. A theory of psychological scaling, 1952, Ann Arbor: University of Michigan Press.Google Scholar
Coombs, C. A theory of data, 1964, New York: Wiley.Google Scholar
Cudeck, R., Cliff, N., & Kehoe, J. TAILOR: A FORTRAN procedure for interactive tailored testing. Educational and Psychological Measurement, 1977, 37, 767–769.CrossRefGoogle Scholar
Cudeck, R., McCormick, D., & Cliff, N. Monte Carlo evaluation of implied orders as a basis for tailored testing. Applied Psychological Measurement, 1979, 3, 65–74.CrossRefGoogle Scholar
Ducamp, A., & Falmagne, J. C. Composite measurement. Journal of Mathematical Psychology, 1969, 6, 359–390.CrossRefGoogle Scholar
Freeman, L. C. Elementary applied statistics, 1965, New York: Wiley.Google Scholar
Glaser, R. Instructional technology and the measurement of learning outcomes: Some questions. American Psychologist, 1963, 18, 519–521.CrossRefGoogle Scholar
Guttman, L. The quantification of a class of attributes: A theory and method of scale construction. In Horst, P. (Eds.), The prediction of peronsal adjustment, 1941, New York: Social Science Research Council.Google Scholar
Hubert, L. A note on Freeman's measure of association for relating an ordered to an unordered factor. Psychometrika, 1974, 39, 517–520.CrossRefGoogle Scholar
Humpheys, L. G. The normal curve and the attenuation paradox in test theory. Psychological Bulletin, 1956, 53, 472–476.CrossRefGoogle Scholar
Keats, J. A. Statistical theory of objective test scores, 1951, Melbourne: Australian Council for Educational Research.Google Scholar
Krus, D. Order analysis: An inferential model of dimensional analysis and scaling. Educational and Psychological Measurement, 1977, 37, 587–601.CrossRefGoogle Scholar
Krus, D., & Bart, W. An ordering-theoretic method of multidimensional scaling of items. Educational and Psychological Measurement, 1974, 34, 525–535.CrossRefGoogle Scholar
Loevinger, J. A. A systematic approach to the construction and evaluation of tests of ability. Psychological Monographs, 1947, 61, (4, Whole No. 285).CrossRefGoogle Scholar
Loevinger, J. A. The technique of homogenous tests compared with some aspects of “scale analysis” and factor analysis. Psychological Bulletin, 1948, 45, 507–529.CrossRefGoogle Scholar
Loevinger, J. The attenuation paradox in test theory. Psychological Bulletin, 1954, 51, 493–504.CrossRefGoogle ScholarPubMed
McCormick, D., & Cliff, N. TAILOR-APL: An interactive computer program for individual tailored testing. Educational and Psychological Measurement, 1977, 37, 771–774.CrossRefGoogle Scholar
McNemar, Q. Psychological statistics, 1949, New York: Wiley.Google Scholar
Reynolds, T. J. The logical fallacy of order analysis. Multivariate Behavioral Research, in press.Google Scholar
Sato, T. S-P Table Analysis-Analysis and interpretation of test scores, 1975, Tokyo: Meiji-Tosho Publishing.Google Scholar
Sato, T., & Kurata, M. Basic S-P score table characteristics. Nippon Electric Co. Research & Development, 1977 (Whole No. 47), 64–71.Google Scholar
Schulman, R. S. Correlation and prediction in ordinal test theory. Psychometrika, 1976, 41, 19–29.CrossRefGoogle Scholar
Schulman, R. S. Individual distributions under ordinal measurement. Psychometrika, 1978, 43, 19–29.CrossRefGoogle Scholar
Schulman, R. S., & Haden, R. S. A test theory model for ordinal measurements. Psychometrika, 1975, 40, 455–472.CrossRefGoogle Scholar
Wherry, R. J., & Gaylord, R. H. Factor pattern of test items and tests as a function of the correlation coefficient: Content, validity, and constant error factors. Psychometrika, 1944, 9, 237–244.CrossRefGoogle Scholar