Hostname: page-component-745bb68f8f-kw2vx Total loading time: 0 Render date: 2025-01-07T19:16:56.202Z Has data issue: false hasContentIssue false
  • English
  • Français

Ordinal Test Fidelity Estimated by an Item Sampling Model

Published online by Cambridge University Press:  01 January 2025

Norman Cliff  and
John R. Donoghue
Norman Cliff*
Affiliation:
University of Southern California
John R. Donoghue
Affiliation:
Educational Testing Service
*
Requests for reprints should be sent to Norman Cliff, Department of Psychology, MC-1061, University of Southern California, Los Angeles, CA 90089.
Get access Rights & Permissions [Opens in a new window]

Abstract

A test theory using only ordinal assumptions is presented. It is based on the idea that the test items are a sample from a universe of items. The sum across items of the ordinal relations for a pair of persons on the universe items is analogous to a true score. Using concepts from ordinal multiple regression, it is possible to estimate the tau correlations of test items with the universe order from the taus among the test items. These in turn permit the estimation of the tau of total score with the universe. It is also possible to estimate the odds that the direction of a given observed score difference is the same as that of the true score difference. The estimates of the correlations between items and universe and between total score and universe are found to agree well with the actual values in both real and artificial data.

Keywords

test theoryitem samplingordinal methodsitem universe
Type
Original Paper
Information
Psychometrika , Volume 57 , Issue 2 , June 1992 , pp. 217 - 236
Copyright
Copyright © 1992 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

Part of this paper was presented at the June, 1989, Meeting of the Psychometric Society. The authors wish to thank several reviewers for their suggestions. This research was mainly done while the second author was a University Fellow at the University of Southern California.

References

Birnbaum, A. (1968). Some latent trait models and their uses in inferring an examinee's ability. In Lord, F. M., Novick, M. R. (Eds.), Statistical theories of mental test scores (pp. 397–472). Reading, MA: Addison-Wesley.Google Scholar
Charlin, V. L. (1987). Implementation and inferential issues of a multiple regression system with ordinal variables. Unpublished doctoral dissertation, University of Southern California.Google Scholar
Cliff, N. (1977). A theory of consistency of ordering generalizable to tailored testing. Psychometrika, 42, 375–401.CrossRefGoogle Scholar
Cliff, N. (1979). Test theory without true scores?. Psychometrika, 44, 373–393.CrossRefGoogle Scholar
Cliff, N. (1986). Ordinal analogues of multiple regression. 1986 Proceedings of the Social Statistics Section of the American Statistical Association (pp. 320–324). Washington, DC: American Statistical Association.Google Scholar
Cliff, N. (1989a). “Multiple regression” with ordinal variables. Unpublished manuscript.Google Scholar
Cliff, N. (1989). Ordinal consistency and ordinal true scores. Psychometrika, 54, 75–91.CrossRefGoogle Scholar
Cliff, N., & Charlin, V. L. (In press). Variances and covariances of Kendall's tau and their estimation. Multivariate Behavior Research.Google Scholar
Donoghue, J. R. (1990). Properties of ability estimates derived from logistic item response theory and ordinal item sampling test theory. Unpublished doctoral dissertation, University of Southern California.Google Scholar
Donoghue, J. R., & Cliff, N. (In press). An investigation of ordinal true score test theory. Applied Psychological Measurement.Google Scholar
Douglas, G. A., Wright, B. D. (1986). The two-category model for objective measurement, Chicago: University of Chicago, Department of Education, MESA Psychometric Laboratory.Google Scholar
Gulliksen, H. (1950). Theory of mental tests, New York: Wiley.CrossRefGoogle Scholar
Guttman, L. (1944). A basis for scaling qualitative data. American Sociological Review, 9, 139–150.CrossRefGoogle Scholar
Guttman, L. (1950). The basis for scalogram analysis. In Stouffer, S.A. (Eds.), Measurement and Prediction (pp. 60–90). Princeton, NJ: Princeton University.Google Scholar
Hawkes, R. J. (1971). The multivariate analysis of ordinal data. American Journal of Sociology, 76, 908–926.CrossRefGoogle Scholar
Hettmansperger, T. P. (1984). Statistical inference based on ranks, New York: Wiley.Google Scholar
Holland, P. W. (1990). The Dutch identity: A new tool for understanding item response theory. Psychometrika, 55, 5–18.CrossRefGoogle Scholar
Kendall, M. G. (1970). Rank correlation methods 3rd ed.,, New York: Hafner.Google Scholar
Loevinger, J. A. (1947). A systematic approach to the construction and evaluation of tests of ability. Psychological Monographs, 61, (Whole No. 285).CrossRefGoogle Scholar
Loevinger, J. A. (1948). A technique of homogeneous test evaluation compared with some aspects of “scale analysis”. Psychological Bulletin, 45, 507–529.CrossRefGoogle Scholar
Lord, F. M. (1968). An analysis of the verbal Scholastic Aptitude Test using Birnbaum's three-parameter logistic model. Educational and Psychological Measurement, 28, 989–1020.CrossRefGoogle Scholar
Lord, F. M. (1975). The “ability” scale in item response theory. Psychometrika, 40, 205–217.CrossRefGoogle Scholar
Lord, F. M., Novick, M. R. (1968). Statistical theories of mental test scores, Reading, MA: Addison-Wesley.Google Scholar
Mislevy, R. J. (1987). Recent developments in item response theory with implications for teacher certification. Review of Educational Research, 14, 239–275.Google Scholar
Mokken, R. J. (1971). A theory and procedure of scale analysis, Hawthorne, NY: Mouton & Co.CrossRefGoogle Scholar
Mokken, R. J., Lewis, C. (1982). A nonparametric approach to the analysis of dichotomous item responses. Applied Psychological Measurement, 6, 417–430.CrossRefGoogle Scholar
Reckase, M. D. (1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9, 401–412.CrossRefGoogle Scholar
Reynolds, T. J., Sutrick, K. H. (1986). Assessing the correspondence of one or more vectors to a symmetric matrix. Psychometrika, 51, 101–112.CrossRefGoogle Scholar
Roskam, E. E., Jansen, P. G. W. (1984). A new derivation of the Rasch Model. In deGreef, E., Buggenhaut, J. (Eds.), Trends in mathematical psychology (pp. 293–307). Amsterdam: Elsevier.CrossRefGoogle Scholar
Samejima, F. (1979). A new family of models for the multiple-choice item, Knoxville, TN: University of Tennessee.CrossRefGoogle Scholar
Schulman, R. S., Haden, R. L. (1975). A test theory for ordinal measurements. Psychometrika, 40, 329–340.CrossRefGoogle Scholar
Smith, R. B. (1974). Continuities in ordinal path analysis. Social Forces, 53, 200–229.CrossRefGoogle Scholar
Somers, R. H. (1974). Analysis of partial rank correlation measures based on the product-moment model: Part One. Social Forces, 53, 229–246.CrossRefGoogle Scholar
Stout, W. (1987). A nonparametric approach for assessing latent dimensionality. Psychometrika, 52, 589–617.CrossRefGoogle Scholar
Stout, W. (1988). A nonparametric multidimensional IRT approach with applications to ability estimation and test bias, Champaign, IL: University of Illinois.Google Scholar
Stout, W. (1990). A new item-response theory modeling approach with applications to unidimensionality assessment and ability estimation. Psychometrika, 55, 293–325.CrossRefGoogle Scholar
Thissen, D., Steinberg, L. (1984). A response model for multiple choice items. Psychometrika, 47, 501–519.CrossRefGoogle Scholar
Thissen, D., Wainer, H. (1982). Some standard errors in item response theory. Psychometrika, 45, 397–412.CrossRefGoogle Scholar
Yen, W. (1986). The choice of scale for educational measurement: An IRT perspective. Journal of Educational Measurement, 23, 299–325.CrossRefGoogle Scholar