Hostname: page-component-745bb68f8f-l4dxg Total loading time: 0 Render date: 2025-01-07T18:12:53.092Z Has data issue: false hasContentIssue false
  • English
  • Français

A Rapid Non-Parametric Estimate of Multi-Judge Reliability

Published online by Cambridge University Press:  01 January 2025

Desmond S. Cartwright
Desmond S. Cartwright*
Affiliation:
University of Chicago
Get access Rights & Permissions [Opens in a new window]

Abstract

A technique is presented for obtaining a rapid estimate of reliability between judges, with special reference to qualitative judgments. It is shown that reliability and discrimination are independent and that estimates of both are needed. A method of obtaining an independent estimate of multi-judge discrimination is developed. It is shown that the size of item-samples is specified by the latter method. Tests of significance for both reliability and discrimination are described.

Type
Original Paper
Information
Psychometrika , Volume 21 , Issue 1 , March 1956 , pp. 17 - 29
Copyright
Copyright © 1956 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

This technique was developed in connection with research at the Counseling Center, University of Chicago. This investigation was supported by a research grant (PHS M 903) from the National Institute of Mental Health, of the National Institutes of Health, Public Health Service. Acknowledgment is made to Dr. Lyle V. Jones, University of Chicago, for valuable discussions relevant to the technique. Responsibility rests, however, entirely with the writer.

References

Bliss, C. I. A chart of the chi-square distribution. J. Amer. statist. Assoc., 1944, 39, 246–248CrossRefGoogle Scholar
Fisher, R. A., Yates, F. Statistical tables for biological, agricultural and medical research 4th Ed., New York: Hafner, 1953Google Scholar
Friedman, M. The use of ranks to avoid the assumption of normality. J. Amer. statist. Assoc., 1937, 32, 675–701CrossRefGoogle Scholar
Garner, W. R., Hake, H. W. The amount of information in absolute judgments. Psychol. Rev., 1951, 58, 446–459CrossRefGoogle Scholar
Guetzkow, H. Unitizing and categorizing problems in coding qualitative data. J. clin. Psychol., 1950, 6, 47–583.0.CO;2-I>CrossRefGoogle Scholar
Kogan, L. S., Hunt, J. McV. Problems of multi-judge reliability. J. clin. Psychol., 1950, 6, 16–193.0.CO;2-D>CrossRefGoogle Scholar
Pitman, E. J. G. Significance tests which may be applied to samples from any populations. III. The analysis of variance test. Biometrika, 1937, 29, 322–335Google Scholar
Tukey, J. W. Discussion of symposium. J. clin. Psychol., 1950, 6, 61–743.0.CO;2-S>CrossRefGoogle Scholar
Whitworth, W. A. Choice and chance, New York: Stechert, 1934Google Scholar
Wilcoxon, F. Some rapid approximate statistical procedures, New York: American Cyanamid Co., 1949Google Scholar