Hostname: page-component-5f745c7db-q8b2h Total loading time: 0 Render date: 2025-01-06T07:32:55.920Z Has data issue: true hasContentIssue false
  • English
  • Français

Inferential Procedures for Multifaceted Coefficients of Generalizability

Published online by Cambridge University Press:  01 January 2025

Marsha L. Schroeder  and
A. Ralph Hakstian
Marsha L. Schroeder
Affiliation:
University of British Columbia
A. Ralph Hakstian*
Affiliation:
University of British Columbia
*
Requests for reprints should be sent to A. Ralph Hakstian, Department of Psychology, University of British Columbia, Vancouver, B.C., Canada V6T IY7.
Get access Rights & Permissions [Opens in a new window]

Abstract

A two-facet measurement model with broad application in the behavioral sciences is identified, and its coefficient of generalizability (CG) is examined. A normalizing transformation is proposed, and an asymptotic variance expression is derived. Three other multifaceted measurement models and CGs are identified, and variance expressions are presented. Next, an empirical investigation of the procedures follows, and it is shown that, in most cases, Type I error control in inferential applications is precise, and that the estimates are relatively efficient compared with the correlation coefficient. Implications for further research and for practice are noted. In an Appendix, four additional models, CGs, and variance expressions are presented.

Keywords

generalizability theorycoefficient of generalizabilityreliabilitymultifaceted measurement designssampling variance
Type
Original Paper
Information
Psychometrika , Volume 55 , Issue 3 , September 1990 , pp. 429 - 447
Copyright
Copyright © 1990 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The research reported herein formed part of a doctoral dissertation conducted by Marsha Schroeder (Schroeder, 1986), under the direction of Ralph Hakstian, at the University of British Columbia. We acknowledge with thanks the contributions to this research of Todd Rogers and James Steiger. We are also very indebted to an mous reviewer who provided some important clarifications in connection with two of the models considered. Some support for this research was provided by a grant to Ralph Hakstian from the Natural Sciences and Engineering Research Council of Canada.

References

Box, G. E. P. (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems, II. Effects of inequality of variance and of correlations between errors in the two-way classification. Annals of Mathematical Statistics, 25, 484–498.CrossRefGoogle Scholar
Brenan, R. L., Harris, D. J., & Hanson, B. A. (1987). The bootstrap and other procedures for examining the variability of estimated variance components in testing contexts. Paper presented at the annual meeting of the National Council on Measurement in Education, Washington, D.C.Google Scholar
Chalmers, D. J., & Knight, R. G. (1985). The reliability of ratings of the familiarity of environmental stimuli: A generalizability analysis. Environment and Behavior, 17, 223–238.CrossRefGoogle Scholar
Collins, J. R. (1970). Jackknifing generalizability, Boulder, Colorado: University of Colorado.Google Scholar
Cronbach, L. J., Gleser, G. C., & Nanda, H., Rajaratnam, N. (1972). The dependability of behavioral measurements, New York: Wiley.Google Scholar
Feldt, L. S. (1965). The approximate sampling distribution of Kuder-Richardson reliability coefficient twenty. Psychometrika, 30, 357–370.CrossRefGoogle ScholarPubMed
Feldt, L. S. (1969). A test of the hypothesis that Cronbach's alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363–373.CrossRefGoogle Scholar
Feldt, L. S. (1980). A test of the hypothesis that Cronbach's alpha reliability coefficient is the same for two tests administered to the same sample. Psychometrika, 45, 99–105.CrossRefGoogle Scholar
Fleiss, J. L. (1971). On the distribution of a linear combination of independent Chi squares. Journal of the American Statistical Association, 66, 142–144.CrossRefGoogle Scholar
Fleiss, J. L., & Shrout, P. E. (1978). Approximate interval estimation for a certain intraclass correlation coefficient. Psychometrika, 43, 259–262.CrossRefGoogle Scholar
Gillmore, G. M., Kane, M. T., & Naccarato, R. W. (1978). The generalizability of student ratings of instruction. Journal of Educational Measurement, 15, 1–15.CrossRefGoogle Scholar
Hakstian, A. R., & Schroeder, M. L. (1986). Inferential procedures for generalizability coefficients. Paper presented at the annual meeting of the Society of Multivariate Experimental Psychology, Atlanta, Georgia.Google Scholar
Hakstian, A. R., & Whalen, T. E. (1976). AK-sample significance test for independent alpha coefficients. Psychometrika, 41, 219–231.CrossRefGoogle Scholar
Katerberg, R., Smith, F. J., & Hoy, S. (1977). Language, time, and person effects on attitude scale translations. Journal of Applied Psychology, 62, 385–391.CrossRefGoogle Scholar
Knuth, D. E. (1968). The art of computer programming: Vol. 2. Seminumerical algorithms, Reading, MA: Addison-Wesley.Google Scholar
Kraemer, H. C. (1981). Extensions of Feldt's approach to testing homogeneity of coefficients of reliability. Psychometrika, 46, 41–45.CrossRefGoogle Scholar
Leone, F. C., & Nelson, L. S. (1966). Sampling distributions of variance components. I. Empirical studies of balanced nested designs. Technometrics, 8, 457–468.CrossRefGoogle Scholar
Marascuilo, L. A. (1966). Large sample multiple comparisons. Psychological Bulletin, 65, 280–290.CrossRefGoogle ScholarPubMed
Nussbaum, A. (1984). Multivariate generalizability theory in educational measurement: An empirical study. Applied Psychological Measurement, 8, 219–230.CrossRefGoogle Scholar
Paulson, E. (1942). An approximate normalization of the analysis of variance distribution. Annals of Mathematical Statistics, 13, 233–235.CrossRefGoogle Scholar
Rao, C. R. (1973). Linear statistical inference and its applications 2nd ed.,, New York: Wiley.CrossRefGoogle Scholar
Rouanet, H., & Lépine, D. (1970). Comparison between treatments is a repeated measurement design: ANOVA and multivariate methods. British Journal of Mathematical and Statistical Psychology, 23, 147–163.CrossRefGoogle Scholar
Satterthwaite, F. E. (1941). Synthesis of variance. Psychometrika, 6, 309–316.CrossRefGoogle Scholar
Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2, 110–114.CrossRefGoogle ScholarPubMed
Schroeder, M. L. (1986). Inferential procedures for multifaceted coefficients of generalizability, Vancouver, Canada: University of British Columbia.Google Scholar
Schroeder, M. L., Schroeder, K. G., & Hare, R. D. (1983). Generalizability of a checklist for assessment of psychopathy. Journal of Consulting and Clinical Psychology, 51, 511–516.CrossRefGoogle ScholarPubMed
Shavelson, R. J., & Webb, N. M. (1981). Generalizability theory: 1973–1980. British Journal of Mathematical and Statistical Psychology, 34, 133–166.CrossRefGoogle Scholar
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.CrossRefGoogle ScholarPubMed
Smith, P. L. (1978). Sampling errors of variance components in small sample multifacet generalizability studies. Journal of Educational Statistics, 3, 319–346.CrossRefGoogle Scholar
Webb, N. M., Shavelson, R. J., Shea, J., & Morello, E. (1981). Generalizability of General Education Development ratings of jobs in the United States. Journal of Applied Psychology, 66, 186–192.CrossRefGoogle Scholar
Woodruff, D. J., & Feldt, L. S. (1986). Tests for the equality of several alpha coefficients when their sample estimates are dependent. Psychometrika, 51, 393–413.CrossRefGoogle Scholar