Hostname: page-component-745bb68f8f-mzp66 Total loading time: 0 Render date: 2025-01-07T18:59:09.325Z Has data issue: false hasContentIssue false
  • English
  • Français

Bias in Estimation of Misclassification Rates

Published online by Cambridge University Press:  01 January 2025

Shelby J. Haberman
Shelby J. Haberman*
Affiliation:
Educational Testing Service, Princeton
*
Requests for reprints should be sent to Shelby J. Haberman, Mailstop 12T, Educational Testing Service, Rosedale Road, Princeton, NJ 08541. E-mail: shaberman@ets.org
Get access Rights & Permissions [Opens in a new window]

Abstract

When a simple random sample of size n is employed to establish a classification rule for prediction of a polytomous variable by an independent variable, the best achievable rate of misclassification is higher than the corresponding best achievable rate if the conditional probability distribution is known for the predicted variable given the independent variable. In typical cases, this increased misclassification rate due to sampling is remarkably small relative to other increases in expected measures of prediction accuracy due to samplings that are typically encountered in statistical analysis.

This issue is particularly striking if a polytomous variable predicts a polytomous variable, for the excess misclassification rate due to estimation approaches 0 at an exponential rate as n increases. Even with a continuous real predictor and with simple nonparametric methods, it is typically not difficult to achieve an excess misclassification rate on the order of n−1. Although reduced excess error is normally desirable, it may reasonably be argued that, in the case of classification, the reduction in bias is related to a more fundamental lack of sensitivity of misclassification error to the quality of the prediction. This lack of sensitivity is not an issue if criteria based on probability prediction such as logarithmic penalty or least squares are employed, but the latter measures typically involve more substantial issues of bias. With polytomous predictors, excess expected errors due to sampling are typically of order n−1. For a continuous real predictor, the increase in expected error is typically of order n−2/3.

Keywords

penalty functionentropyconcentrationlarge-sample approximation
Type
Original Paper
Information
Psychometrika , Volume 71 , Issue 2 , June 2006 , pp. 387 - 394
Copyright
Copyright © 2006 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Anscombe, F.J. (1956). On estimating binomial response relations. Biometrika, 43, 461–464.CrossRefGoogle Scholar
Bahadur, R.R., & Ranga Rao, R. (1960). On deviations of the sample mean. Annals of Mathematical Statistics, 31, 1015–1027.CrossRefGoogle Scholar
Chernoff, H. (1952). A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. Annals of Statistics, 23, 493–507.CrossRefGoogle Scholar
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.CrossRefGoogle Scholar
Cronbach, L.J., & Gleser, G.C. (1965). Psychological tests and personnel decisions. Chicago, IL: University of Illinois Press.Google Scholar
Feng, X., Dorans, N.J., Patsula, L.N., & Kaplan, B. (2003). Improving the statistical aspects of e-rater superscript registered: Exploring alternative feature reduction and combination rules. Princeton, NJ: Educational Testing Service. Technical Report RR-03-15Google Scholar
Gilula, Z., & Haberman, S.J. (1995). Dispersion of categorical variables and penalty functions: Derivation, estimation, and comparability. Journal of the American Statistical Association, 90, 1447–1452.CrossRefGoogle Scholar
Gilula, Z., & Haberman, S.J. (1995). Prediction functions for categorical panel data. Annals of Statistics, 23, 1130–1142.CrossRefGoogle Scholar
Goodman, L.A., & Kruskal, W.H. (1954). Measures of association for cross-classifications. Journal of the American Statistical Association, 49, 732–764.Google Scholar
Haberman, S.J. (1982). Analysis of dispersion of multinomial responses. Journal of the American Statistical Association, 77, 568–580.CrossRefGoogle Scholar
Haberman, S.J. (1982). Measures of association. In Kotz, S., & Johnson, N.L. (Eds.), Encyclopedia of statistical sciences (Vol. 1, pp. 130–137). New York: Wiley.Google Scholar
Hambleton, R.K., Swaminathan, H., & Rogers, H.J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.Google Scholar
Lachenbruch, P.A. (1968). On expected probabilities of misclassification in discriminant analysis, necessary sample size, and a relation with the multiple correlation coefficient. Biometrics, 24, 823–834.CrossRefGoogle Scholar
Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
Savage, L. (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66, 783–801.CrossRefGoogle Scholar
Stephan, F.F. (1945). The expected value and variance of the reciprocal and other negative powers of a positive bernoullian variate. Annals of Mathematical Statistics, 16, 50–61.CrossRefGoogle Scholar