In Chapter 12, the author discusses approaches to judging the effectiveness of both criterion-referenced and norm-reference performance assessments. The chapter includes guidelines for helping create or evaluate performance assessments for particular contexts. It also describes how to use statistical techniques for the same purpose. The chapter presents actual ratings from a classroom-based group oral discussion test and shows how teachers used statistics to determine both score dependability and reliability. The author discusses how to calculate coefficient agreement to help determine the dependability of the assessment and Cronbach’s Alpha to help determine its reliability. A major point of the chapter is that when certain conditions exist, test users can exploit an assessment to determine test takers’ mastery of language criteria (criterion-referenced purpose) and to compare their abilities (norm-referenced purpose). The author provides an appendix that shows readers how to use Excel software to calculate the statistics in the chapter.

In Chapter 9, the author discusses both content and statistical approaches to evaluating and revising criterion-referenced assessments at both the item and test levels. The chapter includes guidelines for judging criterion-referenced item types and the content of specific items. The chapter also discusses standard setting and takes readers through an example of how to use the Angoff method for this purpose. It introduces descriptive statistics for criterion-refenced assessments, including median, mastery rate, item facility, and B-Index, a statistic that provides an indication of how well an item distinguishes among test takers who have mastered and test takers who have not mastered the targeted ability of the assessment. The chapter concludes with an explanation of how to calculate and interpret dependability for a classroom vocabulary test. It includes an appendix which shows how to use Excel to calculate each of the statistics that the chapter introduces.

Diagnostic reliability receives cursory attention in the literature with wide ranges of kappa coefficients often interpreted as “adequate.” Moreover, the vast majority of personality disorder (PD) kappa estimates, including those reviewed by Flory (this chapter), are derived from the audio/video recording method. This method results in inflated estimates of diagnostic reliability, which provides limited insight into whether patients would receive the same diagnosis at different hospitals or clinics and whether researchers are studying similar patients. A more rigorous and ecologically valid method for assessing diagnostic reliability is the test-retest method. Although echoing some of Flory’s points, the authors here disagree with her assertion that categorical PDs demonstrate acceptable reliability. In fact, the reliability of categorical PDs assessed using the test-retest method is far lower than the DSM-5 conceptualization would indicate. Moreover, the test-retest diagnostic reliability of categorical PDs fails to achieve minimal benchmarks established in the “normal” personality literature, indicating critical problems for the DSM-5 categorical model. If the goal is to provide optimal patient care and advance clinical science, then adopting the trait-based dimensional model of personality pathology from section III of DSM-5 is necessary.