MP015: Daily encounter cards: evaluating the quality of documented assessments

Abstract

Introduction: In response to concerns in the literature over the quality of completed work-based assessments (WBAs), faculty development and rater training initiatives have been developed. The Completed Clinical Evaluation Report Rating (CCERR) was designed to evaluate these interventions by providing a measure of the quality of documented assessments on In-Training Evaluation Reports (ITERs). Daily Encounter Cards (DECs) are a common form of WBA used in the Emergency Department setting. A tool to evaluate initiatives aimed at improving the quality of completion of this widely used WBA is also needed. The purpose of this study was to provide validity evidence to support using the CCERR to assess the quality of DEC completion. Methods: This study was conducted in the Department of Emergency Medicine at the University of Ottawa. Six experts in resident assessment grouped 60 DECs into three quality categories (high, average, poor) based on their perception of how informative each DEC was for reporting judgments of the resident’s performance. Eight clinical supervisors (blinded to the expert groupings) scored the 10 most representative DECs in each group using the CCERR. Mean scores were compared using a univariate ANOVA to determine if the CCERR was able to discriminate DEC quality. Reliability for the CCERR scores was determined using a generalizability analysis. Results: Mean CCERR scores for the high (37.3, SD=1.2), average (24.2, SD=3.3), and poor (14.4, SD=1.4) quality groups differed (p<0.001). A pairwise comparison demonstrated that differences between all three quality groups were statistically significant (p<0.001), indicating that the CCERR was able to discriminate DEC quality as judged by experts. A generalizability study demonstrated the majority of score variation was due to differences in DECs. The reliability with a single rater was 0.95. Conclusion: There is strong validity evidence to support the use of the CCERR to evaluate DEC quality. It can be used to provide feedback to supervisors for improving assessment reporting, and offers a quantitative measure of change in assessor behavior when utilized as a program evaluation instrument for determining the quality of completed DECs.