MP002: Beyond rater cognition: the impact of supervisor continuity on the quality of documented work-based assessments

Abstract

Introduction: Barriers to completing high quality work-based assessments (WBAs) include relational factors such as the episodic and fragmented interaction that often exists between clinical supervisors and trainees. In an effort to increase supervisor-trainee continuity, the Department of Emergency Medicine at the University of Ottawa created Clinical Teaching Teams (CTT) in which a resident and clinical supervisor work matched shifts together throughout the year. The aim of this study was to determine the impact of supervisor-trainee continuity on the quality of assessments documented on Daily Encounter Cards (DECs). Methods: DECs completed by 20 clinical supervisors were collected and sorted into three groups representing differing degrees of supervisor-trainee continuity (Group 1: CTT emergency resident; Group 2: non-CTT emergency resident; Group 3: non-CTT off-service resident). DECs were scored using the Completed Clinical Evaluation Report Rating (CCERR), a 9-item instrument that has been shown to have reliable ratings and the ability to discriminate the quality of completed DECs. Scores were analyzed using a univariate ANOVA with “mean CCERR score” as the dependent variable and “continuity group” and “supervisor” as between-subject variables. The relationship between CCERR scores and number of CTT encounters over time was examined using a repeated measures ANOVA with “encounter number” as the within-subject factor. Results: Mean CCERR scores for the CTT (21.0, SD=5.8), non-CTT (21.9, SD=4.2), and off-service (20.7, SD=4.0) groups differed (p=0.019). A subsequent pairwise comparison demonstrated a statistically significant difference in means between the non-CTT and off-service groups (p=0.04); however, this 1.2 difference on the 45-point CCERR scale is unlikely to be of any educational significance. The number of repeated encounters did not have a statistically significant effect on CCERR scores (p=0.43) indicating that DEC quality did not improve with greater supervisor-trainee interaction. Conclusion: DEC quality as scored by the CCERR was low for all three groups. Increasing supervisor continuity alone did not result in higher quality assessments of clinical performance. Additional research focusing on the educational alliance that develops between supervisor and trainee may hold greater promise.