This article discusses subspecialty Canadian neurosurgeons’ perceptions of entrustable professional activities (EPAs) assessments and variabilities prior to the implementation of the Competence by Design (CBD) system in Canada. Vascular neurosurgeons were asked to reflect on how they would evaluate and give feedback to neurosurgery residents concerning the EPA “Performing surgery for patients with an intracranial aneurysm.” Interviews were transcribed and analyzed using a deductive approach. Themes were derived from these interviews and reflected on the subjectivity and biases present in the EPA assessment forms. Indeed, faculty may require more training in the transitioning to a CBD evaluation system.

Even before starting your evening shift you know it's going to be busy. Ambulances are lined up in front of the hospital, and the charge nurse already seems stressed out. The senior Emergency Medicine (EM) resident is standing in the physician office, ready to start her shift as well. You have worked with her a few times during this rotation. She is competent, you trust in her management plans for all her individual patients. Together you both review the patient tracker: a variety of patient presentations ready to be seen, plus an additional 20 patients in the waiting room. Negotiating the learning objective for the shift, the resident indicates that she would like to work on more efficiently managing patient flow and the administration of the emergency department (ED). But…isn't that a skill you just learn from experience? You wonder what evidence-informed strategies might exist for training her for this next step.

Introduction: Competency-based medical education (CBME) relies on pragmatic assessment to inform trainee progression decisions. It is unclear whether face-to-face workplace-based assessment (WBA) scoring by faculty reflects their true perception of trainee competence, as many factors influence individual assessments. To better defend competence committee decisions, it is critical to understand how accurately WBAs reflect the faculty's honest perception of resident competence and entrustment. Methods: To best capture faculty perception of trainee competence, we created a periodic performance assessment (PPA) tool for anonymous faculty assessment of residents after repeated clinical interactions. PPA surveys were distributed to full-time EM faculty at a single Canadian FRCPC-EM training site. Faculty were asked to score residents on entrustable professional activities (EPAs) based on encounters over the previous 6-months, and were advised that all data would be anonymized. All WBA scores for FRCPC-EM residents (N = 21) were collected from the 6-months preceding PPA completion. Analysis compared paired WBA and PPA entrustment scores for an individual resident, faculty, and EPA using Wilcoxon Signed Ranks tests and Spearman correlations. Data were analyzed across faculty, EPAs, and both faculty and EPA. Results: About half (17/33) of all invited full-time EM faculty participated. Overall, anonymous PPAs had a significantly lower mean score compared to face-to-face WBAs (3.61-3.69 vs. 3.92-4.06, p < 0.001 for all) across all groupings. Individual WBAs had a low-moderate correlation with individual PPAs (rho = 0.44). When scores were averaged across 1) faculty or 2) EPA, there was an increase in correlation, but it remained moderate (rho = 0.53 and 0.54, respectively). When scores were averaged for an individual resident across 3) faculty and EPA, there was a strong correlation between WBA and PPA (rho = 0.86). Conclusion: There is only moderate correlation between an individual faculty's WBAs and their anonymous longitudinal entrustment for a given resident on a specific EPA. These results may signal caution when interpreting WBA scores in the context of high stakes decisions. Aggregated scores from multiple faculty and/or multiple EPAs substantially increased the correlation between WBA and PPA. These findings highlight the importance of using aggregated WBA scores across multiple assessors and EPA for high-stakes resident progression decisions, to minimize the noise and bias in individual assessment.

Introduction: Emergency department (ED) crowding and increased patient load has been shown to have an impact on physician decision making and patient mortality. As the volumes in Canadian EDs increase, so does the need to effectively prepare new learners for the challenges ahead. This study aims to determine which level of training varying teaching techniques should be employed to educate Emergency Medicine (EM) residents about ED management and flow in the age of competency based medical education. Methods: We designed a survey that contained a previously derived list of ED flow and management teaching strategies. We piloted and edited the survey based on feedback from operations and educational experts. A total of 21 teaching techniques were included in the final survey ranging from didactic teaching sessions to experiential techniques such as residents running the department with supervision. Then, we invited members of the Royal College of Physicians and Surgeons of Canada EM specialty committee, the Canadian Association of Emergency Physicians Education Scholarship Section, and the Canadian EM Simulation Educators Collaborative to participate in our survey. We analysed the results using simple descriptive statistics. Results: A total of 21 EM (38% female, 62% male) educators from 11 programs (78% of Royal College Training sites) responded to the survey, representing 7/10 provinces, with a mean years-in-practice of 15.2 years (SD 9.7). All respondents were involved in resident education; 66% had a current formal educational role, such as Program Director. Results showed a universal trend towards teaching flow and management skills later in residency. Participants endorsed 35.93% of teaching strategies for the “Core of discipline” and 39.65% for the “Transition to practice” stages of training. Didactic and observational techniques were occasionally considered acceptable at earlier training stages, whereas experiential teaching techniques were skewed towards the later stages of residency. Conclusion: EM educators from across Canada believe that most teaching techniques for flow are better suited for the later stages of residency training, with didactic techniques more suitable earlier on. This work will inform faculty development on managerial/leadership skills teaching in the ED.

Introduction: The specialist Emergency Medicine (EM) postgraduate training program at Queens University implemented a new Competency-Based Medical Education (CBME) model on July 1 2017. This occurred one year ahead of the national EM cohort, in the model of Competence By Design (CBD) as outlined by the Royal College of Physicians and Surgeons of Canada (RCPSC). This presents an opportunity to identify critical steps, successes, and challenges in the implementation process to inform ongoing national CBME implementation efforts. Methods: A case-study methodology with Rapid Cycle Evaluation was used to explore the lived experience of implementing CBME in EM at Queens, and capture evidence of behavioural change. Data was collected at 3- and 6- months post-implementation via multiple sources and methods, including: field observations, document analysis, and interviews with key stakeholders: residents, faculty, program director, CBME lead, academic advisors, and competence committee members. Qualitative findings have been triangulated with available quantitative electronic assessment data. Results: The critical processes of implementation have been outlined in 3 domain categories: administrative transition, resident transition, and faculty transition. Multiple themes emerged from stakeholder interviews including: need for holistic assessment beyond Entrustable Professional Activity (EPA) assessments, concerns about the utility of milestones in workplace based assessment by front-line faculty, trepidation that CBME is adding to, rather than replacing, old processes, and a need for effective data visualisation and filtering for assessment decisions by competency committees. We identified a need for administrative direction and faculty development related to: new roles and responsibilities, shared mental models of EPAs and entrustment scoring. Quantitative data indicates that the targeted number of assessments per EPA and stage of training may be too high. Conclusion: Exploring the lived experience of implementing CBME from the perspectives of all stakeholders has provided early insights regarding the successes and challenges of operationalizing CBME on the ground. Our findings will inform ongoing local implementation and higher-level national planning by the Canadian EM Specialty Committee and other programs who will be implementing CBME in the near future.

Introduction: The field of Clinical Informatics (CI) and specifically the electronic health record, has been identified as a key facilitator to achieve a sustainable evidence-based healthcare system for the future. International graduate medical education programs have been challenged to ensure their trainees are provided with appropriate skills to deliver effective and efficient healthcare in an evolving environment. This study explored how international Emergency Medicine (EM) specialist training standards address training in relevant areas of CI. Methods: A list of categories of CI competencies relative to EM was developed following a thematic review of published references documenting CI curriculum and competencies. Publically available, published documents outlining core content, curriculum and competencies from international organizations responsible for specialty graduate medical education and/or credentialing in EM for the United States, Canada, Australasia, the United Kingdom and Europe. These EM training standards were reviewed to identify inclusion of topics related to the relevant categories of CI competencies. Results: A total of 23 EM curriculum documents were included in the thematic analysis. Curricula content related to critical appraisal/evidence based medicine, leadership, quality improvement and privacy/security were included in all EM curricula. The CI topics related to fundamental computer skills, computerized provider order entry and patient-centered informatics were only included in the EM curricula documents for the United States and were absent for each other organization. Conclusion: There is variation in the CI related content of the international EM specialty training standards which were reviewed. Given the increasing importance of CI in the future delivery of healthcare, organizations responsible for training and credentialing specialist emergency physicians must ensure their training standards incorporate relevant CI content, thus ensuring their trainees gain competence in essential aspects of CI.