CLINICIAN'S CAPSULE
What is known about the topic?
Little is known about how the Royal College Emergency Medicine (RCEM) residency programs are currently selecting their residents.
What did this study ask?
How are RCEM residency programs currently selecting their residents?
What did this study find?
Heterogeneity exists between programs concerning the value of various selection elements and only 1 of 13 programs has performed a process evaluation.
Why does this study matter to clinicians?
This study characterizes the landscape of RCEM residency selection, improving transparency and acting as a driver for process improvement initiatives.
INTRODUCTION
In Canada, emergency medicine consistently ranks among the most competitive residency specialties with only 0.58 positions available per applicant.Reference Green, Jones and Thomas1,2 Unfortunately, little is known about how Royal College Emergency Medicine (RCEM) residency programs are currently selecting their residents. In general, the process of selection is complex, relying on a series of imperfect criteria without an established gold standard.Reference Van Meter, Williams, Banuelos and Carlson3–Reference Bandiera, Abrahams and Ruetalo5 Amid talented applicant pools where differences between candidates are often subtle, the process of selection can pose an even greater challenge.
In the United States, there have been multiple studies that have helped identify the most important application elements in the U.S. residency match.Reference Green, Jones and Thomas1,Reference Crane and Ferraro6–Reference Negaard, Assimacopoulos, Harland and Van Heukelom8 Across these studies, objective criteria, including clerkship grades and United States Medical Licensing Examination (USMLE) scores, were consistently considered to be important factors. In Canada, we do not have a standardized test, such as the USMLE, to act as a benchmark for student aptitude. Our medical school grading system is “pass” or “fail” at a majority of schools.Reference Glauser9 As such, students and selection committees alike have relatively fewer objective data points to consider. Which criteria Canadian residency programs are relying on remain unclear, creating the potential for an incongruent process both within and across specialties.Reference Wycliffe-Jones, Hecker and Schipper10 Moreover, medical students often identify that they find the selection process to be ambiguous and too heavily reliant on subjective criteria.Reference Ryan11 Before improvements can be made, however, our current selection practices must first be adequately characterized. To our knowledge, no other study has sought to achieve this characterization at a national level.
The objective of our study is to develop a better understanding of the Canadian residency selection process in emergency medicine through the use of a national program director survey. Specifically, we aim to identify which elements of the Canadian Residency Matching Service (CaRMS)12 application are most valued by program directors. This will add transparency to the selection process for future candidates and provide valuable information to selection committees who will be able to evaluate how their selection practices differ from those of other Canadian programs and established best practice guidelines.Reference Bandiera, Abrahams and Cipolla13,Reference Roger, Epstein and Bandiera14 Information gained will fill a void in the Canadian residency selection literature and act as an important driver for future process improvement initiatives.
METHODS
Study design
An online survey was distributed to all Royal College of Physicians and Surgeons of Canada emergency medicine programs via email using the modified Dillman's Tailored Design Method.Reference Dillman15 The initial email contained information about the purpose and goals of the study, as well as the survey link. The survey was hosted on the SurveyMonkey Team Advantage platform. Up to three reminder emails were sent, each spaced by a two-week time interval. Research was conducted between March and May of 2019. An ethics waiver was obtained from the Ottawa Health Science Network Research Ethics Board.
Population
Surveys were distributed to program directors and assistant program directors from all 14 Canadian RCEM programs. Our goal was to receive a response from each program to ensure a balanced national perspective. If both the program director and the assistant program directors from a single program submitted a survey, only the program director survey was used for data analysis.
Survey design
The survey content was developed by the study authors (one resident and two clinician educators)Reference Sherbino, Frank and Snell16 in consultation with three members of the residency selection committee of the University of Ottawa, Department of Emergency Medicine. Important elements of the application file, interview, and ranking process were identified and ultimately used to develop the questions for our survey. Text boxes for narrative comments were provided to allow respondents to elaborate or qualify their responses. The preliminary survey was piloted on the Canadian College of Family Physicians Emergency Medicine (CCFP-EM) certificate program directors at the University of Ottawa for clarity and comprehensibility, and further refinements to the survey were made based on pilot feedback. In its final version, our survey consisted of 22 items containing both open and closed-ended elements (Appendix A). Questions addressed the following six domains: application file, letters of reference, elective selection, interview, rank order, and selection process evaluation. The importance of each element was measured using a five-point Likert scale (1 = not at all important, 5 = very important).
Data analysis
Simple descriptive statistics were used to report the survey results. Percentages, means, and standard deviations were calculated for Likert scale responses; percentages were calculated for questions requiring the selection of a single drop-down answer choice. Data analysis was conducted using Microsoft Excel for Mac version 16.34. Open-ended questions were categorized into themes by the study authors if two or more respondents answered similarly. Only responses that could be categorized into themes were included in the results section. Narrative responses that clarified a closed-ended question were not mandatory fields and therefore do not necessarily reflect consensus opinion. Because of this, these data were not included in the results section and only used in the discussion if they were felt to enrich the dialogue.
RESULTS
In total, 13 of 14 Canadian RCEM residency programs completed our survey for an aggregate response rate of 92.9%. Program directors accounted for 76.9% of responses with the remainder from assistant program directors.
Table 1 displays the relative importance of each of the elements of the CaRMS application file. Letters of reference, curriculum vitae (CV), and the personal letter were identified as the three most important elements. Of these, letters of reference were most frequently identified as the most important. Respondents did not feel strongly that a candidate's research experience had to be specifically in the field of emergency medicine.
Table 1. Relative importance of CaRMS application file elements
Table 2 displays the relative importance of different characteristics of letter of reference authors. Having a high level of familiarity with the applicant, specializing in emergency medicine, and having an academic centre-based practice were the three most important characteristics; of these, having a high level of familiarity with the applicant was most frequently identified as the most important. The author being well known in the emergency medicine community was considered significantly less important by respondents. If an applicant completed an elective at their institution, 38% of respondents indicated that a letter written by a faculty member of their own program was important or very important.
Table 2. Key characteristics of the preceptor writing a letter of reference
The majority of respondents (76.9%) indicated that spending 26–75% of CaRMS elective time in emergency medicine demonstrated commitment to the specialty. Only a single respondent indicated that completing an elective at their institution was important or very important, and all respondents stated that a candidate could be invited for an interview if they did not do an elective at their institution.
When the interview stage is reached, 76.9% of respondents stated that the interview was weighted at least as much as the paper application in determining final rank order. Over half, 53.8%, stated that the interview was weighted more heavily; 76.9% of programs conducted structured interviews, where each candidate is asked the same set of questions in the same order; and less than half of these programs used a standardized rubric to evaluate candidates’ interview responses.
Table 3 demonstrates the relative importance of select interviewee qualities. Demonstrating honesty and sincerity with answers, providing clear and articulate answers, and offering unique or refreshing answers were identified as the three most important interview elements. Of these, demonstrating honesty and sincerity with answers was rated as the most important by the majority of respondents (61.5%).
Table 3. Relative importance of select interviewee qualities
When final candidate scores are established for both the paper application and the interview, 100% of program respondents acknowledged that further adjustment is made to the rank order list before final submission. All program respondents agreed that a candidate's perceived program ranking did not affect their ranking of the candidate.
When asked what the single most important thing that a prospective candidate could do to improve their chances of matching to an emergency medicine residency program, respondents varied greatly in their responses. Being strong clinically (23.1%) and demonstrating interest in a particular program (15.4%) were the most cited themes.
The majority (69.2%) of respondents indicated that their strongest residents were “often” or “almost always” the highest-ranked candidates in their respective CaRMS application year. Only one of the programs (7.7%) reported ever having completed a formal evaluation of their CaRMS selection process to determine effectiveness.
For an in-depth question by question breakdown of quantitative responses, please refer to Appendix A. For a snapshot of key takeaways for programs, students, and references, refer to Table 4.
Table 4. Key takeaways for programs, students, and references
DISCUSSION
Interpretation of results
The information gained from this study helps characterize the landscape of the RCEM residency selection process. We identified significant heterogeneity between programs concerning the importance of various elements of the application file where standard deviations approached and often exceeded one point (five-point Likert scale). Although generally being considered of low importance overall, transcripts and dean's letters demonstrated notably high variability, suggesting that they should not be entirely neglected. The understanding of a program director regarding the relationship between the CaRMS ranking of their residents and future performance was also highly variable and possibly relates to an overall lack of regular process/outcome evaluation. Recent calls for regular assessment of process outcomes have been published in Best Practices in Application and Selection (BPAS) for residency programs in Canada.Reference Bandiera, Abrahams and Cipolla13,Reference Roger, Epstein and Bandiera14 However, in our study, only 1 of 13 programs had ever performed a formal evaluation of their selection process. As such, outcomes evaluation of Canadian emergency medicine selection processes represents a much needed area of future study. The ultimate goal of selection should be to effectively identify the candidates who most wholly possess the characteristics that a given program desires. These characteristics may vary somewhat between programs but should be made explicit and the processes used to select them internally standardized.Reference Bandiera, Abrahams and Cipolla13
Letters of reference emerged as the most important element of a candidate's application file. Our data would suggest that applicants can improve the perceived quality of their reference letters by performing electives at academic hospitals and by asking for reference letters from emergency medicine preceptors who know them well, rather than targeting a well-known name in the emergency medicine community. The commonly held belief that the “name” of the writer matters above all else has been discredited. Programs can assist candidates with achieving improved familiarity by scheduling students with the same preceptor over multiple shifts. For preceptors, indicating how many students/residents they typically work with and providing a ranking relative to other trainees were helpful for several respondents. Certain respondents also felt that frequently reusing the same letters or ranking multiple applicants in the same high percentile detracted from the quality of a letter.
There is a growing body of evidence suggesting that non-structured or “narrative” letters of reference are highly subjective, inconsistent for the range of qualities described, and demonstrate suboptimal inter-rater agreement among readers.Reference Langdorf, Cheaito, Epter and Kazzi17,Reference Fortune18 Including a relative ranking of a candidate, as suggested previously, could help facilitate comparisons between applicants and improve alignment with the BPAS recommendation of striving to incorporate more objective assessment strategies.Reference Bandiera, Abrahams and Cipolla13 A potentially more robust solution could be to transition to standardized letters of reference (SLOR). SLORs consist of a set of standardized domains that can be tailored to a given specialty's unique features.Reference Langdorf, Cheaito, Epter and Kazzi17 In general, SLORs have been shown to allow for more direct comparisons between candidates and improved inter-rater reliability,Reference Langdorf, Cheaito, Epter and Kazzi17–Reference Hartman, Lefebvre and Manthey19 likely improving the overall utility of the letter of reference as a selection instrument.
Consistent with other studies, research experience ranked lower among selection criteria with only 38.5% of respondents identifying it to be important or very importantReference Green, Jones and Thomas1,Reference Crane and Ferraro6,Reference Negaard, Assimacopoulos, Harland and Van Heukelom8 ; what is clear from other studies is that perceived commitment to a specialty is a factor often considered by selections committees,Reference Crane and Ferraro6,Reference Hartman, Lefebvre and Manthey19 and research is one possible way to demonstrate this. Another is through elective choice. In our study, spending anywhere from 26–75% of CaRMS elective time in emergency medicine demonstrated commitment to the specialty. However, the Association of Faculties of Medicine of Canada has recently enacted a policy, effective beginning with the Class of 2021, that restricts student electives to a maximum of eight weeks in any single entry-level discipline, such as emergency medicine.20 Therefore, we recommend that students spend the maximum allowable CaRMS elective time in emergency medicine to demonstrate commitment to the specialty.
The majority of respondents reported that the interview is weighted more heavily than the application file in determining final rank order, consistent with much of the residency selection literature.Reference Crane and Ferraro6,Reference Negaard, Assimacopoulos, Harland and Van Heukelom8 Structured interviews where each candidate is asked the same questions in the same order were used in 76.9% of programs. Less than half (46.2%) use a standardized framework to evaluate responses. This raises some concerns about the reliability of the interview process. Although there is no definitive interview format that is considered to be the gold standard, it is generally well accepted that unstructured interviews are highly subjective and demonstrate low reliability.Reference Prideaux, Roberts and Eva4,Reference Stephenson-Famy, Houmard and Manyak21–Reference Gardner, Onofrio and Dunkin23 On the other hand, evidence for the validity and relative reliability of structured interviews has been previously shown.Reference Prideaux, Roberts and Eva4,Reference Blouin, Day and Pavlov24 One study demonstrated that implementing a half-day course that teaches the fundamentals of structured interviewing can substantially improve interviewer knowledge and inter-rater agreement, even among experienced interviewers.Reference Gardner, Onofrio and Dunkin23 Despite this, the literature continues to demonstrate a poor association between the interview and future resident performance.Reference Stephenson-Famy, Houmard and Manyak21,Reference Kenny, Mcinnes and Singh22 Additional research is needed to better understand what interview factors (e.g., format, content, method) are most helpful in identifying candidates who will later become successful residents.
Strengths and limitations
To the best of our knowledge, our study is the first to perform a comprehensive analysis of the Canadian RCEM residency selection process. The 92.9% response rate is a notable strength and helps ensure a balanced national perspective. The retrospective nature of the study is one potential limitation as certain aspects of our data will be subject to recall bias. Another limitation is that data collection using a survey instrument is inherently subject to a degree of self-report and social desirability bias. Having focused exclusively on Canadian emergency medicine residency programs, our results may not be generalizable to other specialties or geographic regions. Finally, as we focused exclusively on program directors and assistant program directors, our data may differ somewhat from the opinions of other selection committee members.
Study implications and future directions
The information gained from this study has characterized the landscape of the Canadian residency selection process in emergency medicine. It immediately adds transparency to the process for prospective candidates allowing for increased confidence when deciding where to focus their attention. For selection committees, the data provide an easy way to identify where their practice differs from other Canadian programs/selection best practices.Reference Roger, Epstein and Bandiera14,Reference Paterson, Hartmann, Woods, Martin and Thoma25 Now that we have a firm understanding of what factors are currently driving selection decisions, this study sets the stage for future research that aims to identify selection methods that are more predictive of future resident performance. With the transition to competency-based medical education and the resultant shift to more comprehensive resident assessments, the current climate is timely to conduct such research.
CONCLUSION
We have identified the most important elements of the residency application in emergency medicine: letters of reference, CV, and interview performance. Medical students should target letters of reference from preceptors who know them well, programs should provide opportunities for students to experience multiple shifts with the same preceptors, and referees should provide context regarding how a student compares to others. Going forward, residency programs are encouraged to conduct regular reviews of their selection process to ensure internal consistency and fairness to applicants.
Supplemental material
To view supplementary material for this article, please visit https://doi.org/10.1017/cem.2020.457.
