Ascertaining top evidence in emergency medicine: A modified Delphi study

Abstract

Objectives

The application of evidence-informed practice in emergency medicine (EM) is critical to improve the quality of patient care. EM is a specialty with a broad knowledge base making it daunting for a junior resident to know where to begin the acquisition of evidence-based knowledge. Our study’s objective was to formulate a list of “top papers” in the field of EM using a Delphi approach to achieve an expert consensus.

Methods

Participants were recruited from all 14 specialty EM programs across Canada by a nomination process by the program directors. The modified Delphi survey consisted of three study rounds, each round sent out via email. The study tool was piloted first with McMaster University’s specialty EM residents. During the first round, participants individually listed top papers relevant to EM. During the two subsequent rounds, participants ranked the papers listed in the first round, with a chance to adjust ranking based on group responses.

Results

A total of eight EM specialty programs responded with 30 responses across the three rounds. There were 119 studies suggested in the first round, and, by the third round, a consensus of>70% agreement was reached to generate the final list of 29 studies.

Conclusions

We produced, via an expert consensus, a list of top studies relevant for Canadian EM physicians in training. It can be used as an educational resource for junior residents as they transition into practice.

Résumé

Contexte

L’application de la pratique fondée sur des données probantes en médecine d’urgence (MU) est cruciale pour l’amélioration de la qualité des soins. La MU est une spécialité qui exige une vaste base de connaissances, tant et si bien que les résidents juniors ne savent pas par où commencerce pour acquérir du savoir fondé sur des données probantes. L’étude visait donc à dresser une liste des « meilleurs articles » dans le domaine de la MU, à l’aide de la méthode Delphi pour permettre l’établissement d’un consensus entre spécialistes.

Méthode

Les participants ont été recrutés parmi les 14 programmes de spécialité en MU, offerts partout au Canada, après un processus de désignation d’experts par les directeurs de programme. L’enquête menée selon une version modifiée de la méthode Delphi consistait en trois tours de vote, chacun tenu par courriel. L’outil d’enquête a d’abord été soumis à un essai chez les résidents inscrits au programme de spécialité en MU de l’Université McMaster. Par la suite, les participants ont dressé individuellement une liste des meilleurs articles en MU au premier tour et, aux deux tours suivants, ordonné les articles soumis la première fois, puis modifié au besoin leur classement selon les réponses du groupe.

Résultats

Des représentants de huit programmes de spécialité en MU ont donné 30 réponses au cours des trois tours de vote. Il y avait 119 études suggérées au premier tour, et le nombre a été ramené à 29 sur la liste définitive, après atteinte d’un consensus>70 %.

Conclusion

Les participants ont établi, par voie de consensus entre spécialistes, une liste des études les plus pertinentes pour les médecins en formation en MU, au Canada. Elle peut servir de ressource didactique aux résidents juniors durant leur passage de l’apprentissage à la pratique.

CLINICIAN’S CAPSULE

What is known about the topic?

EM is a specialty with a broad knowledge base making it daunting for a junior resident to know where to begin the acquisition of evidence-based knowledge.

What did the study ask?

What list of “top papers” was formulated in the field of EM using a national Canadian Delphi approach to achieve an expert consensus?

What did the study find?

A list was produced of top studies relevant for Canadian EM physicians in training.

Why does this study matter to clinicians?

The list produced can be used as an educational resource for junior residents.

INTRODUCTION

The ability to critically appraise and apply evidence to clinical practice is a skill required by emergency medicine (EM) specialists and is an accreditation requirement for residency programs. ^{1 - 4} However, because EM is a generalist specialty, it can be daunting for a junior learner to know where to even begin their acquisition of evidence-informed EM knowledge. There have been many attempts to create lists of sentinel papers for junior learners to use as a springboard to developing an evidence-based clinical practice. Lists of “top cited papers” in EM have been previously compiled, but these lists are often arbitrary and based on the idiosyncrasies and experiences of the curators. ^{5 , 6} In the growing age of FOAM (Free Open Access to Medical Education), many online resources have created lists of “top papers.” ^{7 - 9} These lists can be an excellent starting point but tend to be compiled off of the opinions of one or two authors or are focused on recent publications, often excluding sentinel papers that have been published in the past. ^{10 , 11 , 14}

The Colorado Compendium created a list of 100 sentinel articles for EM residents. ^{12 , 13} However, it reflects the opinions of a single American academic centre. Similarly, the best evidence in emergency medicine (BEEM) score ¹⁵ has been created to define lists of more recent evidence but not used to aggregate papers that have historically changed practice. While other groups have reported aggregating papers via crowdsourcing from senior residents ^{16 , 17} at a national review course, ¹⁸ no systematic survey of Canadian academic EM faculty responsible for training junior emergency physicians has been conducted to date.

The objective of this study was to systematically generate a list of “top EM papers” (i.e., practice changing results based on strong methods) to assist junior Canadian EM trainees in developing evidence-informed clinical practice.

METHODS

Participants were recruited from all Royal College of Physicians and Surgeons of Canada specialist EM programs. All 14 program directors were contacted via email with a description of the study and a request to nominate two EM experts from their institution with an interest or background in emergency medical education or research.

Delphi survey

Our modified, three-round Delphi survey was conducted electronically with the aim of gathering and aligning opinions from a national set of experts with different perspectives with the hope of defining a more generalizable list of initial papers. ^{19 - 21} The Delphi rounds were distributed in 3-week intervals using Google Forms (Google, Mountain View, CA), and delivered via email. The nationwide Delphi was conducted between July and November 2016.

Round 1 consisted of an open-ended questionnaire in which participants were asked to generate a list of what they felt were the most important and influential papers related to the field of EM. Results of Survey 1 were compiled for use in Survey 2. In round 2, participants were asked to rank each paper on the compiled list as Include, Not include, Important but not top priority, or Unfamiliar. Participants were also asked to provide a rationale for their scoring. In round 3, participants viewed the percentage of participants who thought each paper should be included in the final list and the free-text comments. Participants then selected whether they agreed or disagreed with the inclusion. Study results were then categorized into six categories for analysis: 100% inclusion, 70% inclusion, 50% inclusion, primarily “good to know,” no consensus (less than 50% agreement), and primarily exclude or unfamiliar.

Ethics

We received approval from the Hamilton Integrated Research Ethics Board (HIREB) to conduct this study.

RESULTS

Participants

A total of 13 participants participated in all three rounds of the study. Not all participants were involved in each round of the study. Eight of the 14 sites with Royal College training programs in Canada were included. Appendix A lists the breakdown of the sites and participants.

Delphi results

From the first round of the Delphi, a total of 120 papers were nominated (Appendix B). By round 3, there were 10 studies with “100% agreement” for final inclusion. Nineteen other studies included in the final list were endorsed by more than 70% of participants as “must include” (Table 1).

Table 1. Top studies in Delphi survey

DISCUSSION

We identified 29 top papers for junior Canadian EM trainees to use in developing evidence-informed practice. Interestingly, even amongst experts in the field, there is a significant variability in what is considered “must know” studies. This represents a structured attempt with a national consultation of experts to ascertain foundational papers for Canadian trainees.

Although other studies have created similar reading lists for EM, ^{11 - 13 , 15 - 17} this study uses a more robust methodology to systematically create a consensus list with national input. The variability seen in Survey 1 demonstrates how much literature exists that is relevant to the field of EM. In comparing our list with the popularly cited 2016 list from the Academic Life in Emergency Medicine blog, ¹⁴ there is some degree of overlap; 11 studies were included on both lists. Some reasons for discrepancies between the two lists include a higher prevalence of Canadian-based EM literature, newer studies, and a smaller final list for our study. Future directions may include using a similar methodology to create an annual “must read” lists for continuing professional development and revalidation. ^{22 - 24}

Limitations

Firstly, the sample size was much smaller than originally anticipated. Although there is no consistent standard in the literature for a Delphi sample, ¹⁹ a total of 13 participants is within the typical size range for this type of study. We were also able to achieve only 50% representation from a diversity of geographical regions; the exclusive use of an English language survey may have been a barrier with a francophone faculty.

CONCLUSIONS

Using a modified Delphi technique, we aggregated a nationally endorsed list of top papers for junior EM trainees. This list may anchor an evidence-informed reading list for junior EM trainees.