Delving into Delphis

What do the RAND corporation, top literature in emergency medicine, and bougie-assisted cricothyroidotomy have in common? The Delphi method.

INTRODUCTION

In this issue of CJEM, we find two education papers that use the Delphi method, or at least a modification of such, to determine the outcome of their study. The Delphi is the most commonly applied consensus group method, or systematic means to measure and develop a consensus.

The modified Delphi study has become a more frequently used investigatory tool in the last several years, especially in medical education. This method can be especially useful when examining questions requiring expertise, and therefore might be applied to a variety of emergency medicine (EM) questions and problems for which uncertain empirical evidences exist.

WHAT IS THE DELPHI METHOD?

The Delphi method was originally developed to harness the wisdom of a group of experts in a field to determine an anticipated outcome. Usually, it requires an anonymous method to iteratively refine the answer to a question. Key elements of a Delphi method include the assembly of a large and diverse group of experts who anonymously provide their opinions, a mechanism for a structured data collection and feedback, and a facilitator who collects the data and reports them back to the experts for reconsideration and recalibration. In its original implementation by the RAND Air Force corporation of America in the 1950s, this was done via paper-based questionnaires and surveys, but today most of these techniques use some form of an electronic method of data collection and reporting.

The Delphi technique was developed by the RAND Corporation to effectively gather experts’ opinions to predict the probability of enemy attacks in wartime.¹ Since then, the Delphi has been used extensively in multiple fields of study, including medicine, nursing, business, public policy, science and technology, and medical education. In medical education, where decisions must often be made with limited information, the consensus methods, including the Delphi, are used extensively for the purposes of developing or renewing curricula, developing assessment tool items, defining competencies, and more recently to determine entrustable professional activities (EPAs).² In this issue of the journal, the modified Delphi has been used to derive essential steps for a novel technical skill (bougie-assisted cricothyroidotomy) and for educational resource development (top EM papers to teach junior trainees).^{3, 4} These studies provide excellent examples of how the Delphi can be used to collect expert judgements to arrive at a decision that is arguably considered more credible and defensible than a singular opinion.

STRENGTHS AND WEAKNESSES OF THE DELPHI METHOD

Recent studies have demonstrated the variability in how the Delphi method has been applied. Both studies in this issue of CJEM use a modified Delphi, yet there is no clear consensus on the true definition of the “modified Delphi.” This lack of clear definition has led to considerable confusion and substantial variation in the quality of reporting of Delphi studies. For example, a recent review demonstrated that one-third of medical education Delphi studies failed to report that a literature review on the topic of interest had been conducted, and over half failed to report key aspects such as what background information was provided to participants; the response rates for each round; what formal feedback of group rating was shared between rounds; a statement that anonymity was maintained; and a clear definition of consensus.²

One of the drawbacks of the Delphi method is that the underlying assumption that decision-making in groups leads to wider perspectives and, ultimately, better decisions has not been definitely proven. Critics have suggested that consensus methods generate nothing more than collective ignorance or simply reflect a formalized form of peer pressure.¹ In the event of multiple rounds, attrition in participation can lead to a false consensus, because those with dissenting views drop out.

Despite these pitfalls, consensus methods have many potential benefits when rigorously conducted and reported. Table 1 lists potential strengths and weaknesses of the Delphi.

Table 1. Potential strengths and weaknesses of the Delphi method

CAN WE DO BETTER?

Although modified Delphi methods have their benefits, inconsistent definitions and poor reporting can lead to a lack of scientific credibility. Lack of clarity in the reporting of procedures and methodological choices associated with modified Delphi studies can prevent readers from effectively appraising and interpreting findings. In response to this, many “how to” guidelines have been published.^6–8 For a quick reference, we have adapted a list of best practices in Box 1 of this paper.

Box 1. Best practices for conducting a rigorous Delphi study

1. 1) Define the purpose or objective of the study.

2. 2) Outline each step of the process: If modifications were made, provide a rationale for your choices.

3. 3) Describe the selection and preparation of the scientific evidence for the participants.

4. 4) Describe how items were selected for inclusion in the initial questionnaire: Describe the process in sufficient detail.

5. 5) Describe how the participants were selected and their qualifications. Include a description of the facilitator's credentials.

6. 6) Describe the number of rounds planned and/or criteria for terminating the process.

7. 7) Clearly describe how consensus was defined.

8. 8) Report response rates and results after each round.

9. 9) Describe the type of feedback provided after each round.

10. 10) Describe how anonymity was maintained.

11. 11) Address potential methodological issues (e.g., lack of consensus) or limitations in the discussion (e.g., low response rate).

Adapted from Humphrey-Murto S, Varpio L, Gonsalves C, Wood TJ. Using consensus group methods such as Delphi and Nominal Group in medical education research. Med Teach 2017;39(1):14–9.

In addition to rigorously designing, conducting, and reporting Delphi studies, there is a need for further research to understand best practices for defining consensus, identifying participants, exploring the nature of consensus building, determining outcomes, and how to select a consensus method.²

Medical educators have identified Delphi methods as a useful way to approach educational problems that require synthesizing expert opinion to make decisions. The work of the Dharamsi and Bazak teams is a helpful example of different applications of this method and, although generally provides rigorous reporting, could be improved upon. For example, some readers may have desired more information regarding the expert participants in the Bazak study,⁴ and the inclusion of otolaryngology or trauma surgeon participants in the Dharamsi article might have provided a more holistic understanding of key steps in this procedure.³

EM researchers and educators who wish to apply this method to consensus decision-making are encouraged to use a systematic approach to study design and implementation, with careful attention to and reporting of published recommendations to ensure methodological rigor. Although word count limitations may provide a barrier to detailed descriptions of the methods, authors can reference method papers in the citations, or submit more detailed methods for consideration for publications as supplemental material. Methodological rigor and transparent reporting are essential to assure readers that the consensus results are applicable to their environment, and to translate expert opinion into practice.