Introduction
The transition from core surgical training to sub-specialty training at specialty trainee level 3 (first year registrar in the UK) is a significant step in an ENT trainee's progression. Despite the number of applicants declining in recent years, the application and interview process remains competitive and is a challenging hurdle to overcome.Reference Green, Steven and Haddow1
There are criteria set out by the Joint Committee on Surgical Training in the Core Surgical Training curriculum regarding progression to ENT sub-specialty training. They set out a number of ENT-specific clinical skills and procedures that core surgical trainees should be able to manage independently without direct consultant supervision. Core surgical trainees undertake a two-year programme to gain a broad exposure to surgical specialties prior to progressing to specialty trainee level 3. This list of recommended procedures and the level of experience required is set by an expert panel as there is no evidence relating to this area of medical education and training. Similarly, at interview for specialty trainee level 3 ENT positions, applicants gain marks for providing evidence of performing a set list of ENT procedures, their duration of experience within ENT and experience of working in allied specialties, which are demonstrated in Table 1.2
Table 1. Points awarded at ENT specialty trainee level 3 interview for surgical procedures and clinical experience
Maximum portfolio score = 115
Delphi studies are becoming a more widely utilised tool to develop consensus where either research is lacking or where other forms of research methodology are unsuitable. Delphi studies have been used on several occasions in the field of ENT to develop consensus on post-operative follow up, developing an ENT undergraduate curriculum and management of otitis media with effusion in cleft palate patients.Reference Lloyd, Tan, Taube and Doshi3–Reference Powell, El Dean, Carrie, Wilson and Paleri5 Delphi studies, across the specialties, are usually conducted in several rounds and can last for several months largely because of the delays in waiting for responses, attrition of the expert panel and time for analysis.
We developed a novel ‘live’ Delphi study design with the aim of identifying consensus in terms of which skills and competencies any new ENT specialty trainee at level 3 should possess. We hoped to identify a set list of criteria that core surgical trainees could aim for to be deemed competent to progress to a specialty trainee level 3 post. We focused on four key areas: (1) surgical procedures, (2) clinical presentations, (3) experience working in the field of ENT and (4) experience working in allied specialties.
Materials and methods
Conventionally, a Delphi study is conducted remotely via e-mail or post. We conducted our first two rounds at a live event on the same day. This resulted in all three rounds and analysis being undertaken within two weeks. We believe this alteration in methodology increases the efficiency of this technique when compared with other previous Delphi exercises.
This three-round Delphi exercise was conducted as part of a regional ENT conference where there were various stakeholders present from consultants to medical students. We used an online voting mobile-based application (VoxVote, Breda, Netherlands) to gain real-time responses from the audience. The subject of the Delphi exercise was circulated prior to the meeting to allow our respondents time to develop ideas for their responses in round 1. The first round of questions was developed prior to the conference by the investigative team and was designed with open questions to gain broad responses. These questions were presented to the attendees for the first time on the day of the live Delphi exercise and no prompts were used. The first and second rounds were undertaken across two 45-minute sessions: one in the morning and one in the afternoon.
Example questions are: ‘1. Which elective surgical procedures should a new ENT specialty trainee level 3 be able to manage without direct consultant supervision?’ and ‘2. Which emergency surgical procedures should a new ENT specialty trainee level 3 be able to manage without direct consultant supervision?’.
In the second round of the Delphi exercise, we used audience responses from round 1 and presented them back to the group. The aim of the second round was to investigate if there was agreement that a new ENT specialty trainee level 3 should (or should not) be able to manage a procedure or clinical scenario independently without direct consultant supervision or if there was consensus on the amount of experience they should (or should not) possess prior to starting their post. These questions were developed between the two interactive sessions on the day with the first round being undertaken in the morning and the second round in the afternoon. The audience was given a 7-point Likert scale ranging from very strongly disagree to very strongly agree.
The final round was emailed out to participants following the meeting. Respondents who had completed the first two rounds were contacted via e-mail and asked to complete the questions online using a web-based survey tool. We presented the responses that had not reached consensus with results from the first round visible to the panel to see if this altered results. We defined a positive consensus as a median score of at least strongly agree and a negative consensus of at least strongly disagree. There were 39 collaborators who completed all three Delphi rounds.
Following assessment of the final round for consensus, we compared if there were differences in responses between specialty trainees in ENT and consultant ENT surgeons. A Mann–Whitney U test was used to compare Likert scores between specialty trainees and consultants. In order to see if the experience of the respondent impacted the results, we removed medical students and foundation doctors in a subsequent analysis.
There are no reporting guidelines relevant to independent Delphi studies within the Equator network.
Results
The number of respondents by round is as follows: round 1, 61; round 2, 52; and round 3, 39. Respondents by grade after all 3 rounds (39 respondents) are shown in Figure 1.
Fig. 1. Occupation or role within the ENT specialty.
Elective surgical procedures
The following elective procedures were identified during the first round: tonsillectomy, grommets, panendoscopy or direct laryngoscopy, adenoidectomy, aural micro-suction, functional endoscopic sinus surgery, raising neck flaps, nasal cautery, neck dissection, excision of skin lesion, and oesophagoscopy.
Table 2 demonstrates results that reached consensus after rounds 2 and 3. The remaining procedures did not reach consensus.
Table 2. Elective surgical procedures that reached consensus
Emergency surgical procedures
The following emergency surgical procedures were identified during round 1: incision and drainage of peritonsillar abscess, incision and drainage of superficial skin abscess, post-tonsillectomy bleed arrest, pinna haematoma drainage, tracheostomy, removal of foreign body from the ear or nose, suturing of laceration, removal of food bolus, manipulation of nasal fracture, incision and drainage of septal haematoma, cricothyroidotomy, arrest of post-thyroidectomy bleed, and none (do not need to be competent at managing any emergency surgical procedure without direct consultant supervision). Table 3 demonstrates the procedures that reached consensus following rounds 2 and 3. The remaining procedures did not reach consensus.
Table 3. Emergency surgical procedures that reached consensus
Elective clinical presentations
The following elective clinical presentations were identified by the respondents in round 1: chronic rhinosinusitis, hoarse voice, recurrent tonsillitis, otitis media with effusion, chronic otitis media, otitis externa, bell's palsy, vertigo, facial pain, dysphagia, hearing loss, obstructive sleep apnoea, tinnitus, laryngopharyngeal reflux, two-week wait (cancer) referrals and nasal obstruction. Table 4 demonstrates presentations that reached consensus during round 2. No further consensus was reached on the remaining presentations in round 3.
Table 4. Elective clinical presentations that reached consensus
Emergency clinical presentations
These emergency clinical presentations were identified during round 1: acute mastoiditis, food bolus, blocked tracheostomy, upper airway obstruction, deep neck space abscess, epistaxis, tonsillitis, otitis externa, acute sinusitis, peritonsillar abscess, periorbital cellulitis, acute vertigo, neck lump and wax impaction.
Table 5 demonstrates the emergency clinical presentations that reached consensus during round 2. No further presentations reached consensus after round 3.
Table 5. Emergency clinical presentations that reached consensus
Time working in ENT prior to specialty trainee level 3
The following time periods were identified during round 1: time spent should not be a criterion, greater than 4 years, greater than 3 years, greater than 2 years and greater than 6 months. Table 6 demonstrates the time periods that reached consensus during round 2; there were no new time periods that reached consensus after round 3.
Table 6. Time spent in ENT prior to starting specialty trainee level 3
Allied specialty experience prior to specialty trainee level 3
The following specialties were identified during round 1: maxillofacial surgery, plastic surgery, general surgery, general practice, intensive care, cardiothoracics, accident and emergency, neurosurgery, anaesthetics, and no allied specialty required. None of these responses reached positive or negative consensus after round 2 or 3.
Comparison in responses between trainees and consultants
In the majority of categories, there was no difference in the responses between consultants and specialty trainees. However, there was a significant difference in responses for oesophagoscopy (p = 0.047) and removal of food bolus (p = 0.039), where specialty trainees were more in agreement that this should be a procedure a new specialty trainee level 3 should be able to perform independently when compared with their consultant colleagues. Trainees were also more in favour that experience in maxillofacial (p = 0.005) and anaesthetics (p = 0.036) should be gained prior to specialty trainee level 3 when compared with consultants.
Removing medical students and foundation doctors
We felt that a greater cross-section of ENT stakeholders, from a variety of backgrounds and experience, would provide both exposure to a new technique and sufficient responses. However, to demonstrate that seniority would make little impact on the results of this particular question, we removed medical students and foundation doctors from the analysis. After removal of these sub-groups, foreign body removal and prior general practice experience reached negative consensus (changing from disagree to strongly disagree). All the other results were unchanged.
Discussion
We developed a novel technique, a live Delphi study, to seek consensus on the skills and competencies that a new ENT specialty trainee level 3 should possess. This format allowed the Delphi study to be conducted in a reduced time frame when compared with previous studies, a frequently recognised deficiency of this technique.Reference Chae, Kim, Yoo and Lee6 We had a high engagement, compared with previous Delphi studies.
The Delphi technique has been utilised in medical education previously with good effect, but this is the first study to apply to post-graduate surgical training specifically.Reference Rohan, Ahern and Walsh7,Reference Collins, Levy, Stefanidis, Gallagher, Coleman and Cecil8 The progression to specialty trainee level 3 from core surgical training is a critical one because responsibilities and expectations increase. Having clear parameters and targets for clinicians to achieve prior to starting specialty trainee level 3 provides a benchmark to work towards for aspiring future ENT consultants. Previously, these curriculum aims have been set by a specialty-specific educational committee without input from a broader group within the specialty.
This study allowed trainees and consultants to develop consensus as a group without the impact of seniority biases. Consensus, for what presentations or procedures a new ENT specialty trainee level 3 should be able to manage without direct consultant supervision, was mostly achieved amongst emergency presentations and procedures. The reason for this may be that at this level the clinicians’ experience is largely emergency based, and they may not have had significant operating theatre exposure. This may reflect that training time for core trainees is weighted towards emergency presentations and management rather than elective theatre and clinic patients.
There were no differences in responses between the specialty trainees and consultant groups, who formed the majority of our Delphi respondents, apart from for oesophagoscopy and foreign body removal. This suggests that there is differing opinion within these groups for the procedures and presentations surveyed, but the average (median) opinion was similar for both groups.
• A novel live Delphi technique can be adopted to reach consensus over a shorter time frame compared with traditional techniques
• This novel Delphi technique could be adopted in future to shape medical and surgical postgraduate curricula and opinion in other areas of ENT
• This study demonstrated consensus regarding skills and competencies that a new ENT specialty trainee level 3 should possess
• There was largely no significant difference in the opinions of consultants and trainees in this Delphi exercise
• Consensus was reached regarding the minimum experience required in ENT prior to starting ENT specialty trainee level 3 at six months
The Delphi method does have limitations. Delphi studies do not provide re-test reliability, and as such we feel that this method could be applied more widely or in a different region of the UK to assess if similar responses were found. In our study, the first two rounds were conducted in a conference room with delegates responding on mobile devices, and there is a potential therefore that discussion took place between our panel and they were not truly independent. There was a reduction in the number of respondents with each round. This was particularly noticeable between rounds 2 and 3 when the ‘live’ component was removed. There is a trade-off between engagement with such research methods and overwhelming a teaching or academic day, which should be balanced. Undertaking all three rounds on one day will reduce drop-out but is likely to cause fatigue and a negative association with this method in some participants.
Conclusion
We have demonstrated a new technique for developing consensus that could help shape post-graduate surgical training and recruitment into specialty training in a more directed and evidence-based manner. This study specifically investigated presentations and procedures that a new ENT specialty trainee level 3 should be able to manage independently, but similar Delphi approaches could be used to look at a spectrum of supervision for trainees across different experience levels and help shape the surgical curriculum.
Data availability statement
The data that support the findings of this study are available from the corresponding author upon reasonable request.
Competing interests
None declared
Authorship
The following authors contributed in all three rounds of the Delphi study and reviewed the draft manuscript: Thomas Hampton, Showkat Mirza, Haitham Darwich, Carissa Jacobs, Eugene Omakobia, Adam McClean, David Selwyn, Katherine E L Hamlett, Jacob Duffin, Estelle How Hong, Paul Nix, David Crampsey, Amy Pearson, Michael Chu, Anastasha Herman, Djamila Rojoa, Gabrielle Thompson, Bethan Jones, Amy Pearson, Sarah Healy, Lindsey Knight, Rachel Edmiston, Pooja Bijoor, Rajeev Advani, Daniel Lin, Andrew S Lau, Hannah Emerson, Rosie Wright, Rohma Abrar, Brian Goh, Saree Al-awsi, Thida Than, Nasser Ayoubi, Randa Asswad, George McNally, Hannah Emmerson, Joanna Godbehere, Elizabeth L Kershaw, Thida Than, Moustafa M Dawoud and Antonia Tse.
