MTAS fiasco: lessons for psychiatry

Abstract

AIMS AND METHODS

The recent Medical Training Application System (MTAS) has been fraught with problems. A survey of MTAS applicants from two London training schemes and a request for emails from trainees and trainers documenting problems with the system were undertaken with the aim of canvassing the views of candidates and shortlisters/interviewers about MTAS, and providing evidence of the system's failings.

RESULTS

A total of 101 candidates responded to the survey, and the first 92 emails sent to the Royal College of Psychiatrists were analysed. The majority of respondents (73%) were dissatisfied with MTAS and 63% thought the system unfair. UK and European Economic Area (EEA) applicants (71%) were more successful in being shortlisted than non-EEA candidates (48%). The majority of applicants (70%) planned to reapply in subsequent rounds if they failed to secure a training post.

CLINICAL IMPLICATIONS

Both surveys showed clearly that the system was not working well. The trainees and trainers both felt that the system was flawed and that the reasons for the failure included a centralised system without any piloting, and the question design. In addition a lack of resources added to the stress and burden on both trainees and trainers.

The Medical Training Application Service (MTAS) was intended to be the single national application route for all doctors seeking entry into run-through specialist training. It replaced the host of existing separate application processes into senior house officer (SHO) and specialist registrar (SpR) posts. The MTAS rules forced applicants to trade off their commitment to one specialty against the potential appointment to one of four deaneries. There have been serious concerns about whether the system identifies the doctors most suitable for particular specialist training and the fairness of the process.

The MTAS process went live in January 2007 and within the first few hours several problems emerged. These included problems accessing the system, repeated crashing of the system, and an inability to add references by the referees. When shortlisting was announced, the rejection of a number of good candidates across specialties and deaneries indicated that the process had not been as smooth as predicted. There was discussion to determine whether the process was seriously flawed.

When the MTAS Review Group was set up, the medical Royal Colleges were asked to provide definite evidence of inappropriateness of the process. Following an appeal on the Royal College of Psychiatrists’ website for further details, emails were received from trainees and trainers. At the same time, trainees from two south London training schemes who had applied through MTAS were surveyed by means of a questionnaire.

We present the findings of these two separate undertakings, which shared the common objectives of canvassing views of trainees and trainers about the application and shortlisting processes, and highlighting problems experienced.

Method

Trainees from two psychiatry rotations in south London were identified using group email lists from tutors and organisers of local MRCPsych courses. The survey was restricted to those candidates who applied for the first round of shortlisting through MTAS, before the Review Group decided that all trainees should get at least one interview by a different process.

Trainees were sent a questionnaire, which asked them:

1. • to provide basic demographic information

2. • whether they had been successful in shortlisting

3. • to rate on a Likert scale their degree of satisfaction with and sense of fairness of the process

4. • to highlight problem areas encountered (in a free text section)

5. • to indicate what alternative plans they had if they failed to get a specialist training post.

In addition, consultants from one NHS trust (which has trainees/trainers from both training schemes) who had been shortlisters or interviewers were surveyed, and asked to describe any problems they encountered.

Following an email request for trainees and trainers to send details of the problems to the Royal College of Psychiatrist's website, in view of the urgency to respond to the Review Group's need for information, the first 92 emails received on the first day were categorised by K.F. in discussion with D.B. These included comments from both trainees and trainers.

Results

Training scheme survey

In total, 101 trainees responded. Their mean age was 30 years (range 24-49) and they had worked in psychiatry training posts on average for 23 months (range 0-60). Table 1 gives a breakdown of the respondents’ demographic details and their shortlisting success rates. Specialist training 2 and 3 were the most highly applied for levels. Forty-eight per cent of non-European Economic Area (EEA) applicants were shortlisted compared with 71% of candidates from the UK and the EEA. Of those who were not shortlisted, 70% planned to reapply, 16% wanted to do staff grade or locum work and 7% considered changing specialty. Only 7% planned to emigrate, with Australia and the USA being the most popular destinations. Seventy-three per cent of applicants were dissatisfied with the MTAS process, 19% were satisfied and 8% had a neutral view. Sixty-three per cent thought that the process was unfair, 18% thought it was fair and 19% held a neutral view.

Table 1. Candidates’ success rates in shortlisting (n=101)

	Shortlisted
	Yes	No
	%
Total (n=101)	60	40
Level applied for:
ST1 (n=4)	25	75
ST2 (n=49)	61	39
ST3 (n=37)	54	46
ST4 (n=11)	82	18
Nationality
UK (n=36)	61	39
EEA (n=15)	93	7
Non-EEA (n=50)	48	52
Gender
Male (n=54)	57	43
Female (n=47)	62	38

The comments from the free text section were analysed qualitatively. Box 1 highlights the common problem areas identified by the candidates and the shortlisters/interviewers.

Emails sent to the College

In total, 200 emails were sent to the Royal College of Psychiatrists. The first 92 emails were analysed as it became apparent that the emerging themes were consistent and repeating. The problem areas identified were similar to those found in the survey. However, this time they were grouped into broad categories and a quantitative analysis was conducted. The areas of difficulties identified by the respondents, along with the number of respondents, were as follows: initial problems (n=17); difficulties with the application form (n=25); technical problems (n=19) (for example poor access, system crashes); selection (n=77) (i.e. problems with shortlisting); marking (n=30); interviews (n=24); and feedback (n=13).

Lack of information about previous employment featured strongly as a difficulty encountered with the application form. However, by far the greatest number of reported problems was with the selection process: (a) especially, with degrees and publications not being taken into consideration (32 respondents); (b) references not contributing to shortlisting; (c) inability to write proper references; (d) random selection of trainees; and (e) emphasis on creative writing skills rather than skills of achievement.

Box 1. Problem areas identified by the candidates (n=101) and shortlisters/interviewers (n=5) survey

Candidates

1. • Disorganisation of the process

2. • Lack of transparency of the application's marking system

3. • Technical problems

4. • Applicants’ personal circumstances were not taken into account (moving to different geographical areas etc.)

5. • An apparent discrepancy in shortlisting (good candidates not offered interviews etc.)

6. • System seems to favour EEA graduates

7. • Time frame was too rushed

8. • Ever-moving goalposts

9. • Applying for jobs without knowing the terms of the contracts

Shortlisters/interviewers

1. • Lack of time to mark applications (maximum of 4 days)

2. • The volume of applications to mark (up to 700 per shortlister)

3. • Shortlisters were unable to judge a candidate as a whole as they only marked one section on the application

4. • Candidates often put good answers but in the wrong box

5. • Interview scoring scheme was not transparent to interviewers

EEA, European Economic Area.

Discussion

These findings shed light on the potential and real problems faced by the trainees and trainers who were shortlisting candidates. Neither the training scheme sample nor the electronic survey participants are representative of MTAS applicants throughout the UK and across specialities. They were older and more likely to be from a non-EEA country than those in a larger ongoing survey reported elsewhere (Lydall et al, 2007), and had greater shortlisting success (60 v. 37%) than the national average for psychiatry (Shannon, 2007). However, their views probably are representative of most trainees, aside from the issue of seeking further training overseas if not successful through MTAS: 7% of this sample v. 55% reported in a recent survey (British Medical Association, 2007).

There was overall dissatisfaction with the MTAS process; even those who were shortlisted thought the process unfair. Furthermore, the system seems to favour UK and EEA applicants.

The old system may have needed overhaul but it provided much more opportunity and flexibility; candidates could apply to various training schemes at different times of the year, giving a greater chance of working in a chosen geographical area and on a specific rotation, and they could change their specialty mid-training more readily if they felt they had chosen wrongly. There is no doubt that trainees may have been selected in some specialties for training on the basis of patronage but the system that was set to replace it has had serious failures. There is clear evidence that not listening to the profession's concerns, not piloting the selection process in one deanery and a lack of appropriate resources have all contributed to a sense of disenchantment in a large number of trainees. The recent online survey by Lydall et al (2007) has indicated that nearly three-quarters of trainees are feeling low energy levels and half are feeling hopeless about their future. In addition one-third are drinking more and 305 said that they are making more mistakes at work. A large majority (96%) attributed their increased stress levels to MTAS and/or Modernising Medical Careers. In these three samples there appears to be a common theme of loss of control which has led to a sense of abandonment. The lessons from the fiasco are clear - the profession must speak with one voice to highlight the message that there is a problem in initiating new programmes without piloting, information must be made available early and regularly, and training and resources must be made available if any changes are to be introduced.

Conclusions

Although most respondents to this survey acknowledged failures in the old system, ironically a lack of transparency and flawed selection procedures were two of their major criticisms of MTAS. The reluctance to abandon a clearly malfunctioning process and instead trying to patch it up left many doctors pondering potential hidden political motives behind the recent radical changes to medical training.

Declaration of interest

P.W. gave a talk on MTAS funded by Janssen-Cilag; M.M. was an MTAS applicant; K.F. is employed by the Royal College of Psychiatrists; D.B. is Dean of the Royal College of Psychiatrists.