Foundation programme assessment tools in psychiatry

In line with Modernising Medical Careers (Department of Health, 2003), the foundation year programme aims to bridge the gap between undergraduate and specialist training. Psychiatry posts have been incorporated into the second year of this programme, with satisfactory progress of doctors being monitored via a range of workplace-based assessment tools. Learning that occurs in the context of the daily workplace is more likely to be relevant and reinforced, leading to better practice (Davis et al, 1995).

This paper provides an overview for consultants, specialist registrars (SpRs) and staff grade/associate specialists, all of whom may be approached to assess foundation year 2 trainees using these competency-based assessments. Examples of psychiatric settings in which the range of workplace-based assessment tools can be used and a critical review of their usefulness are considered.

The assessment tools

There are four tools that assessors may be asked to complete by foundation year 2 psychiatric trainees:

1. • mini-Clinical Evaluation Exercise (mini-CEX)

2. • case-based discussion (CbD)

3. • mini-Peer Assessment Tool (mini-PAT)

4. • direct observation of procedural skills (DOPS)

Although the trainee holds responsibility for deciding the time of the assessments, where they occur, and selecting their assessors (Davies et al, 2005), they have to complete a minimum number of each during foundation year 2. General instructions for using these tools are available at the Modernising Medical Careers website (http://www.mmc.nhs.uk/pages/assessment)

Box 1. Examples of foundation year 2 mini-Clinical Evaluation Exercise (mini-CEX) in psychiatry

1. • Risk assessment in schizophrenia/case of self-harm

2. • Explaining the treatment options of depression/schizophrenia to a user/carer

3. • Assessing capacity to consent

4. • Cognitive assessment in an older person

5. • Assessing for features of alcohol dependence syndrome and its complications

6. • Eliciting extrapyramidal side-effects of antipsychotic medication and explaining treatment options

7. • Assessing for the physical complications of substance misuse

mini-Clinical Evaluation Exercise (mini-CEX)

This is a 15-min ‘snapshot’ assessment of an observed clinical encounter, adapted from the American mini-CEX, which was originally designed as a replacement for the traditional long case format. It is adaptable to a range of clinical scenarios, producing roughly comparable scores over examiners and settings (Norcini et al, 1997), and has been shown to be a feasible and reliable evaluation tool (Durning et al, 2002). Each mini-CEX should be followed by a 5-min instant feedback by the assessor. Although the difficulty of each mini-CEX is dependent on the patient's complexity, assessors tend to take this into account by overcompensating for patient difficulty (Norcini et al, 2003).

Strengths

Being similar to the MRCPsych objective structured clinical examination (OSCE) format, certain areas lend themselves well to being assessed with this tool (Box 1). Scenarios should be incorporated into daily clinical practice settings such as ward reviews or out-patient clinics. There is scope for other professionals (for example, liaison psychiatric nurses) to act as assessors, while the opportunity for instant feedback can help trainees refine their interview techniques and diagnostic skills by encouraging reflective practice.

Physical examination skills with a relevance to psychiatry can also be assessed. This should encourage assessors to keep their skills up to date, a concern highlighted by Garden (2005).

Weaknesses

Only small aspects of the psychiatric consultation process can be assessed, and there may be a greater variance with psychiatric patients (in terms of rapport, willingness to discuss issues or volunteering information). Concerns have been raised that assessors tend to form limited general impressions of trainees based on their assessment of only one or two objectives (for example clinical skills, professionalism). This may be exaggerated in psychiatry, where verbal communication skills play a key role.

Box 2. Example of case-based discussion questions in psychiatry

Assessment of alcohol misuse for detoxification

1. • How did you assess motivation in the patient?

2. • How did you decide on in-patient v. out-patient detoxification?

3. • What are the goals of treatment, and how does it fit in with the overall management of this patient?

4. • How did you assess for withdrawal symptoms?

5. • How did you calculate the dose of chlordiazepoxide required?

6. • What physical complications have you foreseen, and have you guarded against them? Have you requested any physical investigations, and what are you looking for?

7. • What psychiatric comorbidities have you looked for?

Case-based discussion (CbD)

By focusing on a trainee's case note records, this allows the assessment of the trainee's clinical decision-making, reasoning and application of medical knowledge with actual patients (Brown & Doshi, 2006). It is based upon the concept of ‘chart stimulated recall’, used in the USA and Canada. As each CbD lasts only 15 min, only one aspect of the case (rather than the entire case) should be explored (see Box 2).

Strengths

‘Holistic’ aspects of the treatment of common mental illnesses can be discussed, in relation to what the trainee actually did. It can also be useful to explore the trainee's ‘hierarchy’ of drug management (for example using antidepressants or antipsychotics), and whether it is consistent with the current evidence base. Issues of capacity and consent may be better looked at within a psychiatric context, as can differential diagnoses and underlying aetiological factors.

Case-based discussion can enhance the skills of doctors who may enter other specialties (for example general practitioners, gastroenterologists and endocrinologists who come across comorbid alcohol misuse or complications of obesity), and can encourage good record-keeping.

Weaknesses

The duration is not long enough to explore complex patient's problems, and there can be a danger of the exercise turning into a mini-long-case viva, particularly with assessors who are more familiar with that format. Psychiatric notes are more comprehensive than those of other specialties, so it is often necessary to adapt the questioning for certain cases.

mini-Peer Assessment Tool (mini-PAT)

This is a form of multi-source feedback, and has been adapted from the Sheffield Peer Review Assessment Tool (Archer et al, 2005). It is based on the concept of 360° assessment, in which a trainee seeks feedback about their performance at work from a variety of colleagues, highlighting areas of strength and those in need of improvement (King, 2002). It is a useful way of assessing generic skills (such as communication, team working, teaching and reliability), which indirectly measure performance (Hays et al, 2002). In UK pilots, it has been found to be practical and acceptable to senior house officers in hospital settings (Whitehouse et al, 2002). A variation of this tool (the Team Assessment of Behaviour) is used in some regions.

Strengths

The principles of 360° appraisal are supportive to interprofessional team development (McLellan et al, 2005). It can help foundation year 2 trainees to develop a holistic approach to patient care by playing a part in multiprofessional ward reviews and care programme approach (CPA) meetings.

By assessing aspects of the doctor-patient relationship, this tool assesses the qualities which overlap with attributes of a good psychiatrist (Bhugra & Holsgrove, 2005). Ward reviews or CPA meetings can allow trainees to display their communication skills with patients and carers (for example explaining diagnoses or treatments) in front of other health professionals, some of whom could be selected to complete the mini-PAT forms. Out-patient psychiatric clinic letters also allow a trainee's written communication skills to be assessed, as copies are usually sent to other members of the multidisciplinary team. The use of several assessors in the mini-PAT process leaves the tool less open to bias.

Weaknesses

In the context of the shift system of psychiatric on-call work, and the short duration of each post, how many peer colleagues are in a position to accurately comment on a trainee's performance? Evans et al (2004) have commented on this concern in other specialties. It can also encourage unduly positive feedback in the ‘space for comments’ section, owing to concerns that trainees may recognise anonymous comments highlighting poor performance. The most valid source of ratings for ‘humanistic’ dimensions are patients (Church, 1997), particularly in psychiatry, but they are surprisingly excluded from the assessment process.

Direct observation of procedural skills (DOPS)

This assesses trainees’ practical skills, in a range of pre-determined tasks with a patient. Each DOPS should last no longer than 15 min, followed by 5 min of feedback.

Tasks such as venepuncture (for clozapine blood monitoring or for plasma lithium levels), performing an electrocardiogram (for example prior to considering antipsychotic treatment) or giving electroconvulsive therapy can be readily assessed, placing emphasis on the trainee's communication skills (for example obtaining valid consent and explaining the need for the test) in performing the task. Aside from these examples, this tool currently has limited applicability in foundation year 2 psychiatry posts.

Discussion

There is no nationally agreed undergraduate curriculum in psychiatry to act as a baseline template for foundation year 2 competency assessments, with medical schools having considerable autonomy in how they teach and examine psychiatry. The ‘ten essential shared capabilities’ (Hope, 2004) are an attempt to reach a consensus on what competencies should be core for the entire mental health workforce. This can act as a parallel framework for applying workplace-based assessment tools for foundation year 2 psychiatric trainees.

It is essential that trainees get a fair assessment of their competence, by pitching the tools at an appropriate level, in the context of more generic career development; this requires adequate training of all potential assessors. It would be reasonable to expect a foundation year 2 psychiatric trainee to perform a safe and effective suicide risk assessment, or be able to describe their routine approach to the assessment of pyrexia or confusional state within a psychiatric context. However, it may not be appropriate to expect a detailed knowledge or experience of cognitive-behavioural therapy.

Although foundation year 2 trainees may have more medically oriented approaches to the presentation of common mental health problems (for example assessment and management of substance misuse), the assessment tools can provide an opportunity to reinforce the importance of personal, family, social and cultural factors.

Feedback from these tools should encourage trainees to create self-directed learning plans. However, there are concerns as to whether this actually happens in reality (Norman et al, 2004). Criticisms also exist regarding the vagueness of the scoring systems and the absence of independent assessors, as most assessors are known to the trainee (Rose, 2006).

Exposure to foundation year 2 psychiatry posts offers an opportunity to boost recruitment into our specialty, but evidence suggests that undergraduate experience may be a more positive determining factor (Goldacre et al, 2005). Therefore, these assessment tools should be used as an opportunity to develop more advanced psychiatric competencies in trainees who may enter other specialties.

Workplace-based assessment tools for specialist training

Although the foundation year 2 assessment tools will also be used to assess doctors in specialist training, some of the formats have been adapted by the Royal College of Psychiatrists (for example the mini-CEX has been revised to become the mini-Assessed Clinical Encounter or mini-ACE). Additional tools are also currently being piloted, including the Assessment of Clinical Expertise (ACE), case and journal club presentations, and a patient satisfaction questionnaire (details are available from the Royal College of Psychiatrists’ website: http://www.rcpsych.ac.uk/training/specialtytrainingassess.aspx). Assessors are likely to have to use different sets of workplace-based assessment tools for trainees in their foundation year 2 or specialist training years 1-5.

Conclusions

Foundation year 2 assessment tools provide structured, standardised methods of monitoring doctors’ progress, against a broad range of core competencies. Brown & Bhugra (2005) have highlighted the need for such tools to assess clinical and non-clinical competencies. By grounding these assessments within a relevant workplace context, it is hoped that trainees will retain the skills they learn. As the character Arthur said in Patrick White's The Solid Mandala: ‘I forget what I was taught. I only remember what I’ve learnt.’

Declaration of interest

None.