Systematic review of workplace-based assessments in psychiatry: surgical dissection and recommendations for improvement

Summary

Recent surveys have highlighted widespread criticisms of the use of workplace-based assessments (WPBAs) in psychiatric training. We describe our systematic review of psychiatric WPBAs, including a ‘surgical dissection’ of their format and process. From our review, we identified seven overarching WPBA themes, and have drawn on these to make further recommendations to strengthen the wider acceptability of WPBAs in psychiatric settings. We hope this will encourage further debate on ways of improving these tools, rather than them becoming side-lined as ‘top-down’ tick-box exercises.

The past 5 years have seen major changes to UK psychiatric training, with a move towards competency-based curricula. Many of these changes have occurred within the context of wider national reforms to postgraduate medical education, in response to Unfinished Business, ¹ Tomorrow's Doctors, ² Modernising Medical Careers, ³ The Gold Guide to Postgraduate Specialist Training, ⁴ The Core Curriculum Guide, ⁵ and the establishment of the Postgraduate Medical Education and Training Board (PMETB). By aiming to explore all of Miller's pyramid assessment of competencies, ⁶ including what the trainee actually does in daily clinical practice, workplace-based assessments (WPBAs) have a central role in monitoring the progress of trainees. They also offer a number of other advantages compared with traditional methods of trainee assessment, including: the opportunity to observe real-life ‘long case’ assessments of patients; ⁷ time for immediate trainee debriefing, highlighting their strengths and weaknesses; opportunities for sampling a wide range of clinical scenarios; assessments of a trainee's teaching skills to peers in journal clubs and case presentations; and encouragement of formative learning. ⁸

However, recent surveys have highlighted widespread problems and negative attitudes among trainees and trainers towards the current format of WPBAs. ^9-11 Between a half and two-thirds of trainees feel that WPBAs do not accurately reflect their progress, have questionable validity and reliability, and have no real beneficial effects on supervision, training, clinical practice and confidence. ¹⁰ Although the College's core training WPBA guide ¹² provides greater guidance in mapping scenarios against levels of performance, wider concerns about these tools remain.

Within the context of the current evidence base, drivers to change within postgraduate medical education, and likely future WPBA changes, how can we improve WPBAs to make them more meaningful within psychiatric settings? In order to address this question, we set about to: systematically review the existing evidence relating to WPBAs; record the salient WPBA themes identified from articles meeting the inclusion criteria for our review, including any points made by the authors of those papers; and draw on our systematic review findings to make recommendations for change, to improve WPBAs in psychiatry.

Method

We systematically reviewed the literature with an objective to search for and review the evidence base of WPBAs, primarily focusing on their use within psychiatric settings. We initially searched the four electronic bibliographic databases MEDLINE, EMBASE, CINAHL and PsycINFO (all from 1981 to 2010). Our search terms (English language only) in the title and abstract were: “workplace-based assessment”; “360 degree feedback” OR “multisource feedback” OR “mini peer assessment tool” OR “mini PAT” OR “mini assessed clinical encounter” OR “mini ACE” OR “case based discussion” OR “assessment of clinical expertise”; and “psychiatry”. Results from these searches were combined with AND and duplicate searches removed. Using this search strategy we found 332 articles, which were independently screened by two authors (K.S. and S.P.) on the basis of title, abstract and (where necessary) full texts, and those articles clearly not relevant excluded (e.g. those where WPBAs only occurs as a citation, but without further expansion). We evaluated the remaining articles for quality using our modified version of the Best Evidence Medical Education (BEME) guide ¹³ (Box 1), which was modifed to take into account the relatively few published quantitative comparative studies evaluating WPBAs in psychiatry. Only those articles which met five or more of the quality indicators, from our modified guide, were included, with any discrepancies in the modified BEME score reviewed by one of the consultant authors (A.N.). Using this method for assessing quality, 14 articles satisfied our inclusion criteria: 12 full text articles, including 1 invited commentary ¹⁴ in response to 2 useful WPBA surveys; ^10,11 and 2 published correspondences, 1 with valid statistical information from a regional WPBA survey, ⁹ and 1 by the Dean of the Royal College of Psychiatrists ¹⁵ in response to a regional WPBA survey. ¹⁰

In order to widen our search for further psychiatry-specific papers we also conducted focused database searches of the Royal College of Psychiatrist journals The Psychiatrist (previously Psychiatric Bulletin), Advances in Psychiatric Treatment and the British Journal of Psychiatry (all from January 2000 to June 2010). We used the same search strategy (but excluding ‘psychiatry’ as a search term) as per our electronic database review, with the search being applied in all fields and for all types of published articles within the paper journals. We found 243 articles, which were independently screened as above (including removal of duplicate articles identified from our previous search), and those articles clearly not relevant were excluded. After applying our inclusion criteria (modified BEME score ⩾5), we found three additional articles: two full text articles and one published correspondence ¹⁶ highlighting important issues in response to a regional WPBA survey. ¹⁰

An additional 6 articles were found to meet our inclusion criteria (modified BEME score ⩾5), by searching the references lists of the 17 articles so far included in our systematic review. To be fully inclusive of all national guidelines relating to WPBAs and postgraduate training, we also conducted an electronic website search (all from 2000 to 2010) of: the Royal College of Psychiatrists; PMETB; Department of Health; Modernising Medical Careers; Medical Education England; and the General Medical Council (GMC). We found a further ten articles that met our inclusion criteria using this final search method.

Box 1 Modified version of the Best Evidence Medical Education guide

1. • Psychiatry-specific articles

2. • Peer-reviewed (or equivalent) publications

3. • Published national guidelines

4. • Study subject group appropriate for the study being carried out

5. • Studies with data collection methods that were reliable and valid for the research question and context

6. • Data-set complete, with an acceptable questionnaire response rate

7. • Appropriate methods of analysis (statistical or other)

8. • Article recommendations reproducible and applicable for workplace-based assessments in psychiatry

9. • Article data or discussions justify a clear set of conclusions

10. • UK published articles

Adapted from Buckley et al. ¹³

Overall, 33 articles met our inclusion criteria for this review (Box 2). Two pairs of authors independently extracted information relating to WPBAs from these articles, using a predesigned and piloted template. This included recording: any salient WPBA themes highlighted in each article; any findings (from formal studies, surveys or other methods) and recommendations from the articles’ authors; and any areas of concern for using WPBAs in psychiatric settings. The accuracy of information extraction was double-checked by the corresponding pair of authors, with inconsistencies resolved by review by one of the consultant authors.

Results

Seven overarching themes relating to WPBAs were identified from the articles included in our systematic review, with some articles covering more than one theme.

Assessment of psychiatric competencies

Workplace-based assessments were not specifically designed to assess postgraduate psychiatric competencies in the first place. ^7,18 Therefore, concerns expressed by trainees and trainers ^9-11 should hardly be surprising. Psychiatry remains unique in placing greater emphasis on working in partnership, promoting a longer-term ‘recovery model’, negotiating meaningful goals that are achievable by the patient, promoting safety and positive risk-taking and challenging inequalities. Demonstration of a trainee's ability to devise diagnostic formulations, alongside awareness of differentials and classification systems, remain important core curricula skills. ¹⁸ Likert-type assessment scales fail to capture the complexities of these competencies, consistent with evidence that a checklist approach is not appropriate for assessing higher-level trainees. ¹⁹ This can lead to complex psychiatric skills, that take time to master, being reduced to isolated box-ticking competencies. ¹⁰ Assessors

Box 2 Systematic review of psychiatric workplace-based assessments: articles included in our review

1. • Full text studies/surveys: Archer et al (2010); ¹⁷ Babu et al (2009); ¹¹ Bhattacharya et al (2010); ¹⁸ Hodges et al (1999); ¹⁹ Julyan (2009); ²⁰ Menon et al (2009); ¹⁰ Searle (2008); ²¹ Wilkinson et al (2008) ²²

2. • Full text commentaries: Oyebode (2009) ¹⁴

3. • Full text review articles: Abdulla (2008); ²³ Brown & Cooke (2009); ²⁴ Crossley et al (2002); ²⁵ Faruqui & Ikkos (2007); ²⁶ Fitch et al (2008); ⁷ Holsgrove et al (2009); ²⁷ McIlwrick et al (2006); ²⁸ Miller (1990); ⁶ Naeem et al (2007); ²⁹ Ten Cate (2006); ³⁰ Thompson (2009) ³¹

4. • Published correspondence: Howard & Brittlebank (2010); ¹⁵ Pathan & Salter (2008); ⁹ Sikdar (2010) ¹⁶

5. • National guidelines/papers: Department of Health (2005); ³² Department of Health et al (2007); ⁴ Modernising Medical Careers (2007); ³ Oakley (2009); ³³ Royal College of Psychiatrists (2009); ⁵ Royal College of Psychiatrists (2010); ¹² Royal College of Psychiatrists (2010); ³⁴ Postgraduate Medical Education and Training Board (2007); ³⁵ Postgraduate Medical Education and Training Board (2009); ³⁶ Temple (2010) ³⁷

are less likely to correctly administer a tool that significantly limits their freedom to employ their professional judgement. ²⁷

Although the current WPBA forms do allow for some ‘free text’ comments by assessors, they remain secondary to lists of Likert-scale scores, limiting the opportunity for trainees to gain specific advice on how to improve their performance. Workplace-based assessments must provide relevant feedback to trainees, ⁴ which is a powerful instrument in furthering their personal and professional development. ²⁴ However, even the best-intended feedback may be unhelpful if it is not descriptive or specific enough. ²⁸

Assessing performance against an agreed exit standard for all trainees, including those who have just started in their post, remains problematic. Some assessors judge where trainees should be at their current stage of training, whereas others score against the ‘end-point’ of their training (which is what these tools were originally intended to do). ³⁶ Although WPBA assessor training has the potential to improve standardisation, wider concerns exist if assessors and trainees regard this part of the system to be inherently unfair. ^11,14 How many other professions assess their workforce according to what they should be able to do at the end of the year? Medical trainees are generally competitive, and view low scores as ‘failure’. ³⁶

Core v. higher specialist training assessments

The European Working Time Directive has resulted in concerns about the impact of restricting the range of clinical experiences trainees are exposed to during their training. ³³ Some aspects of the medical curriculum require more rigorous assessment than others, such that a trainee needs to be assessed carrying it out a number of times. ³⁶ Within psychiatry, these include risk assessments, ability to formulate complex diagnostic situations and explaining specific treatments. Concerns continue to be raised by the low success rate of trainees at the MRCPsych Clinical Assessment of Skills and Competencies (CASC) examinations, ¹⁶ where all subspecialty scenarios are assessed. There is a lack of established standards for various training grades, ¹⁰ especially for those in higher (ST4-6) specialist training.

Formative v. summative assessments

Workplace-based assessments should act as supportive learning tools for trainees, primarily designed to assess their readiness for progression to summative tests. ³⁶ However, this is where the paradox lies for both trainers and trainees. If WPBAs are primarily tools for giving formative feedback, ⁵ why are they given significant weight during the Annual Review of Competence Progression (ARCP)? Non-fulfilment of WPBA requirements can result in trainees being failed at their ARCP. ¹⁰

Alternatively, given that WPBAs assess up to the highest level of Miller's pyramid of competencies, ⁶ how can ARCP panels accurately gauge a trainee's progress in the clinical workplace without WPBAs contributing evidence to inform summative decisions? ⁵ By making a clear divide between ‘formative’ and ‘summative’ assessment roles, we can end up under- or overestimating the role of WPBAs in monitoring the progress of trainees.

The role of non-medical assessors

Compared with other medical specialties, psychiatry places greater importance on multidisciplinary team working. Although assessments by non-medical members of teams remain vital to providing a broad psychosocial perspective, devolving trainee assessment to generic mental health workers carries significant concerns. ¹⁰ Trainees perceive non-medical assessors to have significantly less knowledge of WPBAs, and be less likely to assess them accurately. ¹⁰ Limited evidence exists for using assessors who are not senior medical clinicians to assess experienced postgraduate trainees. ²¹ Assessors need to be able to demonstrate their competence in using WPBAs in psychiatry, including their ability to give feedback. ^3,4

360-degree feedback assessments

The concept of 360-degree feedback originated within industrial organisations who wished to improve the leadership qualities of their workforce via self-awareness and positive behaviour change. The mini-Peer Assessment Tool (mini-PAT), a shortened version of the Sheffield Peer Review Assessment Tool (SPRAT), was introduced by PMETB as one of its multisource feedback tools. Whereas some feasibility data are available for the SPRAT, the mini-PAT currently lacks robust data regarding its validity and reliability. ²³ Concerns exist that the validity of multisource feedback tools can be limited by systematic bias from the wide range of assessors, resulting in problems with unregulated self-selection of assessors by trainees. ¹⁷ For core trainees, in the context of their predominant shift system of on-call work and shorter posts, how many team colleagues are in a position to accurately comment on their performance? ²⁹ It can be difficult for team members to confidentially mention their concerns without being easily recognised.

Quality assurance

Standardising WPBA judgements by trainers remains problematic. ^9,11 Even the Assessed Clinical Encounter (ACE), one of the tools that trainees find to be most useful, ¹¹ has weak interrater reliability. ²¹ Although assessor training can reduce the potential ‘hawk’ or ‘dove’ effects of individual assessors, there are wider problems. These include the fears of trainers damaging their trainer-trainee relationship by giving low scores, ²⁴ the ‘acquaintance effect’ where trainees who are well-known to the rater are scored higher, ²¹ and that more senior staff tend to give lower but more accurate ratings compared with less senior staff. ²² We have all come across the core trainee with a portfolio of perfect WPBA scores, baffled by their failure to pass the MRCPsych CASC exam. ^15,16

Consultant supervision

The ‘1 hour per week’ supervision has been the cornerstone of the delivery of direct teaching of psychiatric trainees by their supervisors. For supervisors, the time required to deliver this is estimated to be 0.25 programmed activities per week per trainee, which should be incorporated into their job plans. ³⁴ Supervision and WPBAs can have complementary functions, but the increased emphasis on assessment of competencies may displace other educational needs that would have previously been covered within the ‘protected hour’. ²⁰ Quality of supervision remains the most important factor in determining overall trainee satisfaction. ²⁶

Discussion

In this systematic review, we identified and examined the available current evidence relating to WPBAs and their use within psychiatry. Seven overarching WPBA themes were identified from the reviewed literature, which we have grouped into three areas (Box 3) that pose the greatest challenge in terms of wider acceptability of these tools within psychiatric settings. Our discussion draws on the review findings covering these three grouped areas, making further recommendations for change to help improve the use of WPBAs within psychiatric training.

Box 3 Psychiatric workplace-based assessments (WPBAs): grouping the seven overarching themes identified from our systematic review into areas for improvement

Overarching themes from our systematic review	Areas for improvement
(1) Assessment of psychiatric competencies	Structure and format of WPBAs
(2) Core v. higher specialist training	WPBAs and psychiatric training
(3) Formative v. summative assessments
(4) The role of the non-medical assessor
(5) 360-degree feedback assessments
(6) Quality assurance	Quality assurance and consultant supervision
(7) Consultant supervision

Structure and format of WPBAs in psychiatry

Our review identified widespread concerns regarding the use of multiple Likert-type assessments to assess complex psychiatric skills. The alternative, global marking schemes, can increase the ‘halo effect’, however, they allow expert assessors to ‘weight’ the components of the task in a more situation-specific way. ³¹ There is emerging evidence that narrative information enriches the assessment process, ^35,36 and can improve the aspiration towards ‘excellence’ rather than just ‘competence’. This would also allow supervisors to factor in components such as their ‘professional trust’ of the trainee. ^7,30 Some specialties have already designed their own specific WPBA tools (for example Assessment of Psychotherapy Expertise). The concept of a marking system where scores progressively improve with time is

not adopted in most medical schools across the world, and remains problematic. ^11,14 Our recommendations for change include the following.

1. 1 We support plans to replace the Likert-scale scoring system with a single global marking scheme, as employed in the MRCPsych CASC examinations. However, allowing greater ‘free text’ space, rather than MRCPsych tick-box lists of ‘areas for development’, would encourage assessors to provide more personalised and context-specific feedback to trainees. Examples would include commenting on trainees’ awareness of resource management and National Institute for Health and Clinical Excellence guidelines, their ‘higher order’ thinking in assessing diagnostic uncertainties, and their skills at imparting therapeutic optimism to patients.

2. 2 Trainees would be better informed of their progress if they were predominantly assessed according to their current level of training, with the global exit standard score appearing at the bottom of the WPBA form. This would reduce the risks of trainees clustering their assessments at the end of their post, when their scores may be highest.

WPBAs and psychiatric training

Acquiring competencies in one branch of psychiatry may not generalise to acquiring skills in another context. ²⁷ For example, a trainee may be able to elicit a history of dementia in a generic setting, but may have more difficulty doing so within an intellectual disability context. This problem may be exacerbated by the impact of the European Working Time Directive ³³ and New Ways of Working, ³² and reflected by low MRCPsych CASC pass rates.

Although WPBAs have an important formative assessment role for trainees, they should retain a summative role for ARCP purposes, alongside other portfolio documentation. This dual formative-summative role should be made clear to trainees at the outset. ²⁵

Our systematic review has highlighted the extremely limited evidence base for using non-medical assessors, particularly for more experienced trainees. This is consistent with our findings that non-medical assessors score trainees more generously, limiting the usefulness of WPBA tools.

Predictive validity studies of multisource feedback tools, correlating them with clinical and examination performance, are necessary to establish their longer-term credibility. ²⁷ We have found that mini-PATs have consistently low response rates from multidisciplinary team members. Allowing core trainees to nominate assessors from previous posts within the last year may help to resolve some of these difficulties. However, we question the validity of the mini-PAT in being able to pick up core trainees in difficulty, as they are more likely to be identified by their supervisors through direct observation, or via formal or informal feedback from those working with them. Our recommendations for change include the following:

1. 1 Training programme directors should ensure that all core trainees continue to get opportunities to do a post in neurodevelopmental psychiatry, to allow them to complete two to three direct clinical contact WPBAs in this field prior to entering the MRCPsych CASC.

2. 2 The need to spread out WPBAs should be explicitly incorporated into a trainee's personal development plan at the beginning of each post.

3. 3 The use of non-medical assessors should be restricted to the multisource feedback components of WPBAs. In examining a patient for consideration of selective serotonin reuptake inhibitor (SSRI) antidepressant treatment, can a non-medical assessor accurately assess a trainee's ability to assess for significant medical aspects of the history, enquiring about gastrointestinal bleeding, coagulopathies or liver disease? Are they able to assess for a trainee's ‘enhanced’ competencies, by rewarding a trainee who enquires about chronic pain disorders and over-the-counter medications taken by the patient, as SSRIs can greatly increase the risk of gastrointestinal bleeding if regularly taken with non-steroidal anti-inflammatory drugs.

4. 4 As future GMC revalidation is likely to include some form of direct patient feedback, it would be beneficial to link multisource feedback assessments with validated tools that evaluate patient satisfaction.

Quality assurance and consultant supervision

Our review identified the ongoing concerns of trainees undergoing summative WPBAs predominantly by their own clinical supervisor. The low MRCPsych CASC pass rates would reflect these concerns. Quality assurance of WPBA scoring by assessors could be improved by linking the current core training WPBA guide ¹² with video examples on the Royal College of Psychiatrists Assessments online website (https://training.rcpsych.ac.uk/), or via a College training DVD. However, to improve the overall validity of WPBAs, there needs to be some form of ‘external’ assessment within each post.

Although WPBAs should be incorporated into daily clinical practice, their implementation has placed considerable demands on trainees and supervisors. It remains unclear as to how much, if any, of the supervision ‘hour’ is devoted to completing WPBAs, and if there is individual variation among supervisors. Medical Education England has emphasised the need for commissioner levers to be strengthened to incentivise training. ³⁷ Our recommendations for change include the following:

1. 1 It should be mandatory for each trainee to have one to two ‘external’ direct clinical contact WPBAs in each post. Within our training scheme, we have introduced a system of ‘paired’ supervisors, where core trainees have a number of assessments by a consultant psychiatrist who is not directly supervising them. Alternatively, ‘external’ assessor pools can be built up from neighbouring training schemes in a reciprocal way, although this has resource implications.

2. 2 Time for additional training responsibilities, such as WPBA assessments, must be maintained within consultants’ job plans.

Conclusions

Within the context of changes to the delivery of postgraduate medical education, WPBAs play a central role in the assessment of trainees. Although many of the principles underpinning them were sound, there have been significant limitations in their effectiveness when applied to routine clinical practice. We hope our systematic review and recommendations offer practical ways of improving the structure and administration of these tools so that they become more meaningful longer-term assessments. It also offers opportunities for widening the currently limited evidence base on the use of WPBAs in psychiatry, via piloting of some of our recommendations before any general implementation. Areas for College research could include comparative studies looking at the correlation of original and amended WPBA outcomes with MRCPsych CASC scores (including those in subspecialty stations) for core trainees, and with the recently validated psychiatric patient satisfaction (PatSat) scale for all trainees. ³⁸ As competency-based curricula and their associated assessments develop internationally, there would also be scope for other countries to pilot original and amended UK WPBA formats before any general implementation.

Limitations

We used a modified BEME guide for selecting eligible articles, as we found few published comparative descriptive or observational studies of the impact of WPBAs. We set a relatively low threshold number for our modified BEME guide, to ensure we considered the broadest range of currently published evidence. As we did not include e-letters, unpublished articles or informal evidence from conferences, there is a risk of publication bias. There could also be a risk of selective outcome recording (reporting bias), although we tried to minimise this by using ‘hierarchical’ internal consultant peer review of the information extracted from the articles included in our review. We did not assess for bias in each included article, and not all papers meeting the inclusion criteria in our review were used in constructing our recommendations of improving WPBAs. In designing our recommendations, we did not survey psychiatric trainees, but did incorporate triangulation methodology, utilising the experiences of training programme directors involved in the ARCP process. As the use of WPBAs is still in its infancy, it may be too early to establish clear conclusions as to their efficacy. There may also be future changes to their implementation due to wider financial constraints.