Over recent years postgraduate medical training in the UK has been subjected to radical changes. Workplace-based assessment (WPBA), introduced as part of Modernising Medical Careers, 1 is now an essential element of the assessment framework for trainees. The Royal College of Psychiatrists’ competency-based curriculum was implemented in August 2007, since when all trainees must maintain a competency-based portfolio. Progression up the training ‘ladder’ is dependent on the achievement of defined competencies.
Workplace-based assessment carries a number of potential advantages. 2 Its particular strength lies in the rich variety of feedback it potentially offers as a formative assessment tool and its perceived validity in terms of offering information about actual performance in the workplace rather than in the artificial environment of a summative examination. Reference Tooke3 This competency-based system is intended to identify areas for improvement in the individual trainee, on the basis of supportable and documented evidence. Workplace-based assessment complements the more traditional examination-based assessment of knowledge, and thus affords a more holistic and comprehensive assessment of trainees’ progress.
Numerous disadvantages have, however, been highlighted and this system of assessment has attracted severe criticism. At a basic level it is argued that the very identification of a range of competencies truly representative of the knowledge relevant to the trainee is fraught with error. Reference Mansfield and Burke4,Reference Mitchell, Wolf and Fennel5 There exists a strong subjective component to any judgement as to whether any given competency is achieved or otherwise. It has been pointed out that a competency-based approach leads trainees on a superficial path towards achieving a set of discrete and narrow prescribed skills, with little attention paid to the relationships connecting the individual competencies and the deeper meaning underlying each task. Reference Leung6
There is also the risk that WPBA may be applied inappropriately, with a focus on minimum acceptable standards and resultant implications for the standard of doctors – a culture of mass mediocrity. A review of evaluative studies of competency-based training in industry has found no convincing beneficial effects in terms of motivation and performance, despite an increase in administrative costs. Reference Bates, Flude and Sieminski7 Workplace-based assessment is a relatively new concept in medical training in Britain and there is little research in this area. Evidence indicating general dissatisfaction in the medical community is however, slowly emerging. Reference Wilkinson, Crossley, Wragg, Mills, Cowan and Wade8–Reference Menon, Winston and Sullivan12
The British Medical Association argues that the intrinsic validity and reliability of such an assessment system is influenced by multiple factors. These include the environment in which it is employed, the nature of the competencies it purports to measure, the effectiveness of its implementation, the availability of adequate and appropriate resources and support, and whether it is compatible with and complementary to the larger overall assessment programme. Additionally, WPBA tools have been validated in only a limited number of clinical settings and it is recognised that individual specialties require bespoke tailored assessment programmes in order to provide the best possible training. 13
It is also relevant that this assessment system may not be fully understood by trainees in what is traditionally a highly competitive environment. A poor score at an assessment may create a feeling of failure in a trainee, rather than the process being viewed as a formative experience with the opportunity to identify areas for learning. It is also recognised that by its very nature WPBA carries the potential to stifle aspiration towards excellence, in favour of settling for adequacy. In this context it is relevant that computerised forms can oversimplify judgements that are necessarily complex and thus preclude the identification of excellence. 2
Finally, despite recognition that WPBA is time consuming, little attention has been paid to the considerable resources needed for its effective implementation particularly in the context of educational supervision and feedback. The Postgraduate Medical Education and Training Board trainers’ survey in 2008 identified significant pressures on trainers who lack allocated time within their job plans for the provision of supervision and structured feedback to trainees. 2
Two studies (the trainees’ and trainers’ surveys in Wales) were therefore designed to explore the attitudes of both psychiatric trainees and trainers along with the practical difficulties they face with a view to providing the structured feedback so essential to improving the validity, reliability, relevance and practical benefits of WPBA. The results of the trainees’ survey were published in the Psychiatric Bulletin in December 2009. Reference Menon, Winston and Sullivan12
Method
The objectives of the trainers’ survey were to explore attitudes and perceptions regarding WPBA among psychiatric trainers (consultants) currently supervising trainees under the Modernising Medical Careers training system in Wales, to identify problems currently experienced and to report measures recommended by respondents to streamline and successfully implement WPBA.
The survey was conducted under the auspices of the Welsh Division of the Royal College of Psychiatrists and the Welsh Deanery. The methodology followed was similar to that used for a previous survey of trainees, Reference Menon, Winston and Sullivan12 involving an anonymous questionnaire-based cross-sectional survey with minor variations to the questionnaire as deemed appropriate for this different target group. After clearance from the National Research Ethics Service, the survey was designed in conjunction with a statistician and a clinical research fellow at the University of Cardiff.
It was originally intended to run both trainees’ and trainers’ surveys concurrently. The latter was however, significantly delayed in its launch and also prolonged in its course for several reasons. Difficulties were encountered at the outset in obtaining an accurate, current and comprehensive list of trainers. A list was therefore compiled from several sources: the Welsh Division of the Royal College of Psychiatrists, the Welsh Deanery and individual employing trusts across Wales, and also through personal communication. This process was hampered by data protection issues and the reluctance of some trusts to release email contact details for consultants. Finally, a poor response from the trainers resulted in further delay.
Many questions were designed as consistently positive statements and respondents were asked to indicate their agreement or otherwise on a five-point Likert scale (i.e. 1 – strongly disagree, 2 – disagree, 3 – no opinion, 4 – agree, and 5 – strongly agree). Other questions involved respondents rating any given subject on a five-point scale (i.e. 1– very poor, 2 – poor, 3 – acceptable, 4 – good, and 5 – very good) and the remainder required respondents to select a preferred answer from multiple choices (see online supplement DS1 and DS2 for the full questionnaires).
After a week's ‘pilot’ phase, minor modifications were made to the questionnaire, and the survey was launched in January 2009 via the Bristol Online Survey website (www.survey.bris.ac.uk) and was closed in August 2009. Potential respondents were invited to participate by the Welsh Division of the Royal College of Psychiatrists via a generic email containing a link to the Bristol Online Survey website.
The response rate was low and remained at less than 16% despite several further emails from the Welsh Division. We therefore attempted to obtain the individual email addresses of trainers where possible via trusts and personal contacts and these individuals were then contacted personally via email. As a result the response rate rose to 63%, high enough to draw valid conclusions, although still lower than the 92% response rate of the trainees’ survey. Reference Menon, Winston and Sullivan12
The results from the trainers’ survey were then compared with those from the trainees’ survey. We used SPSS 16.0 for Windows; the data being predominantly categorical, a Pearson chi-squared (χ2) test was employed to explore the significance of any associations between the variables in the two groups. A Fisher's exact test was reported wherever there were less than five observations in a cell.
Results
A total of 104 of 164 invited trainers (63%) responded, the majority (88%) from South Wales. Of these 74 (71%) trainers had more than 5 years’ experience working as a consultant, 75 (72%) had educational supervisor status, 97% had supervised a trainee, although only 59% had supervised a senior trainee. Ninety-three (89%) of the trainers had undertaken a WPBA for a trainee.
Trainers’ perceptions
Main reasons and the most important driving forces behind the introduction of WPBA
The two most prominent perceived reasons were ‘to improve training’ (44%) and ‘politically driven’ (36%), accounting for 80% of respondents (Table 1). Only 8% of trainers considered that it was introduced ‘to improve patient care’, and only 2% of trainers feel that trainers themselves had been instrumental as a driving force behind the genesis of WPBA. The two most commonly perceived driving forces were the Postgraduate Medical Education and Training Board and the government, accounting together for 76% of trainers, whereas the Royal College of Psychiatrists came third at 20%.
WPBA and its introduction
A substantial proportion of trainers expressed no opinion (see online table DS1). Of the remainder, the majority were unimpressed with WPBA as an educational/assessment tool in terms of its underpinning evidence and reliability, although opinions were more favourable with regard to its validity and the propriety of it having been made compulsory. A large majority (71%) opined that assessments are not independent of the assessors’ personal preferences. Concerns were also expressed about the manner of introduction of WPBA and the lack of information, guidance and training.
Effect of WPBA
A large number of trainers (from 19 to 42%) expressed no opinion (online table DS2). The majority of respondents, however, perceived little beneficial effect from WPBA.
Practicalities of assessments
Trainers were divided equally over ease of organisation of assessments and support from colleagues (online table DS3). They were, however, largely positive about ease of access to computers, although many expressed concerns about the impact of assessments on time available for clinical duties.
TABLE 1 Comparison of trainees’ and trainers’ perceptions about the main reasons and important driving forces behind the introduction of workplace-based assessment
| Trainees % (n) |
Trainers % (n) |
|
|---|---|---|
| Main reasons | ||
| To improve training | 43 (35) | 44 (46) |
| To improve patient care | 9 (7) | 8 (8) |
| Failure of the previous training system | 5 (4) | 6 (6) |
| Politically driven | 41 (33) | 36 (37) |
| Do not know | 2 (2) | 7 (7) |
| Most important driving forces | ||
| Royal College of Psychiatrists | 6 (5) | 20 (21) |
| PMETB | 44 (36) | 37 (38) |
| Government/political | 44 (36) | 39 (40) |
| Public demand | 1 (1) | 2 (2) |
| Trainers | 0 (0) | 2 (2) |
| Do not know | 4 (3) | 1 (1) |
PMETB, Postgraduate Medical Education and Training Board.
The assessment tools adopted
A more encouraging picture emerges with regard to the ease of use and relevance of the assessment tools recommended by the Royal College of Psychiatrists (65% and 58% of trainers expressing approval), although only 30% felt that WPBA facilitated training positively. Details are given in online table DS4.
Recording of assessments
‘Acceptable’, ‘good’ and ‘very good’ were considered as satisfactory standards. In total 86%, 82% and 83% of trainers rated the online system as satisfactory in terms of ease of use, reliability and time consumption (online table DS5a); 53% perceived no change, and the remainder felt that the current online system was an improvement over the previous HcAT system (online table DS5b).
Discounting the ‘no opinion’ group, 52% felt that an online system was easier, although 46% of trainers expressed concerns about the reliability of such a system. Opinions were equally divided in terms of compliance and ease of form completion, and 58% felt that an online system would be more time-intensive. Details are given in online table DS5c.
Preferred system for recording assessments
Overall, trainers appeared to prefer an online system of recording assessments (37%), with an additional 31% indicating a preference for a combination system (Table 2).
Perceptions of themselves as assessors
‘Acceptable’, ‘good’ and ‘very good’ were similarly considered as satisfactory standards (online table DS6). The vast majority of trainers rated themselves as satisfactory (availability 98%, willingness to complete assessments 94%, knowledge 86%, computer literacy 93%, understanding of the online system 85%, the ability to assess accurately and impartially 93% and the ability to provide constructive feedback 99%).
Senior trainees being assessed by non-medical professionals
The majority of trainers expressed approval with regard to senior trainees being assessed by non-medical professionals (by nursing staff 65%, by social workers 54%), and less than a third disagreed.
Relevance of WPBA for various training grades
In total 26% of trainers felt that WPBA is more relevant to junior trainees (ST3/CT3 and below). A small minority (2%) expressed the opposite view, namely that WPBA is more relevant to senior trainees (ST4 and above). The majority (57%), however, held that WPBA is relevant to trainees at all stages of training; 15% of trainers expressed no preference.
Overall perceptions about WPBA in its current form
Strikingly 48% and 46% of trainers expressed ambivalence as to whether WPBA in its existing form is being used appropriately and whether it represents an improvement over the previous system, although 40% expressed no opinion about whether WPBA is the way forward (Table 3). A total of 46% felt that WPBA does not accurately reflect trainees’ progress, although 55% felt that WPBA is acceptable to them.
TABLE 2 Comparison of trainees’ and trainers’ preference with regard to a system for recording assessments
| Trainers, % | Trainers, % | |
|---|---|---|
| On paper | 51 | 16 |
| Online | 6 | 37 |
| Either (both are equally good) | 5 | 16 |
| Combination | 38 | 31 |
Comments, criticisms and recommendations by trainers
The survey also incorporated the option for respondents to express, in free text, their views on WPBA. The majority of such comments were unfortunately negative; the comments and concerns expressed are listed in the appendix at the end of this paper. Trainers were similarly invited to offer their recommendations with regard to remedial measures that might improve the WPBA system, these are detailed in Box 1.
Discussion
Perhaps the most interesting aspect of these surveys involves the contrast between the response rates of trainees (92%) and trainers (63%). These figures, however, do not represent the whole picture, since the trainees’ survey was open for a mere 2 months compared with 7 months for the trainers’ survey. The rapidity of response is also relevant; the majority of trainees completed the survey promptly with little requirement for reminders, whereas quite the opposite was the case for the trainers (the response rate stood at 16% after 4 months and as many reminders), indicating the strength of feeling among trainees in this context, compared with perhaps a relative indifference from the consultant body. Consultants have overall responsibility for training and assessing their juniors; it is therefore arguable that such indifference does not bode well for the future of WPBA, especially when one considers that trainers are not perceived as having been a significant driving force behind the introduction of WBPA in the first place.
A considerable number of respondents (more so in the trainers’ survey) selected the ‘no opinion’ option with regard to the various aspects of WPBA with obvious implications upon the significance of the results. This is surprising considering that only 1 trainee (1%) and 11 consultants (11%) had not actually undergone or conducted an assessment. Admittedly some respondents might not have had exposure to certain aspects of WPBA (for instance possibly not having used the online system), in which case a ‘no opinion’ response would be appropriate. For the remainder, however, it seems reasonable to infer that the selection of ‘no opinion’ as a response is possibly reflective of, at best, an ambivalence with regard to WPBA, through a lack of knowledge of relevant issues to, at worst, a regrettable and worrying lack of interest in what is undeniably an integral component of current medical training.
Box 1 Recommendations by trainers
-
• Clear guidance on prescribed standards to be attained by trainees for each stage of training with assessments individually tailored to the placement and level of the trainee.
-
• Methods to standardise assessments between assessors to maximise objectivity of assessments and reduce inter-assessor variability with some form of bench-marked training for workplace-based assessment (WPBA).
-
• Concentrated assessments by fewer ‘specialist assessors’ to increase reliability.
-
• Structured training for assessors (a prominent and consistent theme) with comprehensive specialty-specific training for relevant staff along with booster sessions and opportunities for trainers to share experiences.
-
• More options for outlining opinions rather than tick boxes and meaningful summarisation and interpretation of the results of assessments.
-
• Further development of subcategories to allow for variation between subspecialties in terms of clinical approach, relevant skills, etc. and flexibility to tailor assessment tools as required to individual subspecialties.
-
• Robust research to demonstrate predictive validity of assessments and more meaningful and clear recording of assessments.
-
• Protected time to organise and conduct WPBA along with more support from employing trusts in terms of consultant timetables.
-
• Reduced freedom of trainees to choose their assessors.
-
• Supplementation of WPBA by other methods of assessment (like the ‘old fashioned’ reference) to identify issues missed by formal assessments.
-
• Maintenance of a log book and application of reflective learning principles, rather than becoming overly reliant on WPBA.
-
• Provision of a short report by the educational supervisor based on two observed patient interviews. Supervisors should give an opinion on trainees appetite for work, willingness to involve him/herself in further training opportunities (for example cognitive–behavioural therapy), intellectual curiosity, knowledge, patient management, clinical decision-making, and ability to get on with staff and colleagues. Candidates who are thought to be failing to be offered a meeting with course tutor and the educational supervisor.
TABLE 3 Trainees’ and trainers’ overall perceptions about workplace-based assessment (WPBA) in its existing form and and comparison of their overall perceptions using Fisher's exact test
| Trainees’ attitudes, % (n) (n=80) |
Trainers’ attitudes, % (n) (n=102) |
Comparison of trainees’ and trainers attitudes |
||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| WPBA: | Strongly disagree |
Disagree | No opinion |
Agree | Strongly agree |
Strongly disagree |
Disagree | No opinion |
Agree | Strongly agree |
Fisher exact test |
P |
| Is being used appropriately | 14 (11) | 40 (32) | 28 (22) | 18 (14) | 1 (1) | 2 (2) | 16 (16) | 48 (49) | 34 (35) | 0 | 29.454 | < 0.001 |
| Is acceptable to you as a trainee/trainerFootnote a | 19 (15) | 35 (28) | 24 (19) | 21 (17) | 1 (1) | 1 (1) | 25 (26) | 19 (20) | 52 (53) | 3 (3) | 30.779 | < 0.001 |
| Accurately reflects trainees’ progress | 25 (20) | 43 (34) | 18 (14) | 13 (10) | 3 (2) | 8 (8) | 38 (39) | 30 (31) | 23 (23) | 1 (1) | 14.852 | < 0.001 |
| Is a better system than the previous one | 24 (19) | 35 (28) | 24 (19) | 15 (12) | 3 (2) | 4 (4) | 20 (20) | 46 (47) | 27 (27) | 4 (4) | 25.357 | < 0.001 |
| Is the way forward and should be retained | 30 (24) | 28 (22) | 29 (23) | 11 (9) | 3 (2) | 8 (8) | 19 (19) | 40 (41) | 30 (30) | 4 (4) | 21.061 | < 0.001 |
Results in bold are significant (2-sided test).
a. For trainers n = 103.
The main reasons and driving forces behind the introduction of WPBA
Despite large proportions of both trainees (43%) and trainers (44%) believing that WPBA was introduced with a view to improve training, the new framework has engendered overwhelmingly negative attitudes among both groups (Table 1). This is partly due a common perception (held by 41% of trainees and 36% of trainers) that WBPA is a politically motivated initiative with an unclear rationale and purpose.
It is interesting that only a small minority of trainees and trainers (5% and 6% respectively) feel that WPBA was introduced to overcome failures of the previous training system. Furthermore, only 9% of trainees and 8% of consultants feel that the system was introduced to improve patient care. Interestingly there is a statistically significant difference between the perceptions of the two groups (Fisher's exact value 10.618, associated P = 0.032), possibly due to the fact that a substantially larger proportion of trainers perceive the Royal College of Psychiatrists as an important driving force.
WPBA and its introduction
This overwhelmingly negative attitude is perhaps largely due to the manner in which WPBA was introduced (83% of trainees and 64% of trainers feel that it was not introduced in a well-thought-out manner) (online table DS1). It is also widely perceived as being conceptually flawed, based on scant evidence and of dubious validity as an assessment tool. Indeed, less than half of both groups expressed favourable opinions with regard to the validity of WPBA (42% of trainers, 23% of trainees). A similar picture emerges with regard to the propriety of WPBA having been rendered compulsory from the outset, this being endorsed by 46% of trainers and only 13% of trainees. This obvious lack of enthusiasm from the trainees is with little doubt related to the fact that the entire responsibility of managing one's WPBA portfolio has been placed on the shoulders of the trainee. Indeed, trainees consistently rated WPBA and its manner of introduction much worse than did trainers – and the differences between the groups are statistically significant on all parameters except one (i.e. whether WPBA is backed by good evidence (online table DS1).
Given the universal agreement that objectivity is essential to a fair, valid and reproducible system of assessment, it is interesting that the majority of trainees (81%) and trainers (71%) agree that WPBAs are not independent of assessors’ personal preferences (the difference between the two groups is statistically significant). Such subjectivity may stem from a number of factors including differences between individual trainers’ expectations, a lack of consensus among assessors, a lack of clear guidance on prescribed standards to be attained for each stage of training, inadequate training opportunities for assessors and indeed, an arguably inappropriate assumption that every assessor of any background (let alone a consultant trainer) is possessed of the ability to accurately, objectively and repeatably assess any given trainee at a level appropriate to that trainee's experience. This is possibly a fundamental flaw of any non-structured assessment process in which any given clinical scenario, circumstance and questions posed are not standardised.
Only a small minority in each group (8% and 20% of trainees and trainers respectively) feel that the introduction of WPBA has been accompanied by sufficient training, a significant shortfall given the relevance of such training to the introduction of any novel system.
Effect of WPBA
Trainees were vociferous in expressing overall dissatisfaction with the effects of WBPA, the majority holding the view that WPBA has had no real beneficial effects on supervision, training, clinical practice and confidence (online table DS2). Disappointingly a large number of trainers (from 19 to 42%) expressed no opinion on these four important parameters. Analysis of those who did however, suggests that the majority of trainers agree with the trainees in terms of perceiving little beneficial effect as a result of WPBA, although trainees rate the effects of WPBA far worse than do trainers, with differences that are statistically significant on all parameters except that of supervision (online table DS2). Given the rationale for the introduction of WPBA, the most striking feature here perhaps is that only a minority of trainees (19%) and trainers (33%) feel that WPBA has achieved its objective of improving training.
Practicalities relating to WPBAs
Over 70% of trainees feel that they encounter difficulties organising assessments, compared with 45% of trainers (online table DS3). This result however, requires interpretation in the light of the fact that the entire responsibility for maintaining a performance-based portfolio rests with the trainee, who is obliged to complete a given number of mandatory assessments over a limited time frame, an undeniably stressful task, given that non-fulfilment of these requirements carries the potential for a trainee to be failed at their Annual Review of Competence Progression. As discussed in the previous survey, Reference Menon, Winston and Sullivan12 this process is therefore at considerable risk of degenerating into a tick-box/paper-pushing exercise, which nevertheless by its very nature, is likely to take priority over the acquisition of essential clinical experience, a sentiment expressed by numerous respondents.
Given that assessments being intrinsically time and resource intensive, both groups (70% of trainees, 63% of trainers) agree that without formal allocation of time for the increased levels of supervision and assessment of trainees (integral to the performance based portfolio) there will inevitably be negative implications on the time available for clinical duties.
In total 44% of trainees harbour concerns about lack of support from supervisors and colleagues and 79% have concerns about the lack of access to computer facilities in the context of assessments. We found that 32% of trainers appear to be dissatisfied over the issues surrounding support from their colleagues for WPBA, although trainers largely feel positive about ease of access to computers (56%), in sharp contrast with their trainees (79%).
Yet again, trainees portray a much less complimentary picture about the practicalities associated with assessments than do trainers, with differences that are statistically significant on all parameters with the exception of the impact on time available for clinical duties (online table DS3).
Assessment tools adopted by the Royal College of Psychiatrists
Numerous concerns were raised by trainees with regard to the assessment tools endorsed by the Royal College of Psychiatrists (online table DS4); 58% feel that they do not facilitate training, 46% feel that they are not easy to use and 32% feel that they are not relevant to psychiatry. Among trainers, however, a more encouraging picture emerges with regard to both the ease of use and relevance of the assessment tools (65% and 58% respectively of trainers expressing approval), although the overall view of whether WPBA affects training positively is not so encouraging (30%). Predictably trainees once again rated the assessment tools worse than did the trainers, the differences being statistically significant on two parameters, namely ease of use and facilitation of training (online table DS4).
Recording of assessments
A major source of frustration for trainees involved the online system for recording and storing assessments (HcAT), which was made compulsory from the outset. Immense difficulties rendered HcAT very unpopular – indeed, the majority of the trainees found it unacceptable (online table DS5b). Trainers on the other hand, expressed a considerably more positive outlook on an online system, 86%, 82% and 83% rating this as acceptable or better in terms of ease of use, reliability and time consumption respectively, the differences on all parameters between the two respondent groups being statistically significant (online table DS5a).
Trainers’ opinions were equally divided in terms of compliance and ease of completion of forms, although the majority felt that an online system would be more time intensive. This is in sharp contrast with the trainees, who were largely unimpressed with the online system of recording assessments, on the basis that in their collective opinion, recording assessments online would not render the process of WPBA any easier, compared with a paper-based system.
The trainers’ survey, however, took place some months after the trainees’, by which time HcAT had been abandoned in favour of ‘Assessments Online’, an updated system, introduced in September 2008. The new system appears to have possibly addressed some of the difficulties with HcAT, evidenced by the overall positive response from the trainers, although when asked specifically as to whether the current online system represented a major improvement over the previous system trainers were divided in their opinions, 53% perceiving no change, and the remainder an improvement.
Preferred system for recording assessments
There exists a statistically significant difference between trainees’ and trainers’ preferences with regard to the two methods of recording assessments (χ2 = 41.096 with an associated P<0.001). Over half the trainees preferred a paper-based system of recording assessments (a smaller proportion prefer a combination system), whereas trainers largely preferred an online or combination system (Table 2). The major changes made to the online system during the time period between the two surveys however, arguably render invalid any comparative analysis of the attitudes of the two respondent groups.
Trainees’ perceptions of their assessors and trainers’ perceptions of themselves
For this question, responses of ‘acceptable’, ‘good’ and ‘very good’ were considered as being satisfactory, and ‘poor’ and ‘very poor’ as unsatisfactory. From the results it is evident that trainers almost unanimously rate themselves as satisfactory on the numerous parameters in this context (online table DS6a,b). A lower proportion of trainees, however, rated trainers as satisfactory on all parameters except the ability to assess accurately and impartially. Differences between the two groups are statistically significant on the parameters of accessibility/availability, the ability to provide constructive feedback, knowledge of WPBA, and understanding of HcAT, the differences being most marked on the last two parameters and such discrepancies could prove a stumbling block to the whole process at a fundamental level (online table DS6b).
Trainees’ views with regard to assessors from non-medical backgrounds (who, of relevance, have not undergone such assessments themselves) are however, not so complimentary with an overwhelming majority of trainees allocating poor scores. Worryingly, such assessors were rated very poorly on the three most important parameters, i.e. understanding of WPBA, the ability to assess accurately and impartially and the ability to provide constructive feedback. The views expressed by the consultant body in the context of the propriety of senior trainees being assessed by such individuals are interesting, with only 29% and 32% expressing disapproval at the concept of the assessors being nursing staff and social workers respectively, when one considers that consultants themselves have not been subjected to such assessments by non-medical staff.
Trainees’ and trainers’ overall perceptions about WPBA
Overall, between half and two-thirds of trainees feel that WPBA in its current form is unacceptable to them, does not accurately reflect their progress, is no better than the system adopted prior to Modernising Medical Careers, and is not the way forward and therefore should not be retained (Table 3). This contrasts with a much lower level of disapproval among the consultant body in these contexts, with the notable exception of the accuracy of reflection of trainees’ progress, in which area 46% of trainers expressed disapproval. Perhaps the most striking statistics for the latter group however, are those illustrating high levels of ambivalence as to whether WPBA in its existing form is being used appropriately, whether it represents a better system than the previous one and whether WPBA is the way forward, with 48%, 46% and 40% of trainers expressing no opinion. Despite this, interestingly, 55% of trainers feel that WPBA is acceptable to them. The two groups displayed statistically significant differences between their overall perceptions about WPBA on all the above parameters, the trainees’ collective opinion being worse than that of trainers (Table 3).
It is debatable as to whether individuals from non-medical backgrounds (who do not undergo a system of regular appraisal themselves) are qualified to assess the clinical acumen of postgraduate trainees. As with any principle, there are exceptions – few would argue that highly trained professionals such as psychotherapists and psychologists would not be qualified to assess trainees in their own areas of expertise. Apart from these exceptions however, the old adage ‘the eye does not see what the mind does not know’ would certainly apply to such assessments. Indeed, it would intuitively appear inappropriate that non-medical members of staff, with little understanding of medical training and the standards acceptable/mandatory at each level over a wide range of competencies should be involved in making formal assessments of medical personnel. Certainly there is little published evidence attesting to the efficacy, validity and propriety of doctors being assessed by non-medically qualified individuals.
Study strengths and limitations
With due consideration given to the intrinsic limitations of the study design, the remarkably high response rate from the trainees reflects the validity of the results obtained (in that they are truly representative of the views of Welsh psychiatric trainees) while also reflecting the strength of feeling among trainees on this subject. As a corollary, the response rate from trainers, although significantly lower and less impressive, is nevertheless respectable in the context of a cross-sectional survey and of sufficient magnitude to reasonably draw conclusions from. Furthermore, the trainees’ survey was conducted approximately a year after the introduction of WPBA and the trainers’ survey a few months later. Both surveys therefore involved respondents who had acquired first-hand practical experience with this system of assessment.
Questionnaire-based surveys are intrinsically prone to the limitations imposed by the narrow scope and potential inadequacy of the spectrum of selectable responses for any given question. The views of participants might therefore be inadequately expressed, because the questions and the spectrum of selectable answers might be incompatible with respondents’ personal views. These surveys sought to address this issue by giving respondents the opportunity to state, in free text, any further views they wished to express. This study design affords the advantage of a reduced level of ‘observer bias’ since the anonymity enjoyed by the respondents removes the constraints of social niceties and political correctness with regard to expressing their views freely.
Finally, the time interval between conducting the survey and publication of the findings carries the potential, albeit small, to detract from the relevance and applicability of the study results by the time of publication, although this would arguably apply to any study that finds its way into print.
Implications
Both surveys highlight concerns with regard to the current competency-based assessment framework that persist 2 years after its introduction. Dissatisfaction is widespread among trainees, the general consensus being that WPBA has proved a paper exercise no better than the previous training system that, while increasing their workload, has added little real value to their training. Trainers on the other hand, appear more diverse with regard to their views and opinions, although considerable numbers are similarly dissatisfied. Indeed, the most striking feature is that only a minority of the respondents (19% of trainees, 33% of trainers) feel that WPBA has achieved its objective of improving training.
The WPBA, in its current form, is plagued by numerous problems, including a generalised ambiguity surrounding assessments, a lack of clear and defined standards, a reluctance to award poor ratings to underperforming trainees and variability between assessors. There appears to be an almost unanimous perception that WPBA has not been accompanied by sufficient training – this is particularly surprising, considering the high profile enjoyed by this new initiative and the importance of education and training in this context. The implications are obvious when one considers that flawed assessments by incompetent and ineffectual assessors inaccurately reflect trainees’ abilities and performance, a point highlighted by the fact that trainees adjudged as competent in clinical skills at their WPBAs (and who therefore pass their Annual Review of Competence Progressions) often do not perform at levels consistent with their WPBA ‘scores’ at the Royal College's Clinical Assessment of Skills and Competences exam, where the pass rate has recently dropped to below a third. Reference Sikdar14 Not uncommonly therefore, is encountered the ‘trainee with a portfolio of perfect WPBA scores, baffled by their failure to pass the Clinical Assessment of Skills and Competences exam’. Reference Howard and Brittlebank15
Structured training of assessors will be fundamental to addressing these weaknesses and to optimising the validity and objectivity of assessments. Encouragingly, the Royal College of Psychiatrists is taking measures to address concerns highlighted in recent surveys, including the provision of training for medical assessors. 16 This will, however, need to extend to assessors from all backgrounds – and undeniably, the provision of robust structured training to innumerable assessors from various disparate (non-medical) backgrounds will prove a logistic nightmare. Equally significant is current legislation that mandates the undertaking of formal training in order to maintain assessor status. 16 With little incentive for non-medical assessors to undergo such training, the implications are obvious.
Finally, ingrained perceptions are difficult to dispel – trainees (and perhaps trainers too) tend to view assessments as summative exercises rather than the formative experiences they are designed to be. It is not uncommon for trainees to view WPBAs as mini-examinations; poor ratings are therefore demoralising, so much so that trainees may defer assessments and/or preferentially seek out ‘soft’ assessors, who are more likely to be generous with their scoring. Remedying this will require a sea change at a very fundamental level for both assessors and assessees. Both groups of respondents in this survey have made a number of pertinent and relevant recommendations that merit careful consideration, coming from individuals at the ‘coal face’ with practical experience of WPBA.
The question as to whether WPBA has been incontrovertibly proven to be ‘fit for purpose’ therefore remains unanswered. The persistence of serious widespread and ongoing concerns 2 years after the introduction of WPBA (by which time one could reasonably expect that any teething problems would have been resolved) raises the stark question as to whether the fundamental problem involves the flawed implementation of an otherwise excellent system or alternatively whether WPBA, at its very heart, is fundamentally flawed. Can the ineffective original online system and the hurried implementation of WPBA be blamed entirely for the poor uptake and negative perceptions or are we ignorant, or turning a blind eye, to deeper problems intrinsic to WPBA? It is vital that relevant research in this context be supported to assess this basic question, along with ongoing stringent quality-assurance measures to constantly review and improve psychiatric training.
Appendix
Comments and criticisms by trainers.
‘It does not really inform us about how well are the trainee and training. It is just a form completing exercise.’
‘They are easy to complete but are essentially a meaningless bureaucratic exercise.’
‘I think the tools used are confusing. Despite some guidance I am not sure I am assessing in line with peers.’
‘We need a complete rethink of what we are trying to achieve.’
‘Pen and paper assessments are still much more immediate in the thick of clinical practice.’
‘The system has not be fully thought through and introduced in a chaotic manner without enough preparation. Improvements in all the above areas need to be made for it to be potentially useful.’
‘Due to the infrequency of ever being in a clinical setting at the same time as a trainee (i.e. they are rarely in out-patients or ward rounds) my knowledge of their abilities or otherwise is restricted to what they tell me they are doing and the WPBA seems to me to be a fig-leaf compared to actually seeing a trainee at work.’
‘I have attended WPBA training at Royal College of Psychiatrists tutor meetings and witnessed total disagreements between different assessors over whether a trainee (on DVD) had done well or not, with no way proposed to resolve the assessors differences. This hardly bodes well for consistency of training.’
‘ScrapWPBA and examine trainees in exams with external examiners. Trainees should keep a log book of clinical activity that is signed off. Trainees should be assessed by experienced doctors who have passed the MRCPsych; not any old MDT [multidisciplinary] member who has not been assessed themselves.’
‘May not be perfect but at least it is some sort of assessment which is more than we had before when people could just drift through training for years until problems were identified. The main problem is that WPBA only assess if trainees possess certain skills and can perform certain tasks – not whether they will actually do so in a real (i.e. not tested) environment.’
‘My concern is the tendency to turn this into a tick box exercise rather than look at the evidence base for how they should be used and their purpose. Further education of both trainers and trainees into their use and ways to improve their reliability and validity might be helpful to overcome this.’
‘Some guidance or consensuses from the college on rating trainees is necessary as in my experience trainer's have different expectations. I have been to training sessions but no consensus seems to exist as yet.’
‘No follow-up training or guidance since initial brief training. Doctors are doing the best they can but inevitably will be developing different ideas about what constitutes competency at certain levels of training.’
‘I always have to fill them in my personal time each evening as I have no time to complete during working day. Trainees are so busy trying to fill in forms, attending one course or another, or on rest periods due to EWTD [European Working Time Directive], their exposure to patients and continuity of treatment has been diluted and therefore their training is not as good. As for being a consultant after 6 years of this kind of run through training, I personally think this will be insufficient to prepare the trainees for life as a consultant.’
‘The online system is difficult to achieve when computer monitors are scarce. Guidance and training of assessors needs to improve – the current system makes results very unreliable. The short assessments are not really appropriate for senior trainees who need to be assessed on how they tackle difficult problems over time and manage a team. WPBA should be supplemented by other methods of assessment (like the old fashioned reference) which may perhaps be less reliable but might pick up issues missed by formal assessments and identify high flyers.’
‘The current system of WPBA is very difficult to bench mark against the average trainee. The level of knowledge acquired in our placement is so much greater for our trainees than for those who haven't done the placement. On this basis, it is frequently only possible to give the highest mark. The different types of assessment are also difficult to differentiate for our placement. All patients are seen at length and detailed discussion takes place with the trainee regarding each one. Trying to shoe horn what the trainee actually does (knowledge/skills attitudes acquired, etc.) into the boxes seems quite pointless and irrelevant. Of course we do it – as our excellent trainees cannot progress without it! However, it seems like a politically driven necessity introduced at a whim and without any evidence for its applicability. I'd recommend that it's replaced by something more individually tailored to both the trainee and the placement.’
Acknowledgements
We are indebted to all the trainees and trainers who gave of their precious time to participate in this survey. We also thank Dr Helen Mathews and Mrs Siobhan Conway at the Welsh Division of the Royal College of Psychiatrists and Mrs Elenor Williams at the Wales Deanery for their invaluable support. We are grateful to Mr Paul Buckland for his support with regard to hosting the survey online and to Dr Marianne van den Bree and Dr James Walters at the University of Cardiff for their help with designing and validating the questionnaires. We also thank Mr Kerry Rees, Dr Seth Mensah and Dr Nimalee Kanakkahewa for their statistical advice and input.
Open access
eLetters
No eLetters have been published for this article.