Letter to the Editor,
Psychiatry trainees need to pass examinations to progress through training. We have set up progress tests to assist trainees in the hope of improving exam performance to support career progress. Preliminary data is presented here.
Progress tests assess against the standard expected for the ultimate aim (e.g. graduating from medical school, or final membership exams), without this, benefits of progress testing are lost (Dijksterhuis et al. Reference Dijksterhuis, Scheele, Schuwirt, Essed, Nijhuis and Braat2013). In undergraduates, there is improvement in scores with each year of seniority (Van der Vleuten et al. Reference Dijksterhuis, Schuwirt, Braat and Scheele1996). For undergraduates in USA, progress testing is associated with higher pass rates on the national licensing examination (Norman et al. Reference Norman, Neville, Blake and Mueller2010). Progress tests allow knowledge to grow and be retained for longer (Schuwirth & van der Vleuten, Reference Schuwirth and van der Vleuten2012). This is likely due to progress testing encouraging deep learning; with superficial learning being more likely when exams are set at the end of a specific course (Van der Vleuten et al. Reference Dijksterhuis, Schuwirt, Braat and Scheele1996). Progress testing in postgraduate medical education may re-ignite the drive for developing knowledge, an important part of the expert problem-solver, in the face of a curriculum which focuses on performance through implementation of work-based assessments, 360° feedback and portfolios (Dijksterhuis et al. Reference Van der Vleuten, Verwijnen and Wijnen2009).
This pilot study aims to evaluate the reliability of a written progress test, and assess the ability to distinguish between year of training, for psychiatry trainees in one region in the United Kingdom.
Following ethics approval, we invited Year 1 and Year 2 core psychiatry trainees in 2016 to sit the written progress test. Trainees were given three hours to complete 200 multiple-choice questions which reflect the MRCPsych examination format from the Royal College of Psychiatrists, UK. The questions were written by the course module leads who have detailed knowledge of the syllabus, and these were further validated by the Deputy Program Director (G.T.).
After completing the progress test, trainees were given their scores as an overall percentage, as well as scores in different subject areas (e.g. psychopharmacology, psychopathology) to aid them in understanding areas of relative strength and weakness. They were also given anonymized scores for their peers, allowing them to make comparisons between their peers.
Reliability was calculated with Cronbach’s α, to establish the internal consistency of each exam. Student’s t-test was used to compare test scores for Year 1 core trainees against Year 2 core psychiatry.
In total, 27 trainees took the progress test, 12 were Year 1, and 15 were Year 2 trainees. The maximum possible score was 204. The mean score was 88.5 (s.d.=22.4), range 46–127.5. Cronbach’s α=0.85. The mean scores for Year 1 trainees was 78.8, and for Year 2 trainees was 96.2, t=2.11, p=0.047, d=0.82 (d=effect size).
The written progress test is reliable. This reflects that the exam items are closely related and consistent. The results suggest that the progress test has the ability to distinguish between trainees of differing seniority. However, it is not known if the Year 2 core trainees were simply a stronger cohort than the Year 1 trainees, or if the results suggest there has been development and learning of the more experienced trainees. If the latter, it is not clear if the progress test has contributed to any development or learning. Progress tests could potentially be a tool to help improve trainee development and exam performance, but further research is required.
Acknowledgements
The authors would like to thank all the trainees who participated. In addition, the Medical Education Manager (Sam Abbott) and the Admin Team at North Manchester General Hospital.
Financial Support
No funding was awarded for conducting this study.
Conflicts of Interest
The authors have no conflicts of interest to declare.
Ethical Standards
Ethical approval was granted by Health Education England Research Governance Committee. We ensured that trainees were aware that participation was optional, and they could withdraw up until their test paper was submitted. Trainees were made aware that their data may be used anonymously for publication.
