Evaluating and enhancing supervisee competence is a key function of supervision and can be aided by the use of direct assessments of clinical competence, e.g. the Cognitive Therapy Scale – Revised (CTS-R). We aimed to review the literature regarding inter-rater reliability and training on the CTS and CTS-R to present exploratory data on training raters to use this measure. We employed a systematic review. An exploratory study evaluated the outcomes of a CTS-R supervisor training workshop (n = 34), including self-reported familiarity with and confidence in using the tool, and inter-rater consistency on three CTS-R subscales, pre- and post-training. CTS and CTS-R inter-rater reliability was variable, with evidence of rater training enhancing reliability, although the form, duration and frequency of such training is unclear. The exploratory study found that supervisors rated themselves as more familiar with and confident in using the CTS-R at the end of training compared to at the beginning. However, inter-rater reliability was poor at the beginning and end of the training. Rating competence requires supervisors to make qualitative judgements, which is inherently variable. Training raters has been shown to improve rater reliability, although this was not demonstrated in the exploratory study. Practice implications and future research priorities are identified.

The Cognitive Therapy Scale is a well-established tool for assessing skills in delivering cognitive therapy, but has been subject to criticism. It has recently been updated by two groups, producing the Revised version or CTS-R (Blackburn, James, Milne and Reichelt, 2000) and a version designed for therapy of psychosis, the CTS-Psy (Haddock et al., 2001). The present study made a direct comparison of these scales to evaluate their inter-rater reliability, the extent to which they measure the same therapist qualities, and their utility for assessment of skills in trainee therapists working with different client-types. Twenty-six trainees submitted tapes of therapy with clients suffering either personality disorder or psychosis. Each tape was rated by two independent assessors on each of the two scales. Results suggest the scales are both fairly easy to use and produce highly similar estimates of student competence. Client diagnosis has no significant influence on the scores obtained by the therapist. However, inter-rater reliability is relatively low. It is concluded that safeguards are needed where these scales are used as a training outcome measure.

The existing scale for assessing competence in cognitive therapy (CTS) dates from 1988 and only the previous version of 1980 has been validated to any extent. A revised version, the CTS-R, was devised to improve on the CTS by: eliminating overlap between items, improving on the scaling system, and defining items more clearly. Kolb's well-known educational model was used as a guideline. In the new 14-item scale, three new items measure general therapeutic flair, the facilitation of emotional expression, and therapist's non-verbal behaviours (optional). We hypothesized that the CTS-R would prove more user friendly and demonstrate satisfactory reliability and validity. Twenty-one mental health professionals undergoing training in cognitive therapy provided 102 video-tapes of therapy with 34 patients, reflecting three stages of therapy. The tapes were rated by four expert raters, in a balanced design. The CTS-R showed high internal consistency and adequate average inter-rater reliability. Reliability for individual items varied widely among pairs of raters. Validity was demonstrated by improved ratings of competence for trainees who saw patients early and later during the course of training. Although raters found the CTS-R a more useful tool than the CTS and satisfactory reliability and validity were demonstrated, more refinement is needed in item definition. The study has led to modifications in the CTS-R, which are in the process of evaluation.