The effective delivery of digital CBT: a service evaluation exploring the outcomes of young people who completed video conferencing therapy in 2020

Abstract

Despite its impressive evidence base, there is a widening access gap to receiving cognitive behavioural therapy (CBT). Video conferencing therapy (VCT) offers an effective solution for logistical barriers to treatment, which has been salient throughout the Coronavirus pandemic. However, research concerning the delivery of CBT via VCT for children and young people (CYP) is in its infancy, and clinical outcome data are limited. The aim of this service evaluation was to explore the effectiveness of a VCT CBT intervention for CYP referred from Child and Adolescent Mental Health Services (CAMHS) in the UK. A total of 989 records of CYP who had completed CBT via VCT in 2020 with Healios, a digital mental health company commissioned by the National Health Service (NHS), were examined to determine changes in anxiety, depression and progress towards personalised goals. Routine outcome measures (ROMs) were completed at baseline and endpoint, as well as session by session. Feedback was collected from CYP and their families at the end of treatment. There was a significant reduction in symptoms of anxiety and depression and significant progress towards goals, with pre- to post-effect sizes (Cohen’s d) demonstrating medium to large effects (d=.45 to d=−1.39). Reliable improvement ranged from 31 to 80%, clinical improvement ranged from 33 to 50%, and 25% clinically and reliably improved on at least one measure; 92% reported that they would recommend Healios. This service evaluation demonstrates that Healios’ CBT delivered via VCT is effective for CYP receiving it as part of routine mental health care.

Key learning aims

1. (1) To consider whether CBT can be effectively delivered in routine care via VCT.

2. (2) To explore whether CBT delivered in routine care via VCT is acceptable to children, young people and their families.

3. (3) To reflect on the benefits of VCT and the collection of a variety of ROMs via digital platforms.

Introduction

In 2020, NHS Digital (2020) reported that one in six (16%) children of school age (5 to 16 years) were identified as having probable mental health problems, an increase from one in nine (10.8%) reported in 2017. If left untreated, these psychological problems can cause distress that often continues into adulthood (Copeland et al., 2015; Woodward and Fergusson, 2001). Despite this evident need, up to 80% of children and adolescents who suffer from internalising disorders (anxiety, depressive and somatic symptoms) do not receive appropriate treatment (Bekker et al., 2017; Creswell et al., 2014; Collins et al., 2004; Cobham, 2012). CBT is an effective treatment for a number of psychological conditions in children and young people (Wergeland et al., 2020), including anxiety disorders, obsessive-compulsive disorder (OCD) and depression (David-Ferdon and Kaslow, 2008; Ishikawa et al., 2007; Kreuze et al., 2018; Oud et al., 2019; Reynolds et al., 2012). Furthermore, CBT has been shown to be effective for behavioural and emotional problems for adolescents who receive it within routine care (Walter et al., 2018). Despite its impressive evidence base, of those who do receive treatment, many are not offered evidence-based therapies such as CBT (Comer and Barlow, 2014; McHugh and Barlow, 2010).

There are many barriers children and young people face in accessing mental health treatment. Barriers identified in a recent systematic review by Radez et al. (2020) included perceived stigma and embarrassment, limited mental health knowledge, difficulties trusting an unknown person, logistical barriers, and financial costs associated with treatment. Children and families may also experience additional difficulties; for example, parents may need to organise childcare for siblings whilst a child accesses treatment (Carpenter et al., 2018). Waiting times for child and adolescent mental health services (CAMHS) further pose a significant barrier to children and young people accessing mental health services in a timely manner (Children’s Commissioner, 2020), with pre-pandemic median average waiting times of two months (up to six months in some cases) to start treatment (Education Policy Institute, 2020), which is double the recommended four-week standard set out by the UK Government. The Education Policy Institute (2020), an independent research institute promoting high-quality education outcomes for children and young people, also reported that approximately one-quarter (26%) of children and adolescents’ referrals to specialist mental health services were not accepted in 2018–2019, primarily due to not meeting diagnosis eligibility criteria. This leaves a plethora of children not accessing the mental health treatment they require.

Innovative and improved ways of communicating with young people are needed to reach the 80% of children and young people not accessing therapy (Bekker et al., 2017). In particular, CBT-based alternatives to traditional clinic-based treatment may be effective at managing and preventing mental health problems (Bekker et al., 2017). One way to increase access to psychological interventions is through the use of video conferencing teletherapy (VCT), delivering therapy sessions via a live video meeting between the client and the therapist, through the medium of computers, tablets or mobile phones. Clients can access therapy sessions from their own private space, for example their home or school. VCT can overcome potential barriers to therapy, including geographic location, time and convenience, and limited access to transport (Carpenter et al., 2018; Harvey and Gumport, 2015; Murphy et al., 2020). VCT is an effective method of delivering treatment for a range of mental health difficulties (Osenbach et al., 2013; Peñate and Fumero, 2016; Wade et al., 2020), with symptom reduction reported as comparable to in-person therapy (Norwood et al., 2018). Recent systematic reviews of VCT by Berryhill et al. (2018, 2019) noted that the majority of studies reported significant reductions in symptoms of depression and anxiety, and studies that compared VCT with in-person therapy found no significant differences.

However, less is known about the efficacy and effectiveness of VCT with children and young people. Norwood et al. (2018) did not identify any studies with children and/or adolescents, and the systematic reviews by Berryhill et al. (2018, 2019) only included four studies (7%) which involved children and adolescents. These consisted of one pilot study (McLellan et al., 2017; n=16), two case studies (Ojserkis et al., 2014; n=1; Shealy et al., 2015; n=1) and one randomised trial study (Nelson et al., 2003; n=28).

As VCT has been used extensively to provide continued access to treatment during the global pandemic (Huscsava et al., 2020; McKenny et al., 2021; Wade et al., 2020), there is an urgent need for more research and evaluation of VCT for children and adolescents using appropriate outcome measures. Initial reports regarding the transition to telehealth during the COVID-19 pandemic are encouraging (Nicholas et al., 2021), but as far as the authors are aware, no large naturalistic evaluations of CBT interventions delivered via VCT have been conducted in populations of children and young people who are receiving it as part of routine out-patient mental health care.

Most existing studies use nomothetic approaches focusing on symptomatic improvement, but there is increasing recognition that young service users primarily value outcome measurement that is meaningful and tailored to their individual priorities, such as the development of autonomy and identity (Green et al., 2014; Stasiak et al., 2013). A Norwegian study (Lavik et al., 2018; p. 1) concluded that: ‘“Good outcomes” in youth mental health services should be understood as recovery oriented, sensitive to developmental phases, and based on the personal goals and values of each adolescent client’. Furthermore, young people are more likely to engage with therapy when the measures being used to determine the effectiveness of therapy are meaningful to them (Childs et al., 2013). A recent paper by Krause et al., 2021a) reported that consistent cross-domain improvement was observed in only 16% of 1641 cases from Child and Adolescent Mental Health Services (CAMHS) in England. Of the sample, 24% of young people with reliably improved symptoms reported no reliable improvement in functioning, and 35% reported meaningful goal progress but no reliable symptom improvement. However, 70% had improved on their goal-based outcomes (GBO), which appear to encapsulate meaningful change to the client in a way that symptom-based measures do not. Krause et al. (2019, Krause et al., 2021a) call for the adoption of more comparable outcome measures and recently published recommendations on routine outcome measures (ROMs) for children and young people with anxiety, depression, obsessive compulsive disorder, and post-traumatic stress disorder (Krause et al., 2021b). They suggest that future research should investigate ROMs in a multi-dimensional fashion, in order to be able to pick up on the nuances of how and when treatment is successful, in order to determine what the recipe is for therapeutic success (Krause et al., 2019).

Outcome data in VCT are in their infancy within the child and adolescent population. Therefore, the aim of this service evaluation was to explore the effectiveness and acceptability of CBT delivered via VCT within the Healios service, which is delivered as part of routine NHS out-patient mental health care for children and young people in the UK. This service evaluation uses ROMs to capture change in anxiety, depression, emotional wellbeing and progress towards personalised goals following the intervention.

Method

Ethical considerations

As service evaluations are excluded from ethical review in the United Kingdom (UK) (NHS Health Research Authority, 2020), we consulted the UK-based Health Research Authority (HRA) online decision-making tool, which confirmed that ethical review was not required. This decision was further reviewed by Healios’ Information Governance team, who approved the study and confirmed that ethical review was not required, provided that all data were pseudo-anonymised for analysis.

Study design

We undertook a service evaluation for our Child & Young Person’s (CYP) CBT pathway using pseudo-anonymised clinical records with ROMs and satisfaction measures being used to determine effectiveness and acceptability, respectively.

Sample

This sample consisted of 989 children and young people who received at least one session of CBT with Healios, an online digital mental health company commissioned by the NHS, to deliver therapy sessions to children and young people referred by CAMHS, between 2 January 2020 and 31 December 2020. Mean age at referral was 14.90 years (SD=2.20, 67% females). The children and young people had not formally been diagnosed with anxiety or depression; however, scores on the Young Person’s Clinical Outcomes in Routine Evaluation (YP-CORE; Twigg et al., 2009) demonstrated that 83% of the sample were within the clinical range at baseline, 61% of the sample scored in the clinical range at baseline on the Revised Children’s Anxiety and Depression Scale (RCADS-47; Chorpita et al., 2000) and 41% were within the clinical range at baseline on the RCADS-25 (Spence, 1997).

All children and young people in this sample met Healios’ inclusion criteria: access to the internet; a good level of conversational English (unless a translator was arranged by NHS referrers); aged 7 years or older. Referrals who met the following criteria (Healios’ exclusion criteria) were not accepted on to the service: active self-harm requiring medical intervention; suicidal intent within the last three months; co-morbid diagnosis of psychosis; an eating disorder as primary need; known trauma; known current alcohol or drug use; currently undergoing criminal investigation due to offending behaviour; current child protection concerns/investigation; young people with autistic spectrum conditions (ASC) where anxiety is part of ASC and speech and language therapy/occupational therapy might be a priority.

Evaluation design

Aim (i) – to consider whether CBT can be effectively delivered in routine care via VCT

In line with Healios’ protocols, online versions of the following routine outcome measures were completed to evaluate the service:

Outcome measures

Revised Children’s Anxiety and Depression Scale (RCADS; Chorpita et al., 2000; Spence, 1997). The RCADS is a youth self-report questionnaire with subscales for anxiety and depression. The RCADS-47 (Chorpita et al., 2000) is a 47-item questionnaire, and the RCADS-25 (Spence, 1997) is a shortened (25-item) version of the RCADS-47. Questions are scored on a 4-point Likert scale ranging from 0 to 3, with a maximum raw score of 141 on the RCADS-47 and 75 on the RCADS-25. The clinical cut-off for the RCADS is a computed t-score >70 (Chorpita et al., 2000). Healios’ measures procedures changed during 2020, with RCADS-47 replacing RCADS-25. Cronbach’s alpha (α) for the RCADS-25 (α=.92; Muris et al., 2002), and the RCADS-47 (α=.96; Donnelly et al., 2019) demonstrate ‘excellent’ internal consistency. The RCADS has ‘good convergent validity’ (Bouvard et al., 2015; Esbjørn et al., 2012; Donnelly et al., 2017, as cited by the Child Outcomes Research Consortium, n.d).

Young Person’s Clinical Outcomes in Routine Evaluation (YP-CORE; Twigg et al., 2009). The YP-CORE is a self-report questionnaire for young people with 10 items scored on a 5-point scale covering frequency of anxiety, depression, trauma, physical problems, functioning and risk to self ‘over the last week’, which can be used as a session-by-session monitoring tool. Scores can range from 0 to 40, with a clinical cut-off score of 14 (Wolpert et al., 2016). This measure has ‘good’ internal consistency (α=.85, Twigg et al., 2009). The YP-CORE has ‘considerable clinical face value, is valid and reliable, and is sensitive to change’ (Twigg et al., 2009, as cited in Twigg and McInnes, 2010).

Goal-based outcomes (GBO; Law, 2019; Law and Jacob, 2015). GBO assess progress on up to three idiosyncratic therapy goals rated from 0 (no progress) to 10 (goal achieved), with goal progress being tracked between first rating and final rating. Goal-based outcomes demonstrate ‘good’ internal consistency (α=.83, Pearce et al., 2017).

Aim (ii) – to explore whether CBT delivered in routine care via VCT is acceptable to children, young people and their families

Friends and Family Test (FFT; NHS England, 2014). The FFT is a tool to measure recommendation of services to family and friends together with qualitative feedback, regarding what the service did well and opportunities for improvement. It was administered to all children, young people and their families at the end of treatment.

Procedure

Sessions were usually held weekly and included completion of ROMs, with session-by-session measurements taken for the YP-CORE and GBO. Baseline and end-point measures were taken for the RCADS. Goals were typically set within the first three sessions, in accordance with guidance from Law and Jacob (2015) and Law (2019). Goals were rated in each session by the child or young person in collaboration with their clinician. Following the course of treatment, children, young people and their families were invited to complete the FFT. All ROMs and questionnaires were completed via Healios’ secure online portal.

Intervention

This project was run in collaboration between Healios and 23 NHS Children and Adolescent Mental Health Services (CAMHS). Healios is a digital mental health company established in 2013 to deliver online therapeutic services. Services are commissioned by the NHS and provided free at the point of delivery. Videoconferencing CBT sessions were delivered via Healios’ secure online portal (‘Panacea’) and were recorded with client and clinician consent. Patient ROMs, video sessions and notes were stored on the secure online platform and all data were pseudo-anonymised for analysis. Within online sessions, Healios clinicians utilised multimedia, semi-structured clinical materials known as ‘decks’ which were developed in-house. These ‘decks’ were shared online during sessions, enabling children and young people to engage with virtual whiteboards, emojis and sliding scales in real time. Decks were used to guide therapeutic conversations, presenting different elements of CBT with specific content at the clinician’s discretion, depending on the individual needs of the client. The standard 10-session Healios CBT package typically included content such as: formulation, setting goals, coping strategies, psychoeducation, exposure tasks, managing unhelpful thoughts, problem solving and relapse prevention. The CBT decks provide structure and promote fidelity to our National Institute for Health and Care Excellence endorsed CYP CBT manual (NICE, 2017). Although most children and young people receive the 10-session intervention, additional sessions can be offered if required, and fewer sessions can also be provided. All CBT sessions were conducted by accredited CBT therapists or clinical psychologists qualified to deliver CBT, all of whom received individual and group supervision on a weekly basis.

Analyses

Data analyses were conducted in Excel (Microsoft 365, Apps for business, version 2110, Build 13628.20380) and JASP (version 0.14). Descriptive statistics and paired t-tests were performed to investigate change in ROMs between baseline and post-intervention. Cohen’s d effect sizes were also calculated. Paired data were based on the first and the last scores if measurements were taken at multiple time points. Baseline values were only used if taken in sessions 1 to 3. Post-intervention measures were only included if taken from session 5 onwards. In order to determine the effectiveness of the intervention, children and young people who had received fewer than five sessions were removed from the analysis.

Cohen’s d pre-to-post effect sizes with 95% confidence intervals (CIs) were computed for symptom measures and progress towards goals. Cohen’s d effect sizes can be interpreted as small (d=0.2), medium (d=0.5), and large (d=0.8) based on benchmarks proposed by Cohen (1988).

The reliable change index (RCI) was used to estimate whether the change in score was great enough to be considered statistically significant on an individual level, with cases being classed as ‘reliably improved’, ‘reliably deteriorated’ or ‘no reliable change’ (Jacobson and Truax, 1991). The RCI for each measure was calculated using the standard deviation from the current sample and Cronbach’s alpha reliability from previous research. RCI calculations were performed using the Acorn Organization Inc. RCI calculator (http://www.psychoutcomes.org/OutcomesMeasurement/ReliableChangeIndex).

Previous research has defined the RCI for GBO as movement of ±2.45, aggregated across three goals (Edbrooke-Childs et al., 2015). However, in the current service evaluation only one goal was selected per person for analysis, due to the way in which the data extraction was coded. In order to not over-estimate our reliable improvement we first selected any goal rating where the client had deteriorated, otherwise the first goal that showed improvement was selected. We were unable to change this data extraction algorithm; however, we ensured that we did not over-estimate goal improvement in our sample by computing an alternative, more conservative meaningful reliable change index (RCI of 2.6 points rather than 2.45 points) that was not restricted to having to have an aggregated score of at least three goals. This procedure minimised any potential for bias and ensured scientific rigour.

Clinical and reliable improvement occurred when the change in scores on the measure indicates both reliable change and clinical improvement (i.e. crossing the clinical threshold to move from the clinical to non-clinical range). For a case to be considered clinically improved using multiple measures, we followed the standard recommendations (Gyani et al., 2013; Wolpert et al., 2016) whereby children and young people needed a baseline measure above the clinical threshold in at least one measure that subsequently moves into the non-clinical range at final measure, with all measures being below the clinical threshold at end-point. For clinical and reliable improvement on multiple measures, the client’s scores had to show both clinical and reliable improvement on at least one measure, with no clinical or reliable deterioration on any measure and all measures finishing below the clinical threshold. A quantitative analysis of satisfaction with the service was also performed using the FFT data, and a further analysis was conducted to explore if scores on a symptom measure were associated with acceptability of the intervention.

Results

Data were available for n=989 clinical records. The mean number of sessions held was 7.81 (range 1–21; mode = 10). If baseline measures were collected after session 3, these scores were not included in the analysis (RCADS-25: 29; RCADS-47: 36; YP-CORE: 65), leaving n=967 (98%) client records with a baseline in at least one measure. Final measure scores that were collected prior to session 5 were not included in the analysis (RCADS-25: 13; RCADS-47: 5; YP-CORE: 131; GBO: 22). After removing these scores, there were 717 (74%) children and young people who had a paired score in at least one measure. Healios’ measures procedures changed during 2020, replacing the RCADS-25 with the RCADS-47. As 10 children and young people had baseline and final scores on both the RCADS measures, their RCADS scores were subsequently removed from the dataset.

The difference between baseline and final measure scores were converted into z scores, and any score that was ±3 was determined to be an outlier. A number of outliers were detected in the data (RCADS-25: 4; RCADS-47: 1; YP-CORE: 5; GBO: 3) and further investigation revealed that one client record had corrupt scores for the YP-CORE which were subsequently removed from the dataset. The paired samples t-test analysis was run with and without outliers, and as these analyses were virtually identical, all subsequent analyses included outliers. All other assumptions for paired samples t-tests were met.

Paired samples t-tests were conducted on pre- and post-intervention scores (paired scores) for each measure (see Table 1). These indicated a statistically significant change in the desired direction across all measures, demonstrating a medium to large effect in symptom measures, and a large effect for goals.

Table 1. Reliable change, pre- and post-intervention means, standard deviations, t-scores, significance and effect sizes across the four routine outcome measures (n=989)

RCI (Cronbach’s alpha) and reference: ^a11.7 (.92), Muris et al. (2002); ^b8.6 (.96), Donnelly et al. (2019); ^c8.8 (.85). Twigg et al. (2009); ^d2.6 (.83), Pearce et al. (2017); CI, confidence interval; LL, lower limit; UL, upper limit.

A sub-analysis of those with a pre- and post-intervention score (paired score) who were clinical at baseline was conducted (see Table 2). This analysis revealed reductions on symptom measures and larger effect sizes when compared with the total sample analysis reported above.

Table 2. Reliable change, pre- and post-intervention means, standard deviations, t-scores, significance and effect sizes across three routine outcome measures for those who had a baseline score within the clinical range for RCADS-25, RCADS-47 or YP-CORE (n=538)

RCI (Cronbach’s alpha) and reference: ^a11.7 (.92), Muris et al. (2002); ^b8.6 (.96), Donnelly et al. (2019); ^c8.8 (.85), Twigg et al. (2009); CI, confidence interval; LL, lower limit; UL, upper limit.

Reliable and clinical improvement

Figure 1 shows the results of all measures combined. On at least one ROM, n=967 (98%) children and young people had baseline scores, and n=717 (74%) of these children and young people had a paired score, demonstrating a high completion rate of baseline and paired measures. Of those with a paired score, n=512 (71%) reliably improved on at least one ROM, whilst not reliably deteriorating on any other measure.

Figure 1. Flow diagram to illustrate clinical and reliable change across all measures (RCADS-25; RCADS-17; YP-Core; GBO).

The statistics for clinical improvement and clinical and reliable improvement are based on the subgroup of children and young people who had a baseline score above the threshold for a clinical disorder (n=538; 75%). Clinical improvement was determined by movement from above the clinical threshold at baseline, to below the clinical threshold post-intervention on at least one measure, with all measures being below the clinical threshold at final measure. On this basis, n=166 (31%) of children and young people showed clinical improvement after completing a course of CBT, with n=135 (25%) showing both reliable and clinical improvement. Of the total sample (n=717), n=46 (9%) showed evidence of clinical deterioration on at least one measure.

Table 1 provides an overview of each measure including its associated RCI, the number of those with paired scores, reliable improvement and deterioration. The majority (80%) showed reliable improvement on personalised goals as measured by GBO. Thirty-four (5%) young people reliably deteriorated on one or more measures and did not show reliable improvement on any measure. A sub-analysis of those who started in the clinical range on the YP-CORE, RCADS-25 or RCADS-47 revealed a larger proportion of children and young people demonstrated reliable improvement than the total sample, with 48% reliable improvement on the RCADS-25 in the sub-analysis compared with 31% in the total sample (see Table 2).

The results in terms of clinical improvement on each measure (except for GBO, which does not have a clinical cut-off) are provided in Table 3. Over a third of our sample demonstrated clinical improvement. Despite fewer baseline scores starting in the clinically significant range, a larger percentage of cases clinically improved on RCADS-25 (50%). The penultimate row in Table 3 represents the number of people who showed both reliable and clinical improvement in each measure.

Table 3. Clinical improvement (‘recovery’) across the four routine outcome measures (n=989)

* Percentages are based on those who started in the clinical range. RCI (Cronbach’s alpha) and reference: ^a11.7 (.92), Muris et al. (2002); ^b8.6 (.96), Donnelly et al. (2017); ^c8.8 (.85), Twigg et al. (2009).

A further investigation of those that clinically deteriorated on at least one symptom measure (n=46) revealed that eight (17%) reliably deteriorated on all measures. However, 32 (70%) reliably improved on at least one other measure, and three (7%) clinically improved on one other measure, whilst three (7%) showed no clinical or reliable change on any other measure.

Family and Friends Test (n=374)

In order to measure acceptability of VCT, children, young people and their parents/carers were invited to complete the FFT at the end of the intervention. The data from the 374 respondents were linked to 329 cases (of whom 297 had completed at least 10 sessions). Of these respondents, 344 (92%) reported that they would recommend Healios (241, 64% would ‘certainly recommend’ Healios; 103, 28% would ‘partially recommend’ Healios); 24 (6%) said that they ‘would not recommend’ Healios, and six (2%) were unsure whether they would recommend Healios.

FFT and YP-CORE

We explored whether acceptability of VCT (as measured by the FFT) was associated with scores on the YP-CORE (see Table 4). Of those who completed the FFT and had a paired score on the YP-CORE (n=304), the majority would recommend Healios (n=283, 93%), despite n=192 (63%) showing no reliable change on the measure. Of those who reliably deteriorated on the YP-CORE (and had responded to the FFT survey, n=10), the majority (n=9, 90%) reported they would recommend Healios, and nobody in this category stated they would not recommend Healios.

Table 4. Family and Friends Test (FFT) and reliable improvement on YP-CORE (n=304)

Discussion

This service evaluation has demonstrated that CBT delivered via videoconferencing for children and young people in routine care can be acceptable and effective. The majority of children and young people, seen pre- and during the pandemic, reliably improved on their idiosyncratic therapeutic goals, and 31–38% had a reliable reduction in reported symptoms of anxiety and depression. Effect sizes were comparable to those reported in Walter et al. (2018), who found CBT to be effective in treating adolescents with mental health conditions in routine clinical care when provided in-person.

All final mean scores moved in the desired direction, with final mean scores on the RCADS-25 and RCADS-47 below the clinical threshold at the end of the CBT intervention. It was encouraging to note that the measure which is particularly personally meaningful to children and young people, the GBO (which tracks progress towards their individualised goals), demonstrated the largest reliable improvement (80%) and the largest effect size (d=−1.39), which corroborates findings in the wider literature such as Krause et al., 2021a, 2021b).

The sub-analysis of those that were clinical at baseline on at least one symptom measure demonstrated greater effect sizes than our total sample analysis. This is in line with Walter et al. (2018) who also found that effect sizes increased when only analysing those who started above the clinical threshold. It should also be noted that there was greater reliable improvement for those who started in the clinical range on these measures. This is unsurprising as children and young people who start in the clinical range have higher initial scores and therefore a greater likelihood of improvement (Wolpert et al., 2016).

Our clinical and reliable improvement (‘reliable recovery’) rates are consistent with previous research which demonstrates a modest improvement in outcomes for young people receiving treatment as part of routine out-patient mental health care (Bear et al., 2020; Warren et al., 2010; Wolpert et al., 2016; Wolpert et al., 2012).

It should be noted that whilst clinical improvement for individual measures ranged from 33 to 50%, only 25% achieved clinical and reliable improvement on at least one ROM (ranging from 27 to 37%). This is comparable to the 27% ‘reliable recovery rate’ (clinical and reliable improvement) reported by Wolpert et al. (2016) in a cohort of 5896 children with paired measures who received routine out-patient mental health treatment. Understanding ‘recovery’ or ‘clinical improvement’ is challenging, especially if multiple ROMs are used, and this analysis only involves the subgroup who had initial scores above the clinical threshold. This may increase the likelihood of this metric being affected by natural variation and ‘regression to the mean’, as initial higher scores demonstrate a greater opportunity for improvement (Wolpert et al., 2016).

It is also interesting to note that, to our knowledge, other papers do not report clinical deterioration (the movement of those who are below the clinical threshold at baseline into the clinical range at subsequent measurements). We think it is helpful for routine service evaluations to include these data and to consider and investigate the factors related to this. Our analyses investigating those that clinically deteriorated on at least one symptom measure found that 70% of that sample demonstrated reliable improvement on at least one other measure, suggesting that clinical deterioration does not necessarily mean no progress was made during the intervention. Rather, this reflects that different constructs are being measured, and that change is multi-dimensional (Krause et al., 2021a). In particular, symptom improvement and goal attainment are distinct measures of therapeutic change, and require very different foci for therapy, and it is clear that most young people value the latter (Lavik et al., 2018).

The results from this service evaluation also demonstrate that children and young people found the intervention highly acceptable, and the majority would certainly recommend it. Furthermore, our explorative analysis of the FFT and the measure with the most paired scores (YP-CORE) revealed that, of those who had no reliable change (n=192) and those that reliably deteriorated on this measure (n=10), the majority (n=182, 90%) reported they would still recommend Healios. This suggests that children and young people may be gaining value from the intervention that is not reflected by symptom reduction, and despite not evidencing progress on this measure the intervention was still highly acceptable.

The high level of acceptability echoes previous research reporting high levels of overall satisfaction in families that received VCT (Carpenter et al., 2018; Pesämaa et al., 2004). These high rates of acceptability may be due to the advantages of VCT reported in the wider literature. For example, interventions can be accessed from the young person’s home, with no need to travel, and VCT overcomes geographical barriers, and reduces costs and time (Thomas et al., 2021). VCT can be economical for families including reducing costs associated with transport, parking and childcare for siblings (McKenny et al., 2021). Moreover, Radez et al. (2020) reported that young people wished to access therapy in an autonomous manner and were concerned about stigma, and both of these concerns are reduced by VCT. They also noted that some young people found it hard to talk about their feelings in-person and may prefer computerised psychological treatments. VCT may further bring some benefits for clinicians. A recent study (McKenny et al., 2021), investigating clinicians’ views of conducting therapy online (due to social distancing measures in 2020), found that VCT also gave the clinician an opportunity to gain insight into the client’s home, which was noted to be beneficial in facilitating the therapeutic process.

Whilst other services have recently moved online due to the COVID-19 pandemic, Healios is a digital company and therefore the resources for clinicians to use during clinical sessions are specifically designed to work online. For example, our CYP CBT pathway clinicians have access to interactive slides which allow children and young people to not only engage verbally via the video link, but also with whiteboards, emojis and sliding scales. This form of VCT may be particularly salient to children and young people who may sometimes struggle to communicate verbally about their anxieties and emotions. Moreover, as a digital company, all ROMs were already available for online completion, which may have enabled us to collect a high proportion of paired data. This appears to be in line with young people’s preferences – for example when comparing electronic outcome measures with paper-based versions for children and adults with asthma, Bushnell et al. (2003) reported that 77% of young people in their study preferred electronic outcome measures. Similarly, Hall et al. (2014) reported that children, young people and their families responded positively to electronic measures within CAMHS services.

However, our online ROMs collection procedures could also have resulted in some children and young people completing the measures without sufficient care and attention; e.g. they may have rushed their answers in order to access a session for which they were running late. We are therefore focusing attention on encouraging clinicians to better integrate ROMs within the clinical process, using them to collaboratively monitor progress.

These results may have been impacted by the COVID-19 pandemic. Recent research has demonstrated that living through lockdowns and social distancing measures has had a detrimental effect on the mental health of children and young people (O’Sullivan et al., 2021; Ravens-Sieberer et al., 2021). Although therapy may have been of benefit, the environmental constraints may have limited the improvements that some young people could achieve.

Implications and recommendations

This retrospective service evaluation utilising multiple ROMs has highlighted that delivering CBT for children and young people via video conference can be acceptable and effective within routine clinical practice.

This evaluation will inform the development of the Healios outcome strategy and the development of data informatic systems which allow us to link individual outcomes at the level of subscales to process and input factors such as demographics and symptom profile. It will also help us to identify the optimal combination of measures to use to assess outcomes, including the use of clinician- and parent-reported outcomes.

An immediate benefit of this service evaluation has been the identification of variability in practice regarding the collection of outcome measures, which has led to training and supervision initiatives: in particular, the need to collect baseline ROMs and set goals prior to session 3, and to make collection of paired scores more robust. Moreover, a qualitative investigation of the FFT on this cohort is underway to further inform and develop our services.

Limitations

Firstly, it has to be acknowledged that service evaluations do not have any comparison conditions. The lack of a comparison group means we cannot conclusively determine that the changes observed were due to the treatment administered and therefore comparison with previous literature needs to be done with caution.

Furthermore, the current service evaluation did not include clinician assessed outcomes, parent/guardian assessed outcomes, or teacher ratings. Outcomes were therefore based solely on the children and young peoples’ self-reported change, and consequently lacked multi-informant data and the combination of multiple perspectives recommended when monitoring outcomes (Terrelonge and Fugard, 2017). Goal-based outcomes were, however, collaboratively rated during therapeutic sessions. We hope to include clinician and parent ratings in future research, and aspire to integrate parent/guardian feedback in our delivery.

It should be noted that we used previously published Cronbach’s alpha scores to calculate our reliable change indices because we did not have individual item scores to calculate internal reliability within the current sample. We calculated all RCIs for consistency as previously published RCIs were only available for some of the measures. Nevertheless, our reliable change indices were comparable to RCIs that have been published (Twigg et al., 2016; Wolpert et al., 2016), giving us confidence that our RCI calculations were sufficient.

Finally, our service evaluation included all the children and young people who had been referred and had received at least one session, which means that we are not able to consider those that might have been referred but not engaged with any sessions. There was also a high level of missing data (n=272), which could mean we are not sufficiently capturing the true effectiveness of this intervention. However, this service evaluation has high ecological validity, and a comparatively high completion rate of paired ROMs (ranging from 46 to 96%) when compared with Morris et al. (2021), where only 3% of CAMHS cases completed paired measures (SDQ) in a study examining 28,382 clinical records, and Gibbons et al. (2021) who reported 46–60% paired outcome data in at least one measure (RCADS and SDQ; n=2246).

As we defined the final measure as taken from session 5 onwards, this sample will include some young people who completed therapy but did not require the 10 sessions routinely allocated, as well as people who did not fully complete treatment. The dataset did not allow us to explore this further, but this may have had an impact on the results, in particular on the YPCORE and GBO session-by-session measures. Future research could investigate changes over time using these measures, factors that appear to lead to premature termination of treatment and ways to encourage engagement.

Conclusion

To our knowledge this is one of the first naturalistic studies to demonstrate that CBT delivered via interactive VCT is effective in reducing symptoms of anxiety and depression in young people receiving it as part of routine mental health care. Previous research trials have demonstrated that VCT is an efficacious way to deliver therapy, and that the reduction in symptoms of anxiety and depression is comparable to in-person treatment (Berryhill et al., 2018; Berryhill et al., 2019; Norwood et al., 2018). Moreover, our results demonstrate that VCT is acceptable to children, adolescents and their families. This may be a potential way to offer an effective solution to the widening gap children and adolescents experience in accessing evidence-based psychological treatments.