Hostname: page-component-cd9895bd7-mkpzs Total loading time: 0 Render date: 2024-12-26T03:30:45.518Z Has data issue: false hasContentIssue false

Internet-Delivered Cognitive Therapy for Social Anxiety Disorder: A Development Pilot Series

Published online by Cambridge University Press:  16 May 2013

Richard Stott
Affiliation:
Institute of Psychiatry, Kings College London, UK
Jennifer Wild
Affiliation:
University of Oxford, UK
Nick Grey
Affiliation:
South London and Maudsley NHS Foundation Trust, UK
Sheena Liness
Affiliation:
Institute of Psychiatry, Kings College London, UK
Emma Warnock-Parkes
Affiliation:
Institute of Psychiatry, Kings College London, and University of Oxford, UK
Siobhan Commins
Affiliation:
Institute of Psychiatry, Kings College London, UK
Jennifer Readings
Affiliation:
Institute of Psychiatry, Kings College London, UK
Georgina Bremner
Affiliation:
Institute of Psychiatry, Kings College London, and University of Oxford, UK
Elizabeth Woodward
Affiliation:
University of Oxford, UK
Anke Ehlers
Affiliation:
University of Oxford, UK
David M. Clark*
Affiliation:
University of Oxford, UK
*
Reprint requests to David M. Clark, Department of Experimental Psychology, University of Oxford, South Parks Road, Oxford OX1 3UD, UK. E-mail: david.clark@psy.ox.ac.uk
Rights & Permissions [Opens in a new window]

Abstract

Background: Randomized controlled trials have established that individual cognitive therapy based on the Clark and Wells (1995) model is an effective treatment for social anxiety disorder that is superior to a range of alternative psychological and pharmacological interventions. Normally the treatment involves up to 14 weekly face-to-face therapy sessions. Aim: To develop an internet based version of the treatment that requires less therapist time. Method: An internet-delivered version of cognitive therapy (iCT) for social anxiety disorder is described. The internet-version implements all key features of the face-to-face treatment; including video feedback, attention training, behavioural experiments, and memory focused techniques. Therapist support is via a built-in secure messaging system and by brief telephone calls. A cohort of 11 patients meeting DSM-IV criteria for social anxiety disorder worked through the programme and were assessed at pretreatment and posttreatment. Results: No patients dropped out. Improvements in social anxiety and related process variables were within the range of those observed in randomized controlled trials of face-to-face CT. Nine patients (82%) were classified as treatment responders and seven (64%) achieved remission status. Therapist time per patient was only 20% of that in face-to-face CT. Conclusions: iCT shows promise as a way of reducing therapist time without compromising efficacy. Further evaluation of iCT is ongoing.

Type
Accelerated Publication
Creative Commons
Creative Common License - CCCreative Common License - BYCreative Common License - NCCreative Common License - SA
The online version of this article is published within an Open Access environment subject to the conditions of the Creative Commons Attribution-NonCommercial-ShareAlike licence . The written permission of Cambridge University Press must be obtained for commercial re-use.
Copyright
Copyright © British Association for Behavioural and Cognitive Psychotherapies 2013

Introduction

Social anxiety disorder is common (Kessler et al., Reference Kessler, Berglund, Demler, Jin, Merikangas and Walters2005), typically starts in childhood or adolescence (Kessler et al., Reference Kessler, Berglund, Demler, Jin, Merikangas and Walters2005) and is one of the most persistent of the anxiety disorders in the absence of treatment (Bruce et al., Reference Bruce, Yonkers, Otto, Eisen, Weisberg and Pagano2005). Clearly, there is a need for effective and efficient psychological treatments. Randomized controlled trials (RCTs) have established that individual cognitive behaviour therapies (Clark et al., Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006; Ledley et al., Reference Ledley, Heimberg, Hope, Hayes, Zaider and Dyke2009), group cognitive behaviour therapies (Alden and Taylor, Reference Alden and Taylor2011; Hofmann, Reference Hofmann2004; Hope, Heimberg and Bruch, Reference Hope, Heimberg and Bruch1995; Rapee, Abbott, Baillie and Gaston, Reference Rapee, Abbott, Baillie and Gaston2007), exposure therapy (Clark et al., Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006; Hope, Heimberg and Bruch, Reference Hope, Heimberg and Bruch1995), interpersonal psychotherapy (Stangier, Schramm, Heidenreich, Berger and Clark, Reference Stangier, Schramm, Heidenreich, Berger and Clark2011) and brief psychodynamic therapy (Leichsenring, Reference Leichsenring2012) are all effective when compared with no treatment. Several of the cognitive-behaviour therapies (Clark et al., Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006; Heimberg et al., Reference Heimberg, Dodge, Hope, Kennedy, Zollo and Becker1990, Reference Heimberg, Liebowitz, Hope, Schneier, Holt and Welkowitz1998; Davidson et al., Reference Davidson, Foa, Huppert, Keefe, Franklin and Compton2004; Rapee, Gaston and Abbott, Reference Rapee, Gaston and Abbott2009) have also passed a stricter, comparative efficacy test in the sense that they have been shown to be superior to one or more alternative psychological and/or drug treatments.

Although effective psychological treatments exist, many individuals with social anxiety disorder do not receive them. Poor recognition of social anxiety disorder by health professionals, a scarcity of suitably qualified therapists, and the initial reticence of some people with social anxiety disorder to disclose personal information to strangers are all likely to contribute to the current under-utilization of treatment. The last two factors (insufficient therapists and patient reticence) could partly be overcome by the development of effective internet-based versions of CBT for social anxiety disorder. Although internet-based psychological treatments tend to be more effective when accompanied by regular support from a clinician (Palmqvist, Carlbring and Andersson, Reference Palmqvist, Carlbring and Andersson2007) the total amount of therapist time is typically much less than that involved in traditional face-to-face therapy. An individual therapist could therefore treat considerably more patients using internet-based CBT. Social psychological research suggests that the internet could also be an effective way of starting therapy with individuals who are initially fearful of the face-to-face contact involved in traditional treatment. In particular, Weidman et al. (Reference Weidman, Fernandez, Levinson, Augustine, Larsen and Rodebaugh2012) reported that, compared to low socially anxious individuals, high socially anxious individuals feel more comfortable self-disclosing on the internet and less comfortable self-disclosing face-to-face.

Researchers in Sweden (Andersson, Carlbring, Furmark, Hedman and colleagues) and Australia (Titov and colleagues) have already developed internet-delivered CBT programmes for social anxiety that have proved appealing to patients and have been shown to be superior to no treatment (Andersson, Carlbring and Furmark, Reference Andersson, Carlbring and Furmark2012; Berger, Hohl and Caspar, Reference Berger, Hohl and Caspar2009; Carlbring et al., Reference Carlbring, Gunnarsdottir, Hedensjo, Andersson, Ekselius and Furmark2007; Furmark et al., Reference Furmark, Carlbring, Hedman, Sonnenstein, Clevberger and Bohman2009; Hedman et al., Reference Hedman, Furmark, Carlbring, Ljótsson, Rück and Lindefors2011; Titov, Andrews, Schwencke, Drobny and Einstein, Reference Titov, Andrews, Choi, Schwencke and Mahoney2008; Titov, Andrews and Schwencke, Reference Titov, Andrews and Schwencke2008; Titov, Andrews, Choi, Schwencke and Mahoney, Reference Titov, Andrews, Schwencke, Drobny and Einstein2008; Titov, Andrews, Johnston, Robinson and Spence, Reference Titov, Andrews, Johnston, Robinson and Spence2010). Two trials (Andrews, Davies and Titov, Reference Andrews, Davies and Titov2011; Hedman et al., Reference Hedman, Furmark, Carlbring, Ljótsson, Rück and Lindefors2011) have also reported that the improvements obtained with these internet-based therapies are not significantly different from those obtained with group CBT. While these results are extremely encouraging, there is scope for further evaluation and development of internet-delivered treatment. With respect to evaluation, there is a need to demonstrate that internet-delivered CBT is more effective than an internet-delivered placebo intervention. There is also a need to compare internet treatment with individual CBT, as recent studies suggest that individual treatment may be more effective than the group CBT programmes to which internet treatments have so far been compared. Turning to development, it is notable that existing internet-based CBT programmes have particularly focused on psycho-education, repeated exposure to feared situations, and cognitive restructuring using thought records. While these procedures are used in many face-to-face CBTs, they are not the main focus of individual cognitive therapy (CT) based on the Clark and Wells (Reference Clark, Wells, Heimberg, Liebowitz, Hope and Schneier1995) model. That treatment mainly focuses on other procedures (such as video feedback, attention training, behavioural experiments and dropping safety behaviours, memory discrimination and re-scripting) that are intended to reverse the maintaining factors specified in the model. The present study reports the development and piloting of a version of CT that implements all its key procedures via the internet.

We were interested to develop an internet version of CT as face-to-face CT is one of the first choice treatments recommended by NICE (Pilling et al., in press) and has shown particularly clear evidence of comparative efficacy. In particular, six RCTs conducted in the UK, Germany and Sweden have shown face-to-face CT to be superior to exposure therapy (Clark et al., Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006), two forms of group CBT (Mortberg, Clark, Sundin and Wistedt, Reference Mortberg, Clark, Sundin and Wistedt2007; Stangier, Heidenreich, Peitz, Lauterbach and Clark, Reference Stangier, Heidenreich, Peitz, Lauterbach and Clark2003), interpersonal psychotherapy (Stangier et al., Reference Stangier, Schramm, Heidenreich, Berger and Clark2011), brief psychodynamic psychotherapy (Leichsenring, Reference Leichsenring2012), selective serotonin re-uptake inhibitors (Clark et al., Reference Clark, Ehlers, McManus, Hackmann, Fennell and Campbell2003), medication-based treatment-as-usual (Mortberg et al., Reference Mortberg, Clark, Sundin and Wistedt2007), and pill placebo (Clark et al., Reference Clark, Ehlers, McManus, Hackmann, Fennell and Campbell2003).

Normally face-to-face CT is delivered in weekly therapy sessions over a period of 3–4 months and involves a total of around 20 hours of therapist contact. In the standard version of the treatment (Clark et al., Reference Clark, Ehlers, McManus, Hackmann, Fennell and Campbell2003, Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006, Reference Clark, Wild, Grey, Stott, Liness and Deale2012) therapy sessions are 90 minutes long. Sessions of this length are recommended to ensure that therapists can regularly conduct behavioural experiments (both in the office and outside) and have sufficient time to discuss the results of the experiments with their patients. A shorter session (55–60 minutes) version of CT has also been evaluated in trials (Leischenring, Reference Leichsenring2012; Mortberg et al., Reference Mortberg, Clark, Sundin and Wistedt2007; Stangier et al., Reference Stangier, Heidenreich, Peitz, Lauterbach and Clark2003, Reference Stangier, Schramm, Heidenreich, Berger and Clark2011). Although both versions have shown good comparative efficacy, inspection of the pre–post change on the Liebowitz Social Anxiety Scale (Liebowitz, Reference Liebowitz1987) suggests that the standard session version is associated with approximately 50% more change than the shorter session version. In developing an internet version of the treatment, we aimed to come close to the larger improvements associated with the standard version.

Method

Participants

A development cohort of 11 patients meeting DSM-IV (American Psychiatric Association, 2004) criteria for social anxiety disorder worked through the internet programme. They were recruited via a NHS clinic for patients with anxiety disorders. Seventeen patients with social anxiety disorder were assessed for possible inclusion in the development cohort. Reasons for exclusion were: social anxiety disorder not the main problem (1 patient); unable to commit to accessing the programme every week (2 patients); and borderline personality disorder (3 patients). Diagnostic interviews used the Anxiety Disorders Interview Schedule (ADIS; Brown, Di Nardo and Barlow, Reference Brown, Di Nardo and Barlow1994) for the Diagnostic and Statistical Manual of Mental Disorders (4th ed: DSM-IV) to identify social anxiety disorder and the Structured Clinical Interview for DSM-IV Axis I (SCID-I; First, Spitzer, Gibbon and Williams, Reference First, Spitzer, Gibbon and Williams1995) and Axis II disorders (SCID-II; First, Gibbon, Spitzer, Williams and Benjamin, Reference First, Gibbon, Spitzer, Williams and Benjamin1997) to identify co-morbid conditions. Diagnostic interviews were conducted by four clinical psychologists with extensive training in using the ADIS and SCID.

Internet programme description

The internet programme was developed with the following objectives in mind: (a) to provide patients with a secure, intuitive, motivating and effective website available 24–7 through which to engage in a course of cognitive therapy, with no requirement to meet their therapist in person; (b) to implement all key features of the Clark and Wells (Reference Clark, Wells, Heimberg, Liebowitz, Hope and Schneier1995) CT programme, with therapist support being provided electronically (by phone or e-mail); (c) to substantially reduce overall therapist time whilst increasing the frequency and availability of the therapist to provide guidance, feedback and encouragement; (d) to creatively utilise modern technology in the delivery of certain specific features of the treatment. Further details of the treatment implementation are outlined below. A short video illustration of the internet programme can be viewed at: http://youtu.be/rXXOOSkA0qg

Modules

The programme is delivered in a series of modules that were originally developed for a self-study assisted version of the treatment and have been further developed for use via the internet. The modules typically comprise educational text, patient testimonies, video illustrations, case examples, questions for patients to consider, and information boxes for them to record their answers, various types of monitoring sheets, behaviour experiments and other homework assignments. The modules fall into three categories: (1) Nine core modules that are assigned to all patients. These are “Introducing the Treatment”, “Getting Started”, “Feeling Self-Conscious”, “Safety Behaviours”, “My Safety Behaviour and Attention Experiment”, “Watching Your Conversation Videos”, “Behavioural Experiments”, “Getting Out of Your Head and Into the World”, and “My Therapy Blueprint”. (2) Problem-specific modules that the therapist only releases if they are relevant to the individual's particular fearful concerns. These are the following (with percentages of patients to whom they were assigned): “Going over Social Situations after they have Happened” (82%); “Updating my Key Images and Memories” (82%); “Feeling Boring” (64%); “Having Conversations” (64%); “Feeling Stupid” (45%); “Feeling Responsible for Others’ Enjoyment” (45%); “Worrying in Advance” (36%); “Blushing” (36%); “Sweating” (9%) and “Shaking” (0%). (3) Modules that focus on common obstacles to progress and are released by the therapist if required. These are (with percentages of patients to whom they were assigned): “Self-Esteem” (64%), “Giving Myself Credit” (64%), “Decatastrophizing” (36%) and “Managing My Mood” (0%).

Model conceptualization

Development of an individual version of the Clark and Wells (Reference Clark, Wells, Heimberg, Liebowitz, Hope and Schneier1995) model with the therapist is a key first step in face-to-face CT. The internet programme uses the same questions that a therapist would ask and guides the patient through a stepwise process that results in the generation of a graphical representation of the model. The patient is encouraged to review, edit or elaborate the model at any time, and can print it out for future reference.

Attention and safety behaviours experiment

In order to demonstrate to patients the adverse effects of self-focused attention and safety behaviours, face-to-face CT uses an in-session experiential exercise in which patients engage in a social task while alternately focusing on themselves and using safety behaviours or focusing externally and dropping key safety behaviours (see McManus et al., Reference McManus, Clark, Grey, Wild, Hirsch and Fennell2009 for an evaluation of this procedure). The internet programme implements the experiential exercise in full. A new dedicated module entitled “My Attention and Safety Behaviours Experiment” guides the patient through a series of brief webcam link-ups with another person. The programme automatically tabulates the patient ratings of key variables (e.g. subjective anxiety, beliefs about how one came across) and highlights differences between the conditions, as would an experienced therapist.

Video feedback is a powerful way of correcting negative self-perceptions of social behaviour and is used extensively in face-to-face CT. It has also been built into the internet version. The interactions in the attention and safety behaviours experiential exercise (and other webcam chats) are recorded so that the patient can play them back. A dedicated module entitled “Watching your Conversation Videos” coaches patients in how to review recordings of their social interactions in a manner that maximizes the chance that they will notice discrepancies between their negative self-perceptions and their actual performance. The programme facilitates learning by automatically tabulating pre- and postvideo feedback ratings. In addition, patients can opt to use their webcam to practise giving presentations to a video recording of an audience. A choice of nine audiences, varying in terms of the number of audience members (from 3 to 25) and their friendliness (from smiling and welcoming to bored and fidgety) can be selected by the patient each time they give a presentation. The programme records the presentation and allows the patient to view it afterwards.

Attention training

Systematic training in externally focused, non-evaluative attention is a prominent part of face-to-face CT. The internet programme includes two modules: (“Feeling Self Conscious” and “Getting Out of Your Head and into the World”) that help patients to learn how to focus externally in social situations. The former provides information about the negative effects of self-focused attention and encourages a shift to more externally focused attention. The latter provides a comprehensive training in externally focused attention and an “attention gym” for further practice.

Behavioural experiments are a key intervention in the Clark and Wells (Reference Clark, Wells, Heimberg, Liebowitz, Hope and Schneier1995) approach and are used in most face-to-face CT sessions. The experiments invariably require patients to engage in feared situations or activities. In face-to-face therapy, the therapist can relatively easily grade tasks and encourage the patients through modelling and other means. Helping patients to overcome their fears about behavioural experiments is more of a challenge in an internet programme. To help overcome this problem, we created with actors a large number of short videos (average 5 min) that illustrate how to set up and conduct behavioural experiments to test most fearful predictions. The videos are embedded in the relevant modules. A dedicated section of the website provides forms that allow patients to plan their experiments in advance and to record the outcomes and learning points. A running log of completed experiments is accessible at all times.

Discrimination training and imagery rescripting

Some patients are able to identify early, socially traumatic experiences that seem to be related to their current concerns and negative self-images. In face-to-face CT the therapist has the option of using one of two techniques to break the link between the past and present in such instances. The simplest is “discrimination training”. When this is not appropriate, or does not work, the more complex technique of “memory rescripting” (see Wild and Clark, Reference Wild and Clark2011) is available. Both techniques are available in the internet programme in the “Updating Your Key Images and Memories” module.

Blueprint

A core module entitled “My Therapy Blueprint” is released towards the end of therapy. The module helps patients to bring together the major points they have learned while working through the programme and to plan for how they would use this information to deal with any future setbacks (relapse prevention).

Score graphs

Each week patients complete a range of questionnaires covering social anxiety, other symptoms and relevant process variables (see details below). The programme automatically scores and graphs the questionnaires so patients can monitor their own progress.

Library

The programme contains a resource library where patients can access and watch again any of the videos that appear in modules. They can also read surveys of other people's attitudes to issues that they are concerned about (for example, blushing). The surveys help patients discover that other people are generally less critical of signs of anxiety and less perfectionistic about social performance than they would assume.

Therapist support

Patients are assigned a named therapist who supports them as they progress through the programme. They see a photograph of their therapist on the programme's main screen. When logged onto the programme, therapists can view all the modules that their patients have worked on, including the information patients have entered (such as negative thoughts, planned and completed behavioural experiments, conclusions following video feedback, and questionnaire scores). This enables therapists to make judgements about what patients have, or have not, understood and to follow through with suggestions. The therapist has several options for communicating with the patient. A within programme secure messaging system allows therapist and patient to correspond electronically. A once a week phone call is scheduled, usually lasting 10–15 minutes. Therapists may also send short mobile SMS messages to remind and encourage the patient about a particular assignment, when patients have signed up for this feature. Automatic SMS messages can also be programmed. Although the total amount of time that therapists spend communicating with a particular patient is much less than in face-to-face CT, the frequency of contact is often higher as therapists will usually communicate with their patients several times each week. It is hoped that this feature will ensure that patients feel closely supported and regularly encouraged, even though they are working on the internet. This way of working allows therapists considerable flexibility in terms of how they plan their day and manage their case-load.

Four clinical psychologists (JW, RS, EW-P, NG) supported patients through the internet treatment. All had extensive prior training and experience with face-to-face CT for social anxiety disorder.

Measures

Social anxiety symptoms

The primary outcome measure was the self-report version of the Liebowitz Social Anxiety Scale (LSAS; Baker, Heinrichs, Kim and Hofmann, Reference Baker, Heinrichs, Kim and Hofmann2002). Secondary measures of social anxiety were the Social Phobia Inventory (SPIN; Connor et al., Reference Connor, Davidson, Churchill, Sherwood, Foa and Weisler2000) and the Social Phobia Weekly Summary Scale (SPWSS; Clark et al., Reference Clark, Ehlers, McManus, Hackmann, Fennell and Campbell2003).

Social anxiety process measures

Several unpublished measures (Clark, Reference Clark2005) covering central processes in cognitive models of social anxiety (Clark and Wells, Reference Clark, Wells, Heimberg, Liebowitz, Hope and Schneier1995; Rapee and Heimberg, Reference Rapee and Heimberg1997) were administered and used to help guide therapy, as in face-to-face CT. The Social Cognitions Questionnaire (SCQ) is a 22-item scale measuring both frequency and belief ratings for a range of typical cognitions in social anxiety. The Social Behaviour Questionnaire (SBQ) is a 29-item scale measuring the frequency by which the patient engages in a range of common safety-seeking behaviours. The Social Attitudes Questionnaire (SAQ) is a 50-item scale measuring the conviction with which the patient endorses a range of attitudes often held by people with social anxiety.

General mood

Depression and general anxiety were assessed with respectively the Patient Health Questionnaire-Depression scale (PHQ-9; Kroenke, Spitzer and Williams, Reference Kroenke, Spitzer and Williams2001) and the GAD-7 (Spitzer, Kroenke, Williams and Löwe, Reference Spitzer, Kroenke, Williams and Löwe2006).

As mentioned above, therapists have full access to the patient's current and historical responses on all measures. This information can help the therapist decide which optional modules to release, as well as their order of priority. It can also inform the messages that therapists send to their patients. For example, this may be to recommend a particular behavioural experiment to test a certain cognition, or provide encouragement or feedback on a patient activity such as dropping a particular safety behaviour.

Criteria for treatment response and remission

Response

Patients were classified as treatment responders if they showed an improvement of greater than 31% on the LSAS between pretreatment and posttreatment. In a large scale (n = 1178) study, Bandelow, Baldwin, Dolberg, Andersen and Stein (Reference Bandelow, Baldwin, Dolberg, Andersen and Stein2006) showed that this definition is equivalent to the definition of responder status used in most medication trials. The latter generally use the Clinical Global Impression-Improvement (CGI-I) scale and define treatment responders as individuals who are rated as 2 “much improved” or 1 “very much improved”. Bandelow et al. (Reference Bandelow, Baldwin, Dolberg, Andersen and Stein2006) reported a high correlation (r = 0.85) between percentage change on the LSAS and CGI-I. The 31% cut-off on the LSAS identified as many responders as a CGI-I rating of 1 or 2.

Remission

Patients were classified as remitted if they met Jacobson and Truax's (Reference Jacobson and Truax1991) criteria for reliable and clinically significant change on the LSAS. This is a two-fold criterion. Pre–post change must both exceed the measurement error of the scale and move the individual into the distribution of the non-clinical population (mean ± 2 SD). Using Fresco et al.'s (Reference Fresco, Coles, Heimberg, Liebowitz, Hami and Stein2001) data for a non-clinical population, this equated to a pretreatment to posttreatment fall of at least 12 points on the LSAS and the posttreatment LSAS score of 38 or less.

Results

Characteristics of patients

Patients’ mean age was 33.1 years (SD 5.9) and the mean duration of the social anxiety disorder was 15.1 years (SD 9.4). All patients (100%) had the generalized subtype of social anxiety disorder and met the English Improving Access to Psychological Therapies (IAPT) caseness criteria (see Department of Health, 2011) on the SPIN. However, only six (55%) met the general IAPT caseness criteria (a PHQ-9 score of 10 or above, and/or a GAD-7 score of 8 or above at pretreatment). This reflects the well-publicized weakness of the general IAPT caseness criteria for identifying individuals with social anxiety disorder. Fifty-five percent of patients were male and 82% were Caucasian. Seventy-three percent were in employment (full or part-time), 18% were unemployed, and 9% were students. None were married, 45% were cohabiting, and 55% were living alone. Twenty-seven percent had school education only, 45% had obtained an undergraduate qualification, and 27% had obtained a Masters level degree. Sixty-four percent met diagnostic criteria for another axis-I disorder (36% current, 27% past only). Comorbid axis-I disorders were major depressive disorder (36%), past major depressive disorder (27%), alcohol abuse (9%), health anxiety (9%) and posttraumatic stress disorder (9%). Fifty-five percent met criteria for one or more personality disorders, which were avoidant (55%) and paranoid (9%). Forty-five percent of the group had previously sought treatment in connection with their social anxiety. Previous treatments comprised counselling (18%), cognitive-behavioural guided self-help (9%), long-term psychodynamic psychotherapy (9%), hypnotherapy (9%) or selective serotonin reuptake inhibitor (SSRI) medication (9%).

How patients used the internet programme

Patients spent a mean of 13.7 weeks in treatment (SD 4.0) during which time they spent a total of 34.9 hours on the website (SD 20.3). They planned a mean of 19.0 behavioural experiments (SD 18.1) using the website's dedicated behavioural experiment planner, and completed 17.9 behavioural experiments (SD 18.1). All patients used the web-chat facility for the attention and safety behaviours experiment that occurred early in therapy, and 55% used the facility on at least one other occasion. All patients reviewed their recordings with the aid of the “Watching Your Video” module. Thirty-six percent used the webcam presentation facility at least once in order to video-record, and subsequently view themselves speaking to a virtual audience. Patients messaged their therapist a mean of 19.3 times (SD 11.3). Nobody dropped out of treatment.

Therapist activity

Therapists sent an average of 30.8 (SD 16.5) secure messages to their patients through the website during the course of treatment. They also made a mean of 7.3 phone calls to each patient (SD 3.9), totalling 108 minutes (SD 59.1), over the course of treatment. They sent a mean of 12.2 mobile SMS messages to patients (SD 12.9), which were a mixture of personalized SMS messages and automatically scheduled SMS reminders. The mean time that therapists spent supporting each patient during treatment was 232 minutes (SD = 114.8 mins), which is 3.87 hours. This compares with a mean therapist time of 19.14 hours (1148 minutes) in a comparable course of face-to-face cognitive therapy (Clark et al., Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006).

Clinical outcome

Table 1 shows the outcome measures pre- and post-therapy. On the main outcome measure, which was the Liebowitz social anxiety scale (LSAS), patients achieved a mean drop of 40.2 points (SD 19.9). This represents a pre–post effect size of 1.64, which is considered large. The LSAS change is within the range of those reported for face-to-face CT in published trials. The two trials (Mortberg et al., Reference Mortberg, Clark, Sundin and Wistedt2007; Stangier et al., Reference Stangier, Schramm, Heidenreich, Berger and Clark2011) that used 55–60 minute therapy sessions both reported a mean LSAS improvement of 30 points for face-to-face CT. The two trials (Clark et al., Reference Clark, Ehlers, McManus, Hackmann, Fennell and Campbell2003, Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006) that used longer (75–90 min) sessions reported mean LSAS improvements of 43 and 46 points respectively.

Table 1. Outcome and process measures at pre- and posttreatment

Notes: Paired samples t-tests were conducted on all measures from pre- to posttreatment (n = 11). LSAS = Liebowitz Social Anxiety Scale (Liebowitz, Reference Liebowitz1987); SPIN = Social Phobia Inventory (Connor, Davidson, Churchill, Sherwood, Foa and Weisler, Reference Connor, Davidson, Churchill, Sherwood, Foa and Weisler2000); SPWSS = Social Phobia Weekly Summary Scale (Clark et al., Reference Clark, Ehlers, McManus, Hackmann, Fennell and Campbell2003); SCQ-F = Social Cognitions Questionnaire – Frequency subscale; SCQ-B = Social Cognitions Questionnaire – Belief subscale; SBQ = Social Behaviour Questionnaire; SAQ – Social Attitudes Questionnaire; PHQ-9 = Patient Health Questionnaire (Kroenke, Spitzer and Williams, Reference Kroenke, Spitzer and Williams2001); GAD-7 = Generalized Anxiety Disorder Screener (Spitzer, Kroenke, Williams and Löwe, Reference Spitzer, Kroenke, Williams and Löwe2006).

p < .10. *p< .05. **p< .01. ***p < .001.

Nine patients (82%) were classified as treatment responders and seven (64%) achieved remission status, using Jacobsen and Truax's (Reference Jacobson and Truax1991) criteria for reliable and clinically significant change on the LSAS. Six patients (55%) also met IAPT criteria for reliable recovery from social anxiety disorder using the SPIN and PHQ-9. With these criteria (see Department of Health, 2011) patients are classed as a recovered case if they score above the clinical threshold (19) for the SPIN at pretreatment, improve by at least 10 points on the SPIN and fall below the clinical threshold for both the SPIN and PHQ-9 at posttreatment.

Improvements on the secondary social anxiety outcome measures (SPIN and SPWSS) were also large. Effect sizes were 2.06 and 3.17 respectively. The SPWSS has been used in several trials of face-to-face CT (Clark et al., Reference Clark, Ehlers, McManus, Hackmann, Fennell and Campbell2003, Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006; Mortberg et al., Reference Mortberg, Clark, Sundin and Wistedt2007). The change in SPWSS observed with internet CT (2.5 points) was again within the range of that observed with face-to-face CT in the trials.

Process outcome

Table 1 also shows the process measures at pre-therapy and post-therapy. Significant and large improvements were observed on each process measure with effect sizes ranging from 1.68 (for safety behaviours) to 2.28 (for social anxiety related negative assumptions).

Discussion

Creating an internet-delivered version of cognitive therapy for social anxiety disorder was a challenge. Only a few years ago it would not have been possible. However, the recent roll-out of high speed broadband to many homes, as well as businesses, has enabled the internet to support features that are very helpful for delivering the treatment. Video-feedback can now be achieved on the internet through the use of a video chat facility with built in recording and playback features. We also found that the problem of motivating patients to conduct key behavioural experiments without their therapist being present could be at least partly overcome by the extensive use of video clips that illustrated how to plan and conduct the relevant behavioural experiments. Patients reported that the numerous video clips of actual patients who had successfully completed CT were also very helpful for engaging them in the treatment and helping to keep them motivated. However, as the pilot series progressed it became increasing clear to the team that regular, brief contact with a therapist via messaging and telephone was important for helping to ensure that the momentum of therapy was maintained. This is in line with the experience of other researchers (Palmqvist et al., Reference Palmqvist, Carlbring and Andersson2007).

The improvements in social anxiety and related process measures that were associated with internet treatment in the pilot cohort were within the range of those that have been reported for face-to-face CT in RCTs. Encouragingly, the mean change (40 points) on the main social anxiety measure (LSAS) was close to that observed with the standard 90 minute session protocol (45 points on average) and somewhat larger than that observed with the shorter session (55–60 min) protocols. However, the pilot cohort is small and there was no control condition. To more clearly establish the value of internet-delivered CT a randomized controlled trial is now underway in which internet-delivered CT is being compared with face-to-face CT and a no treatment (wait-list) control condition.

The pilot cohort completed all of the recommended outcome measures for the English Improving Access to Psychological Therapies (IAPT) initiative. Up to now, IAPT services have generally defined clinical caseness and recovery using a combination of the PHQ-9 and the GAD-7. However, the latest version of the IAPT Data Handbook (Department of Health, 2011) recommends that for social anxiety disorder the combination of the SPIN and the PHQ should be used to define clinical caseness at pretreatment and recovery at posttreatment. In particular, individuals should be classed as social anxiety disorder cases at pretreatment if they score above the clinical/non-clinical cut-off on the SPIN (irrespective of PHQ score) and should be subsequently defined as recovered if they score below the clinical cut-offs on both the SPIN and PHQ at post-treatment. Our findings support the recommendations of the IAPT Data Handbook. At pre-treatment, diagnostic interviews confirmed that all patients in the cohort met DSM-IV criteria for social anxiety disorder. Classification using the SPIN also coded all patients as cases. However, only six (55%) patients met the combined PHQ and GAD caseness criteria at pretreatment, presumably because extensive avoidance behaviour prevented some of the patients from scoring high on anxious affect (as assessed by the GAD). There was also fairly good agreement between our recovery criteria, which were based on applying Jacobsen and Truax's (Reference Jacobson and Truax1991) Reliable and Clinically Significant Change criteria to the LSAS, and the IAPT recovery criteria using the SPIN and PHQ: seven patients (64%) were classified as recovered with the former, and six (55%) were classified as recovered with the latter.

One of the main motivations for developing internet CT is a desire to reduce the amount of therapist time that is required to treat a patient so that scarce therapist time can be deployed more efficiently. The average amount of therapist time per patient in standard CT is approximately 19 hours (Clark et al., Reference Clark, Ehlers, Hackmann, McManus, Fennell and Grey2006), whereas the average time therapists spent supporting patients through the internet treatment in the present study was only 3.87 hours. All other things being equal, this means that a therapist could treat approximately five patients with internet CT in the time that would be needed to treat one patient with face-to-face CT. However, this may be an over-simplistic calculation as internet treatment involves a change in the way that therapists deploy their time, as well as an overall reduction in the total amount of time per patient. In face-to-face CT therapists must be available for sessions at a pre-arranged time each week and need to block out 90 minutes plus some time for preparation and writing up notes. In internet treatment, therapists make brief phone calls at pre-arranged times but they have considerable flexibility about when they send electronic messages to patients. As the messaging in the programme is not live chat, therapists can log-on to the site to review a patient's progress and send brief messages of encouragement whenever they wish. This gives therapists greater flexibility in how they manage their time. It was our impression that brief but often (i.e. several times a week) messaging was considered particularly helpful by patients.

Although internet-delivered CT requires less therapist time than face-to-face CT, the amount of time that a patient devotes to therapy may not be less. Indeed, the amount of time that patients spent on the website (35 hours) was almost twice as long as the time they would have spent in face-to-face therapy sessions. On top of these commitments, patients in both internet and face-to-face CT will commit many hours to homework assignments, including conducting behavioural experiments in everyday social situations. It is unclear whether these additional assignments take more or less time in internet CT.

The fact that there were no drop-outs in our pilot cohort suggests that internet treatment is acceptable to at least a subset of patients. Further research is required to determine how large that subset is and how the experience of internet CT compares with that of face-to-face CT. It seems likely that some patients will prefer face-to-face therapy and/or find that they cannot progress with some of the challenging behavioural experiments that are an integral part of treatment without the presence of a therapist. On the other hand, some patients may be reluctant to start therapy if it involves extensive initial face-to-face sessions and may be more likely to engage with an internet-based approach. Now that an internet version of CT has been developed, we hope to be able to clarify who is particularly likely to benefit from the approach. In the meantime, the very encouraging results observed with the pilot cohort suggest that internet delivered CT is likely to be a useful addition to existing treatment approaches.

Acknowledgements

This research was supported by grant 069777 from the Wellcome Trust and by the NIHR Biomedical Research Centre at the South London and Maudsley NHS Foundation Trust and Kings College London and the NIHR Oxford Cognitive Health Clinical Research Facility.

References

Alden, L. E. and Taylor, C. T. (2011). Relational treatment strategies increase social approach behaviors in patients with Generalized Social Anxiety Disorder. Journal of Anxiety Disorders, 25, 309318. doi: S0887-6185(10)00208-2 [pii] 10.1016/j.janxdis.2010.10.003CrossRefGoogle ScholarPubMed
American Psychiatric Association (2004). Diagnostic and Statistical Manual of Mental Disorders (4th edn). Washington, DC: American Psychiatric Association.Google Scholar
Andersson, G., Carlbring, P. and Furmark, T. (2012). Therapist experience and knowledge acquisition in internet-delivered CBT for social anxiety disorder: a randomized controlled trial. PLoS One, 7, e37411. doi: 10.1371/journal.pone.0037411CrossRefGoogle ScholarPubMed
Andrews, G., Davies, M. and Titov, N. (2011). Effectiveness randomized controlled trial of face to face versus internet cognitive behaviour therapy for social phobia. Australian and New Zealand Journal of Psychiatry, 45, 337340. doi: 10.3109/00048674.2010.538840CrossRefGoogle ScholarPubMed
Baker, S. L., Heinrichs, N., Kim, H.-J. and Hofmann, S. G. (2002). The Liebowitz social anxiety scale as a self-report instrument: a preliminary psychometric analysis. Behaviour Research and Therapy, 40, 701715.CrossRefGoogle ScholarPubMed
Bandelow, B., Baldwin, D. S., Dolberg, O. T., Andersen, H. F. and Stein, D. J. (2006). What is the Threshold for Symptomatic Response and Remission for Major Depressive Disorder, Panic Disorder, Social Anxiety Disorder, and Generalized Anxiety Disorder? (Vol. 67). Memphis, TN, ETATS-UNIS: Physicians Postgraduate Press.CrossRefGoogle ScholarPubMed
Berger, T., Hohl, E. and Caspar, F. (2009). Internet-based treatment for social phobia: a randomized controlled trial. Journal of Clinical Psychology, 65, 10211035. doi: 10.1002/jclp.20603CrossRefGoogle ScholarPubMed
Brown, T. A., Di Nardo, P. A. and Barlow, D. H. (1994). Anxiety Disorders Interview Schedule for DSM-IV. Albany, NY: Graywind Publications.Google Scholar
Bruce, S. E., Yonkers, K. A., Otto, M. W., Eisen, J. L., Weisberg, R. B., Pagano, M., et al. (2005). Influence of psychiatric comorbidity on recovery and recurrence in generalized anxiety disorder, social phobia, and panic disorder: a 12-year prospective study. American Journal of Psychiatry, 162, 11791187.CrossRefGoogle ScholarPubMed
Carlbring, P., Gunnarsdottir, M., Hedensjo, L., Andersson, G., Ekselius, L. and Furmark, T. (2007). Treatment of social phobia: randomised trial of internet-delivered cognitive-behavioural therapy with telephone support. British Journal of Psychiatry, 190, 123128.CrossRefGoogle ScholarPubMed
Clark, D. M. (2005). Three questionaires for measuring central constructs in the cognitive model of social phobia. Unpublished manuscript. Kings College London, UK.Google Scholar
Clark, D. M., Ehlers, A., Hackmann, A., McManus, F., Fennell, M. J. V., Grey, N., et al. (2006). Cognitive therapy and exposure plus applied relaxation in social phobia: a randomised controlled trial. Journal of Consulting and Clinical Psychology, 74, 568578.CrossRefGoogle Scholar
Clark, D. M., Ehlers, A., McManus, F., Hackmann, A., Fennell, M., Campbell, H., et al. (2003). Cognitive therapy versus fluoxetine in generalized social phobia: a randomized placebo-controlled trial. Journal of Consulting and Clinical Psychology, 71, 10581067.CrossRefGoogle ScholarPubMed
Clark, D. M. and Wells, A. (1995). A cognitive model of social phobia. In Heimberg, R., Liebowitz, M., Hope, D. A. and Schneier, F. R. (Eds.), Social Phobia: diagnosis, assessment and treatment (pp. 6993). New York: Guilford Press.Google Scholar
Clark, D. M., Wild, J., Grey, N., Stott, R., Liness, S., Deale, A., et al. (2012). Doubling the clinical benefit of each hour of psychotherapy: a randomized controlled trial of self-study assisted cognitive therapy for social anxiety disorder. Manuscript submitted for publication.Google Scholar
Connor, K. M., Davidson, J. R. T., Churchill, L. E., Sherwood, A., Foa, E. and Weisler, R. H. (2000). Psychometric properties of the social phobia inventory (SPIN): new self-rating scale. British Journal of Psychiatry, 176, 379386.CrossRefGoogle ScholarPubMed
Davidson, J. R., Foa, E. B., Huppert, J. D., Keefe, F. J., Franklin, M. E., Compton, J. S., et al. (2004). Fluoxetine, comprehensive cognitive behavioral therapy, and placebo in generalized social phobia. Archives of General Psychiatry, 61, 10051013.CrossRefGoogle ScholarPubMed
Department of Health (2011). The IAPT Data Handbook: guidance on recording and monitoring outcomes to support local evidence-based practice. London: DoH. Available at www.iapt.nhs.ukGoogle Scholar
First, B. M., Gibbon, M., Spitzer, R. L., Williams, J. B. W. and Benjamin, L. S. (1997). User's Guide for the Structured Clinical Interview for DSM-IV Axis II Personality Disorders: SCID II. Washington, DC: American Psychiatric Press.Google Scholar
First, B. M., Spitzer, R. L., Gibbon, M. and Williams, J. B. W. (1995). User's Guide for the Structured Clinical Interview for DSM-IV Axis I Disorders: SCID-I clinician version. Washington DC: American Psychiatric Press.Google Scholar
Fresco, D. M., Coles, M. E., Heimberg, R. G., Liebowitz, M. R., Hami, S., Stein, M. B., et al. (2001). The Liebowitz Social Anxiety Scale: a comparison of the psychometric properties of self-report and clinician-administered formats. Psychological Medicine, 31, 10251035.CrossRefGoogle Scholar
Furmark, T., Carlbring, P., Hedman, E., Sonnenstein, A., Clevberger, P., Bohman, B., et al. (2009). Guided and unguided self-help for social anxiety disorder: randomised controlled trial. The British Journal of Psychiatry, 195, 440447. doi: 10.1192/bjp.bp.108.060996CrossRefGoogle ScholarPubMed
Hedman, E., Furmark, T., Carlbring, P., Ljótsson, B., Rück, C., Lindefors, N., et al. (2011). A 5-year follow-up of internet-based cognitive behavior therapy for social anxiety disorder. Journal of Medical Internet Research, 13, e39. doi: v13i2e39 [pii] 10.2196/jmir.1776CrossRefGoogle ScholarPubMed
Heimberg, R. G., Dodge, C. S., Hope, D. A., Kennedy, C. R., Zollo, L. J., and Becker, R. E. (1990). Cognitive behavioural group treatment for social phobia: comparison with a credible placebo control. Cognitive Therapy and Research, 14, 123.CrossRefGoogle Scholar
Heimberg, R. G., Liebowitz, M. R., Hope, D. A., Schneier, F. R., Holt, C. S., Welkowitz, L. A., et al. (1998). Cognitive behavioral group therapy vs phenelzine therapy for social phobia: 12-week outcome. Archives of General Psychiatry, 55, 11331141.CrossRefGoogle ScholarPubMed
Hofmann, S. G. (2004). Cognitive mediation of treatment change in social phobia. Journal of Consulting and Clinical Psychology, 72, 393399.Google ScholarPubMed
Hope, D. A., Heimberg, R. G. and Bruch, M. A. (1995). Dismantling cognitive-behavioural group therapy for social phobia. Behaviour Research and Therapy, 33, 637650.CrossRefGoogle Scholar
Jacobson, N. S. and Truax, P. (1991). Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 1219.CrossRefGoogle ScholarPubMed
Kessler, R. C., Berglund, P., Demler, O., Jin, R., Merikangas, K. R. and Walters, E. E. (2005). Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry, 62, 593602.CrossRefGoogle ScholarPubMed
Kroenke, K., Spitzer, R. L. and Williams, J. B. W. (2001). The PHQ-9. Journal of General Internal Medicine, 16, 606613. doi: 10.1046/j.1525-1497.2001.016009606.xCrossRefGoogle ScholarPubMed
Ledley, D. R., Heimberg, R. G., Hope, D. A., Hayes, S. A., Zaider, T. I., Dyke, M. V., et al. (2009). Efficacy of a manualized and workbook-driven individual treatment for social anxiety disorder. Behavior Therapy, 40, 414424. doi: 10.1016/j.beth.2008.12.001CrossRefGoogle ScholarPubMed
Leichsenring, F. (2012). The Social Phobia Psychotherapy Research Network (SOPHONET): structure and preliminary results. Paper presented at the “Milestones in Research and Practice” Psychotherapy Congress, Medical University of Hanover, Germany. April.Google Scholar
Liebowitz, M. R. (1987). Social phobia. Modern Problems in Pharmacopsychiatry, 22, 141173.CrossRefGoogle ScholarPubMed
McManus, F., Clark, D. M., Grey, N., Wild, J., Hirsch, C., Fennell, M., et al. (2009). A demonstration of the efficacy of two of the components of cognitive therapy for social phobia. Journal of Anxiety Disorders, 23, 496503.CrossRefGoogle ScholarPubMed
Mortberg, E., Clark, D. M., Sundin, O. and Wistedt, A. A. (2007). Intensive group cognitive treatment and individual cognitive therapy vs. treatment as usual in social phobia: a randomized controlled trial. Acta Psychiatrica Scandinavica, 115, 142154. doi: Doi 10.1111/J.1600-0447.2006.00839.XCrossRefGoogle ScholarPubMed
Palmqvist, B., Carlbring, P. and Andersson, G. (2007). Internet-delivered treatments with or without therapist input. Expert Review of Pharmacoeconomics Outcomes Research, 7, 291297. doi: 10.1586/14737167.7.3.291CrossRefGoogle ScholarPubMed
Pilling, S., Mayo-Wilson, E., Mavranezouli, I., Kew, K., Taylor, C. and Clark, D. M. (in press). Recognition, assessment and treatment of social anxiety disorder: summary of NICE guidance. British Medical Journal.Google Scholar
Rapee, R. M. and Heimberg, R. G. (1997). A cognitive-behavioral model of anxiety in social phobia. Behaviour Research and Therapy, 35, 741756.CrossRefGoogle ScholarPubMed
Rapee, R. M., Abbott, M. J., Baillie, A. J. and Gaston, J. E. (2007). Treatment of social phobia through pure self-help and therapist augmented self-help. British Journal of Psychiatry, 191, 246252. doi: 10.1192/bjp.bp.106.028167CrossRefGoogle ScholarPubMed
Rapee, R. M., Gaston, J. E. and Abbott, M. J. (2009). Testing the efficacy of theoretically derived improvements in the treatment of social phobia. Journal of Consulting and Clinical Psychology, 77, 317327.CrossRefGoogle ScholarPubMed
Spitzer, R. l., Kroenke, K., Williams, J. W. and Löwe, B. (2006). A brief measure for assessing generalized anxiety disorder: the gad-7. Archives of Internal Medicine, 166, 10921097. doi: 10.1001/archinte.166.10.1092CrossRefGoogle ScholarPubMed
Stangier, U., Heidenreich, T., Peitz, M., Lauterbach, W. and Clark, D. M. (2003). Cognitive therapy for social phobia: individual versus group treatment. Behaviour Research and Therapy, 41, 9911007.CrossRefGoogle ScholarPubMed
Stangier, U., Schramm, E., Heidenreich, T., Berger, M. and Clark, D. M. (2011). Cognitive therapy vs interpersonal psychotherapy in social anxiety disorder: a randomized controlled trial. Archives of General Psychiatry, 68, 692700. doi: 10.1001/archgenpsychiatry.2011.67CrossRefGoogle ScholarPubMed
Titov, N., Andrews, G., Choi, I., Schwencke, G. and Mahoney, A. (2008). Shyness 3: randomized controlled trial of guided versus unguided Internet-based CBT for social phobia. Australian and New Zealand Journal of Psychiatry, 42, 10301040. doi: 905606265 [pii] 10.1080/00048670802512107CrossRefGoogle ScholarPubMed
Titov, N., Andrews, G. and Schwencke, G. (2008). Shyness 2: treating social phobia online: replication and extension. Australian and New Zealand Journal of Psychiatry, 42, 595605. doi: 794820392 [pii] 10.1080/00048670802119820CrossRefGoogle ScholarPubMed
Titov, N., Andrews, G., Schwencke, G., Drobny, J. and Einstein, D. (2008). Shyness 1: distance treatment of social phobia over the Internet. Australian and New Zealand Journal of Psychiatry, 42, 585594. doi: 794820327 [pii] 10.1080/00048670802119762CrossRefGoogle Scholar
Titov, N., Andrews, G., Johnston, L., Robinson, E. and Spence, J. (2010). Transdiagnostic Internet treatment for anxiety disorders: a randomized controlled trial. Behaviour Research and Therapy, 48, 890899. doi: S0005-7967(10)00110-5 [pii] 10.1016/j.brat.2010.05.014CrossRefGoogle ScholarPubMed
Weidman, A. C., Fernandez, K. C., Levinson, C. A., Augustine, A. A., Larsen, R. J. and Rodebaugh, T. L. (2012). Compensatory internet use among individuals higher in social anxiety and its implications for well-being. Personality and Individual Differences, 53, 191195. doi: 10.1016/j.paid.2012.03.003CrossRefGoogle ScholarPubMed
Wild, J. and Clark, D. M. (2011). Imagery rescripting of early traumatic memories in social phobia. Cognitive and Behavioral Practice, 18, 433443. doi: 10.1016/j.cbpra.2011.03.002CrossRefGoogle ScholarPubMed
Figure 0

Table 1. Outcome and process measures at pre- and posttreatment

Submit a response

Comments

No Comments have been published for this article.