Recovery Star: validating user recovery

Abstract

Aims and method

Mental Health Recovery Star is a multifaceted 10-item outcomes measure and key-working tool that has been widely adopted by service providers in the UK. We aimed to explore its factorial validity, internal consistency and responsiveness. Recovery Star readings were conducted twice with 203 working-age adults with moderate to severe mental health problems attending a range of mental health services, and a third time with 113 of these individuals.

Results

Mental Health Recovery Star had high internal consistency and appeared to measure an underlying recovery-oriented construct. Results supported a valid two-factor structure which explained 48% of variance in Recovery Star ratings data. Two Recovery Star items (‘relationships’ and ‘addictive behaviour’) did not load onto either factor. There was good statistically significant item responsiveness, and no obvious item redundancy. Data for a small number of variables were not normally distributed and the implications of this are discussed.

Clinical implications

Recovery Star has been received enthusiastically by both mental health service providers and service users. This study provides further evidence for its adoption in recovery-focused mental health services and indicates that items relating to addictive behaviour, responsibilities and work could be further developed in future.

The concept of recovery and the use of recovery-oriented models have been under a growing focus of mental health service providers. Led by service users, the recovery approach was developed in the USA and emphasised the importance of lived experience over psychiatric symptomatology. ¹ Recovery stands in contrast to medical models that focus on treatment and cure; it implies maintenance of a person-focus rather than an illness-focus, and a reliance on philosophical conviction as much as on scientific evidence. ² The Sainsbury Centre for Mental Health has emphasised the role of personal agency and self-management in recovery, the importance of social roles and communities, and self-discovery. ³ The recovery approach has gained considerable support in the UK from the Care Services Improvement Partnership, the Royal College of Psychiatrists and the Social Care Institute for Excellence, ⁴ the National Institute for Mental Health in England, ⁵ and by the campaigning mental health charity Rethink. ⁶ Recovery-focused mental health services are currently running across the National Health Service (NHS). ⁴

Measuring recovery

A number of tools and measures have been developed to facilitate the use, and to measure the outcome, of recovery-oriented interventions. Campbell-Orde et al ⁷ reviewed research relating to the psychometric properties of nine measures of individual recovery and four properties of recovery-promoting environments and showed great variation in terms of their development. Recovery-focused outcome tools commonly aim to measure concepts such as hope, quality of life, treatment satisfaction, empowerment, overcoming ‘stuckness’, learning, well-being, and relationships. ⁸

Mental Health Recovery Star origins

The Mental Health Recovery Star was commissioned in the UK by the voluntary sector umbrella body the Mental Health Providers Forum (MHPF). ^9,10 Its development has been largely qualitative and undertaken with reference to the academic literature on recovery-focused approaches and to published service user accounts. Considerable user involvement and extensive mental health service user feedback were integral to the development of the tool. Alongside managers and front-line workers, a total of 114 service users were intimately and actively involved in workshops to develop the tool and in trialling and revising its first and second versions. ¹¹ Although it was developed as an outcomes tool with the aim of enabling recovery-focused organisations to work with service users to measure and summarise change, in many organisations the momentum to use Recovery Star has been provided by the enthusiasm of mental health service users and their key workers, who value its role in understanding the current position, planning the actions needed to progress the recovery journey and demonstrating progress across ten domains (Box 1). As such it has potential to motivate users and workers in striving towards recovery and to play an active role in it, and thus is not intended simply as an inert measurement tool. Recovery Star is predicated on an underlying model of a ‘ladder of change’ comprising five stages: being stuck, accepting help, believing, learning and self-reliance. The tool has been widely adopted by mental health service providers in the UK, ¹² although some concerns have been raised about the lack of data on its psychometric properties. ^13,14

Study aims

The aim of our study was to explore the psychometric properties of Recovery Star to inform training and further development. Specific objectives were to ascertain whether items on the tool measured a single underlying construct relating to recovery (internal consistency); to identify the nature of any underlying factors (factor validity); to identify any item redundancy; and to identify whether Recovery Star detects reported change over time (responsiveness).

Method

Measures

Recovery Star comprises ten dimensions that were identified as central to the recovery journey (Box 1). Each item is rated jointly by service user and key worker on a 10-point scale, which relates to the underlying five-stage ‘ladder of change’ model and specifically to the service user's own subjective sense of their position in relation to the ladder rather than as an objective measure of severity. Each stage relates to two steps on the ladder of change (i.e. scores 1 and 2 relate to the stuck stage, 3 and 4 to the accepting help stage, and so on). Recovery Star is intended both as an outcomes measure, in that it aims to quantify the service user's movement along the recovery pathway across the course of service contact, and a key-working tool, in that it guides and informs that collaborative work.

Setting and participants

Participating organisations entered routinely collected and anonymised Recovery Star and demographic data into an internet-based database between November 2008 and July 2010. Recovery Star projects were mainly run by small to medium-size community-based teams (e.g. community mental health teams) in NHS mental health trusts, private healthcare organisations and charitable agencies offering a range of services including day centres, supported housing, psychological therapies, social activities and substance misuse services. Service users were mainly adults of working age presenting with moderate to severe mental health problems.

Box 1 The ten areas of the Recovery Star

1. 1 Managing mental health. This is about learning how to manage yourself and your symptoms and building a satisfying and meaningful life which is not defined or limited by them.

2. 2 Physical health and self-care.This is about how well you look after yourself-taking care of your physical health, keeping clean, how you present yourself, being able to deal with stress.

3. 3 Living skills.This is about the practical side of being able to live independently-shop andcook for yourself, deal withneighbours and people who visit, keeping your place clean and tidy.

4. 4 Social networks.This is about your social networks and being part of your community including volunteering or classes, being part of a club or society, school or faith organisation, or groups of friends.

5. 5 Work.This is about you and work – whether you want to work, knowing what it is you would like to do, having the skills and qualifications to get the work you want and finding and keeping a job; or volunteering or other work-like activity may be a goal.

6. 6 Relationships.This is about the important relationships in your life. We suggest you choose one relationship where you would like things to be different. Whoever youchoose, it is about having the amount of closeness that you want, which is something that you decide.

7. 7 Addictive behaviour.This is about any addictive behaviour such as drug or alcohol use, or gambling, food or shopping. It is about how aware you are of any problems you have in this area and whether you are working to reduce the harm they may cause.

8. 8 Responsibilities.This is about meeting your responsibilities in relation to the place where you live including things like paying the rent, getting on withneighbours or fellow residents. It also covers breaking the law.

9. 9 Identity and self-esteem.This is about how you feel about yourself and how you define who you are. It is about getting to the point where you have a sense of your own identity.

10. 10 Trust and hope.This is about your sense that there are people you can trust and there is hope for your future. It is about trusting in others, yourself and having faith in life.

Adapted from Mental Health Providers Forum. ¹²

Procedure

Recovery Star and demographic data were extracted from the MHPF database of those individuals aged 18-65 years who had completed two or more readings of the Recovery Star, either alone or in collaboration with a project worker, at least 42 days apart (n = 203). Of these, 113 (55.9%) had completed a third Recovery Star at least 84 days after the first reading. A total of 113 completed Stars were omitted from the analysis because they had been completed by a project worker alone.

Analysis

Data were transferred into SPSS 16.0.1 for Windows. Item and total mean scores and standard deviations were calculated. Descriptive data were generated to describe the sample, including significant deviations from normal Gaussian distribution. Cronbach's α was calculated as an indicator of internal consistency; exploratory factor analysis (principal axis factoring, varimax rotation with Kaiser normalisation) was conducted; tests of item redundancy (item-item correlation using Pearson product-moment statistic (r)) and responsiveness (repeated measures t-tests, Cohen's d) were performed.

Results

Descriptive data

In total, 203 pairs of Recovery Star ratings were gathered from 27 participating projects (range 1-43 pairs, median = 29). Item response rate was 100%. Mean age at completion of the first reading was 46.7 years (range 21.2-64.0, s.d. = 10.2). There were 110 males (54.2%) and 93 females (45.8%) in the sample. The ethnic background of 92% of people was recorded as ‘White’. At baseline, Recovery Star was usually completed jointly by a service user and project worker (94.6%), and occasionally by a service user alone (5.4%). Mean baseline ratings for each item and for total score are presented in Table 1. Item means lay close (s.d. = 2.5) to the midpoint (5.5), although the mean score for ‘responsibilities’ approached the upper limit of this boundary. The score for all ten items ranged from 1 to 10. The skewness statistic indicated that the distribution of two items (‘responsibilities’ and ‘addictive behaviour’) was significantly negatively skewed (two standard errors of skewness), with scores bunched on the high end of the scale, and that of one (‘work’) was significantly positively skewed, with scores bunched on the low end.

TABLE 1 Recovery Star responsiveness, reading 1 to 2 (n = 203)

	Mean (s.d.)
Domain	Baseline reading	Second reading a	Change, mean (95% CI)	Statistical test
Trust and hope	5.6 (2.3)	6.0 (2.2)	0.32 (0.08-0.55)	t =—2.60, d.f. = 202, P<0.05
Managing mental health	5.5 (2.0)	5.9 (2.1)	0.39 (0.18-0.61)	t =—3.30, d.f. = 202, P<0.001
Physical health and self-care	5.8 (2.3)	6.2 (2.3)	0.43 (0.17-0.69)	t =—3.30, d.f. = 202, P<0.001
Identity and self-esteem	5.6 (2.3)	6.0 (2.3)	0.40 (0.14-0.65)	t =—3.09, d.f. = 202, P<0.01
Social networks	5.2 (2.3)	5.8 (2.3)	0.59 (0.33-0.84)	t =—4.54, d.f. = 202, P<0.001
Living skills	6.5 (2.5)	6.7 (2.5)	0.23 (0.03-0.48)	t =—1.75, d.f. = 202, P=0.08
Relationships	5.3 (2.8)	5.6 (2.7)	0.30 (0.00-0.60)	t =—1.94, d.f. = 202, P=0.05
Work b	4.7 (2.5)	5.0 (2.7)	0.22	Z=—2.62, P<0.01
Responsibilities b	7.6 (2.6)	7.7 (2.5)	0.08	Z=—0.72, P=0.47
Addictive behaviour b	6.8 (3.1)	7.0 (3.1)	0.15	Z =—1.53, P=0.13

a. Mean (s.d.) duration between readings, 157 days (87.2).

b. Non-parametric equivalent of repeated measures t-test used due to non-Gaussian distribution.

Internal consistency and factor validity

All items correlated at statistically significant levels (range r = 0.50-0.77) with the total score. Cronbach's α = 0.85 represented very good internal consistency. Exploratory factor analysis indicated that two underlying factors comprising five and three items accounted for 48.0% of the variance in scores (Table 2); two items (‘relationships’ and ‘addictive behaviour’) did not load on to either factor. Eigen values showed that factor 1 explained 41.1% of the variance in the data and factor 2 explained 6.9%. Item loadings in the two extracted factors all exceeded 0.5. Internal consistency was very good for factor 1 (Cronbach's α = 0.85) and acceptable for factor 2 (Cronbach's α = 0.73).

TABLE 2 Recovery Star factor analysis (n = 203)

Item	Factor 1 loading	Factor 2 loading
Trust and hope	0.75
Managing mental health	0.78
Physical health and self-care	0.63
Identity and self-esteem	0.74
Social networks	0.55
Responsibilities		0.65
Work		0.62
Living skills		0.62

Item redundancy

High internal consistency on a scale can indicate item redundancy if individual items simply measure the same facet through repetition, which is usually indicated by item-item correlation in excess of 0.7. In contrast, inter-item correlation of less than 0.3 may indicate redundancy due to the non-homogeneity of test items. ¹⁵ In this study almost all items correlated with one another at levels exceeding chance, but no item-item correlation exceeded the 0.7 threshold. The exceptions related to the ‘addictive behaviour’ item, which did not correlate significantly with three items (‘managing mental health’, ‘relationships’, and ‘identity and self-esteem’).

Responsiveness

The skewness statistic indicated distribution within the normal range for seven items and for these, responsiveness was measured using the parametric repeated measures t-test; the non-parametric equivalent was used for items with non-Gaussian distribution. Table 1 shows that there was statistically significant change (indicating positive movement on the recovery journey) between the first two readings of the Recovery Star for seven out of ten items. The available third readings of the Recovery Star indicated similar positive results (Table 3) for the majority of items. However, the item relating to ‘responsibilities’ showed no change at either second or third reading. Where significant, outcome effect size was typically small between first and second measurement (Cohen's d =0.1-0.26) and medium between first and third readings (Cohen's d =0.2-0.4). Those who completed the Recovery Star on three occasions were less far along their recovery journey in relation to ‘self-care’ (x¯ = 6.1) than those who only repeated the Star on two occasions (x¯ = 5.4, t = –2.34, d.f. = 201, P<0.05), but there was no other significant difference between the two groups.

TABLE 3 Recovery Star: responsiveness, reading 1 to 3 (n = 113)

	Mean (s.d.)
Domain	Baseline reading	Second reading a	Change, mean (95% CI)	Statistical test
Trust and hope	5.9 (2.3)	6.0 (2.1)	0.46 (0.11-0.81)	t =—2.58, d.f. = 112, P<0.05
Managing mental health	5.7 (1.9)	6.3 (2.0)	0.64 (0.31-0.97)	t =—3.84, d.f. = 112, P<0.001
Physical health and self-care	6.1 (2.0)	6.7 (2.2)	0.55 (0.21-0.89)	t =—3.19, d.f. = 112, P<0.001
Identity and self-esteem	5.6 (2.1)	6.2 (2.2)	0.54 (0.18-0.90)	t =—2.98, d.f. = 112, P<0.01
Social networks	5.3 (2.3)	6.2 (2.1)	0.88 (0.56-1.21)	t =—5.33, d.f. = 112, P<0.001
Living skills	6.6 (2.3)	7.1 (2.3)	0.50 (0.18-0.81)	t =—3.13, d.f. = 112, P<0.01
Relationships	5.4 (2.7)	5.7 (2.7)	0.42 (0.11-0.86)	t =—1.94, d.f. = 112, P=0.06
Work b	4.8 (2.4)	5.1 (2.7)	0.27	Z=—1.52, P=0.13
Responsibilities b	7.9 (2.3)	7.9 (2.3)	0.02	Z=—0.26, P=0.80
Addictive behaviour b	6.8 (3.0)	7.3 (2.9)	0.48	Z =—2.38, P<0.05

a. Mean (s.d.) duration between readings, 271.1 days (101.1).

b. Non-parametric equivalent of repeated measures t-test used due to non-Gaussian distribution.

Discussion

The routinely collected data presented in this study represent the first quantitative investigation of the psychometric properties of the Recovery Star, and the results are encouraging. The tool had very good internal consistency (Cronbach's α>0.8), little obvious item redundancy, and most item scores moved in a positive direction over time. However, as would be expected from a tool that was designed from the bottom-up to meet user need and to serve a dual purpose as a key-working tool, there were some issues related to the expectations that researchers may have in terms of the psychometric properties of the tool. This study was performed on routinely recorded data with few demographic or other research data to draw on and therefore has some important limitations. We hope that discussion of these will inform any future development of the Recovery Star.

Limitations

Cronbach's α of 0.85 for the instrument suggested a single underlying recovery-related construct, and exploratory factor analysis indicated that this comprised two factors with good (factor 1) or acceptable (factor 2) internal consistency of their own. These factors did not map precisely onto previous, narrative descriptions of the key components of recovery, for example those by the Sainsbury Centre for Mental Health. ³ However, discussion among the researchers led us to believe that there was some rational connection between the grouped items, with factor 1 representing ‘internal management and personal relationships’ and factor 2 representing ‘external management and external relationships’. The greatest amount of variance was explained by factor 1, which included the item ‘social networks’, thus both endorsing and reflecting explanations of recovery as a deeply personal journey of self-discovery that is situated within a social context. ^1-4,16,17 Two items did not load onto either factor (‘relationships’ and ‘addictive behaviour’). The ‘relationships’ item might, superficially, be reasonably expected to fall under the ‘internal management and personal relationships’ factor but it is not intended to refer to the individual's personal relationships in general, which are adequately covered under ‘social networks’, but rather to a single dyadic and essentially intimate relationship. Other authors have noted that intimate relationships may belong to a separate aspect of recovery distinct from socialisation within society because people with mental illness can find that establishing a life partnership is a challenge of a different order to that of managing employee and peer relationships. ¹⁸ Recovery Star was designed with users to reflect their experience of recovery and not to adhere to a predetermined statistical factor structure. A lack of precise concordance between the actual factor structure of Recovery Star and a theorised structure of the wider recovery concept was therefore predictable. Our analysis may aid in developing or devising items to more accurately reflect the underlying statistical properties of recovery for research purposes.

Data distribution for three items was non-Gaussian (Tables 1 and 3) suggesting that there may need to be some calibration if a version of the tool for research purposes were to be developed. Although ‘addictive behaviour’ and ‘responsibilities’ were negatively skewed because a relatively large number of participants identified themselves as being at an advanced stage on this part of the recovery journey, significant numbers still identified themselves as having needs in these areas (25% of participants ‘being stuck’ or ‘accepting help’ on the ‘addictive behaviour’ item, and 15% at a similar stage on the ‘responsibilities’ item). Therefore, the tool did retain clinical utility and is not redundant in that sense. Similarly, distribution of scores for the item ‘work’ was positively skewed, indicating a preponderance of individuals rating themselves as at an early stage of their recovery in this area. We note that the pinnacle of the ladder for this item refers to self-reliance and that the accompanying descriptor refers to ‘work’ rather than to ‘meaningful activity’ as in some other recovery-oriented measures (e.g. Recovery Assessment Scale ¹⁹ ). Some may view this as politically motivated, for example as an attempt to move people through the healthcare system and back into paid employment. However, far from this, the item apex was oriented to paid employment as opposed to voluntary or other non-paid activities during tool development precisely because user consultees wanted recognition for their desire to be in meaningful, paid employment with appropriate support rather than be diverted into voluntary activities. The ‘work’ item proved to be one of the least amenable to change, particularly over the longer term, indicating that a number of people have long-term needs for ongoing support and, for them, there may be a need to clarify the item descriptor to allow for a range of interpretations of ‘work’.

Our study was based on routinely collected data from users of community mental health services and it is possible, therefore, that items with non-Gaussian distribution within this sample may not be replicated elsewhere. For example, forensic mental health service users frequently have needs related to reoffending, ²⁰ violence and self-harm ^21-23 and substance misuse. ²⁴ There is growing demand for recovery-oriented practice in these settings ²⁵ and a need to translate recovery principles into practice for these populations. ²⁶ Items such as responsibilities and addictive behaviour could be supplemented with additional items which reflect these needs, although it will be important to maintain congruence with the positive orientation of the recovery approach, including the underlying ladder of change model, in any future developments.

The responsiveness of items was demonstrated by statistically significant change of small effect size between first and second readings of the Recovery Star. This may simply represent the relatively short period between ratings (mean 157 days) and a consequent lack of time for service users to recognise their own progress in this aspect of their recovery. This hypothesis is consistent with data from individuals who completed a third rating an average of 9 months after baseline, when slightly greater medium effect-sized changes were demonstrated. The data probably fairly reflect the slow gains made by individuals with chronic problems living in the community, and longer-term studies are required.

The study used Recovery Star and brief demographic data that were collected during the course of routine clinical practice across multiple organisations and thus indicated a potential lack of standardisation in data collection. However, most participating organisations received training in completion of the Recovery Star provided by MHPF. Little is currently known about the retest or interrater reliability of the tool or about its external validity, and we await progress on these issues. Importantly, Recovery Star was able to detect statistically significant reported change between measurements and the change was greater (medium effect size) over longer periods. No other measures were collected that could have tested convergent and divergent validity and clinical details about the participants were unavailable. Further research on the psychometric properties of Recovery Star is clearly warranted, but this study provides good preliminary evidence for its adoption in recovery-focused mental health services.