Carers' and Users' Expectations of Services – User version (CUES–U): A new instrument to measure the experience of users of mental health services

Abstract

Background

No existing instrument measures all or even most of the issues considered important by users of mental health services.

Aims

To develop and test a self-assessment instrument to enable users of mental health services to rate their experience across the range of domains that they consider to be important.

Method

Relevant domains were identified and a new instrument was drafted and field tested to examine its psychometric properties.

Results

The 17-item, self-rated Carers' and Users' Expectations of Services – User version (CUES–U) appears acceptable to most service users. Its items have reasonable test–retest reliability and a ‘total CUES–U score’ correlates significantly with a total score of the Health of the Nations Outcome Scales (Spearman's ρ=0.42; P<0.01).

Conclusions

The development and testing of CUES–U suggest that it might be feasible to apply a self-rated measure of the expectations and experience of users of mental health services.

The Department of Health in England intends to evaluate mental health services “against the aspirations and experience of its users” (Department of Health, 1997). Surveys of service users will be conducted locally as a function of clinical governance and by the new Commission for Health Improvement as it inspects mental health services (Department of Health, 1998). Although there are tested instruments for measuring aspects of the experience of service users, including quality of life (Oliver et al, 1996), needs (Phelan et al, 1995), problems (Wing et al, 1996) and satisfaction with services (Ruggeri & Dall' Agnola, 1993), none address all or even most of the issues that are important to service users. This report describes the development and piloting of a brief self-assessment instrument to enable users of mental health services to rate their experience across the range of domains that they consider to be important.

METHOD

The work involved collaboration between three research organisations representing psychiatry (Royal College of Psychiatrists' Research Unit), mental health nursing (Royal College of Nursing Institute) and social work (University of East Anglia School of Social Work) and an organisation representing service users and their carers (National Schizophrenia Fellowship).

The methods and results are described more fully in the final report of the project team to the Department of Health (Lelliott et al, 1999).

Identification of domains

A comprehensive literature search was undertaken to identify domains that might be included in such a measure. The search had two strands. Published reports were identified by a systematic search of electronic databases (MEDLINE, HEALTHStar, PSYCHLIT, PSYCINFO and EMBASE), by ‘cascade’ searches through the reference lists of identified papers and by direct approaches to others working in this field. Unpublished or ‘grey’ literature was identified by writing to more than 300 national and local user and carer organisations in the UK. Two types of material were sought: instruments that measured needs, problems, quality of life or satisfaction of service users; and reports of surveys or other research about the views of service users.

In parallel with the literature search, the research worker employed by the National Schizophrenia Fellowship (J. H.) ran two focus groups of, and conducted seven in-depth semi-structured interviews with, users of mental health services. These users were all people who had experience of working with other service users, for example as advocates, and so had a wider knowledge of the experience of people with a mental illness.

Development of the instrument

The large number of possible domains, identified by the literature search and interviews with service users, were mapped and grouped into the smallest possible number of items without losing definition or meaning. Because the instrument was intended to measure states that are relatively enduring, each item was introduced with a ‘normative’ statement. This described what a service user should expect to be the case for the issue if it did not constitute a problem. The wording of these normative statements was modified in response to the comments of an advisory group of service users and through the use of Flesch scores to increase their readability (Flesch, 1948).

A scale with many points and attempts at overprecision would be more difficult to use. Therefore, after reading each normative statement, the person is asked to respond to two simple questions, each with a three-point scale. Part A asks how the person's situation compares with that described by the normative statement (‘as good as this’/‘worse than this’/‘very much worse than this’) and Part B asks whether the person is satisfied with the issues described (‘yes’/‘unsure’/‘no’). There is also space for a free-text response to each item (Part C).

Eighty-two service users completed the first full draft of the instrument and provided structured feedback. All but one reported that the instructions and language used in the normative statements were always or usually clear. Fifty-one (71%) stated that the pilot version covered all the domains that they considered to be important. Twenty-one (29%) took less than 15 min to complete the instrument, twenty-seven (38%) took between 15 and 30 min and five (7%) took more than 45 min. Fifty-four people (75%) thought that the instrument was about the right length and thirteen (18%) thought that it was too long.

The instrument was redrafted to take account of the feedback from those involved in the pilot and an analysis of the inter-relationships between items. The version used in the field tests, Carers' and Users' Expectations of Services — User version (CUES—U), is outlined in the Appendix.

Field testing

Four hundred and forty-nine service users from 32 locations in England, Northern Ireland and Wales participated in the main field trials. Data collection was coordinated by people who were working for statutory mental health services (127 returns) or for local voluntary sector services (322 returns). Although not selected in any random or systematic way, the participants were all users (mainly long-term) of local mental health services managed by these local agencies.

The results of the first rating made by all participants (time 1) were included in the analyses of the internal psychometric properties of CUES—U. Ninety-nine service users also made a second rating between 2 and 14 days after the first (time 2). These results were used to examine test—retest reliability. The time interval is that recommended by Streiner & Norman (1995).

A sub-study was conducted separately from the main field trials. In this, a rating of the Health of the Nation Outcome Scales (HoNOS; Wing et al, 1996) was made by a mental health professional who knew the person well at the same time as a service user completed CUES—U. Eighty-four pairs of ratings were collected.

RESULTS

Identification of domains

A full account of the literature review is given in the final report (Lelliott et al, 1999).

In summary, people who use mental health services often emphasise and value different aspects of their health and social function than do mental health care professionals. They appear to place less emphasis on symptom reduction than they do on improvements in other areas of their lives. These include: work, or other meaningful daytime activity; financial security; suitable and comfortable accommodation; choice and control over where they live; and the establishment and maintenance of relationships.

Certain qualities of mental health services are also important, including: accessibility and availability; the provision of information about services and treatments; continuity, particularly in terms of establishing and maintaining relationships with individual care workers; ‘seamlessness’ in terms of care provided by different service facilities and agencies; and choice of treatment and care and of who acts as the keyworker. People with a severe mental illness also value access to physical health care services that take their needs seriously.

Service users also value certain attributes in health and social care workers: courtesy; respectfulness; honesty; openness; friendliness; informality; empathy; non-judgemental attitude; caring nature; reliability; punctuality; willingness to share information and decisions and to give practical help and support.

Main field trials

The mean age of the 449 participants was 42 years (range 18-78); 53% were men and 91% were White. About 5% of the data items were missing from the schedules returned. Each analysis included all valid cases.

Table 1 summarises the responses to the Part A and Part B questions for the 16 CUES—U items. The proportion of participants who rated their situation as being as good as the normative statements (Part A questions) ranged from 50.8% to 76.6%. The proportion who expressed satisfaction with their situation ranged from 39.8% to 73.6%. Correlations between Part A and Part B questions were generally high (Spearman's ρ=0.67-0.86). However, for all items there were some service users whose response to the two questions appeared contradictory. For 15 of the 16 items, fewer people responded positively to Part A questions than to Part B questions (i.e. expressed dissatisfaction with their situation despite reporting that it was as good as the normative statement).

Table 1 Response to Carers' and Users' Expectations of Services — User version (CUES—U) Part A and Part B questions at time 1 (n=449)

Item	Part A questions (%)				Part B questions (%)
	As good as this 0	Worse than this 1	Very much worse than this 2	Missing	Satisfied 0	Unsure 1	Dissatisfied 2	Missing
1. Where you live	75.1	18.5	4.0	2.4	72.0	12.8	13.4	1.8
2. Money	64.8	24.3	7.6	3.3	60.6	9.4	28.2	1.8
3. Help with finances	70.4	20.3	4.2	5.1	63.3	16.6	16.3	3.8
4. How you spend your day	71.5	19.8	5.3	3.3	57.5	17.4	22.4	2.7
5. Family and friends	57.0	29.4	9.6	4.0	52.8	20.4	24.4	2.5
6. Social life	53.5	33.6	8.9	4.0	39.8	20.6	37.6	2.0
7. Information and advice	58.4	25.4	11.8	4.5	52.1	17.9	25.5	4.5
8. Access to mental health services	51.9	28.7	14.0	5.3	49.7	19.2	27.5	3.6
9. Choice of mental health services	51.9	30.1	12.7	5.3	46.5	19.7	29.3	4.5
10. Relationships with mental health workers	63.7	20.7	9.6	6.0	63.8	14.8	16.3	5.1
11. Consultation and control	65.0	17.1	10.5	7.3	59.1	14.3	20.8	5.8
12. Advocacy	62.6	20.9	6.5	10.0	56.4	18.6	15.9	9.2
13. Stigma and discrimination	50.8	29.0	13.8	6.5	45.4	18.6	30.6	5.4
14. Medication	65.7	19.6	7.1	7.6	58.2	19.0	17.4	5.4
15. Access to physical health services	76.6	14.0	4.5	4.9	73.6	10.1	12.3	4.0
16. Relationships with physical health workers	73.7	13.4	5.6	7.3	69.6	11.2	21.5	6.5

Part A questions

A principal components analysis was conducted of the Part A questions for the 16 items using a covariance matrix extraction method (Norman & Streiner, 1994). The Kaiser—Meyer—Olkin measure of sampling adequacy (KMO), for which summary values were 0.9, indicated that all 16 items should be included. Varimax rotation yielded three factors with an eigenvalue greater than unity. These accounted for 53% of the variance (Factor 1, 20%; Factor 2, 17%; Factor 3, 16%). Table 2 shows the loadings of the individual items onto each factor where coefficients were greater than 0.4.

Table 2 Components derived from a principal components analysis of Carers' and Users' Expectations of Services — User version (CUES—U) Part A questions at time 1 (n=449)

Item	Component
	1	2	3
1. Where you live	0.42
2. Money			0.76
3. Help with finances
4. How you spend your day			0.63
5. Family and friends			0.61
6. Social life			0.81
7. Information and advice	0.56
8. Access to mental health services	0.80
9. Choice of mental health services	0.84
10. Relationships with mental health workers	0.76
11. Consultation and control	0.53	0.53
12. Advocacy		0.53
13. Stigma and discrimination	0.45	0.45
14. Medication		0.69
15. Access to physical health services		0.65
16. Relationships with physical health workers		0.62

Part B questions

A principal components analysis of Part B questions at time 1 yielded three rotated factors with eigenvalues greater than unity that accounted for 50% of the variance (Factor 1, 24%; Factor 2, 15%; Factor 3, 11%). Again, the KMO indicated that all items should be included. The structure was quite similar to that of the factors derived from the Part A questions (Table 3).

Table 3 Components derived from a principal components analysis of Carers' and Users' Expectations of Services — User version (CUES—U) Part B questions at time 1 (n=449)

Item	Component
	1	2	3
1. Where you live
2. Money		0.71
3. Help with finances	0.58
4. How you spend your day		0.69
5. Family and friends		0.58	0.50
6. Social life		0.79
7. Information and advice	0.73
8. Access to mental health services	0.68
9. Choice of mental health services	0.79
10. Relationships with mental health workers	0.70
11. Consultation and control	0.60		0.41
12. Advocacy	0.60
13. Stigma and discrimination			0.65
14. Medication			0.59
15. Access to physical health services
16. Relationships with physical health workers	0.45

Test—retest reliability

Table 4 shows the intraclass correlation coefficients, for the 16 items for both Part A and Part B questions, between time 1 and time 2 for the 99 people who made two ratings. Coefficients are good (0.61-0.80) for nine of the Part A and eleven of the Part B questions and moderately good (0.41-0.60) for six of the Part A and five of the Part B questions (Landis & Koch, 1977). The exception is Part A of the item relating to medication.

Table 4 Intraclass correlations for time 1 to time 2 (an interval of 2-14 days) Carers' and Users' Expectations of Services — User version (CUES—U) Part A and Part B questions (n=99)

Item	Intraclass correlation coefficients
	Part A	Part B
1. Where you live	0.59	0.76
2. Money	0.61	0.74
3. Help with finances	0.58	0.57
4. How you spend your day	0.68	0.63
5. Family and friends	0.75	0.76
6. Social life	0.62	0.73
7. Information and advice	0.62	0.72
8. Access to mental health services	0.70	0.78
9. Choice of mental health services	0.61	0.58
10. Relationships with mental health workers	0.55	0.63
11. Consultation and control	0.54	0.52
12. Advocacy	0.58	0.67
13. Stigma and discrimination	0.62	0.58
14. Medication	0.37	0.53
15. Access to physical health services	0.73	0.67
16. Relationships with physical health workers	0.48	0.64

Comparison with HoNOS

The HoNOS have 12 items rated 0 (no problem) to 4 (very severe problem). The items cover a range of problems of behaviour, impairment, symptoms and social function. The mean total HoNOS score for the 84 service users in this sub-study was 12.3 (95% CI 11.0-13.7), which is comparable to that reported in the HoNOS field trial (Wing et al, 1996).

Although HoNOS and CUES—U are quite different in structure and mode of application, there are three HoNOS items that have approximate counterparts in the Part A question of five of the CUES—U items: the HoNOS item 9 (problems with relationships) correlated significantly with CUES—U items 5 (family and friends) (Spearman's ρ=0.27; P < 0.05) and 6 (social life) (0.26; P < 0.05); HoNOS item 11 (problems with living conditions) with CUES—U item 1 (where you live) (0.31; P < 0.01); and HoNOS item 12 (problems with daily occupation) with CUES—U item 4 (how you spend your day) (0.33; P < 0.01). A ‘total CUES—U score’ for Part A questions, created by adding responses to all 16 items, correlated significantly with the total HoNOS score (Spearman's ρ=0.42; P < 0.01).

Ease of use

Three-quarters of the participants (n=335) stated that they completed CUES—U without help from another person. Common reasons why help was sought were: difficulty in understanding the format, questions or words; difficulty with reading and writing; visual impairment; and lack of confidence.

DISCUSSION

There may be occasions when the professional's view of what is a problem or a desirable outcome should override that of the service user; for example, when people report themselves to be satisfied with very poor life circumstances because they have become accustomed to poor conditions (Lehman, 1996). However, these occasions should be exceptions. There are two good reasons for paying close attention to the issues that service users consider to be important. First, quality of life, as perceived by the recipient of care, should be the final criterion by which the quality of that care is judged (Wing, 1978). Second, paying attention to the issues considered important by a service user is likely to increase the extent to which he or she is prepared to engage with care (Bowling, 1991).

The purpose of developing CUES—U was to produce an instrument that can measure issues considered important by service users from their perspective. It was anticipated that uses for such an instrument might be:

1. (a) to enable service users to inform health and social care workers of their experience of mental illness and its consequences; this might be in the context of individual care planning or more generally in educating mental health and social care workers about the service-user perspective;

2. (b) to enable service users to participate more actively in audit and evaluation of mental health services; this could be extended to audit and evaluation that is conducted by service users and might range from evaluation of a service to evaluation of a team or even of the care delivered by an individual practitioner;

3. (c) to provide information that, through aggregation, brings the service-user perspective to the activities of service planning and commissioning.

For an instrument to meet this specification it must be easy to use by the majority of service users, it should have good coverage of the issues considered important by service users and ratings should not be unduly influenced by transient influences on subject state, such as short-term changes in mood. The extent to which ratings on such an instrument should be consistent with some other independent and ‘objective’ measure is debatable. Differences in perception, between service users and professionals, about what constitutes a desirable state or outcome is one important justification for the development of such a measure.

To what extent does CUES—U meet this specification?

Ease of use and acceptability

The CUES—U is a self-rated measure and so does not commit the time of mental health care professionals, except for the time taken to encourage its use. Feedback from the pilot indicates that the structure, layout and wording of CUES—U are generally clear and acceptable to service users and that it can be completed quite quickly. About one-quarter of people in the field trial sought help with its completion. However, the type of help needed usually could be provided by a friend, carer or advocate.

There is an inevitable tension between the need for a general tool to enable comparisons between the experience of service users interacting with different services, or living in different geographical areas, and for one that measures the perspective of a particular individual or a specific group. Only a few people whose first language is not English participated in the study and CUES—U has not been translated into any other language. The great majority of those participating in the field trials were White and CUES—U needs to be tested further by people from minority ethnic groups.

Coverage of relevant domains

During the development phase, information was gathered from a variety of sources (literature reviews of surveys and other instruments, focus groups, interviews and consultation) to ensure that the resulting items covered the important domains. The results of the piloting suggested that this had been achieved to a large extent. The factors derived from the principal components analysis, which might be summarised as ‘quality of interactions with mental health workers’, ‘sense of alienation’ and ‘finance, daytime activities and social relationships’, are recognisable ‘clinical’ concepts.

The most notable omission from CUES—U is of an item (or items) relating to symptoms of mental illness (e.g. depressed mood, hallucinations, delusions, etc.). These did not figure prominently in the literature as issues that service users wanted to be addressed, nor during the process of asking service users their views, and so were not included. The CUES—U measure might be used alongside instruments that gather information about symptoms, or items relating to symptoms might be added through further development of the CUES—U measure.

How stable are CUES—U ratings?

Good test—retest reliability is an important property for any instrument that is intended to measure status or outcomes for service users and carers. It is therefore encouraging that, for all but one of the 32 questions relating to the 16 items, the correlations between ratings at two time points, 2-14 days apart, are moderately good or better. The ratings of CUES—U therefore do appear to measure states that are not influenced unduly by, for example, short-term fluctuations in the raters' emotional state. The exception is the Part A question relating to medication. It appears that people's perception of the benefits of their medication and their experience of side-effects is more subject to rapid change or fluctuation.

Do CUES—U ratings reflect severity?

The CUES—U is a new type of scale and there is no gold standard against which to compare it. The sub-study comparing CUES—U with HoNOS suggests that CUES—U scores do reflect severity. As a practitioner-rated measure, HoNOS offers an independent perspective on this.

The CUES—U has not been tested yet for its sensitivity as a measure of the outcome of care.

Do CUES—U items need to have three parts?

Part A questions for each item ask how well the person's situation compares with a standard descriptive statement. The purpose is first to focus attention on the specific issues to which the item refers, and second to increase the consistency of the person's response. Part B questions ask about how satisfied the person is with the issues to which the item refers. Although there are strong correlations between Part A and Part B questions for all items, some people do respond differently to the two questions. The most common situation is for a person to express dissatisfaction despite having rated the situation as comparing favourably with the descriptive statement. It is hypothesised that Part A questions might be more useful as a measure of state (and therefore possibly of outcome when repeat ratings are made after a period of care) and Part B questions as a vehicle for aiding communication between service user and practitioner, especially during care planning.

The third part to each item (Part C) is a free-text section. This proved popular with those who completed CUES—U during both pilot and field trials. It records information about the individual's situation that cannot be captured by the ‘tick-box’ nature of Parts A and B. It is hoped that this information will support communication with practitioners and inform care planning. The Part C item also might be of value to identify specific issues either for a particular group of service users or about a particular service.

In conclusion, the development and testing of CUES—U have demonstrated the feasibility of applying a self-rated measure of the expectations and experience of users of mental health services. However, more work is needed to explore potential uses of the instrument. Test of CUES—U as an outcome measure would require its application to a cohort of service users before and after a substantial period of health and social care. Assessment of its usefulness in service evaluation would require a test of whether CUES—U ratings reflect differences between services, or even developments over time within a service, for instance in the quality and extent of community care.

Clinical Implications and Limitations

CLINICAL IMPLICATIONS

1. • The Carers' and Users' Expectations of Services — User version (CUES—U) measures the domains considered important by users of mental health services.

2. • The CUES—U is self-rated. Its items have reasonable test—retest reliability, suggesting that they measure states that are quite stable. The CUES—U scores also reflect severity as rated by clinicians using the Health of the Nation Outcome Scales (HoNOS).

3. • The CUES—U has the potential to support individual care planning, audit, service evaluation and service planning.

LIMITATIONS

1. • About one-quarter of service users will seek help in completing the CUES—U. The type of help sought can generally be offered by friends, family or carers.

2. • The CUES—U has not been tested as an outcome measure.

3. • The CUES—U has not been tested extensively by people from minority ethnic groups.

APPENDIX

The CUES—U items and normative descriptions

The complete CUES—U instrument is protected by Crown copyright, and may be obtained from: Research Department, NSF, 30 Tabernacle Street, London EC2A 4DD.

1. 1. Where you live The place you live in should meet your individual needs. You should not have to worry about having to move out, and it should not be too out-of-the-way. You should be able to come and go when you want, be alone when you want and not be harassed by the people you live with, by staff or by neighbours.

2. 2. Money You should have enough money to pay bills, stay out of debt and not miss meals. You should not have to feel isolated or cut off from society because of lack of money.

3. 3. Help with finances Many people find that they need help with claiming benefits, filling in forms and working out how to manage their money. You should get as much help as you need to do these things.

4. 4. How you spend your day You should have the opportunity to spend your day in some form of regular and meaningful activity. This could be working, studying, training or going to a day centre or day hospital.

5. 5. Family and friends Mental illness can affect a person's relationships with the people that he or she cares most about. You should be able to maintain good relationships with the people closest to you.

6. 6. Social life You should have the opportunity to mix with people and form new friendships and relationships. To make this possible, you should have enough money, access to transport if you need it and the use of a telephone.

7. 7. Information and advice You should be given as much information as you want or need about the services and treatments available to you, about the Mental Health Act and how it works and about the mental health system generally. Some people find it helpful to have someone like them (such as another service user or a member of the same community) to explain things to them. The information you are given should be clear and easy to understand, and should be available as and when you need it.

8. 8. Access to mental health services You should be able to get help from your local mental health services when you need it, throughout the week, at any time of the day or night.

9. 9. Choice of mental health services A range of services should be available to you, and you should be able to choose those that closely match your needs, including complementary/alternative therapies, counselling and psychotherapy. You should have a choice about the mental health workers you meet with regularly (for example, being able to choose their gender or ethnic background) and be able to change workers if you do not get on.

10. 10. Relationships with mental health workers Doctors, nurses, social workers and other mental health workers should show you respect, be honest with you and discuss things with you in a way in which you can understand. They should keep information about you confidential or ask your permission before passing it on to others. If they pass on information, it should be accurate and save you from having to repeat yourself to new mental health workers.

11. 11. Consultation and control Mental health workers should not pressurise you to do anything that you do not want to, or take decisions on your behalf without getting your permission first. Even if you have been ‘sectioned’, people should show you respect, listen to you and take your opinions seriously.

12. 12. Advocacy You should be able to put your views across to people in authority. This can be difficult for several reasons, such as the effects of medication, if English is not your first language or if the situation is frightening or intimidating. If you want, you should have somebody (an advocate) to help or support you, or speak for you. You should feel that this person really understands what you want and genuinely represents your views when he/she speaks on your behalf.

13. 13. Stigma and discrimination You should feel safe and other people should not harass, exploit, victimise or be violent towards you. You should not experience stigma or discrimination at home, at work or from mental health workers, police or any other section of the community. People should not discriminate against you because of race, culture, religion, gender, sexual orientation, physical or mental disability or for any other reason.

14. 14. Your medication/drug treatment Medication should be given only to relieve the symptoms of mental ill health and to reduce your distress. All medication can have unwanted effects, but these should not cause more disruption to your life than improvement.

15. 15. Access to physical health services You should be able to get the treatment and care that you need for your physical health when you need it, whether you are in hospital or living at home. You should be able to be registered with a general practitioner and have regular check-ups from a dentist. You should have access to other types of care, such as opticians, chiropodists, physiotherapists and so on.

16. 16. Relationships with physical health workers The people who give you physical health care should listen to you, show you respect and take your condition seriously.

17. 17. Other issues There may be other issues that are important to you, in addition to those that the questionnaire has already asked about. Please use the space below to write down anything else that is important to you, that you would like help with or that you would like to change.