Leaflets and other information packages are regarded as integral to educational strategies designed to promote physical health (Reference CoulterCoulter, 1998). These packages provide patients with information to support their participation in treatment decisions and in the management of their health problems (Reference CoulterCoulter, 1997). Much work goes into the production of educational materials. However, to be effective they must contain comprehensible, relevant and accurate information. Unfortunately, few patient information materials currently meet these standards and many leaflets are inaccurate or misleading (Reference CoulterCoulter, 1998).
This study evaluates the quality and usefulness of educational materials about schizophrenia that were regularly used by mental health professionals working at a large primary care trust in Scotland. This evaluation was important because psychoeducation is clearly recommended as part of the basic psychosocial management of schizophrenia (Scottish Intercollegiate Guidelines Network, 1998). Also, a baseline audit of trust practice suggested a need for greater rigour in the provision of information and education to patients, families and carers.
Various tools have been advocated to enhance the quality of health information (Reference Entwistle, Watt and DavisEntwistle et al, 1996; Reference Jadad and GagliardiJadad & Gagliardi, 1998; Reference Silberg, Lundberg and MusacchioSilberg et al, 1997). The NHS Direct website identifies the Discern questionnaire as the first standardised index of the quality of consumer health information (Reference CharnockCharnock, 1998). Discern has undergone detailed development and review and particularly assesses the quality of written information about treatment choices. However, there are no publications regarding its use in the assessment of educational materials about mental health topics. This study had the following objectives:
-
(a) To evaluate the quality and usefulness of educational materials about schizophrenia used regularly by staff at the Greater Glasgow Primary Care Trust.
-
(b) To document the strengths and weaknesses of all the packages reviewed.
-
(c) To comment on the acceptability and reliability of the Discern tool in assessing mental health information.
Method
The study began with a stocktake exercise to collate materials that were currently used by mental health professionals working in community and in-patient settings. Locality teams were contacted and asked for information on and copies of educational materials being used within their service. A list of the educational materials provided was compiled and sent back to the same staff seeking confirmation that the data-set was complete.
Each piece of educational material was initially assessed to determine if it included comments on treatment choices and whether it was widely available to staff in other trusts. Two independent raters then assessed each publication using the Discern questionnaire. This comprises 16 questions rated on a 1-5 scale. The questionnaire is divided into three sections. Section one (eight questions) concerned reliability, section two (seven questions) measured the quality of information on treatment choices and section three (one question) was an overall rating based on the preceding 15 questions. The scores for each section and the overall rating ranged from 1-5. Ratings using Discern are classified as good, fair or poor according to the following criteria:
Good — score = 4 or 5. The publication rated highly on the majority of questions. A high overall score indicates the publication is a useful and appropriate source of information about treatment choices.
Fair — score = 3. The publication rated high and low on a similar number of questions, or most scores fell in the mid-range. A moderate overall rating suggests the material is a useful source of information on treatment choices but has some limitations. Additional supporting information would be required.
Poor — score = 1 or 2. The publication rated low on most questions. A low overall quality rating indicates the publication is poor quality — it has serious shortcomings and is not a useful or appropriate source of information about treatment choices. It is unlikely to be of benefit and should not be used.
Interrater reliability was recorded and a κ score calculated. All scored publications were summarised for strengths and weaknesses. The overall findings and ratings were compiled into a table and fed back to the clinical teams.
Results
Fifteen publications were identified as being in regular use in the trust for patients with schizophrenia, their families and carers. Four were excluded from further assessment. Two publications were linked specifically to pharmacological products and were not considered suitable for generic use. Two other packages were excluded as they were used as part of a local psychoeducation project on schizophrenia, were undergoing review and were not widely available outside of the trust. Eleven publications were independently scrutinised using the Discern tool. Interrater reliability for overall rating was very good, with a κ of 0.87 (P<0.0001). A review of ratings of each Discern question showed the largest difference in ratings (only two marks on the item rating scale) occurred on only one occasion.
As shown in Table 1 none of the publications achieved either a maximum score or a minimum score, indicating that every publication had at least some good aspects but also some limitations. Only two documents showed disparities in the ratings of section one (reliability of information) and section two (quality of information on treatment choices) of the Discern questionnaire. The first (Reference Wilkinson and KendrickWilkinson & Kendrick, 1996) scored 3.3 for reliability and 5.0 for quality of information. The second (PatientWise, 1997) scored 1.9 and 3.2 on sections one and two, respectively. Based on the Discern ratings, four publications currently in use could be recommended with confidence and two recommended for withdrawal. The remaining five publications got a moderate or fair score, indicating some benefits if used in conjunction with other educational materials.
Table 1. Quality of educational materials about schizophrenia
| Title of publication | Overall rating | Quality of publication |
|---|---|---|
| Caring and Coping (National Schizophrenia Fellowship, 1996 (NSF)) | 4 | Good |
| A Carer's Guide to Schizophrenia (Wilkinson & Kendrick, 1995) | 4 | Good |
| Schizophrenia: The Forgotten Illness (SANE, 1993) | 4 | Good |
| Coping with Schizophrenia (Prelapse, 1995) | 4 | Good |
| Schizophrenia: Notes for Relatives and Friends (Reference Leff, Berkowitz and Eberlein-FriesLeff et al, 1991) | 3 | Fair |
| Schizophrenia (NSF, 1998) | 3 | Fair |
| Schizophrenia (NSF & Royal College of Psychiatrists, 1998) | 3 | Fair |
| Schizophrenia: Changing Minds (Royal College of Psychiatrists, 2000) | 3 | Fair |
| Talking about Schizophrenia (Health Education Board for Scotland, 1994) | 3 | Fair |
| Understanding Schizophrenia: A Guide for Sufferers and Carers (Lilly Psychiatry, 1997) | 2 | Poor |
| Medical and Health Information for Patients: Schizophrenia (PatientWise, 1997) | 2 | Poor |
Interestingly, there were three weaknesses common to all documents: most failed to provide details of any external appraisal of the information presented, sources of information were not adequately described or dated and areas of uncertainty were not clearly acknowledged. Other recurring weaknesses were failure to provide references, inadequate descriptions of the risks of treatment and failure to consider the ‘no treatment’ option. Last, the date of publication was only printed on four of the documents.
Discussion
This exercise to evaluate the quality and usefulness of the main schizophrenia educational material in the trust showed that most of the material was at least of fair quality. Although the overall score of each publication does not provide any great insight into the strengths and weaknesses of a publication, the Discern questionnaire can provide a greater level of detail if item scores are reviewed.
Having used Discern, the documents that scored most highly shared the following characteristics:
-
• the documents started with clear aims on what the publication was about, what it was to cover and who it was for
-
• information was provided that was specific and relevant to its target audience
-
• sources of information and dates of publications were clear
-
• details of other sources of appropriate information on the topic were provided
-
• the mechanism of how treatment actually works, and its risks and benefits were detailed
-
• information was objective and uncertainties were acknowledged
-
• the consequences of not having treatment were detailed
-
• choices in treatment were highlighted
-
• the idea of shared decision-making was promoted.
The Discern questionnaire was relatively easy to use and interrater reliability for assessments was good. The learning time was also acceptable, taking about 1-2 hours initially to become familiar with the rationale and layout of the document. However, rating each document took 1-3 hours as the reader has to become very familiar with the publication being rated to ensure each Discern question was rated as directed. There are also one or two limitations of the Discern document. First, the developers clearly state that it cannot be used to assess the scientific quality or accuracy of the evidence on which a publication is based — this means information has to be checked against other sources. Second, Discern is described as a stand-alone tool, allowing judgement of a publication without specialist knowledge of the topic under review. Having used the tool, this is something that the project group would not entirely agree with. For example, knowledge on the possible mechanisms of action of certain treatments is required in order to assess if a document described this fully. The same applies to judgements of the range, benefits and risks of treatments. However, these limitations aside, this is a robust tool that can be used to evaluate medical and psychiatric educational materials.
Finally, it is important to note that a major advantage of using the Discern questionnaire and then providing detailed feedback on ratings of reliability and quality of information, is that staff and service users were able to make informed choices about which educational materials to access. This approach avoids the problems that can arise if staff perceive practice guidelines as too prescriptive. To ensure that assessment of the quality of publications used is maintained in the future, one member of staff from each locality team has been trained in the use of Discern so that they can evaluate the quality of new educational materials and provide feedback to their team members.
Qualitative details on the strengths and weaknesses of each document reviewed are available from the first author.
Open access
eLetters
No eLetters have been published for this article.