Clinical Guidelines in Mental Health II: A guide to making NICE Guidelines

This is the second in a series of three articles about clinical guidelines in mental health. In the first article (Kendall et al, 2004), we introduced the National Collaborating Centre for Mental Health (NCCMH) and its guidelines work programme commissioned by the National Institute for Clinical Excellence (NICE); the possible uses of clinical guidelines; and the different guideline products generated alongside each NICE guideline.

In this article we define clinical guidelines, and describe the process of NICE guideline production as undertaken by the NCCMH with examples from mental health guidelines, either completed (schizophrenia and eating disorders) or nearing completion (depression, eating disorders and self-harm).

What are clinical practice guidelines?

Clinical practice guidelines are systematically developed statements that assist clinicians and patients in making decisions about appropriate treatments for a specific condition (Mann, 1996). NICE clinical practice guidelines are evidence-based; which means that the advice contained therein must be derived from the best research evidence available, using predetermined and internationally agreed methods (AGREE: www.agreecollaboration.org).

The aim is to determine which treatments have the best evidence for their effectiveness, for which group of service users with a particular condition; and to determine this by interrogating the evidence in a systematic, critical and unbiased way.

The scope of NICE guidelines

Once a guideline has been commissioned, NICE, the Department of Health, the relevant National Collaborating Centre and a group of registered national stakeholder organisations (pharmaceutical companies, professional organisations and service user and carer organisations registered with NICE for each guideline) work jointly to agree the limits, or scope, of the guideline. Clearly, to appraise the evidence for the treatment of a condition, particularly for diagnostic categories as broad and heterogeneous as schizophrenia or depression, there needs to be agreement about precisely which treatments for which diagnostic group(s) are to be evaluated.

If the scope includes evaluating all possible treatments for all people with schizophrenia, for example, the time it would take to review all the evidence would probably take near to five years. By the time it was finished, significant parts of the guideline would be out of date and would need to be reviewed again. On the other hand, if the scope were too narrow, the guideline would only be applicable for a small number of service users and a limited range of treatments. In other words, the scope must be limited to allow the completion of a guideline for as broad a group of people as can be addressed within a 2-year period.

The Guideline Development Group

After agreeing the scope, the NCCMH, with advice from the parent professional organisations and the NCCMH reference group, appoint the Guideline Development Group (GDG) chair. The chair must have a good working knowledge of the condition and its treatments, have experience of managing multidisciplinary groups, and be able to lead the writing of the guideline - a sizable task, especially for the larger guidelines.

In addition, the GDG needs at least one national expert for each treatment modality included in the scope - usually pharmacological, psychological and service-level interventions. In addition, primary care representation is normally included.

Two service users and a carer ensure a strong service user focus, and help the GDG to address, for example, outcomes relevant to service users (such as quality of life and the experience of care), as well as outcomes mainly important to professionals (relapse rates, symptoms and suicide rates). Also to help steer the guideline towards everyday clinical practice, enthusiastic and engaged clinicians with a good grasp of evidence-based practice are also recruited. To date, almost all of those asked to join a GDG have willingly accepted.

The clinical questions the guideline should answer

With scientific, health economic and technical support from the NCCMH, the GDG starts the work by formulating the most important clinical questions the guideline should attempt to answer. The questions agreed upon should address all the treatment modalities and service user groups included in the scope, even if there may be little or no available evidence to answer them.

Clinical questions about interventions are structured in a way that facilitates a systematic examination of the evidence from randomised controlled trials (RCTs). So, each question must identify the service user population (P); the intervention (I) to be evaluated; the service or care with which the intervention is to be compared - the comparator (C); and the outcomes (O) against which the intervention and comparator will be compared. This is easily remembered as PICO. For example: The PICO framework is not so useful for questions relating to diagnosis or prognosis, but these questions should still be clear and specific.

For people with schizophrenia (P), do family interventions (I), when compared to other psychological treatments (C), decrease the likelihood of relapse (O)?

Searching for the evidence

Guided by the clinical questions, and having carefully defined the interventions and comparators to be addressed, the NCCMH review team and the GDG develop comprehensive search strategies of all relevant electronic databases (e.g. MEDLINE, EMBASE etc). The use of marker papers (to check the effectiveness of the search strategies) and hand-searching of specified journals are also undertaken for completeness. Usually, many thousands of studies are identified and collated at this stage.

In the search for high quality systematic reviews and RCTs, pre-agreed criteria of quality are used that address key aspects of trial methodology such as randomisation, allocation concealment, adequate description of withdrawals, and blinding. Studies of low quality, those that have incomplete or unusable data and those that do not clearly define populations or interventions are excluded after independent examination by at least two reviewers, double-checked with GDG experts where necessary.

Usually the number of studies suitable for inclusion for further analysis will be reduced substantially, and in some areas there may be no RCT level evidence remaining. On the other hand, if a recent high-quality systematic review can be identified (such as a Cochrane review), a further search for more recent RCTs may be sufficient to capture all relevant randomised controlled studies.

Analysing the evidence

If a systematic review and/or randomised studies of high quality can be identified, the data are extracted and loaded into Review Manager (Cochrane Collaboration, 2003) for meta-analysis. By pooling data on larger numbers of service users than would be found in a single RCT, the GDG is more likely to be able to answer the clinical questions, or at least to answer those questions with greater reliability.

For ease of analysis and collective discussion by the whole GDG, both the results from individual studies and the pooled data are shown as easy-to-understand graphs known as forest plots (Lewis & Clarke, 2001). As a visual display, these are much easier to understand for most statistically (relatively) naïve members of the GDG, and encourage broad participation in appraising the evidence. A forest plot showing the impact of family interventions, when compared to ‘all other interventions’, on relapse rates for people with schizophrenia, is shown in Figure 1.

Fig. 1. Forest plot of family interventions versus other psychological interventions for relapse.

Answering the questions: developing evidence statements

The clinical questions are then used to guide examination of the pooled data and to generate evidence statements. Evidence statements are developed using the same (PICO) pattern as the clinical questions. An example evidence statement for family interventions in the treatment of schizophrenia derived from the forest plot above is as follows:

For people with schizophrenia, there is clear evidence that family interventions, when compared to all other interventions (including ‘ standard care’), decrease the likelihood of relapse up to 15 months’post-treatment follow-up (N=5; n=305; RR=0.67, 95% CI 0.52-0.88). ¹

Each evidence statement is assigned a grade, depending upon the source of evidence used, adapted from an internationally agreed convention (Eccles & Mason, 2001). For the schizophrenia guideline, evidence derived from the meta-analysis of at least 3 RCTs, or from a single RCT with at least 300 participants in the trial, was allocated level Ia, whereas evidence statements derived from a single RCT of fewer than 300 participants was given a Ib grading.

In examining the evidence, two key issues emerge when comparing one treatment with another. First, is the difference in outcomes between the experimental intervention and the comparator statistically significant? Second, if the difference in outcomes is statistically significant, then is the difference in treatment effects clinically significant? In other words, is there reliable evidence that an intervention is better than the comparator, and is that difference clinically valuable (taking into account differences in side-effects and the acceptability of treatment)?

In the depression guideline (National Collaborating Centre for Mental Health, 2004a ), for example, comparing the efficacy of selective serotonin reuptake inhibitor (SSRI) antidepressants with placebo in the treatment of mild to moderate depression, there was a statistically significant difference, with SSRIs being superior in reducing depression scores; but the mean reduction in depression scores associated with SSRIs was only marginally greater than for placebo. The GDG considered the additional benefit from SSRIs to be too small to be of clinical significance.

A different problem arises when the evidence suggests that there is no difference between a treatment and its comparator. Here, the key issue is whether or not there is reliable evidence of no difference between the intervention and the comparator (evidence of absence); or is the evidence inconclusive (absence of evidence)? In this situation, the size of the confidence intervals is used to differentiate. Figure 2 shows the algorithm used for the schizophrenia guideline to help develop evidence statements accordingly.

Fig. 2. Guideline statement decision tree for the Schizophrenia Guideline. CI, confidence interval; ES, effect size.

When randomised trial evidence is lacking

Sometimes, no high-quality RCT evidence is available to address the clinical questions. This usually necessitates further searches for lower levels of evidence; then, through a process of informal consensus, narrative reviews are undertaken led by the relevant expert. In some instances, the GDG may have to depend upon published expert consensus statements or on the collective experience of GDG members. It is important to note that the RCT evidence can be absent for a number of different reasons. In mental health, it is commonly because genuine consent to take part in a trial may not be possible; for example in the use of rapid tranquillisation for behavioural disturbance (National Collaborating Centre for Mental Health, 2003). Alternatively, and just as commonly, it may be that the research has never been done.

Very occasionally, the lack of RCT evidence is the result of the unquestionable efficacy of a drug, making it unethical to undertake an RCT. In reviewing the efficacy of naloxone in the treatment of opioid overdose for the self-harm guideline, no randomised placebo-controlled efficacy trials could be found (National Collaborating Centre for Mental Health, 2004b ). The very rapid onset of action following intravenous injection of naloxone in reversing respiratory depression (and thereby saving lives) makes comparison with placebo unethical.

From evidence statement to clinical practice recommendation

Evidence statements are probabilistic statements that indicate the likelihood that a treatment will alter the outcomes for a particular population of people when compared to another intervention or control condition. Clinical practice recommendations are advisory statements about the value of an intervention for a particular group of people, based upon how confident we are in the evidence statements upon which the recommendations are based. And just as with evidence statements, recommendations should aim to address all the clinical questions.

To go from the evidence statements to recommendations about clinical practice, the GDG take a broad view of all the possible comparative outcomes for a treatment, including the likelihood of side-effects and the acceptability of treatment. For family interventions in the treatment of schizophrenia, the GDG made the following recommendation on the basis of its efficacy and acceptability in a wide range of studies:

Family interventions should be available to the families of people with schizophrenia who are living with or are in close contact with the service user (A) (National Collaborating Centre for Mental Health, 2003, p.107)

The evidence for family interventions also allowed the GDG to make a number of further recommendations, specifying subgroups of people who do well with family interventions, such as people who have recently relapsed, or those who have persisting symptoms after an acute episode has subsided. Moreover, additional recommendations were generated from the meta-analysis about the format of treatment, the minimum duration of treatment, and the minimum number of sessions likely to be needed to reduce relapse rates.

Recommendations are graded according to their evidential source so that busy clinicians do not have to examine the evidence base underpinning each recommendation. If a recommendation is based upon RCT evidence (i.e. level I evidence), then it is given an ‘A’ grading: whereas ‘ B’ graded recommendations are derived from non-randomised studies (evidence levels II and III), or based upon extrapolations from RCT data.

If good-quality clinical studies are lacking, then expert opinion is used to develop grade ‘C’ recommendations. Finally, the GDG often generates good practice points (GPPs) to address areas of practice where evidence is unlikely to be forthcoming or when value-based recommendations are deemed desirable by the GDG. An example of the latter would be the need for comprehensive assessment, the provision of information for service users and carers or the development of advance directives.

Integrating the recommendations: developing the ‘NICE guideline’

When all the recommendations have been agreed by the GDG, a series of care pathways are developed tracing the routes and contexts within which service users and carers could receive treatment and help within the NHS. For most mental health guidelines, this will include pathways through primary, community, secondary and tertiary care. Recommendations can then be inserted into the care pathways at the most appropriate points. Sometimes this will necessarily lead to a limited repetition of recommendations when the same treatment can be provided in different settings, for example when primary care and emergency department clinicians provide comparable treatments (such as the physical treatments for injuries caused by self-cutting).

The integrated care pathway with all the recommendations inserted (but without supporting evidence) forms the ‘summary of recommendations’ chapter in the full guideline. This chapter is then extracted unchanged to form the ‘NICE guideline’ for that condition. The version for service users, carers and the public is a plain English adaptation of the NICE guideline.

Other guideline products

For all guidelines currently under development or recently published, 10 key recommendations are selected by the GDG and listed at the beginning of the NICE guideline, and represent the main headlines for that particular guideline. The key recommendations are also used to develop an audit tool.

To help simplify the guideline for general NHS use, and to help develop a systematic approach to local implementation, each guideline also contains clinical algorithms and additional care pathways. These act as maps for clinicians and managers to assist in the local planning and delivery of care consistent with the NICE guideline. More concise versions of each new guideline for use as quick reference guides are also being developed.

When there is insufficient evidence to determine with confidence the value of an intervention, or when recommendations are only supported by expert consensus, the GDG make recommendations for research to fill the ‘ evidence gap’. For family interventions, the evidence for whether it should be used for people with schizophrenia in the first episode was insufficient to make recommendations, either positive or negative. The GDG therefore developed a research recommendation about the use of family interventions in this context.

Consultation

From the scope to the first complete draft of a guideline, with all its products and versions should take between 12 and 18 months, depending upon the size and compass of the disorder, the treatments reviewed and the breadth of the scope. The first drafts of guideline products are then submitted to NICE, placed on the NICE website and sent to all registered stakeholders, special advisers and experts suggested by the GDG. Consultees are given a month to read and return comments on each of the products.

After collating all the responses from stakeholders and others, the guideline developers (GDG and NCCMH team) are required to respond to all comments and suggestions, and make changes to the guideline where necessary, within a month of receipt. This is no small task. Larger guidelines can generate well in excess of 200 pages of tabulated and cross-referenced comments, each and every one of which must have a detailed response, with any accepted changes to the guideline being cross-referenced to all relevant guideline products. Throughout, changes made to any one guideline product must also be made to all other guideline products to ensure that all guidance is consistent.

At the end of the first consultation phase, consultees are sent responses to their comments from the guideline developers, along with a full set of modified guidelines with changes clearly indicated. In this second round of consultation, consultees can only comment on how the guideline developers have responded in the first consultation phase and whether the consultees are satisfied with the changes made to the guideline products. Again, a month is allowed for comments to be returned to NICE and passed on to the NCCMH, and a further month is given for the GDG and NCCMH to prepare and tabulate their response to consultees’ second round of comments and to make any further alterations to the guideline products.

At the end of this lengthy and labour intensive process, all guideline products, with tables of comments and responses, all cross-referenced to the appropriate guideline products, for both the first and second round of consultation, are submitted to a NICE Guideline Review Panel (GRP). Each NCC has an independent GRP with expertise in evidence-based practice, guidelines and direct experience of the relevant area of healthcare. Their role is to ensure that the guideline developers have responded appropriately to stakeholder comments and made any necessary changes, particularly focusing upon differences and disagreements between stakeholder comments and the responses made by the guideline developers.

When the GRP is satisfied with the final guideline products, the comments and the responses, the guidelines are finally reviewed by the NICE Guidelines Executive and signed-off (or not) for publication; and 2 years’ work for a sizable number of people comes to an end, only to be restarted some 2-4 years later.

In the last article of this series, we will focus on the schizophrenia guideline, its contents, how the guideline relates to other national initiatives, and the experience of developing the first NICE guideline.

Materials

All NICE guidelines referred to in this article are available on the NICE website. Paper versions of the NICE schizophrenia and eating disorders guidelines and the patient version can be ordered from the NHS Response Line on 0870 1555 455 (quoting number NO176 and NO177). The full Schizophrenia Guideline can be purchased from Gaskell (http://www.rcpsych.ac.uk/publications/gaskell), and the full eating disorders guidelines can be purchased from The British Psychological Society (http://www.bps.org.uk).

Declaration of Interest

The NCCMH is funded by NICE. NIMHE funded the production of the CD-ROM version of the training website for schizophrenia. T.K. and S.P. are in receipt of grants from the Department of Health.