The practice of evidence-based psychiatry today

Summary

Research that should change clinical practice is often ignored for years. Fifty-five new trials are reported in medicine every day. Psychiatrists need to have some way to sift, digest and act on new research to benefit their patients. This article outlines the key elements of evidence-based psychiatry in an attempt to address these needs. Initially, an evidence-based approach is about asking an answerable question, searching for the evidence and then critically appraising the available information for its validity and relevance. The fourth step involves integrating the evidence with clinical expertise and the patient's values. The whole idea is to improve real-life patient care. This article outlines a number of the tools needed to overcome some of the common problems that psychiatrists face when trying to find and use published research results to help solve patients' problems.

By 1972, Archie Cochrane of the Cochrane Library had become concerned that many decisions about medical treatments were still based on a ‘hunch’ or a ‘gut instinct’ (Cochrane 1989). Intuition, Cochrane felt, should be informed by the best available research evidence.

Throughout its history, psychiatry has been characterised by ideology and controversy. The specialty was often perceived as inefficient. Owing to concerns, psychiatry was one of the first clinical disciplines to undertake randomised controlled trials and exhibit an explicit enthusiasm for experimental design (Geddes 2000).

There is now an extraordinary amount of evidence in mental healthcare, but practitioners need to be able to find this information quickly and easily. The main barriers to using current research evidence in clinical practice are time and skill. It is difficult to remain knowledgeable about current research, and the up-to-date psychiatrist remains a mythical figure. Psychiatrists need to have some method to sift, digest and act on new research to benefit their patients.

Evidence-based psychiatry

The concept of evidence-based medicine was first used by David Sackett at McMaster University in Canada in the early 1990s. Evidence-based medicine is about the ‘conscientious, explicit and judicious use of current best evidence in making decisions about individual patients’ (Sackett 1996). The approach essentially involves integrating individual clinical expertise with the best available evidence from current research.

There are a number of key elements in practising evidence-based medicine in psychiatry. It is possible to summarise the approach in four simple steps (Box 1). The first step is to ask an answerable question generated by a patient’s condition, the next is to search for the answer, and the third is to critically appraise the retrieved evidence for its validity and relevance. The fourth step involves practitioners making a clinical decision by integrating the evidence with their own expertise and also the patient’s values and preferences. The whole idea is to improve patient care in real-life settings. The first issue then in putting an evidence-based approach into practice is to formulate a focused clinical question in response to a problem or scenario that is encountered in clinical practice.

BOX 1 Steps in evidence-based medicine

1. 1 Formulate an answerable question

2. 2 Find the evidence

3. 3 Appraise the evidence

4. 4 Apply the evidence

Formulating answerable questions

Professor Sackett (1996) suggests that asking a well-built clinical question is a fundamental skill. It is always best to convert a knowledge gap into an answerable, explicit question that in turn generates a set of key words that will help to search for the evidence.

Many issues in daily practice involve a treatment question about how to select a therapy that, it is hoped, will not do more harm than good.

A practical example

Let us suppose that a patient asks whether talking therapy is better than tablets for depression. The question is triggered by the needs of an individual and can be built systematically by developing a number of key words. As the issue here concerns treatment, it fits neatly into a format that will incorporate a population, an intervention, a comparison and an outcome.

There are four parts to any question’s anatomy and the question is best constructed in the well-known PICO format (Heneghan 2006) (Box 2). The terms identified from this question formulation will form the best search strategy. In the case of our patient who wants to know whether psychotherapy is better than medication for low mood, the question may contain four key elements: the patient is an adult with depression; the intervention could be psychotherapy; the comparison might be medication; and the outcome may be recovery from depression. This question can now guide our search for the best evidence to answer the patient’s query.

BOX 2 The PICO question format

According to this approach, the question should include:

• Patient’s diagnosis

• Intervention or treatment of interest

• Comparison or control intervention

• Outcome of interest

(Heneghan 2006)

Finding the evidence

The next task is to find the best evidence quickly and efficiently. Medical literature doubles every 10 years and can lead to information overload; reviewing all of the available literature is not practical. Searching skills, then, are necessary for every clinician hoping to stay up to date. Finding the best evidence in psychiatry requires knowledge of the most appropriate information sources and the best ways to search them (De Brun 2009).

The first principle of good searching involves using the best key word or phrase. Once the key words are identified, a note can be made of related terms or relevant synonyms. The clinical issue can then be explored by searching the literature, such as journal articles that are indexed in medical databases and other online resources (Box 3).

BOX 3 Important electronic databases

1. • Cochrane Library (www.thecochranelibrary.com)

2. • Cochrane Central Register of Controlled Trials (CENTRAL) (http://onlinelibrary.wiley.com/o/cochrane/cochrane_clcentral_articles_fs.html)

3. • MEDLINE (www.nlm.nih.gov/bsd/pmresources.html)

4. • Embase (www.embase.com)

5. • PsycINFO (www.apa.org/pubs/databases/psycinfo/index.aspx)

The internet

Frequently, a patient’s first port of call is the internet. Results from an internet search, however, vary in content, authorship, currency and, most important, quality. The web is a vast collection of information with no guarantee of reliability or accuracy. Internet users rarely go past the first page of hits but popularity is not the same as reliability. There is little empirical evidence supporting the value of using general internet search engines to identify potential studies in the billions of pages of the web (Eysenbach 2001).

Online clinical databases such as MEDLINE and Embase contain references to journal articles that have been organised for easy retrieval and are a much better bet. These ‘bibliographic’ databases address similar topics, but from a different point of view, so whereas there is some overlap, it is essential to search more than one database to be comprehensive (De Brun 2009). In terms of online clinical databases, the Cochrane Library is the best place to start.

The Cochrane Library

The Cochrane Library is a collection of databases that contain high-quality, independent evidence to help healthcare decision-making. Cochrane overviews, made up of systematic reviews and meta-analyses, represent the highest level of evidence on which clinical treatment decisions can be based.

A systematic review is a summary or overview of what is known and what is not known about a topic. A meta-analysis is simply a synthesis of the results of the systematic review. These overviews are known as ‘secondary evidence’ because they draw together a wide range of primary research in an accessible and usable form.

Users can search the whole Cochrane Library or individual databases within it, which may contain overviews or the primary randomised controlled trials. The full text of the article is often available to print. Importantly, the Cochrane Collaboration has developed a growing database called the Cochrane Central Register of Controlled Trials, known as CENTRAL. The emergence of this database is significant – it is now considered to be the best single source of reports of trials (Lefebvre 2008).

After CENTRAL, two databases are generally considered to be the most important sources to search for clinical trials: MEDLINE and Embase. These are often available through national provision or licensed to an individual hospital and are generally considered to be the key international, general healthcare databases (Lefebvre 2008).

MEDLINE and PubMed

MEDLINE is the major bibliographic database for biomedical literature. The database contains citations and abstracts from journals published in the USA, the UK and many other countries. (A citation is the name and date of an article; an abstract is a brief description of the report.)

MEDLINE, which is US-based, contains more than 18 million references to journal articles from the mid-1940s onwards (US National Library of Medicine 2011). PubMed is the free version of MEDLINE and in some ways is preferable to it. PubMed includes up-to-date citations not yet indexed by MEDLINE and also has records from a wider scope of journals.

In most cases, results from searching these databases are available as only a brief abstract, but a growing number of records are now linked to free, full-text articles.

Embase

Embase has more than 24 million records from 1974 onwards. This international database is the European version of MEDLINE and contains abstracts on medicine and pharmacology. Currently, over 7500 journals are indexed (Embase 2011).

There is considerable overlap, but a third of journals indexed by MEDLINE are not covered by Embase. The two databases will return a similar number of relevant citations, but not necessarily the same ones. A systematic review has shown that as little as 30% of all known, published, randomised controlled trials were identifiable using MEDLINE (Dickerson 1994). As a result, a comprehensive search requires that both databases be searched.

PsycINFO

PsycINFO is another important database for psychiatry and contains records from 1887 onwards. Produced by the American Psychological Association, this database contains abstracts of articles from both psychology and psychiatry.

Building a search strategy

Online electronic databases allow thousands of articles to be searched relatively quickly. Many hospitals now have good internet access and the ability to search these databases effectively is an important aspect of evidence-based medicine.

It is generally recommended that doctors become familiar with the Cochrane Library and MEDLINE. The Cochrane Library databases are available free, as is the user-friendly PubMed, the most widely searched database for biomedical literature.

Once the key words have been decided and the appropriate databases selected, it is time to run the search.

Our practical example

Our question is about a treatment, either medication or psychotherapy, and so the first port of call is an overview of randomised controlled trials in the Cochrane Library. The next form of evidence to be looked at is the individual randomised controlled trials themselves in the Cochrane Library and in MEDLINE.

Clearly we cannot type the whole detailed clinical problem into a search engine like Google. If we enter ‘Is psychotherapy better than medication in helping a young man become less depressed?’, it is likely that we will retrieve no results. It is better to type specific key words such as, ‘adult’, ‘depression’, ‘treatment’ and ‘remission’.

For a comprehensive search, it is important that all of the alternative terms, spellings and synonyms are also used.

The process of building a search strategy involves the use of Boolean operators (Box 4). These combine search terms and make them more relevant to the research question. They include combining terms such as ‘AND’ (which narrows the search) and ‘OR’ (which broadens it). The use of Boolean operators enables users to find the most relevant references to quickly answer a clinical question.

BOX 4 Boolean searches

A Boolean search uses the ‘operators’ AND, OR and NOT to refine a search:

1. • ‘depression’ AND ‘cognitive therapy’ – this will find results that mention both depression and cognitive therapy

2. • ‘depression’ OR ‘depressive disorder’ – this will find pages that mention either depression or depressive disorder

3. • ‘depression’ NOT ‘medication’ – this will find pages that mention depression but not medication

Our practical example

In our example, we can type in ‘depression’ OR ‘depressive disorder’ AND ‘cognitive therapy’ AND ‘medication’ AND ‘remission’ to find research that will answer our question.

Free text v. controlled language

Free text is known as ‘natural language’ and is the natural way to search databases using everyday words such as ‘depression’ or ‘talking therapy’. However, free text can retrieve many irrelevant articles.

A more effective way of searching is to use the ‘controlled language’ that has been developed specifically to search a particular database. Each article in the database is allocated several medical subject headings (MeSH terms; www.nlm.nih.gov/mesh/meshhome.html) to identify essential themes, such as ‘psychotherapy’ or ‘clinical depression’. This aids easier retrieval of articles on a topic while keying in fewer search terms. However, indexing is influenced by human error, so a combination of free-text searching and controlled-language searching is advised to prevent relevant articles being missed.

Refining search results

A search strategy can be sensitive or specific. A sensitive search picks up many articles relevant to a clinical question, but also a lot of unwanted material. A specific search will retrieve more relevant articles, but some important reports may be missed. The best way to deal with this is to start with a broad search and then narrow it if necessary.

Another way to limit a search is by publication type. Using the ‘Publication Type’ field in MEDLINE, users can search for a ‘meta-analysis’, a ‘randomised control trial’ or a ‘practice guideline’. In other words, the search can be limited to a particular study design. Users can then click on the title of the retrieved report to see the abstract or indeed the full-text version. Full text is now available electronically from an increasing number of journals. Using the ‘Results Manager’ at the bottom of the page there are options to print, email or save the results.

So far, the initial steps of the search process have involved formulating a clear, answerable question, selecting relevant databases and then identifying appropriate search terms and synonyms for the search. Using key words and free-text terms, combined with Boolean operators, it is then possible to search the Cochrane Library, MEDLINE, Embase and PsycINFO efficiently. This search strategy should identify articles relevant to the clinical question.

The search strategy may yield a number of articles, but perhaps only one is relevant. The next step is to appraise this article for its validity and clinical usefulness.

Appraising the evidence

The quality of the evidence found is often variable. Putting unreliable evidence into practice could lead to harm or a waste of resources. An evidence-based approach can help to simplify and clarify a complex clinical article by providing a framework for assessing the validity and relevance of a study.

Critical appraisal is a key component of evidence-based practice. It is also an important exam topic and is increasingly relevant for trainers and clinicians. In appraising any study it is important to consider how well the research was designed. Naturally, in deciding how much weight to attach to the conclusion of a study, it is important to reflect on how well or rigorously that study was performed. There are a number of essential steps to critically appraise an article found following a search (Box 5). Essentially, readers want to decide whether the evidence found is valid or close to the truth. They also want to assess the report to see whether it is important or clinically useful and whether the results can help their patient.

BOX 5 Questions in critical appraisal

1. • Are the results valid?

2. • Are the results important?

3. • Are the results helpful?

Are the results of the study valid?

Randomisation

Initially, practitioners need to ensure that the researched question was focused in terms of the population studied, the intervention used and the outcomes measured. Next, they need to assess whether the groups in the study were randomised. The major reason for randomisation is to create two groups that are similar at the start of the trial. To reduce bias or error as much as possible, the decision as to which treatment a patient receives, such as a talking therapy or an antidepressant, should be determined by random allocation.

A randomised controlled trial provides the best evidence because randomisation spreads all confounding variables, even unknown ones, evenly among the study groups. It generates two groups that are equal in all important ways. In asking whether the evidence about a treatment is valid, readers may find that a study falls at the first hurdle if there was no randomisation of patients to the different treatment arms.

Allocation concealment

Doctors who are entering patients into a trial may unconsciously distort the balance between the groups. As a result, it is best that the randomisation list be concealed from the allocating clinicians. This is known as ‘allocation concealment’ and it is important to look for this in critically appraising any randomised controlled trial (Heneghan 2006). The concealment of randomisation occurs before patients are enrolled in the study, whereas masking happens afterwards.

Masking

Masking (‘blinding’) of participants, outcome assessors and others to treatment status should be looked for in the report of the study. Ideally, patients and doctors should not know who is receiving the treatment. Observer bias may occur when assessors unconsciously bias their assessment of outcomes if they are aware of who is receiving the treatment of interest. Of course, it is not always possible to mask patients because of the nature of the treatment (such as cognitive therapy). However, it may be possible to mask the assessors to the patients’ treatment allocation.

Accuracy of outcomes measures

Another major validity issue is the accuracy of the outcomes measures used. Generally, the more objective the outcome measure, the better. It is also important that the patients were analysed in the groups to which they were assigned and that the trial lasted long enough to allow the outcome of interest, such as a reduction in symptoms of clinical depression, to become manifest.

Patient loss

Next, it is important to see whether all of the patients were accounted for at the end of the trial. If many individuals dropped out then the study results would be in doubt. In a good quality study, there should be a low loss to follow-up. If less than 80% are adequately followed up, the results might be considered suspect.

Flaws

All studies are flawed in some way in terms of the issues just raised, but what is the impact of these flaws? It is important to conceptualise the study as a whole, not just the component parts. Will the combined impact of the flaws substantially change the overall result? Answering this question allows practitioners to judge the overall quality of the study before going on to decide on its importance.

Are the results important?

Two things to consider here are how large the treatment effect is and how likely it is that the effect is due to chance alone (Akobeng 2005). The treatment effect in a randomised controlled trial may be reported in various ways, such as absolute risk, relative risk, odds ratio and number needed to treat. A large treatment effect may be more important than a small one.

The observed treatment effect may be due to systematic bias, chance or, ideally, the actual impact of the treatment. Once bias has been excluded, within reason, by deciding that the trial is valid, it is time to consider whether the results might be a chance effect, rather than the direct result of the treatment itself.

Statistical significance

Statistical significance refers to the likelihood that the results of a study are not due to chance alone. P-values and confidence intervals can be used to assess statistical significance. The P-value allows us to consider the probability that the results are a chance effect. By convention, P = 0.05 is set as the threshold for statistical significance. Thus, a P -value that is less than 0.05 is often reported as ‘statistically significant’: the treatment effect is unlikely to have happened by chance. This P -value is interpreted as being small enough to justify rejection of the null hypothesis that there is no difference in effectiveness between, in our patient’s case, psychotherapy and medication.

Our practical example

For our question, we need to assess whether the result (showing the superiority of cognitive–behavioural therapy) could still have occurred by chance even if there was no difference between cognitive therapy and medication. As well as the actual results, the study should report a measure of the likelihood that our result could still have occurred by chance, even if cognitive–behavioural therapy were no better than antidepressants.

Confidence intervals

Next, practitioners need to view the confidence interval. A confidence interval is a range of values within which it is possible to be reasonably sure that the true effect of the treatment actually lies. A confidence interval (CI) describes the uncertainty inherent in a best guess at the treatment’s effect. A 95% CI is often interpreted as indicating a range within which it is possible to be 95% confident that the true treatment effect lies. If the confidence interval is narrow, the effect of the treatment is known more precisely. Intervals that are very wide indicate greater uncertainty and are less clear on the precision of a given result.

If the confidence interval for a mean difference includes 0, or for the odds ratio includes 1, then it is not possible to demonstrate a difference between the groups being compared and the result is ‘not statistically significant’. If however, the confidence interval excludes a mean difference of 0 or an odds ratio of 1, than the result is regarded as statistically significant and not likely to be due to chance.

Are the results relevant?

Once systematic bias and chance have been ruled out, the appraiser can move on to quantifying the benefit of the treatment. This can be done by seeing if there is a significant difference in event rates between the two groups, in our case, those who received cognitive therapy and those who got medication. The number needed to treat† is regarded as the most useful measure of the benefit of a treatment. It shows the average number of patients who need to be treated to achieve one good outcome over a given time.

Next, it is important to ask whether there are compelling reasons why the valid, important evidence found should not be applied to a particular patient presenting.

Our practical example

For our question, we need to be confident that the patients in the study, severely depressed in-patients, are not in some way fundamentally different to out-patients. The treatment (here cognitive–behavioural therapy) also needs to be available in our service.

Applying the evidence

Finally, it is especially important to incorporate the patient’s values and preferences into any decision on a course of action such as cognitive therapy or antidepressant medication, because both of these approaches can elicit considerable ‘allegiance bias’ from the patient. Eliciting the patient’s preferences needs to be done in the context of a therapeutic alliance. A central feature of evidence-based medicine concerns the integration of research evidence with clinical experience and also the patient’s values.

Some clinical questions are formidable and valid evidence can be scarce. It is easy to rubbish a report but a study, despite its shortcomings, may represent the best evidence to date. A report may offer some guidance and, sometimes, any guidance about a treatment is welcome.

In an evidence-based approach, doctors need to be able to formulate questions that are answerable, find the evidence quickly, appraise the evidence effectively and then integrate the evidence with clinical experience and the patient’s preferences. Once they have decided that the evidence is valid and important, they then decide whether they can apply it to their patient. If the critical appraisal, however, indicates a fatally flawed article, then they can consider ignoring the article and looking elsewhere. If the article is seriously flawed then the effect it describes may not appear in the patient in the way described in the study.

Continuing professional development

In psychiatry there is a gap between research and treatment, which manifests as unwarranted variations in clinical practice. The inevitable result of these variations is that some patients are not receiving the best psychiatric care.

The practice of evidence-based medicine involves a process of life-long, self-directed learning in which caring for patients creates the need for important information about clinical and other healthcare issues. Secondary journals that summarise the available evidence are part of an array of resources that help psychiatrists to use the strategies of evidence-based medicine as part of continuing professional development. The methodology of reviewing evidence has also advanced greatly with the production of high-quality overviews by the Cochrane Library, together with well-presented clinical practice guidelines. Increased access to the electronic databases containing these systematic reviews and guidelines can also aid continuing professional development.

Evidence-based medicine harnesses the latest advances in clinical epidemiology, biostatistics and information science to produce a coherent and comprehensive approach that allows psychiatrists to base their practice on the best available evidence as part of continuing medical education. Evidence-based medicine uses the most effective strategies for changing professional practice (Geddes 1998). It enables psychiatrists to become self-directed, problem-based, adult learners who recognise gaps in knowledge, locate the best evidence, critically appraise and then integrate it with their clinical experience. The adoption in 1999 of a critical review paper into the membership exam of the Royal College of Psychiatrists is one of the most unequivocal recognitions of the central importance of skills in evidence-based medicine by any of the medical Royal Colleges.

Evidence-influenced psychiatry provides doctors with the tools needed to overcome some of the common obstacles that they face when trying to find and use published reports to help solve patient problems. Using an evidence-based approach certainly is not easy. It requires the rigorous scrutiny of unfamiliar methodology and involves the questioning of assumptions and a commitment to believing the evidence, even when it contradicts a cherished belief. The approach induces a sense of humility.

MCQs

Select the single best option for each question stem

1. 1 Evidence-based medicine does not emphasise:

   1. a critical appraisal

   2. b formulating an answerable question

   3. c applying evidence to individual patients

   4. d searching bibliographic databases

   5. e gut-feeling.

2. 2 Important electronic databases include:

   1. a Facebook

   2. b PubMed

   3. c Google

   4. d Textbooks

   5. e Bing

3. 3 Important questions in critical appraisal include:

   1. a Was there randomisation?

   2. b Were the readers masked?

   3. c Was a cookbook approach taken?

   4. d Do the results fit in with your gut feeling?

   5. e Do the results conform to your initial hunch?

4. 4 Names closely associated with current evidence-based practice include:

   1. a Sigmund Freud

   2. b R. D. Laing

   3. c David Sackett

   4. d Wilhelm Reich

   5. e Carl Jung.

5. 5 Asking answerable questions includes:

   1. a basing the question on general, abstract problems

   2. b using the PICO format

   3. c eliminating key words from the search strategy

   4. d omitting the intervention in a therapy question

   5. e avoiding the outcome measure in therapy questions.

MCQ answers

1

e

2

b

3

a

4

c

5

b