Chapter Eight - Aligning evidence for use in decisions: mechanisms to link collated evidence to the needs of policy-makers and practitioners

Summary

A major challenge in linking conservation science and policy is deciding how, and when, to offer relevant science to decision-makers to have the greatest impact on decisions. This chapter argues it is a question of alignment – of selecting the right knowledge to address the needs of decision-makers, ensuring that knowledge is accessible to them, and articulating it within their decision-making processes. The chapter describes three mechanisms to enhance this alignment: decision support tools; active knowledge exchange mechanisms; and large-scale scientific assessments. For each, we provide examples and draw out guidelines regarding circumstances in which the mechanism is likely to be most effective. No single mechanism is consistently best at aligning evidence with policy and practice. Each has strengths and weaknesses, and can be applied in different circumstances and at different scales. The chapter ends with a call for these mechanisms that link synthesised evidence with policy and practice decisions to be funded sufficiently, alongside environmental research, to enable adherence to core values of salience, legitimacy, credibility and transparency.

8.1 Introduction

We should not be surprised by the scale of the challenge when trying to link a body of scientific knowledge to the complex, shifting and seemingly unpredictable world of policy, or to the massively decentralised, globally distributed world of conservation practice (Young et al., 2014). One side of the challenge is developing a consensual understanding of the science itself. By nature, scientific knowledge is continually progressing, with theories, empirical data and new interpretations emerging all the time. Even within a single discipline, it can be hard to convey what is known at a particular point in time, and this often involves presenting different scientific viewpoints. For instance, there is substantial variation around the world in public health advice regarding alcohol consumption, with ‘safe’ limits in the UK being 50% of those in the USA (Wood et al., 2018). In conservation, the challenge is even greater, as relevant research cuts across the natural, physical and social sciences.

The other side of the challenge is working out how, and when, to offer relevant scientific knowledge to decision-makers, in order to have the greatest impact on the decisions being made. This is the focus of our chapter. We argue that it is a question of correct alignment: of selecting the right knowledge to address the needs of decision-makers, ensuring that knowledge is accessible to them, and articulating it within their decision-making processes.

First, we consider how well current efforts to synthesise evidence in conservation align with the needs of decision-makers. Then we describe three mechanisms that might be used to enhance the alignment of available knowledge with decision-making, starting at small local scales and moving to the global scale: decision support tools, active knowledge exchange and large-scale scientific assessments. For each mechanism, we provide examples and draw out general guidelines regarding the circumstances in which it is likely to be most effective.

8.2 How well do current evidence synthesis activities align with policy and practice needs?

When scientific evidence is needed for decision-making, the process of obtaining and analysing the evidence is often demand-led. An organisation faced with a difficult management or policy decision will undertake or commission a review to answer a specific question. For example, the UK Government Department of Environment, Food and Rural Affairs (Defra) commissioned a review of evidence on the status of pollinators (Vanbergen et al., 2014) before designing the National Pollinator Strategy for England (Defra, 2014). When this happens, the evidence synthesis is well-aligned with the policy and practice needs, summarising relevant material that can be found in the time available. However, it also puts immense time pressure on the evidence synthesis process, because decision-making can only happen once the evidence has been reviewed. This tends to lead to the selection of evidence synthesis methods such as rapid evidence assessments, traditional non-systematic literature reviews and expert consultations, which are not the most rigorous or unbiased approaches available (Dicks et al., 2017).

The Collaboration for Environmental Evidence (www.environmentalevidence.org) and the Conservation Evidence project (www.conservationevidence.com) aim to address the needs of conservation practitioners and policy-makers with more rigorous methods of knowledge synthesis, namely systematic reviews, systematic maps (Collaboration for Environmental Evidence, 2013; see also Chapter 7) and subject-wide evidence syntheses (Sutherland et al., 2019b; see also Chapter 4). They do so by actively involving stakeholders in the selection of topics to synthesise and the collation and subsequent evaluation of the evidence found (Dicks et al., 2016; Haddaway et al., 2017).

To evaluate the overall success of this alignment effort, we recently asked how well evidence collated by the Conservation Evidence project on the subject of sustainable food production matched the priority knowledge needs of decision-makers. Five independent exercises (Pretty et al., 2010; Dicks et al., 2013a, 2013b; Ingram et al., 2013; Jones et al., 2014), involving 240 people from across business, practice, policy-making and academia, had generated 286 priority questions faced by decision-makers. We sorted these into five categories, following the Driver–Pressure–State–Impact–Response (DPSIR) framework (Maxim et al., 2009). This conceptual framework describes interactions between society and the environment in a way that is meaningful for policy. Social and economic developments (Driving Forces, D) exert Pressures (P) on the environment and, as a consequence, the State (S) of the environment changes. This leads to Impacts (I) on ecosystems, human health and society, which may elicit a societal Response (R) that feeds back on D, S or I. We added a category for questions about underlying science that did not fit the DPSIR categories (Figure 8.1).

Figure 8.1 Categorisation of 286 priority questions identified by stakeholders as relevant to sustainable food production (Pretty et al., 2010; Dicks et al., 2013a, 2013b; Ingram et al., 2013; Jones et al., 2014) according to the Driver–Pressure–State–Impact–Response framework. Examples of questions are provided for each category. The extracted segment represents questions already answered by evidence summaries provided by the Conservation Evidence project.

Of all the priority questions, 189 (66%) were about responses (R), which are the focus of the Conservation Evidence project. Evidence had already been summarised that could help answer 35 of these questions (12% overall; Smith et al., 2015; Sutherland et al., 2019a).

In a similar vein, Cook et al. (2013a) investigated the contribution of systematic reviews to conservation decision-making, finding that 35% of the 43 reviews considered practical on-the-ground management, while most addressed interventions relevant to policy. Cook et al. (2013a) argued that the benefits for conservation could be significantly enhanced by increasing the number of systematic reviews focused on questions of direct management relevance.

These two analyses show there is some alignment between high-quality evidence synthesis methods and the needs of conservation practitioners and policy-makers, but it could be improved. Below, we provide a series of examples of mechanisms to enhance this alignment at a range of scales.

8.3 Decision support systems

Decision support systems are tools designed to assist decision-makers, for example, by visually or numerically illustrating different possible outcomes to a question, or leading users through logical decision steps (Dicks et al., 2014). Often software-based, they represent a link between relevant science and decision-making (Dicks et al., 2014; Figure 8.2). Decision support systems are useful for incorporating evidence into decisions related to a specific question that has been widely and repeatedly addressed. It is also important that the evidence can be converted into simple numerical or visual formats.

Figure 8.2 A schematic showing how scientific information could support environmental decision-making (Dicks et al., 2014). The triangle on the left shows an evidence hierarchy, in which summaries, such as those produced by the Conservation Evidence project, integrate evidence from across studies and systematic reviews, and form the basis for information flowing into decision support systems. In these circumstances, environmental decisions (shown by the ‘Decision’ diamond on the right) are based on the best-available evidence, combined with the expertise and local knowledge of the practitioner or policy-maker (described by the ‘Experience’ box). Dashed lines illustrate bypass routes currently taken to inform environmental decisions.

There are many decision support tools available covering various aspects of environmental science. For instance, Zasada et al. (2017) identified 60 research projects funded between 2002 and 2013 under the European Commission’s 6th and 7th Framework Programmes that had developed decision support tools for landscape and environmental management. Of these, only 61% still existed in 2014, and only half were updated after the projects that developed them ended, although this seems a pre-requisite for ongoing use. The uptake of decision support systems depends on a range of factors, including ease of use, performance, whether they are recommended by peers and the level of marketing (Rose et al., 2016). Uptake can be enhanced by ensuring that users are closely involved in the conception and design of the tools (Rose et al., 2018).

While decision support systems are often designed by researchers as a way of incorporating scientific knowledge into practice, most are based on one particular model, study or approach to a scientific question and represent a ‘bypass’ of the evidence hierarchy (Figure 8.2 and see Dicks et al., 2014). There are only a few examples where they represent the best-available scientific knowledge, based on rigorous synthesis of evidence.

One such decision support tool is the online biodiversity metric incorporated into the Cool Farm Tool (available at www.coolfarmtool.org), which provides scores for the likely benefits for biodiversity of a range of farm management actions. The actions that are included are selected according to a combination of expert judgement and assessments of summarised evidence conducted by the Conservation Evidence project. Each farm management action is assigned scores reflecting the benefit for overall biodiversity, and also for 11 species groups (e.g. woodland birds, beneficial invertebrates), weighted according to the evidence. Actions that are strongly supported by the evidence provided by the Conservation Evidence syntheses (Sutherland et al., 2019a) are scored more highly than those for which effectiveness is not known.

Another example is the set of greenhouse gas emission calculators used in agriculture to support mitigation by changing farm management. These tools incorporate models of greenhouse gas emissions and carbon storage according to vegetation type and farming practice (Richards et al., 2016). These calculators combine empirical models with emission factors collated by the Intergovernmental Panel on Climate Change (see ‘National and International Scientific Assessments’). Although the outputs from these tools are only as good as the data that they are based on, new information can be added to improve their performance as it becomes available. For example, Richards et al. (2016) demonstrated that two widely used software tools tend to overestimate emissions from smallholder farms in tropical environments, but suggest that this is probably due to a systematic bias in literature, with most data coming from temperate regions, rather than bias in the models themselves. As empirical data are included from a wider range of environments, more accurate disaggregated emissions factors will become available for different parts of the world. If the decision support systems are maintained and updated, this new knowledge will directly influence decision-making at farm level.

8.4 Active knowledge exchange mechanisms

Active knowledge exchange mechanisms are the most diverse alignment mechanism of the three considered in this chapter. Our concept is similar to that of ‘boundary organisations’ identified by some other authors (Guston, 2001; Cook et al., 2013b), in that they operate in both scientific and practical spheres, but retain distinct lines of accountability to both groups. They can take a variety of institutional forms, from a dedicated, self-funded or government-funded organisation to a network of people working together across organisations (see also Chapter 13).

The reputation of such a body depends on its ability to produce or broker knowledge that is salient, credible and legitimate (Cash et al., 2003; Sarkki et al., 2015) while maintaining transparency. Credibility refers to the scientific adequacy of the technical evidence and arguments. Salience is the relevance of the brokered knowledge to the needs of decision-makers. Legitimacy reflects the perceptions that the production of information has been respectful of stakeholders’ divergent values and beliefs, unbiased in its conduct and fair in its treatment of views and interests. Achieving all these values requires adequate attention to governance from the outset.

Here, we provide examples of knowledge exchange mechanisms operating at a subnational scale, related to a particular environmental issue or landscape (Wadden Sea case study); at a national or international scale but restricted to environmental science (EKLIPSE mechanism); and at a national or international scale ranging across all scientific knowledge (European Scientific Advice Mechanism, and UK Parliamentary Office of Science and Technology).

8.4.1 Management of the Wadden Sea

At a subnational scale, van Enst et al. (2016) provided a detailed case study of three contrasting knowledge exchange mechanisms that have been important in aligning scientific evidence with policy and management decisions around the Wadden Sea, a shallow estuarine sea in the Netherlands. Competing cockle-fishing, gas extraction and biodiversity conservation interests generate continuous debate over the scientific knowledge, and the strategic use or misuse of such knowledge has played a pivotal role in disputes (Floor et al., 2013). Knowledge exchange mechanisms were devised to improve the transparent use of evidence. Two of the knowledge exchange mechanisms were government-funded: the Wadden Academy, a science-led organisation that oversees monitoring and data-gathering, and the Netherlands Commission for Environmental Assessment, which produces official reports. The third, IMSA Amsterdam, is a commercial think-tank and consultancy, focused on mediating between stakeholders, science and policy. These three organisations worked together to improve the salience, credibility and legitimacy of the scientific knowledge that was available, allowing it to be influential in decision-making related to the cockle-fishery and gas-exploitation controversies. Their efforts ultimately reduced conflict and improved environmental outcomes for the Wadden Sea, for example by enabling more sustainable fishing methods to be adopted (van der Molen et al., 2015; van der Molen, 2018).

8.4.2 The EKLIPSE mechanism

Knowledge exchange mechanisms focused on one environmental issue can develop deep, long-term relationships between a core set of stakeholders and researchers. When operating across many different issues at national or international scale, relationships with experts and other stakeholders are generally short-term and must continually be re-established as the topic of interest to policy changes. One possible approach to this is provided by the EKLIPSE mechanism (Watt et al., 2018; www.eklipse-mechanism.eu), which engages relevant actors from science, policy and society to identify evidence relevant to European policy. EKLIPSE accepts requests for knowledge synthesis on specific issues from policy-makers and other societal actors. A wide network of knowledge-holders can respond to the request, often through the formation of an expert working group (Wyborn et al., 2018). To give an example, the European Commission requested scientific knowledge on how to evaluate nature-based solutions (solutions inspired and supported by nature) for their ability to enhance sustainability in cities. In response, EKLIPSE convened a pan-European expert group to conduct a rapid evidence assessment and build a framework for evaluating the costs and benefits of nature-based solutions. This was disseminated as a policy report and an open-access scientific paper (Raymond et al., 2017).

8.4.3 The European Scientific Advice Mechanism and UK Parliamentary Office of Science and Technology

At a larger scale, knowledge exchange mechanisms can provide an interface between science and policy across all scientific issues. Usually these are national or international, such as the UK Parliamentary Office for Science and Technology (POST; Norton, 1997) and the European Union Scientific Advice Mechanism (ec.europa.eu/research/sam/index.cfm). At this level, knowledge exchange mechanisms have tended to settle on one particular way of doing things that works. At the POST, for instance, a Board selects subjects for briefing notes, known as POSTnotes, from among ideas gathered from a range of sources, including parliamentarians, the public and other stakeholders (www.parliament.uk/post). POSTnotes are generally researched through a series of interviews with key experts. Almost 600 POSTnotes have been published since 1989, on subjects ranging from the psychological health of military personnel to new plant-breeding technologies. All are freely available online and held in the House of Commons library.

The European Union Scientific Advice Mechanism, on the other hand, responds to requests for advice from the ‘College of European Commissioners’ through a group of government-appointed scientific advisers. It delivers evidence review reports on specific issues, drawing on a network of expertise from more than 100 European scientific academies in over 40 countries (e.g. The Royal Society in the UK, Hungarian Academy of Sciences). For both it and POST, adherence to a clearly defined process is a way of building credibility and assuring transparency. However, it does not necessarily provide the flexibility to address the diversity of issues and problems faced by environmental policy decision-makers.

To summarise, active knowledge exchange mechanisms can have a range of scales, formats and institutional arrangements. This plurality is the best approach to linking science and policy in decision-making contexts, where different types of questions continually arise.

8.5 National and international scientific assessments

A longer-term approach to aligning evidence synthesis with conservation policy decisions involves governments or international bodies mandating large-scale, scientific assessments in broad areas of strong policy interest. Examples include the assessment reports conducted by the Intergovernmental Panel on Climate Change (IPCC; www.ipcc.ch), Intergovernmental Science Policy Platform on Biodiversity and Ecosystem Services (IPBES; www.ipbes.net) and Millennium Ecosystem Assessment (www.millenniumassessment.org; see Chapter 16 for further details of mechanism and function of the Millennium Ecosystem Assessment and the IPBES science–policy platform). These global assessments involve hundreds or even thousands of scientists around the world, including indigenous and local knowledge-holders in the case of IPBES (Sutherland et al., 2014; see also Chapter 16).

Generally, governments define the scope of the assessment and identify or nominate a set of experts to conduct it (IPCC, 2015). The nominated experts form working groups and develop report texts, which are subject to extensive, transparent review, first by other experts and then by governments. Following review, the report texts are converted into concise summary documents (usually called ‘Summary for Policy-makers’), the final text of which is agreed by governments. Each statement in the summary document must be traceable back to the full scientific report and, from there, to individual pieces of research or sources of knowledge. Through this process, science and policy influence one another in a two-way exchange of knowledge over very large temporal and spatial scales.

The IPCC, which has been active for almost three decades, has built a strong reputation for providing an overview of climate science across a range of disciplines, from geophysics to economics. There are now clear links from the scientific understanding of human-induced climate change and its impacts to policies controlling greenhouse gas emissions at national and international levels. Most recently, the Paris Climate Agreement of December 2015 is a global accord under which nations have made pledges and set emissions targets to keep global temperature rise below 2°C (Clemencon, 2016; Tobin et al., 2018). A large quantity of scientific research underlies these policy pledges, which would likely not have happened, or not have been so extensive, without the IPCC assessment process. Forty-five different global climate models are now being used together to link levels of greenhouse gas emissions to long-term global temperature rise under different emissions scenarios (Collins et al., 2013). There is also a plethora of analyses and modelling connecting economic activity to greenhouse gas emissions (e.g. Vandyck et al., 2016) and threshold temperate rises with specific impacts on environments, economies and human well-being (IPCC, 2014).

The Millennium Ecosystem Assessment (2005) was the first global evaluation of the status of ecosystems, and developed the ecosystem services framework for understanding how nature can benefit people. The ecosystem services concept originated in the academic world (Potschin & Haines-Young, 2016), but the Millennium Ecosystem Assessment formalised the thinking, providing a conceptual framework and nomenclature for ecosystem services. Since its publication, a growing number of countries have conducted their own national ecosystem assessments (Schrӧter et al., 2016) and the policy ground is being set for their results to be used in national natural-capital accounting. Both Aichi Biodiversity Target 2 from the Convention on Biological Diversity’s Strategy Plan 2011–2020 (Convention on Biological Diversity, 2010) and Action 5 of the EU Biodiversity Strategy to 2020 (European Commission, 2011) call for biodiversity values to be incorporated into national accounting.

Large-scale assessments are most effective at aligning scientific evidence with decisions when there is a broad issue of strong political interest, such as climate change or biodiversity loss. The assessments are expensive (see Table 8.1), so there must be substantial political commitment and a source of funds over the relatively long term.

Table 8.1 A summary of the costs associated with three mechanisms to align evidence synthesis with policy and practice in the environmental field, compared to the costs of individual evidence synthesis methods

Activity	When to apply	Cost (£)
Mechanisms to align evidence synthesis with the needs of policy and practice
Decision support tools	Specific question, repeatedly addressed	380,000–3.9 million per tool1
Knowledge exchange mechanisms	Many questions arising	600,000 per year2
International assessments	One big, broad issue	~3 million per year3

Individual evidence synthesis methods
Systematic review	Many studies address a single question	19,000–190,0001
Subject-wide evidence synthesis	Multiple sources of relevant evidence exist	Initial cost: 45,000–480,000 Update cost: 20% of initial cost1

¹ Dicks et al. (2014);

² Cost of the EKLIPSE mechanism;

³ www.ipcc.ch; www.ipbes.net.

Given the obvious power of national and international scientific assessments to influence policy, it is now more important than ever to incorporate into them the transparent, unbiased repeatable methods that have been developed for evidence synthesis. Currently, the rigour and reliability of large-scale scientific assessments rely on extensive peer review, rather than systematic searching or careful elicitation methods that reduce bias. Evidence synthesis methods are usually not reported (with some exceptions, such as chapter 6 of the Intergovernmental Science Policy Platform on Biodiversity and Ecosystem Services pollination report; IPBES, 2016). However, such assessments are conducted over long timescales, with the IPCC, for example, producing a global assessment report every 5–10 years. With this amount of time and money available (see Table 8.1) there is a clear opportunity to develop rigorous processes of evidence synthesis within this framework. As a first step, we urge policy-makers and institutions involved in commissioning large-scale scientific assessments to require authors to report their underlying synthesis methods.

8.6 What does it all cost?

The cost of the alignment mechanisms outlined in this chapter varies considerably, both within and among the different activities (Table 8.1). These costs should be interpreted in the context of total spending on scientific research. For example, the budget of the European Commission’s flagship scientific research programme, Horizon 2020, is approximately £8 billion per year.

The organisations that fund research and aspire to be evidence-informed already invest heavily in improving interactions between science, policy and practice. Unfortunately, they frequently fund expensive decision support systems that are not maintained or used a few years later (Zasada et al., 2017) and large-scale reviews or scientific assessments that do not follow clear protocols to reduce bias. The challenge in aligning evidence synthesis with decision-making is not to find the money, but to demand and enable improved rigour and continuity in activities that are already taking place.

No single mechanism will be best for aligning evidence with policy and practice in all contexts. Each has strengths and weaknesses, and can be applied in different circumstances and at different scales. International assessments have redirected policies and scientific endeavour on a very large scale, but would be unlikely to align specific scientific findings with conservation practice at smaller scales. At smaller scales, the potential of decision support systems to incorporate rigorously collated environmental evidence has hardly been tapped.

At every level, mechanisms to link synthesised evidence with policy and practice decisions need to be funded sufficiently to ensure salience, legitimacy, credibility and transparency. These linking mechanisms need access to methods of collating and communicating evidence that are well-developed, transparent and widely understood (Cook et al., 2017; Dicks et al., 2017) and are just as important as the research itself, if not more so.