Using a theory of change to evaluate the impact of a conservation training programme: a practitioner's perspective

Abstract

Training plays a central role in the pursuit of conservation goals, and it is vital to know if it is having the desired effect. However, evaluating the difference it makes is notoriously challenging. Here, we present a practitioner's perspective on overcoming these challenges and developing a framework for ongoing evaluation of a conservation training programme. To do this, we first created a theory of change, describing the pathway of change we expect from training delivery to conservation impact. This provided the clarity and structure needed to identify indicators of change in the short, medium and long term. For data collection, we utilized both quantitative and qualitative methods to provide a more complete understanding of the change expected and capture any that might be unexpected. However, the more time that passes since a training event, the more difficult it becomes to attribute results; in response, we shifted predominantly to the use of qualitative methods to understand the long-term results achieved. After 3 years of implementation, this framework has enabled us to measure the difference our training makes to individuals and their work, and to provide evidence for the contribution it makes to achieving conservation impact. We believe that the lessons learnt can be used to improve the evaluation of training activities across the conservation sector and maximize the impact they achieve.

Introduction

There is a continuous need for training within the conservation sector. It equips conservation practitioners with the skills, knowledge and personal qualities they need to overcome new challenges and achieve positive results. To ensure training is having this effect, it requires systematic monitoring and evaluation. This process creates vital opportunities for learning and adaptation, ensures resources are used to greatest effect, and provides the evidence needed for reporting to donors, supporters and other stakeholders (Stem et al., 2005; Jones et al., 2013). However, evaluating the impact of training is also notoriously challenging and, in reality, there is a need to balance the ideal monitoring and evaluation framework (i.e. one that provides robust evidence of the results achieved), with what it is practically possible to measure, especially given limited conservation resources (Jones et al., 2013).

Training providers face a number of challenges when it comes to evaluating their efforts. Firstly, the results they achieve are dependent on the circumstances of each individual who takes a training course, making it practically impossible to predict what success might look like in every eventuality (Roche, 1999; Ortiz & Taylor, 2009). In some cases, the most important results can also be difficult to quantify, such as potentially pivotal changes in an individual's self-confidence or motivation (James, 2009; Vallejo & When, 2016). Furthermore, the desired conservation impact of training could take decades to achieve, and in this time results will inevitably be shaped by a combination of factors (Roche, 1999; James, 2001, 2009; Ortiz & Taylor, 2009; Simister & Smith, 2010). This makes the attribution of results to a single training event a significant challenge, especially given that experimental evaluation approaches, commonly used to determine causality, are typically not feasible because of the ethical implications of randomly assigning control groups and the practicalities of constructing a credible counterfactual (Roche, 1999; James, 2001).

In many cases, the evaluation of conservation training is limited to the quantity and quality of delivery (Bruyere et al., 2020), and although success stories are commonly used for communication purposes (Conservation Leadership Programme, 2021; Tropical Biology Association, 2021), the methods used by training providers to evaluate the effectiveness of their efforts are not readily available. In the wider education sector, one of the most well-known methods for training evaluation is Kirkpatrick's four-level model (Kirkpatrick & Kirkpatrick, 2006). This offers a structured approach to evaluation by guiding practitioners through four-levels of criteria: reaction (i.e. were trainees satisfied with a training event?), learning (i.e. did they learn anything?), behaviour (i.e. did they do anything differently?) and results (i.e. did training affect the wider system in which they operate?). Similarly, the use of a theory of change is also widely recommended for providing a systematic approach to evaluation in a range of contexts, including training (James, 2001; Stem et al., 2005; Kapos et al., 2008; Kapos et al., 2009; Ortiz & Taylor, 2009; Simister & Smith, 2010). This approach requires training providers to articulate how they expect their intervention to achieve the desired impact through explicit causal pathways and, in doing so, makes it easier to identify measures of success in the short, medium and long term.

However, an evaluation framework that relies solely on measuring pre-defined indicators of success can result in missed opportunities to learn from any unexpected results (Roche, 1999; Watson, 2010; Vallejo & When, 2016), particularly in the context of training, where results can be difficult to predict. To provide a more complete understanding of the results achieved, it is widely recommended to utilize both qualitative and quantitative data collection methods (James, 2001, 2009; Stem et al., 2005; Ortiz & Taylor, 2009; Simister & Smith, 2010; Watson, 2010; Vallejo & When, 2016). A mixed-methods approach can help in understanding the extent to which an intervention is achieving pre-defined indicators of success as well as capture any difficult to define or unexpected results. The use of qualitative methods is also useful for capturing the long-term results of training, where direct measurement and attribution is not possible (James, 2001, 2009; Simister & Smith, 2010).

Durrell Wildlife Conservation Trust (Durrell) is a non-profit wildlife conservation organization, whose mission is saving species from extinction. For over 40 years, a core part of achieving this mission has been to provide training for conservation practitioners, typically from the organization's priority regions, and for aspiring conservationists, including university graduates and early-career individuals. Until recently, attempts to evaluate the difference this training makes have been opportunistic and focused predominantly on the collection of case studies. However, in an increasingly evidence-led sector (Sutherland et al., 2004), we wanted to develop a more systematic approach that would enable us to evaluate and maximize our impact. To do this, we decided to use a theory of change and mixed-methods approach as this gave us an opportunity to clarify the assumed links between training and our conservation mission, better understand the complexities of training outcomes, and develop a framework that we could review and adapt over time.

Given Durrell's long history in conservation training, we believe this is a useful case study for exploring the practicalities of using a theory of change and mixed-methods approach for the evaluation of training impact. Here, we describe the approach taken by Durrell to develop a simple theory of change and associated monitoring plan for its conservation training programme. In addition, we present a subset of preliminary quantitative and qualitative results to share the key lessons learnt in the first 3 years of implementation.

Methods

Developing the evaluation framework

In November 2016, we began the development process by reviewing findings from three retrospective evaluations of Durrell's training programme: an internal evaluation of Durrell's flagship 12-week Endangered Species Management course (Payne, 2015), followed by two Imperial College London MSc projects evaluating 1–12 week courses across Durrell's training programme (Ruzowitzky, 2015; Sawrey, 2015). Collectively, these captured the experiences of 514 individuals who attended a course during 1979–2015. Payne (2015) used a mixed-methods approach to identify the outcomes perceived by past participants and the external factors that helped or hindered their progress. Ruzowitzky (2015) developed an evaluation framework based on Kirkpatrick's four-level model (Kirkpatrick & Kirkpatrick, 2006) to measure the extent to which training influenced their professional development and conservation actions, and Sawrey (2015) developed a theory of change from the perspective of trainers and validated it against the outcomes perceived by trainees. The key results from each (Table 1), provided us with an information base from which to develop a theory of change.

Table 1 A summary of key results from three retrospective evaluations of Durrell's conservation training programme.

To do this, we began by explicitly describing the desired conservation impact of training (to achieve our organizational mission of saving species from extinction) and the key change we expect training to affect for this to be realized (more effective conservation action). From here, we used if–then statements to determine the results required in the short and medium term for these long-term goals to be achieved, and the potential causal links that existed between them. In particular, an increase in self-confidence, motivation and self-efficacy, collectively described as perception of control, proved to be one of the greatest outcomes of training (Payne, 2015; Sawrey, 2015) and an important precursor for increased effectiveness at work (Sawrey, 2015). However, in some cases, unsupportive organizations and a lack of opportunities prevented past participants from applying new skills and progressing in their careers. We knew therefore that an enabling work environment would be an important condition for success. To ensure we considered different perspectives in the development process, we shared a preliminary draft of the theory of change with colleagues across Durrell, including training and non-training staff working in a range of countries, and therefore socio-cultural contexts, before inviting three external experts from the conservation training field to challenge our logic and comprehension of results. During this review process, the role of the support network in achieving impact was repeatedly called into question, with the general consensus being that it can sometimes be influential in the success an individual achieves but is not always required. This debate was also echoed in results from the retrospective evaluations (Payne, 2015; Ruzowitzky, 2015; Sawrey, 2015). In response, we decided to explicitly represent this as a non-essential link in the theory of change.

We used the resulting theory of change (Fig. 1) to guide the development of the monitoring plan by defining at what point in time we expected each result to be achieved and what we needed to know to verify if this had happened or not. To streamline data collection and minimize the risk of survey fatigue, we grouped these information needs into four key intervals: those requiring verification immediately post-training, followed by those requiring verification at 1-, 5- and 10-years post-training. Based on our organizational experience, we considered 10 years to be the minimum amount of time in which we could reasonably expect training to improve the effectiveness of conservation action and therefore benefit species recovery. However, when evaluated retrospectively, response rates dropped significantly among participants who had completed a course > 10 years ago (Ruzowitzky, 2015), making it the most suitable time frame for evaluation. Because of this long time frame, we opted for a pre-test post-test non-experimental evaluation design and used online questionnaires, designed using Smart Survey (2020), to collect the information required at each interval and to establish a pre-training baseline. Online questionnaires proved to be a valuable tool for capturing useful information in all three retrospective evaluation projects (Payne, 2015; Ruzowitzky, 2015; Sawrey, 2015) and in comparison to alternative and complementary tools, such as interviews and focus groups, require minimal resources to implement and analyse, making them the most feasible choice for the scale required.

Fig. 1 The theory of change for Durrell's training programme: after completing a training course, an individual gains new and improved skills and knowledge, increased perception of control and access to ongoing support from Durrell's conservation network. An individual then returns to work and uses the skills and knowledge gained to improve their personal effectiveness. A more effective individual then progresses within their career and/or implements more effective conservation action. To achieve these results, we make the assumption that an individual returns to an enabling work environment; i.e. one that provides them with the opportunities and support needed to succeed (indicated by the grey box surrounding these results). The more effective conservation action taken benefits the recovery of species and/or habitats and ultimately achieves Durrell's conservation mission, to save species from extinction.

For results requiring verification immediately post-training, we developed multi-item Likert scales. For perception of control, this included a series of three multi-item Likert scales, one each for motivation, self-confidence and self-efficacy. To measure change in skills and knowledge, we created a list of seven competencies tailored to the learning objectives of each course written as short ‘How to…’ statements. This number gave us sufficient scope to evaluate the main learning objectives of each course without overloading participants with a long list of questions and risking so-called straight lining, in which respondents lose interest and select the same response for each question. To avoid misinterpretation and improve the accuracy of responses, we also kept the wording of statements as clear and concise as possible, for example ‘How to write a grant proposal’. For each, we developed two Likert scales, one to measure level of knowledge and one to measure level of confidence to apply knowledge. We also expected participants to gain a support network immediately post-training, but because of its non-essential role in the theory of change, and in the interest of minimizing the number of questions, we decided it was not a priority for evaluation at this stage in the development process and did not design an associated indicator.

For results requiring verification at 1-, 5- and 10-years post-training, we increasingly combined quantitative and qualitative methods. For example, to measure personal effectiveness at 1-year post-training we transformed the same list of competencies used to measure skills and knowledge into a multiple response, multiple choice question, asking participants to select which, if any, they had applied in their work and to provide an example if relevant. Similarly, to monitor conservation actions, at 1-, 5- and 10-years post-training we used a multiple response, multiple choice question based on the Conservation Action Classification 2.0 (Conservation Measures Partnership, 2021), to understand the types of actions participants had taken in the previous 12 months followed by an open-ended request for an example. To further verify results against the theory of change as well as capture unexpected and long-term results, we used a simplified version of the most significant change method (Davies & Dart, 2005; Steadman, 2021), asking the open-ended question ‘What is the most significant change you experienced as a result of this course?’ in the immediate post-training questionnaire and ‘What is the most significant change you experienced in the past 12 months?’ in subsequent post-training questionnaires. To better understand the relationship between qualitative results and training, we included a rating scale for participants to report the extent to which training with Durrell influenced the result described, as per the approach used by Payne (2015), Sawrey (2015) and Ruzowitzky (2015). At 1-, 5- and 10-years post-training, we also wanted to understand the extent to which participants experienced an enabling work environment. To do this, we transformed a list of the seven most common barriers to the effective implementation of species recovery plans (Richardson, 2016) into a multi-item Likert scale, asking participants to rate the extent to which each affected their work, followed again by a request for a qualitative example if applicable. Finally, before implementing a full-scale trial, we pre-tested pre- and post-training questionnaires in person with a cohort of international course participants to ensure the language was accessible and completion time reasonable.

Preliminary implementation

During 2017–2019, we trialled pre-, post- and 1-year post-training questionnaires (Supplementary Materials 1, 2 & 3) across five face-to-face Durrell courses (Table 2). We e-mailed pre- and post-training questionnaires to all 88 course participants before and after their course and 1-year post-training questionnaires to the same 88 individuals 12 months later. For survey completion, we assigned each course participant a unique identification code, saved alongside their personal details in a secure database, to be entered at the start of each questionnaire. This allowed questionnaires to be completed and managed anonymously but still matched for analysis. To maximize response rate, we personalized e-mail correspondence, included a clear statement of purpose and sent a reminder one week after the initial request. We received 88 completed pre- and post-training questionnaires (100% response rate) and 58 completed 1-year post-training questionnaires (66% response rate). We summarized results from each training event on an ongoing basis before combining results from all five courses into one dataset to produce an appropriate sample size for statistical analysis and to draw general conclusions about the effectiveness of the training programme as a whole. We used the Wilcoxon signed-rank test in R 3.6.3 (R Core Team, 2020) to test for differences in indicators measured pre- and post-training, assuming statistical significance at P < 0.05. For the analysis of qualitative data collected using the adapted most significant change method (Davies & Dart, 2005), we used a thematic coding approach to identify key words and phrases from each response and group them into distinct themes. We then reapplied these themes to determine the number of responses related to each.

Table 2 An overview of the five Durrell training courses included in preliminary implementation of the evaluation framework.

Results

The evaluation framework

The evaluation framework for Durrell's training programme is underpinned by a theory of change describing the pathway of change we expect an individual to take from completing a training course to achieving conservation impact (Fig. 1). To measure progress against this theory of change, and capture any unexpected results, we use a combination of quantitative and qualitative indicators (Table 3).

Table 3 The monitoring plan for Durrell's training programme.

Preliminary results

Immediate post-training

Results showed that trainees did gain new and improved skills and knowledge immediately after a completing a course, with participants reporting higher levels of knowledge post-training (P < 0.001) and higher levels of confidence to apply knowledge (P < 0.001). When results from each course were summarized independently, we were able to identify which competencies participants were the most and least knowledgeable and confident about (Fig. 2). However, preliminary results did not show the expected increase in all three elements of perception of control, with participants reporting an increase in their level of self-confidence (P < 0.001) and self-efficacy (P < 0.001), but not in their level of motivation (P = 0.37). Instead, over 88% of participants reported feeling highly or extremely motivated both before and after the course (Fig. 3). Similarly, when asked to describe the most significant change experienced as a result of the course, 44% of responses described an increase in self-confidence, whereas only 11% described an increase in motivation. Unexpectedly, 40% of the responses recorded immediately post-training described gaining inspiration and/or a new perspective to be the most significant change experienced. For example, as one participant described, ‘it has allowed me to understand that many are working for the betterment of the environment and that although I come from a small country facing many threats, we are not alone, our conservation action is not in vain’, and another, ‘it has helped me to see the world differently and that there is hope in conservation’.

Fig. 2 Per cent of participants who reported each level of knowledge pre-training compared to post-training for each of the competencies assessed during the Endangered Species Recovery course in 2019.

Fig. 3 Per cent of participants who reported each level of motivation pre-training compared to post-training across all of the courses included in this study.

One-year post-training

One-year after completing a course, results showed that participants were using the competencies gained to improve their personal effectiveness and, following each course, we were able to identify which competencies were being applied by the greatest number of participants and which were being applied by the fewest (Fig. 4). Qualitative examples provided further verification and illustrated the ways in which competencies had been applied, for example, in the 12 months since completing the Durrell Endangered Species Management course, one participant described using the population monitoring skills they gained as ‘I set up the monitoring of a threatened EDGE species’, and another described using the grant writing skills gained as ‘I wrote a proposal for a livelihoods group and they were successful in getting the funding’. Results also showed that training influenced the conservation actions taken by participants. For example, in the 12 months since completing the Facilitation and Communication Skills course, one participant described taking an action somewhat influenced by the training received as ‘I brought together an electric company, NGOs and a governmental institution to agree on the correction of a power line which is dangerous to raptors’, and following the Endangered Species Recovery course, one participant described an action significantly influenced by the training received as ‘I recently led my first offshore island invasive species eradication project’. In total, 58% of participants reported that their ability to take a given conservation action was significantly influenced by the training received, 33% reported it was somewhat influenced and 9% reported it was not at all influenced.

Fig. 4 Per cent of participants applying each competency in the 12 months after completing the Durrell Endangered Species Management course in 2018.

Discussion

Despite the challenges faced, we have shown that it is possible to develop a useful and practical framework for ongoing evaluation of a conservation training programme. Using a theory of change approach has helped us to clarify our assumptions about how training contributes to our conservation mission and identify appropriate measures of success in the short, medium and long term. The use of qualitative and quantitative methods to measure progress against the theory of change has provided us with a richer understanding of what training can achieve and enabled us to capture outcomes that were unexpected and difficult to define. The information gained throughout this process will be used to refine the theory of change, improve the evaluation framework, and increase the effectiveness of Durrell's conservation training programme.

In line with the theory of change, preliminary results show that training is having the desired effect on the skills and knowledge of course participants and enabling them to be more effective at work. Immediately after completing a course, participants had more knowledge of the competencies taught and felt more confident to apply them in their work. Within 12 months, they were then using these competencies to improve their personal effectiveness; e.g. to write successful grant proposals and establish monitoring protocols. Within 12 months of completing a course, participants also reported that their ability to take specific conservation action was influenced by the training they received, including actions to remove threats and recover populations of threatened species. Although more difficult to assess, research has shown that measures of key outcomes such as these serve as powerful predictors of conservation success (Kapos et al., 2009) and have provided us with a reliable indication that training can achieve its goal of saving species from extinction. Quantifying the application of skills and knowledge has also enabled us to identify which competencies are the most useful to participants. For example, competencies relating to facilitation, leadership and project management were frequently reported to be the most applied in the 12 months after completing a course and we have begun expanding our training programme to meet these needs more fully. These findings support those of Barlow et al. (2016) and Englefield et al. (2019), who found project management and leadership skills to be in critical need across the conservation sector. In the future, we plan to broaden our definition and assessment of skills and knowledge to include more of the personal competencies we expect an individual to gain from training (Maggs et al., 2021), in particular for longer, more intensive courses such as the 12-week Durrell Endangered Species Management course.

In contradiction to the theory of change, preliminary results did not show a consistent improvement in all three perception of control elements. Participants reported higher levels of self-confidence and self-efficacy following training (i.e. they felt more confident in themselves and more capable of achieving their goals), but reported similar high levels of motivation both before and after training. In hindsight, this is perhaps not surprising, as we can expect individuals who seek out training to be highly motivated already. However, a combination of these perception of control elements previously proved to be an integral outcome of training (Sawrey, 2015; Sawrey et al., 2019) and warrant further investigation. We will continue to review the role they play within the theory of change and aim to improve the way they are defined and measured. In addition, we intend to explore in more detail the role inspiration plays in the theory of change as preliminary results suggest that feeling more inspired and/or gaining a new perspective is one of the most significant outcomes of training and, as suggested in the wider education literature, this can increase the likelihood of an individual achieving their goals (Milyavskaya et al., 2012).

As with any evaluation, we needed to balance what we ideally wanted to know with what was practically possible to measure, and as a result we recognize a number of limitations with the method used. First is the reliance on self-reporting and its associated response biases, which can lead to inaccurate or false results. In particular, participants might report they feel more confident or more knowledgeable than they actually are because they think it is the correct answer. Second is the increasing risk of selection bias over time, where results become skewed because they only represent a subset of individuals who complete a questionnaire. In particular, participants who have had a negative experience or who did not continue to work in conservation might be less likely to complete a questionnaire and therefore be underrepresented in results. Although these biases cannot be fully overcome, their effect can be mitigated by following good survey design and we therefore consider questionnaires to be capable of generating useful information with minimal resources. Finally, the evaluation framework lacks a control group (i.e. a similar cohort of individuals in the conservation sector that do not receive the same training as course participants) against which we can compare rates of progress over time and truly understand the difference training at Durrell makes. This limits our ability to attribute long-term results and conservation impact to the training an individual receives. However, we have found a pre-test post-test non-experimental design to yield meaningful results and the use of qualitative data collection methods has enabled us to capture illustrative examples of success and determine the extent to which they are influenced by training. In addition, we found that these qualitative stories and the theory of change diagram itself to be valuable in communicating training impact to donors, supporters and prospective course participants.

We hope this work encourages others to adopt a systematic approach to evaluating the impact of training in conservation and we believe that the lessons learnt and the simple theory of change presented here can be used and adapted to facilitate this process. Finally, we hope this work will stimulate further discussion on the topic of impact evaluation in conservation training and encourage others to share their experiences.