Eliciting expert opinion on the effectiveness and practicality of interventions in the farm and rural environment to reduce human exposure to Escherichia coli O157

Summary

Few hard data are available on emergent diseases. However, the need to mitigate and manage emergent diseases has prompted the use of various expert consultation and opinion elicitation methods. We adapted best-worst scaling (BWS) to elicit experts' assessment of the relative practicality and effectiveness of measures to reduce human exposure to E. coli O157. Cattle vaccination was considered the most effective and hand-washing was considered the most practical measure. BWS proved a powerful tool for expert elicitation as it breaks down a cognitively burdensome process into simple, repeated, tasks. In addition, statistical analysis of the resulting data provides a scaled set of scores for the measures, rather than just a ranking. The use of two criteria (practicality and effectiveness) within the BWS process allows the identification of subsets of measures judged as potentially performing well on both criteria, and conversely those judged to be neither effective nor practical.

INTRODUCTION

Newly emergent pathogens in the environment pose important challenges to public health policy. Since 1980, 87 new human pathogen species have been discovered, many of which are associated with animal reservoirs [1, 2] such as Campylobacter, Escherichia coli O157 and Yersinia [3]. Emerging pathogen risks are, by definition, associated with incomplete evidence bases and there is a recurring demand for the scientific community to inform policy-makers about risk management even though systematic evidence is often limited [4].

In the absence of a systematic evidence base, alternative approaches for the management of livestock diseases have been sought. These approaches seek to assist the management of uncertainty [4–6]. Often this involves making recommendations on the best available data, while acknowledging the uncertainty involved in the evidence and hence in the resulting recommendations. Expert opinion is a source of information often sought in this context, and identifying expert consensus regarding risks and the appropriate means of their management represents an attractive option in many situations. However, the appropriate means by which experts are identified and the process by which their opinions are elicited remains contested. In addition, even if the views of the correct experts for consultation are identified, there may be no, or only partial, consensus among them. This is not surprising as the extent of consensus within a scientific community may be related to the level of uncertainty regarding the issue at hand and such uncertainty is endemic to the management of emergent pathogen risks.

A number of methods have been used to elicit expert opinion regarding pathogenic risk. Some studies have employed standard questionnaire approaches, for example, asking expert panels to score a range of measures using Likert scales or similar response formats [7, 8]. Other studies have asked an expert panel to weight the relative importance of risk factors [5, 8]. A number of studies have also employed market research techniques such as conjoint analysis [6, 9] in which respondents complete structured survey tasks (making choices, ranking, etc.), generated via an experimental design, in order to explore trade-offs between candidate measures as a means of prioritizing them (see [6, 10, 11]). Conjoint analysis originates in market research where it is used to derive estimates of individuals' preferences for a product or service, and/or its component characteristics [12]. There are various forms of conjoint analysis [adaptive conjoint, choice-based conjoint (CBC), etc.] but they have in common the understanding that a product comprises a number of attributes the importance of which differs across consumers [13].

The application of these techniques to disease management has involved, inter alia, identifying the relative effectiveness of potential management measures and how they might be bundled into effective interventions. Cross et al. [10], for example, use adaptive conjoint analysis (ACA) to explore the effectiveness of trade-offs between interventions proposed for the control of bluetongue. The ACA approach used by Cross and colleagues is well suited to situations with a small number of control measures, often differing in their levels, which would typically be bundled together by practitioners. It is less suitable when there is a very large set of stand-alone control measures under consideration. In this situation a form of conjoint analysis, termed best-worst scaling (BWS) may offer more potential [14].

We apply the technique to investigate the management of E. coli O157 in agriculture and the wider rural environment. E. coli O157 was reported as a human pathogen in 1982 [15] and a number of outbreaks have subsequently occurred, typically associated with foodborne infection [16, 17]. More recent outbreaks have shown environmental contamination to be an increasingly common pathogen transmission pathway [18–20]. These outbreaks, alongside evidence on shedding levels, persistence in the environment [21, 22] and re-colonization [23] have highlighted the need to focus on environmental aspects of mitigation [24].

Given increased calls in the UK for action on the management of E. coli O157 risk, there is a need to evaluate potential management measures in farm and rural settings. This need for action has become more urgent in light of recent outbreaks such as that at Godstone farm in Surrey in 2009 when 93 people became infected, with 17 developing complications and eight requiring kidney dialysis [25]. Hence there is an imperative to understand, and potentially reconcile, expert opinion on the potential of a range of interventions to reduce the risk of E. coli O157 exposure to humans in an environmental setting.

This study investigates control measures applicable in the farm and rural environment. The scope of the farm control measures was limited to pre-lairage for livestock and the farm gate for vegetables. The orientation on the farm and rural environment is one of the novel aspects of the research since many other studies [26, 27] have focused on risks of exposure and associated control measures occurring later in the food chain, often concerning food processing, storage and handling rather than direct environmental exposure. The considered measures affect both direct environmental human exposure (e.g. via camping, petting farms, etc.) and exposure via food (reducing shedding levels of livestock entering the food chain). The focus on the farm and rural environment served to both address a knowledge gap and to keep the range of control measures considered manageable.

This paper describes and critically reflects upon the use of a novel technique (BWS) to elicit expert opinion regarding the ‘effectiveness’ and ‘practicality’ of measures to manage E. coli O157 risk in the farm and rural environment. It has a methodological focus in terms of outlining and testing a novel approach to expert elicitation. It has a substantive focus also, concerning the relative practicality and effectiveness of O157 control measures, and the potential complementarities and conflicts regarding their performance using these two criteria.

METHODOLOGY

BWS

BWS is a choice-based technique that requires respondents to make repeated choices between sets of options [28]. We first explain the method in general before indicating how it is applied to pathogen management in this study.

In a BWS study respondents are presented with repeated, varying, sets of (typically four or five) items and asked to identify at the extremes of their preferences. For example in a set of four control measures (A–D), the respondent indicates which they consider to be the ‘most effective’ and ‘least effective’ measures. If A is more effective than D then information is obtained for five of the six possible pairs of measures within the set. It is known that A is viewed as more effective than B, C and D (three pairs) and that B and C are more effective than D (two pairs). The only pair on which no information is available is B–C. Similarly, for a 5-item set we gain information on 18/25 paired combinations. The process is repeated with new sets (generated via an experimental design) and in each case choices are made at the extreme using whatever criteria is specified by the researcher. The data are analysed by counting procedures or by estimating choice models to derive importance weights across the items featured in the sets.

The BWS method is typically used when information is sought over a large set of items [29]. It holds a number of advantages over other rating and ranking techniques [30]. First, approaches that use some form of scaling (e.g. scoring an item on a scale of 1–5) assume that the distance properties on the scale are equal. Where scaling properties are unknown the possibility of transforming these into parametric data is not reliable [31]. Second, scales can be interpreted differently between individuals. For instance, one respondent may score an item as a ‘4’ while another scores it a ‘5’ when both respondents rate the item equally but have different conceptions of the value of the scores on the scale. Third, respondents are not necessarily compelled to choose one option over another, and frequently they score all of the items with equal importance [32]. This relates to another problem common to rating/ranking approaches, namely the lack of discriminatory power between items [31]. Finally, interpretation of the scores is comparatively simple as the number of times an item is selected as best, minus the number of times an item is selected as worst, approximates the true scale value [28, 29].

In this study we use the approach in a novel manner. We define the sets in terms of E. coli O157 control measures and use two choice criteria with those sets: effectiveness (most/least effective) and, in a second stage, practicality (most/least practical to implement). The resulting data are then analysed and the perceptions of the interventions' performance in terms of both practicality and effectiveness assessed. We now turn to the research process before setting out the analytical models employed and the results derived from them.

Identification of O157 control measures and expert sample

Intervention inclusion

Measures to reduce human exposure to E. coli O157 relating to the farm and rural environment were identified from peer-reviewed papers and grey literature. The scope of the farming system considered was pre-lairage for livestock and the farm gate for vegetables. The initial list comprised 99 interventions which was deemed excessive for a cognitively bearable BWS exercise and hence an initial expert consultation was undertaken to reduce the set to a more manageable size.

Expert group recruitment

The composition of the expert panel was contingent upon the scope of the interventions selected for evaluation. The panel reflected the study's focus on interventions applicable in the rural and farm environment.

Experts were identified via authorship of relevant publications and from peer networks. A snowball technique was adopted whereby invited experts were also encouraged to suggest other members in their field of expertise who might be willing to participate in the study. Such an approach taps into pre-existing professional expert networks and allows the researchers to distance themselves from the expert selection process [8]. Follow-up contact was made by phone and e-mail to encourage participation. Experts were invited from several different disciplines and academic sectors including public health; veterinary science; food microbiology; agricultural/environmental microbiology; clinical microbiology; land management and pathogen management; risk assessment; communicable disease epidemiology; molecular ecology (details of the expert sample can be seen in Table 1).

Table 1. Categories of respondent expertise invited to participate and their response rates

Expert elicitation round 1: shortlisting

The first round of expert consultation required experts to decide which of the 99 interventions should be retained for further evaluation based upon their effectiveness in reducing human exposure to E. coli O157. Experts classified each intervention as ‘priority retain’, ‘retain’, ‘don't retain’ and ‘don't know’. These votes were then scored with values of ‘2’, ‘1’, ‘–1’ and ‘0’ respectively (the full set of interventions is available upon request).

Expert elicitation round 2: BWS

In the second round respondents were asked to re-evaluate the shortlist of measures from round 1 using the BWS methodology. This involved respondents assessing 12 sets, each containing five measures, with the combinations of measures within sets determined by an experimental design. An example of a BWS set is shown in Table 2. In the first stage respondents indicated the ‘most’ and ‘least’ effective measures in each set they faced. The process was then repeated with the ‘most’ and ‘least’ practical measure chosen in each set.

Table 2. Example of a ‘practicality’ best-worst scaling choice set

Analysis

The BWS data on perceptions of the measures' practicality and effectiveness can be analysed in a number of ways. The first form of analysis involves counting rather than estimation. The analyst calculates on what proportion of occasions it was shown each measure was selected as ‘most’ and ‘least’. A more sophisticated analysis involves estimation of practicality and effectiveness scores via a choice model based on random utility (RU) theory [33] which dominates the empirical analysis of choice in many fields. We briefly explain the approach below.

Faced with a BWS set the respondent is asked to indicate the best and worst performing measure, hence the respondent is choosing the two measures with the maximum difference in performance between them. In a set of K measures there are K(K−1) best-worst combinations. The objective is to retrieve estimates of the sample's performance scores that best explain the observed pattern of best-worst choices, and it is the RU choice model which permits this. While such choice models typically involve people choosing a single, most preferred option in this study we consider scales defined in terms of practicality, and then effectiveness. We consider there to be scale of practicality on which measures can be located and refer to φ_A as the position of measure A on that scale. Respondent n's unobserved practicality score for measure A is given by

(1)

where ∊_nA is an error term, the inclusion of which creates a probabilistic rather than deterministic choice model. The probability of person n choosing any pair of best-worst choices, for example measures A and D, is given by the probability that (P _nA−P _nD) exceeds all other K(K−1) performance differences within the BWS set.

The model is statistically implementable via the assumption that the error term, ∊_nA, has an extreme value type I (Gumbel) distribution. This means that the probability that A and D are chosen as most and least practical, respectively, is given by the standard conditional logit formulation:

(2)

Maximum-likelihood estimation of formula (2) involves retrieval of estimates of the φ performance scores which maximize the likelihood of the observed pattern of best-worst choices being observed. The approach is relative, i.e. the estimated practicality and effectiveness scores are relative to each other on an arbitrary scale, with one measure's performance score normalized at zero for identification purposes.

The conditional logit model in formula (2) does not, as specified, include personal characteristics and as such does not allow for the investigation of heterogeneity among the sample. One could extend the model to include characteristics, for example we could allow, and test whether, specific performance scores differ across respondents of differing expertise or demographic profiles. An alternative approach to the accommodation of heterogeneity is the infinite mixture, or mixed logit, model [34, 35]. In this model the importance scores are assumed to be drawn from a distribution the mean and standard deviation of which are estimated. An attraction of this model is that as well as identifying a point estimate of the mean importance score, estimates can be derived for each survey respondent, conditional on the best-worst choice data and the estimated population parameters. More formally, person n's performance score for measure A (φ_nA) is drawn from a distribution with mean (φ*_A) and standard deviation σ_A. Person n's performance score deviates from the sample mean (φ*_A) via a disturbance term, ∇, where ∇~N(0, 1):

(3)

The derivation of respondent-specific performance scores is particularly attractive in this study given the desire to examine the degree of consensus or disagreement among the expert sample on the performance of the O157 control measures. We refer readers to [34, 35] for more on the estimation of the mixed logit model, noting only that we estimate the model using Bayesian methods [36] with the sampler run for 20 000 iterations (the ‘burn in’) before the parameters were recorded, followed by another 20 000 iterations with which to summarize the posterior distribution of the measures' performance scores.

RESULTS

Expert elicitation round 1

For the first round, 53 experts were contacted and 31 completed the survey giving a response rate of 58·5%. In round 2 the snowball sampling process meant 70 experts were invited to complete the survey, 41 of whom did so, giving a response rate of 60%. Thirty-five of the respondents in round 2 gave details of their expertise and these are described in Table 1. Response rates were higher for agricultural/environmental microbiologists (75%) compared to public health experts (48%) which possibly reflects a greater level of confidence for the latter group ranking interventions primarily located in the rural and agricultural environment.

The resultant rankings and the associated suggestion that the top 30 measures would be taken forward to round 2 for more detailed assessment of their practicality and effectiveness were circulated (full details of the results from this round are available upon request). Respondents were invited to appeal against the inclusion/exclusion of any measures but no such appeals were forthcoming. As a result the top 30 measures (Table 3) were included in round 2 which used BWS.

Table 3. List of top 30 interventions selected in round 1 for further appraisal in round 2

HACCP, Hazard Analysis Critical Control Point; COSHH, control of substances hazardous to health.

BWS

Within both the practicality and effectiveness stages of the BWS survey 41 respondents completed a total of 492 BWS sets, making two choices within each one. We first considered the choice data descriptively rather than via model estimates. These estimates provide a powerful descriptive of the typical assessment of a measure and also the degree of consensus. For example measure 27 (reducing stocking densities by 50%) was ranked least practical in almost 50% of the sets in which it appeared and most practical in less than 5% of the sets. In contrast measure 19 (cattle vaccination) was ranked least practical in 35% of the sets but also most practical in 30% of the sets. Cattle vaccination was voted the most effective measure in 72% of the sets in whch it appeared and was rarely chosen as ‘least effective’.

Assessment of the measures' relative practicality and effectiveness was also investigated via estimation of the choice model described in the Analysis section. Table 4 reports results from this model estimated (separately) on both the practicality and effectiveness BWS data. The results in Table 4 show the sample mean point estimates of the practicality and effectiveness scores for all 30 measures. These estimates have been rescaled to sum to 100 to aid interpretation and comparison across the practicality and effectiveness models. In addition to the point estimates, 95% confidence intervals are provided for each parameter estimate.

Table 4. Sample mean point estimates of the effectiveness and practicality scores for all 30 measures

Cattle vaccination, the removal of animals from the proximity of private water supplies (PWS); the removal of high shedding animals prior to slaughter and the monitoring (and where appropriate treatment) of PWS; were estimated to be the most effective measures. The results indicate that the annual inspections of septic tanks, the daily cleaning of in-house water troughs and the use of double fencing to prevent contact between (sub)herds were, on average, regarded as least effective.

In terms of practicality, encouraging farmers and farm visitors to wash hands following contact with farm animals, keeping livestock away from stores of ready-to-eat crops and a 4-week grazing ban following the spreading of animal waste were seen as the most practical measures. Halving stocking densities, banning manure spreading within 500 m of ready-to-eat crops and using vegetative buffer strips to control contamination of ready-to-eat crops from run-off were seen as the least practical to implement.

The mean practicality and effectiveness scores of the measures are combined in Figure 1. The estimated scores are zero-centred and plotted in practicality and effectiveness space in which the axes represent the mean (zero) practicality and effectiveness scores within the sample. Hence measures plotted above 0 on the y-axis have higher than average effectiveness scores, those with scores over 0 on the x-axis have above average practicality scores. Those measures located in the upper right quadrant are regarded as performing relatively well in terms of practicality and effectiveness, those in the bottom left quadrant perform relatively poorly on both criteria. Measures located in the remaining two quadrants highlight the trade-off between the practicality and effectiveness of some measures. Those interventions located above in the upper right-hand segment of the 2×2 plot might be considered candidate interventions for consideration by policy-makers as they score positively for effectiveness and/or practicality.

Fig. 1. Zero-centred scatterplot of mean effectiveness and practicality scores for the 30 control measures.

As indicated in the Analysis subsection of the Methodology section, the choice model allows the estimation of respondent-specific scores as well as the estimate of the mean score for the sample as a whole. Analysis of these scores for a single measure allows a consideration of the degree of consistency or disagreement among the sample regarding the measure's performance. This is highlighted in Figure 2 which shows the spread of practicality and effectiveness scores for two example measures. There was a high degree of agreement regarding the effectiveness of vaccination to reduce pathogen loads, but opinion appeared to be divided regarding the practicality of its implementation. For some measures there was little consistency regarding either criteria, leading to a more uniform distribution of scores in practicality and effectiveness space as shown for intervention 16 [Hazard Analysis Critical Control Point (HACCP) requirement for manure spreading].

Fig. 2. Respondent-specific practicality and effectiveness scores for two control measures (each symbol denotes the practicality and effectiveness scores of a single expert).

DISCUSSION

This study has proposed and implemented BWS as an expert opinion elicitation tool. BWS has been used to evaluate the practicality and effectiveness of measures to reduce human exposure to E. coli O157 in rural and farm environments. Following an initial shortlisting process 30 measures were included in the BWS survey.

The BWS process, completed online, allowed the involvement of many experts from a diverse range of expertises (public health, environmental microbiology, epidemiology, etc.) in varied, often geographically remote, locations. Asking those experts to simply rank the full set of 30 measures would have been extremely cognitively demanding, hence the use of BWS with the repeated ranking over smaller subsets. The study achieved a high response rate with low levels of dropout mid-BWS process, which we regard as evidence that the process was cognitively bearable and intuitive.

The ex-post analysis employed counts and the estimation of a choice model on the best-worst data which allowed a full, scaled ranking to be derived. The model results include both sample-mean, and respondent-specific, estimates of the measures' practicality and effectiveness. Analysis of the individuals' scores for a single measure allows the degree of consensus or disagreement to be investigated.

We regard the BWS as highly flexible and suitable when evaluation needs to be multi-dimensional. In this case two facets of performance were considered (practicality and effectiveness). This allows the location of each measure in a multi-dimensional performance space (Fig. 1). This 2×2 plot is easily interpretable and provides a powerful stimulus for further discussion, as has been evident from our use of it at, inter alia, disease management workshops and meetings of the Advisory Committee on the Microbiological Safety of Food and meat industry bodies.

As to the substantive results, hand washing was regarded as by far the most practical measure. The separation, in time or space, of livestock waste from water supplies, water courses, ready-to-eat crops and the general public were common to many of the measures seen as most practical and effective. In addition some more technical ‘fixes’ were identified as most effective: concerning livestock and PWS. The three measures viewed as most effective were vaccination, removal of high shedding animals, and monitoring and treatment of PWS.

Some measures which recent research [37] has suggested as promisingly effective in this context were regarded as less effective (requiring in-house water troughs to be cleaned daily, maintaining dry livestock bedding). This raises the issue of managing the tension between a common expert opinion and fragments of research evidence which only some may be party to, or accept. The issue of bedding management was highlighted by the Griffin Report into the Godstone outbreak. Another notable feature of our results is that the prevention of children under the age of 11, and other vulnerable groups, from coming into contact with animals at petting, or public visitor farms was regarded as both highly effective and practical.

We note that the Godstone outbreak took place between the first and second rounds of our elicitation process which may have affected the perceived performance of both this measure and hand washing. It is evident that an empirical microbiological evaluation of the exposure hazard would remain unaffected by such an outbreak and underlines the importance of basing policy decisions on experimental evidence where possible. Any such sensitivity of the experts' effectiveness assessments to a recent or current high profile outbreak (such as Godstone or the 2011 outbreak in Germany) may be regarded as a weakness in any expert elicitation approach. It could, however, be regarded as simply reflecting an updating of knowledge and beliefs among the panel in the light of new evidence provided by such an outbreak.

The identification of hand washing as a practical and effective means of reducing exposure concurs with the findings of the Griffin Report [25]. However, the implementation of the measure raises many issues for further thought and analysis. Questions include how best to design hand-washing facilities into the farm layout and whose responsibility it is to ensure that children visiting open farms wash their hands (Griffin argues it is the parents' responsibility). Further, there is the danger that a sole measure approach could lead to the neglect of other risk factors, for example, a focus on hand washing may be at the expense of other critical control measures, for example the initial faecal contact.

BWS is best suited to the evaluation of large sets of stand-alone measures, as was the case here with 99 candidate measures. If one was taking a much smaller set of measures, with multiple levels, and trying to identify the ‘best’ bundles of those measures then alternative methods (ACA, CBC) would be more suitable. Having identified the best performing measures in the BWS study one could then consider bundling measures. Within such bundles, measures may have purely additive effects; however, in some cases they may enhance or reduce the impact of other measures. This could be assessed by modelling the cross-impact balances of intervention bundles [38].

This relates to a further notable point that, in order to keep the process focused and manageable for respondents, we focused on a single pathogen. There may, however, be spillover effects to other pathogens and policy targets. A next phase could model the potential of measures and bundles to add value through impacts on other target organisms, such as Cryptosporidium and norovirus, or contribute to the meeting of other requirements (e.g. the Water Framework Directive). This would entail re-presenting the finalized intervention list to a different panel of experts whose expertise focused on the secondary target pathogens and viruses. The same BWS methodologies would be employed but the initial research question would ask experts to evaluate the 30 interventions based upon their ability to reduce exposure to Cryptosporidium and/or norovirus.

An additional extension of the research would be to extend the sample beyond research or regulatory experts to, for example, farmers. An assessment of the consistency (or otherwise) of the assessments of the two groups would be revealing. Ultimately it will be farmers' perceptions of the practicality and effectiveness of the proposed control measures that will determine the levels of adoption in situ.