Not All Roads Lead to Rome: A Meta-analysis of Invasive Plant Impact Methodology

Abstract

The negative effects of invasive plant species on native ecosystems, which can be large and long-lasting, are the primary justifications for their research and management. Tremendous effort is focused on quantifying the ecological impacts of invasive plants, though two different methods are primarily used: observational (compare invaded and uninvaded) and removals (compare invaded and invader removal). However, it is unknown whether these methods, which differ in their assumptions and execution, yield similar results, which could affect our ability to draw broad conclusions within and across studies. Therefore, we performed a meta-analysis on 174 studies that described 547 impacts of 72 invasive plants to test the effect of study method, invader cover, and removal period on the direction and magnitude of impact. Overall, by only considering impact magnitude and not direction, both observational and removal methods yielded similar results—invasive plants are changing most aspects of the ecosystem—and the variation among species and study systems was dramatically reduced compared with traditional, directionally focused studies. This is contrary to a similar analysis that did not control for directionality of impacts, which found overall differences in impact depended on methodology. However, even when the effects of study ecosystem, invader life-form, and impact type were accounted for, significant differences occurred between removal and observational studies. Particularly vulnerable systems appear to be those that would be more greatly disturbed by the removal of the target species, such as tree species or invasive plants in riparian areas. Additionally, impact magnitude increased with invader cover and removal time. We confirm that invasive plants impact the systems they invade in a nonuniform manner; however, we suggest some study conditions may be more sensitive to study methodology. Careful consideration should be given as to which methodology is used in the context of the study system.

There has been a recent upsurge of interest in identifying the impacts of invasive plants on ecosystem structure and function to better understand their role in global change. These ecological impacts include effects on plant (Hejda et al. 2009; Vilà et al. 2011), animal (Litt et al. 2014), and soil microbial communities (Allison et al. 2006); biogeochemistry (Wang et al. 2015); and ecosystem function (Charles and Dukes 2007). In response to the exponential growth in invasive species impact studies (Lowry et al. 2013), there has been vigorous debate on how and why researchers choose the species and impacts they study, as large biases exist (Hulme et al. 2013; Jeschke et al. 2014). Despite these limitations, several meta-analyses have elucidated general trends of the ecological effects from invasive plants (Gaertner et al. 2009; Pyšek et al. 2012; Vilà et al. 2011). However, current meta-analytic techniques are limited by the variation in studies that comprise them. To overcome the limitations of meta-analytic approaches, integrative (Barney et al. 2013) and comparative (Blackburn et al. 2014; Kumschick et al. 2015a) frameworks have been proposed, but these have only recently been adopted (e.g., Barney et al. 2015). However, with each incremental advance in our understanding of invasive plant impacts, the same concern has repeatedly been outlined; there is no single method to quantify the impacts of invasive plants (Barney et al. 2013; Kumschick et al. 2015b; Stricker et al. 2015). This is becoming increasingly important, as invasive plant impact information is becoming used in decision-making tools.

Box 1 Management Implications

Invasive plant species are considered important because of the negative ecological impacts they impose on the ecosystems they inhabit. Thus, invasive plant species are managed to either reduce their negative economic and ecological impacts or to restrict/slow their spread. Knowing the ecological impacts that result from the presence of an invasive plant species is critical not only for understanding their role in the ecosystem, but for prioritizing management. Identifying the impacts of invasive plant species constitutes a significant proportion of what invasive plant ecologists do; however, scientists have not come to consensus on the best methodology to measure ecological impacts, resulting in potential over- or underestimation of invasive species’ impacts, depending on the methods chosen. With the current dearth of understanding of invasive plant impacts, if impacts are improperly measured, management decisions can be made based on incorrect information, and more harm than good may be done. Additionally, in some cases the direction of impact (i.e., positive or negative) is important, but in many cases land managers are interested in mitigating the magnitude of impact regardless of the direction. A careful examination of currently used methods helps illuminate the potential differences that heterogeneous methodologies may produce and can help in identifying appropriate impact study design and thus management goals. Our study shows that, in some cases, how impacts are measured can yield different conclusions. Importantly, our study shows that when evaluating a variety of impacts across diverse invasive species, inclusion of impact direction can mask important ecological changes.

The discordance in quantifying invasive plant impacts (1) may have important consequences for our ability to accurately identify impacts; (2) impedes our ability to directly compare impacts across species, sites, and time; and (3) remains an overlooked, yet possibly important source of variation in existing meta-analyses of invasive plant impacts (Barney et al. 2013; Kumschick et al. 2015b). Invasive plant impacts cannot be determined by only studying an invaded landscape. Instead, impacts are estimated as the difference between the invaded landscape and a reference/control site. Unfortunately, different “references” that vary in their assumptions and execution have been used to estimate invasive plant impacts. Optimally, field experiments that use multiple references are ideal; however, this is rarely the case (see Materials and Methods). Therefore, meta-analysis may be an alternative to examine the effect of method on perceived impact. Despite the acknowledgment of the consequences of using several different methods to estimate impact (Kumschick et al. 2015b; Vilà and Weiner 2004), no meta-analysis to date has considered the potential implications of study method on the estimates and conclusions of invasive plant impacts (Liao et al. 2008; Pyšek et al. 2012; Vilà et al. 2011). We addressed these major limitations by using a meta-analysis to answer the question: Do different methods yield different impact results?

Three categories of field methods (discussed in more detail below) are used to estimate the impacts of invasive plants, all of which vary in their references. As White and Walker (1997) alluded in the context of restoration: “Among the problems of individual reference sites is the difficulty of finding or proving a close match in all relevant ecological dimensions.”

The observational method compares the invaded site to a nearby uninvaded reference site, which is an area that is deemed similar in all characteristics except for the absence of the invader. It is assumed that the uninvaded site represents the state of the invaded patch before the invasion (Barney et al. 2015), and thus any measured differences are attributed to the invader. The primary limitations are finding an area near the invasion that has not been invaded and confirming that the uninvaded patch is truly similar to the invaded patch. Despite the assumptions and shortcomings of the observational method, it remains one of the most common and robust methods for estimating impacts (Barney et al. 2015; Kumschick et al. 2015b).

In circumstances in which no nearby ecologically similar uninvaded patch exists (i.e., the site is completely invaded), the only option is to remove the invader in some areas. The removal method compares the invasion to patches within the invasion where the target invader has been selectively removed (i.e., removal reference). This has the advantage of not requiring invaded and uninvaded sites for comparison. Unfortunately, removal of an invasive plant may never return a site to a preinvaded state (Corbin and D’Antonio 2012). So-called legacy effects, which exist when certain ecosystem characteristics do not immediately return to their preinvaded state, have been documented in soil microbial composition (Belnap et al. 2005) and soil nutrients (Marchante et al. 2008), but not always in plant community composition (Tekiela and Barney 2015). In some cases, removing the invader may not lead to a return at all, as is potentially the case in novel ecosystems (Hobbs et al. 2009). Additionally, the act of removing an invasive species is itself a disturbance, the magnitude of which varies among methods (i.e., cutting vs. pulling), which can complicate parsing the effect of the invader from the removal itself. Again, despite these limitations, this technique is also commonly used to estimate invader impacts.

The third method creates a “synthetic invasion” by adding the invasive species of interest to an uninvaded area that is then compared with the nearby uninvaded area. Similar to a removal reference site, temporal issues exist in attempting to quantify impacts in synthetic invasions. Just as legacy effects can persist following invader removal, the magnitude of impacts of an invasive plant can change with time after invasion (Dostál et al. 2013; Iacarella et al. 2015). In other words, with this method, when has sufficient time passed for the synthetic invasion to be considered established and representative? In addition to logistical uncertainties, ethical questions must be considered when adding an invasive plant into an area not previously invaded (Minteer and Collins 2010). For additional review of these methods, see (Kumschick et al. 2015b). Therefore, since these methods vary tremendously in their underlying assumptions, yet are used interchangeably, our primary objective was to compare the impact estimates among these methods using a meta-analysis.

Invasive species can change any ecosystem parameter to be either numerically higher or lower than the reference. This directionality is meaningful when looking at specific species and impacts of interest (Hulme et al. 2013). However, outside of testing specific hypotheses, assigning numerically positive and negative changes as ecologically “good” or ”bad” is difficult at best. Furthermore, when integrating across diverse metrics or species, as in meta-analyses, a focus on impact magnitude best represents the conservation concern posed by invasive plants—Are they changing native systems?—that would otherwise be obfuscated with impact direction (Barney et al. 2013). In other words, the most relevant information when quantifying broad trends in the ecological impacts of invasive species is whether they are changing the parameter of interest. Therefore, our second objective was to identify whether the experimental methods are similar or not if only impact magnitude is considered. In conclusion, we aim to determine whether methodology matters when determining invasive impacts, and whether directionality should or should not be considered.

Materials and Methods

Data were gathered using the Web-based scholarly publication search engines ISI Web of Science (https://apps.webofknowledge.com) and Google Scholar (https://scholar.google.com) in April 2015. We used the search term: invas* AND “plant” AND (“removal” OR “manipulate” OR “effect” OR “impact” OR “restoration” OR “management”). This search term returned 8,256 results in Web of Science and 2,180 results in Google Scholar. Results were sorted by relevance in Web of Science and then articles were systematically screened for experimental designs that recorded quantitative measures of impact and included at least one of the following study methods: (1) observationally compared invaded and uninvaded plots, (2) a removal treatment within the invasion compared with surrounding invasion, (3) an addition treatment where the species of interest was introduced and compared with an uninvaded area, or (4) any combination of these methods. Pure modeling studies and studies that did not explicitly make an invaded-reference comparison were not considered. Additionally, greenhouse studies were excluded because of the limited application greenhouse studies have to measuring invasive plant impact. Because of the volume of articles returned by the broad search terms used, when literature no longer included more than one key word, the literature search was halted. The literature cited section of each article was also scanned for any additional articles with relevant methods. Only primary literature was considered in the final database, but literature cited in reviews, gray literature, and meta-analyses were also scanned for additional primary literature for inclusion.

This search resulted in 112 observational, 62 removal, and 8 addition articles, 9 including both uninvaded and removal methods, which allowed us to compare methods within the studies. Only one study included multiple invasive plants; it was therefore treated like the rest of studies. Due to the low number of articles using the addition method, they were excluded from further analyses. However, this also suggests that addition studies are relatively rare (4% of studies). When multiple impact metrics were measured (e.g., soil organic matter, soil pH, plant richness) within a single study, the article was further divided into individual tests or metrics, and each test was considered independently, as is suggested in Vilà et al. (2011). A total of 299 observational and 208 removal metrics were recorded from the 174 studies. Our database was smaller than Vilà et al. (2011) due to our more stringent search criteria to test specific clearly defined methods.

For each test, the response means, standard errors/deviations, and replications per treatment were recorded directly from the text or tables of the article when possible. In cases in which data could not be obtained directly, either the corresponding author was contacted with a request for the data or software (DataThief, http://datathief.org) was used to extract values from figures. In the case of the same test being performed over time or over a range of invader abundance (i.e., cover or density), the difference between longest time or the highest invader abundance was used. Additionally, if available, data were collected on study length, ecosystem type (dune, forest, grassland, riparian), invader life-form (grass, forb, shrub, vine, tree), and response metric type (animals, ecosystems, microbial, plant, soil) to use as response variables and covariates during analyses. For removal studies, removal type (e.g., herbicide, hand pulling) was not considered, because replication within each method was small. Chi-square tests were performed on contingency tables of method type with either ecosystem type, response type, or life-form to determine whether distributions within method type were similar. These contingency tables showed an unequal distribution of ecosystem type, invader life-form, and response metric among study methods (P<0.0001, P=0.0144, P<0.0001, respectively; Figure 1).

Figure 1 Proportional distribution of studies within each categorization for each study method. Numbers within or above bars denote number of studies in each category. Parenthetical numbers denote the total number of negative, null, and positive studies, respectively, within each method. Asterisks (*) signify significant chi-square tests between method types.

As with other ecological meta-analyses (e.g., Vilà et al. 2011) we calculated Hedges’ d, a measure of effect size that is relatively robust at a small replication size, which is frequently the case with ecological studies. We used Hedges’ d with invaded and uninvaded or removal impact metric values to compute a difference similar to standard statistical tests. However, Hedges’ d normalizes these values to the standard deviation of the study to make studies comparable (Rosenberg et al. 2000). A numerically positive Hedges’ d indicates the invaded impact metric value is greater than the reference value; zero (i.e., zero is within the confidence interval [CI]) means there is no difference; and numerically negative values mean the invaded impact metric value was smaller than the reference value. However, the numerical direction does not accurately reflect ecologically “good” and “bad” impacts. Because meta-analyses combine many different species, systems, and individual metrics, the interpretation of the directionality of Hedges’ d can be abstruse and largely subjective (e.g., negative pH and positive diversity). Thus, to address our second objective, which focuses on impact magnitude, we also calculated an absolute value of Hedges’ d (|Hedge’s d|). This absolute value transformation removes directionality and allows for a more complete accounting of the difference between treatments.

Mixed-effect models were used to allow variation of effect sizes among studies, and due to its robustness when using nonnormal data, DerSimonian-Laird estimator was used at a 95% CI for meta-analyses (Kontopantelis and Reeves 2012). We performed a meta-regression for Hedges’ d and |Hedges’ d| using only study method, or study method, ecosystem, life-form, and response type to test whether study method was a significant factor in describing impact. Additionally, a subgroup analysis was performed for each separate categorization (i.e., ecosystem, response type, life-form) on Hedges’ d to test whether study methods were different within categories. However, only dune (n=51), grassland (n=109), forest (n=228), and riparian (n=78) had adequate replication; therefore, they were the only categories included in the ecosystem subgroup analysis.

Invasive plant cover has also been shown to influence the magnitude of impact for some species (e.g., Greene and Blossey 2011; Jäger et al. 2009). Therefore, meta-regression was also performed on |Hedges’ d| for studies that explicitly listed cover (n=278). As previously discussed, time since removal can also be important due to legacy effects from the invasive plant (Cuddington 2011); therefore, a meta-regression of number of years with the invader removed for removal studies (n=208) was performed. Only |Hedges’ d| was used for these analyses, because increase in magnitude of responses over each variable could be masked by directionality of responses in Hedges’ d. In both cases, ecosystem, life-form, and response metric type were included as covariates. All meta-regressions were performed in OpenMEE (Wallace et al. 2016).

Results and Discussion

Across all studies, the observational method showed that the invaded site was not different from the uninvaded site (i.e., the CI of the effect size overlapped zero). However, removal studies did show a significant overall negative impact of invasion (Table 1; Figure 2). Importantly, the observational and removal methods were also different from each other (Figure 2). In other words, the two methods give significantly different outcomes. However, when the effect size was transformed (|Hedge’s d|) to remove the influence of directionality and focus on change alone, the methods resulted in similar and strong effect sizes. Both showed that the invasion was different from the reference (Table 1; Figure 2), that is, invasive plants strongly changed the invaded systems. Regarding the meta-regression that accounted for ecosystem, response type, and invader life-form, explanatory power increased, and each of the subgroup variables was statistically significant, but importantly, study method no longer was (Table 1).

Figure 2 The overall effect of method type on both Hedges’ d and transformed Hedges’ d (absolute value). Unfilled circles are observational studies; filled circles are removal studies with 95% CIs. Asterisk (*) designates significant difference (95% CI); replications are listed to the left of data.

Table 1 Statistical results of meta-regressions for Hedges’ d and |Hedges’ d| with significant effects bolded.

The effect of study method varied strongly among ecosystems, though with no consistent trend (Figure 3a). The effect size varied between study methods in riparian and grassland ecosystems, but in opposite directions (Figure 3a). However, with |Hedge’s d|, the study methods only differed in the riparian ecosystem, where the observational method had a higher effect size than removal. The only difference between the methods for life-form was in grasses, where removals had a negative effect size while the observational method was not different from zero (Figure 3b). Regardless of method, for most life-forms, the CI of the mean effect size overlapped zero, except shrub observational and grass removal. However, once transformed, all life-forms had large effect sizes, and methods only varied among trees, where tree removal had a larger effect size (Figure 3b).

Figure 3 Subgroup analyses for ecosystem type (A), life-form (B), and response metric type (C) for both Hedges’ d and transformed Hedges’ d (absolute value). Unfilled circles are observational studies; filled circles are removal studies with 95% CIs. Asterisks (*) designate significant differences; replications are listed to the left of data.

Effect sizes for response types varied widely in magnitude and direction, with differences between study methods in microbial and ecosystem impact metrics (Figure 3c). Effect size CIs were different from zero more often for observational than removal and were negative for plant, ecosystem, and animal response types (Figure 3c). When transformed, the effect size was similar for both methods within response types.

The magnitude of the transformed effect sizes was moderately positively correlated with removal period (P=0.0640; R²=6.8; Figure 4a) and strongly positively correlated with invader abundance/cover (P=0.0194; R²=22.7; Figure 4b).Using traditional meta-analytic techniques, the two most common study designs used to study invasive plant impacts gave different results. Despite this dramatic finding, this disparity could be partially explained by the tremendous variation among study systems and the inclusion of directionality not associated with “good” or “bad” impacts, because when the factors were considered, the difference disappeared.

Figure 4 Meta-regressions of removal period (A) and invader percent cover (B) with transformed Hedges’ d. Circle size represents study weight in regression.

The effect of method on invasive plant impacts was strongly dependent on the ecosystem in which the study was conducted, the invader life-form, and the type of impact metric. Furthermore, of the nine studies that used both methods, six found that the removal method had a smaller impact magnitude than observational, while the rest found no difference. However, it is not surprising that individual studies may show different trends than meta-analyses, as the latter must integrate over many different systems, invaders, and impacts. Supporting our results, Hulme et al. (2013) found that the systems, species, and metrics studied to date have been unevenly distributed and haphazardly chosen. In fact, Vilà et al. (2011) also found large variation and few trends in invasive plant impacts aside from those imposed by nitrogen-fixing species.

Despite the acknowledged large variation among studies, our subgroup analyses identified a few areas where study methods give different results. For example, invasive trees and riparian systems showed different impact magnitudes between the methods. This is not surprising in the case of trees, which are often difficult to remove and account for a large proportion of total biomass. Therefore, the larger impact magnitude in tree removal studies could have resulted from the removal itself, and not the presence of the invasive tree—though this would need to be empirically tested. This also points out a limitation in our current analysis. We were not able to further subdivide life-forms into perennial and annual due to limited replication; therefore, grasses and forbs were a combination of perennial and annual species. This integration could have hidden the potential importance of the difficulty of removal for some of the species. Riparian ecosystems also showed variation between study methods: observational studies showed a higher impact magnitude than studies using the removal method. This could be attributed to either lag effects reducing the magnitude of impact from removal studies or an inappropriate selection of observational reference sites, though it is unclear why riparian systems are unique in their response to removals. The answer to our primary question of whether the two primary methods used to assess invasive plant impacts are equivalent is “no”—sometimes. The differences are more nuanced and depend on what invader life-forms, ecosystems, and metrics are being examined. Though we did not identify consistent and egregious differences between methods, there remains enough ambiguity that differences could alter conclusions regarding whether invasive plants are the source of important ecological changes. Thus, we advocate the adoption of consistent and uniform approaches to quantifying invasive plant impacts (e.g., Barney et al. 2015). Much of ecology is moving in a direction to address grand challenges, because among many advantages, such an approach allows for integration across studies using traditional statistical techniques, which are more powerful than meta-analyses. However, this does not address the principal question of why we expect consistent directional trends in impacts across the diversity of species and systems? Outside of a few clear metrics for which we want to know the direction of change (e.g., native diversity), is it reasonable to believe that invasive plants would consistently change any ecosystem parameter in a single direction?

In some cases the direction of change is important to the interpretation of the effect of an invader, such as when specific hypotheses regarding particular ecosystem properties are of interest (Hulme et al. 2014). Certainly the literature is replete with examples of the effects of individual species on individual ecosystem properties (Hulme et al. 2013), which present a series of problems of their own (Barney et al. 2013). However, because meta-analyses broadly integrate species, ecosystems, and metrics, the utility and interpretability of directionality are difficult to interpret (Pyšek et al. 2012). For example, if we are looking for trends in changes to soil organic carbon and Invader 1 increases SOC by 1 unit and Invader 2 decreases SOC by 1 unit, the mean change is zero. We would then conclude that invasive plants do not change SOC, which obscures the fact that both species changed SOC by 1 unit. This highlights one of the limitations of traditional meta-analytic techniques. Thus, our transformed effect size, |Hedge’s d|, represents the mean impact magnitude of invasive plants (Barney et al. 2013).

The advantages of bringing focus to the magnitude of changes are many, the most important of which is a more representative accounting of how invasive species change ecosystem properties. This obviates the difficulty in interpreting whether a reduction in SOC or an increase in soil pH, for example, is ecologically negative or positive—they are changes from the native state. We do not advocate that the transformation proposed here replace traditional meta-analyses, which still have utility, particularly when evaluating specific hypotheses, species effects within a single ecosystem, or global effects on single metrics. For example, one of the primary concerns of invasive species is that they reduce native diversity. In this case, traditional meta-analysis is most appropriate. However, when looking for broad trends in the face of tremendous variation, especially when there is no a priori hypothesis, we promote using directionally agnostic transformed effect sizes.

While most studies do not investigate the relationship between invader abundance and impact (Barney et al. 2013), some have shown a linear increase in magnitude with grass (Olsson et al. 2012), forb (Hulme and Bremner 2005), shrub (Gooden et al. 2009), and tree (Ruwanza et al. 2013) cover. It has also been shown that abundance–impact relationships can be nonlinear (Tekiela and Barney 2015; Toth 2016; Truscott et al. 2008; Young and Schlesinger 2014), and others have shown more complex relationships (Fried and Panetta 2016; Thiele et al. 2011). Here we show that impact magnitude increases with invader abundance across all studies. This supports the calls for impact studies to explicitly account for invader abundance (Barney et al. 2015). Elucidation of abundance–impact relationships has the added conservation benefit of identification of possible management thresholds, which could aid management prioritization (Panetta and Gooden 2017).

We also saw an increase in impact magnitude with increasing time since initial invasive plant removal. This corroborates the hypothesis that legacy effects can exist following invasive plant removal (Corbin and D’Antonio 2012), or indicates that sufficient time is required to recover from the removal disturbance itself. Our results suggest that on average, legacy effects exist in removal studies and should therefore be carefully considered before performing this type of study and when interpreting results.

As we continue to understand and catalog the ecological impacts of invasive plants, we must pay careful attention to the chosen method, which may affect the magnitude, direction, and ability to detect impacts depending on the invasive species and system invaded. When looking for effects of the ecological consequences of invasive species, directionality of impacts should be carefully considered. Additionally, we have demonstrated an effect of invader abundance on impact magnitude—reinforcing the need to explicitly account for invader abundance. Traditional and directionally agnostic meta-analyses elucidate important trends in invasive plant impacts, both of which add to our understanding and search for broad trends. We also caution against using removal methodology when dealing with deep-rooted perennial species unless careful consideration is given to the removal method. Finally, due to the wide variation among studies, we support the need for globally coordinated experiments (e.g., Global Invader Impact Network; Barney et al. 2015) that are designed to reduce variation and increase statistical power to aid in our search for understanding invader impacts.