Retrospective Analyses Are Hard: A Cautionary Tale from EPA’s Air Toxics Regulations

Abstract

Under the 1990 Clean Air Act Amendments, the U.S. Environmental Protection Agency (EPA) was required to establish standards limiting air toxics emissions from industrial plants. This paper examines the effects of five of the largest-cost rules issued by EPA in the initial round of air toxics rulemaking over the 1995 to 2000 period. Our estimates suggest that plants in the printing and publishing and pulp and paper industries realized important reductions in their air toxics emissions in the period between publication of the final rule and the effective date for compliance with the rule – although the reduction in air toxics emissions by pulp and paper mills fell short of EPA’s ex ante projections. However, our estimates also suggest that plants in three other industries – petroleum refining, pharmaceuticals, and wood furniture – achieved little or no additional reduction in air toxics emissions over the compliance period in response to EPA’s rules. Finally, the paper explores steps that EPA should take in setting up future retrospective analyses.

The 1990 Clean Air Act (CAA) Amendments revised the provisions addressing air toxics emissions by requiring the U.S. Environmental Protection Agency (EPA) to set technology-based standards for major sources of air toxics. Congress turned to a technology-based approach because EPA had managed to regulate only a few air toxics and their sources in 20 years of regulation under the 1970 CAA. Technology-based standards were a core piece of the 1977 Clean Water Act, and their implementation over the 1980s was widely viewed as achieving substantial reductions in the industrial discharge of toxics in water. Through the 1990 CAA Amendments to Section 112, Congress hoped to replicate that experience with a widespread initiative to reduce toxic air emissions.

Relatively little retrospective research has evaluated EPA’s industry-specific, technology-based emissions standards for toxic pollutants issued under Section 112. One of the key barriers to conducting retrospective analyses is the absence of data at a sufficiently fine-grained scale to yield meaningful results. Because of the limitations on the availability of data, the retrospective studies of environmental regulations in most cases have been accidents of convenience enabled by the existence of a relevant data set rather than part of a systematic plan.

This study of the 1990s round of air toxics rules issued by EPA pursuant to the 1990 CAA Amendments was undertaken as a part of a broader Resources for the Future (RFF) Regulatory Performance Initiative to develop rigorous retrospective analyses of federal regulations using nonconvenience samples of firm-level data. ² The objective of these studies is to expand the universe of rigorous retrospective analyses, demonstrate a range of approaches to some key methodological issues – for example, baseline construction – that arise in conducting retrospective analyses, and advance the conduct and public discussion of the performance of both federal regulatory programs and individual regulations. In addition, by including EPA’s 1998 pulp and paper Cluster Rule, this study complements EPA’s recent retrospective cost study of the Cluster Rule.

As discussed below, this retrospective study originally sought to investigate an entire “class” of rules to mitigate concerns with selection bias. Unfortunately, it serves more as a cautionary tale on the difficulty of doing retrospective analysis because data limitations severely constrained the scope of the study. We were able to study only 5 of 13 manufacturing industries covered by the technology-based rules in this period, and even for these 5, data issues restricted the quality of the analysis. Section 1 provides some background and summarizes the literature. Section 2 outlines our methodology, and Section 3 presents the results. Section 4 discusses lessons learned and offers recommendations.

1 Background

1.1 Regulatory context

Prior to adoption of the 1990 CAA Amendments, EPA had authority to regulate individual air toxics based on their specific health risks. However, over two decades, EPA regulated only seven air toxics emitted by a few sources. To speed up EPA regulation of air toxics, Congress amended the CAA in 1990 to focus regulation on the variety of air toxics emitted by a source category rather than set standards one chemical at a time. Section 112, as amended, lists 188 air toxics subject to regulation and charges EPA with identifying major source categories and establishing a schedule for regulation. ³ In 1992, EPA identified 174 categories of sources subject to air toxics emissions standards (57 FR 31576). Section 112 also establishes specific requirements for setting technology-based emissions limits, known as maximum achievable control technology (MACT) standards, which are to be based on the average level of control achieved by the best-performing 12% of sources in the relevant industry – the so-called MACT floor. ⁴ These standards are intended to raise all the plants in the industry to the level of control achieved by the best performers rather than force the adoption of exotic and unproven technologies. ⁵

1.2 Literature review

There is a limited retrospective literature examining ex ante and ex post estimates of the benefits and costs of major environmental rules. ⁶ In discussing RFF’s recent Regulatory Performance Initiative, Morgenstern (2015) identifies a set of barriers to conducting such studies, among them the availability of data (particularly microlevel data needed to evaluate individual rules) and the difficulty of constructing a separate, credible counterfactual baseline. Similarly, Kopits et al. (2014), in a recent EPA retrospective study of the costs of five rules – including EPA’s 1998 pulp and paper Cluster Rule – report that the paucity of available cost data and the difficulty of defining a counterfactual baseline were significant barriers to carrying out retrospective analysis. ⁷ Relatively little retrospective research has evaluated EPA’s industry-specific, technology-based emissions standards for toxic pollutants issued under Section 112. One exception is that EPA’s recently completed retrospective cost study includes a study of the cost of its 1998 pulp and paper Cluster Rule (EPA, 2014b ; Morgan Pasurka & Shadbegian, 2014). The EPA study reports:

Our findings suggest EPA’s ex ante cost estimates overstated the costs of both the Cluster Rule and the MACT II rule. Using publicly available data from [the National Council for Air and Stream Improvement], we found that EPA overestimated the capital cost of the Cluster Rule by 30%–100%, depending on the choice of baseline year from which we derived the incremental cost. Among the reasons for EPA’s overestimates of these capital costs are the mills’ use of the clean condensate alternative (CCA), flexible compliance options, extended compliance schedules, site-specific rules, use of equivalent-by-permit, and equipment/mill shutdowns and consolidations. Morgan et al. (2014, 219).

As part of the Regulatory Performance Initiative, Gray and Shadbegian (2015) examined the effect of EPA’s 1998 Cluster Rule on the toxic releases from pulp and paper mills. They report that chloroform releases fell dramatically throughout the sample period, with much of the reduction happening in the 1990s, before the effective date of the Cluster Rule. They find that pulp and paper mills achieved cumulative reductions (including prerule reductions) in chloroform releases of 99%. They also report some reductions in other air toxics, although the reductions are smaller than EPA’s ex ante projections. Reductions in volatile organic compounds (VOCs) are smaller in their fixed effects models for VOC than EPA projected but similar in magnitude to EPA’s ex ante estimate in an alternative model. ⁸ Finally, they report no significant reduction in emissions of fine particulate matter (PM).

In addition, several studies have considered other EPA programs aimed at reducing releases of toxic pollutants into the environment, including the Toxics Release Inventory (TRI) and EPA’s voluntary programs, such as 33/50 and the Common Sense Initiative.

1.2.1 EPA’s Toxic Release Inventory

Beginning in 1988, the TRI program required plants in major manufacturing industries to report annual data on toxic releases. Reported TRI releases (encompassing air emissions, water discharges, and land disposal) decreased by 37% in the first years of the program, from 1988 to 1993, but decreases in reported releases slowed to only a 10% reduction between 1993 and 1998 (Hamilton, 2005).

Overall, studies have yielded mixed results on TRI’s effectiveness in reducing toxic releases, although some studies suggest that the TRI program achieved important reductions in toxic releases in its first years. ⁹ Hamilton (2005, 250) concludes that “one cannot say what fraction of reported reductions in TRI arose from the provision of information rather than from other factors, such as command-and-control regulation or market-related fluctuations in production.”

1.2.2 EPA’s voluntary programs

EPA established the voluntary 33/50 Program in the early 1990s to promote reductions in the releases of 17 target chemicals, with the goal of reducing the total amount of the target chemicals released into the environment (or transferred off-site) by 33% by the end of 1992 and 50% by the end of 1995. ¹⁰ Some studies of the 33/50 Program report that participating firms achieved the expected emissions reductions over the period during which the program was in effect, but other studies have suggested a more limited outcome – or even that the program had little overall effect on toxic emissions. ¹¹ Coglianese and Allen (2003) and Coglianese and Nash (2014) examine other voluntary EPA programs – the Common Sense Initiative (1994–98), National Environmental Performance Track (2000–2009), and Project XL (1995–2003) – that followed 33/50 and conclude that they did not yield significant environmental improvements. ¹²

1.2.3 State programs

To manage toxic chemicals, the states have also pursued both voluntary approaches and more stringent toxics regulation. California and New Jersey, for example, were early movers in setting up programs to reduce air toxics emissions in the 1980s. However, state programs differ substantially in terms of their stringency. ¹³

Unfortunately, there is also a limited literature on the effectiveness of state programs on toxic releases. Bui and Kapon (2012, 43) find “strong evidence that both Federal and state pollution prevention (P2) programs have led to significant reductions in average facility-level toxic releases.” However, these programs were already in place before the period covered by this study. ¹⁴ Bui (2005) finds that refineries in states with more stringent toxics regulation in the form of pollution prevention programs had significantly lower levels of emissions than refineries in states with weaker or no regulation. Bui also reports, though, that reductions in toxic emissions intensity were closely related to traditional command-and-control regulation of nontoxic conventional pollutants. ¹⁵ Finally, Shadbegian and Gray (2006) also have found that over their sample period (1996–2005), states with stronger political support for stringent regulation had lower toxic emissions. ¹⁶

1.3 Available emissions data

Beginning in 1987, the TRI program has required industrial facilities to report releases of toxic pollutants into the air and water and the disposal of toxics as waste in land-base facilities. ¹⁷ We focused on TRI release data for 1993 to 2003. The data set contains annual reporting of air toxics emissions at the plant level. We identified individual facilities under a MACT standard using the Air Facility System (AFS) data retrieved from EPA through the Envirofacts Data Service API (EPA, 2014a ). The AFS lists basic summary information about each facility, including regulations that the plant is expected to meet. We then selected regulated facilities from the TRI using the Facility Registry System (FRS). Both the TRI data files and the FRS are available for download on the EPA website. ¹⁸

The original inventory contained roughly 300 individually listed chemicals, many of which are VOCs. ¹⁹ We focused on the air emissions data for those chemicals subject to the MACT rules included in the original TRI chemical list.

TRI data are limited by reporting thresholds – plants below the thresholds are not required to report. For example, the reporting threshold was 25,000 pounds per year for chemicals used in manufacturing and processing. ²⁰ In addition, TRI also limits reporting to facilities with 10 or more full-time employees (EPA, 1998a ). EPA also adopted a short reporting form in 1995 known as the Alternate Threshold Certification Statement, or Form A, for chemicals where the total release is less than 500 pounds and the total manufacture, process, or “otherwise use” of the chemical is less than 1 million pounds (EPA, 1998a ). In using Form A, a plant certifies that its total annual release of the chemical does not exceed 500 pounds. Firms using Form A do not provide quantitative information, such as a specific amount of the release, or apportion the release across media. ²¹

TRI data have been used in a number of studies (e.g., Bui, 2005; de Marchi & Hamilton, 2006; Kraft Stephan & Abel, 2011) to evaluate the effects of the TRI initiative and other programs on toxic emissions behavior. However, even while relying on TRI data, these studies have recognized certain issues:

• ∙ TRI release data are self-reported, and generally the data are based on engineering calculations rather than monitored emissions.

• ∙ Changes to reporting in 1991 (triggered by the Pollution Prevention Act) resulted in a substantial increase in reported releases in subsequent years.

• ∙ Both additions and deletions to the list of chemicals and substances occurred over the period; more than 200 chemicals were added in 1995 alone.

• ∙ TRI reporting thresholds limit coverage to facilities with 10 or more full-time employees and to facilities with releases above specified thresholds for manufacture, process, or “otherwise use”.

• ∙ The use of Form A beginning in 1995 reduced the availability of quantitative release reports for toxics below the Form A thresholds. ²²

• ∙ Chemical releases are reported in pounds, without accompanying information on the toxicity of the chemicals.

Several studies have also raised concerns about the underreporting of releases in the TRI database. Tietenberg and Wheeler (1998, 10) point out that “firms have incentives to mislead the public, either by overstating their environmental accomplishments or by selective omission (noting the positive outcomes and ignoring or burying the negative ones).” Surveys of reporting plants in the early years of TRI indicate that a significant fraction of reported reductions mixed real and paper changes (Poje & Horwitz, 1990; EPA, 1993). Dudley (1999) suggests that individual facility reports may contain such large errors that the data may be unreliable for site-specific analysis. On the other hand, EPA (1998d ) reports that by the mid-1990s, more than 80% of surveyed facilities used an appropriate method to estimate releases.

Our study focuses on TRI release data for 1993 to 2003. We selected this period in part in the hope of skirting the problems identified above with the firm-specific release data from the early years of the TRI program. Nevertheless, our study is limited to coverage of a subset of plants – those plants consistently reporting their releases to TRI – in each of the covered industries.

1.4 Air toxics rules for manufacturing

We started with 21 MACT rules issued between December 1994 and December 1998. We excluded the rules for nonmanufacturing source categories established during this period – for example, gasoline distribution, marine vessel loading operations, shipbuilding and ship repair, and off-site waste recovery operations – because they are not manufacturing operations and are generally likely to have dispersed, fugitive emissions that are difficult to monitor. ²³ We also excluded three source categories – flexible polyurethane foam production, basic liquid epoxy resins, and elastomer production – that apply to specific chemical manufacturing processes that would likely be part of a much larger multiproduct chemical plant. ²⁴ After excluding these eight categories, we were left with 13 MACT rules addressing manufacturing industry categories.

Because of data limitations, we were forced to narrow the focus of our study to rules for five manufacturing industries emitting hazardous air pollutants (HAPs): ²⁵ petroleum refining, pharmaceuticals, printing and publishing, pulp and paper, and wood furniture. ²⁶ We found that small sample size effectively ruled out analysis for 8 of the 13 manufacturing rules. ²⁷ In terms of cost, the 5 rules included in our analysis were projected to incur the largest annualized costs for the MACT rules issued by EPA in the study period (Table 1). ²⁸

Table 1 EPA estimates of HAP emissions reductions and costs for five industry categories.

Sources: EPA (1995a , b , c , 1996, 1998b , c , 1999a , 2000).

EPA (2000) provided estimates of the projected HAP emissions reductions and rule costs for the five industries covered by this review (Table 1). In a separate 1999 report to Congress, EPA provided higher cost estimates for three of these rules. EPA projected VOC reductions of 252,000 tons per year (or 60%) for refineries (60 FR 43248) and of 450,000 tons per year (or 45%) for pulp and paper mills (63 FR 18575). ²⁹

Three of the rules covered by this review – those for petroleum refineries, pharmaceuticals, and pulp and paper – were designated as major, with a Regulatory Impact Analysis containing benefits estimates (60 FR 43245). For petroleum refineries, EPA estimated benefits of $109 million per year arising from the projected reduction in VOC emissions, using benefit transfer values developed by the Office of Technology Assessment (60 FR 43245). ³⁰ For pharmaceuticals, EPA estimated that the annual benefits from the air standards would range from $3.9 million to $67 million per year (63 FR 50410). ³¹ For the pulp and paper Cluster Rule, EPA estimated annual benefits ranging from a negative $1040 million to a positive $1054 million per year. The negative benefits estimate arose because EPA also projected a small increase in PM emissions and an increase in sulfur dioxide ( $\text{SO}_{2}$ ) emissions of 105,000 tons per year. With the final Cluster Rule, however, EPA also issued an additional proposed rule, designated MACT II, to limit PM and $\text{SO}_{2}$ emissions (as cobenefits) from boilers located at these mills, largely eliminating any negative benefits attached to the 1998 Cluster Rule.

2 Methodology

We used a modified event study approach because we are looking for a marked change in the targeted emissions over the period of EPA rulemaking. Based on our review of the literature, we believe this is a reasonable approach because the effects of TRI and the voluntary 33/50 Program on toxic releases from manufacturing plants had largely run their course by 1995, when EPA began issuing the MACT round of air toxics rules.

We used two OLS estimation methods – a difference-in-difference model (Model 1) and a first-difference estimate in toxic discharges (Model 2) – to explore the change in air toxics emissions. For the pharmaceuticals and pulp and paper industry categories, we aggregated emissions across all organic HAPs on the original TRI list. For the other three industry categories, we aggregated a shorter list of organic HAPs specifically identified in the rule’s preamble as the targeted focus of the MACT rule because plants release different mixes of toxics. ³²

For each industry, we used a balanced panel, restricting coverage to firms reporting for each of the years (Model 1) or periods (Model 2). ³³ We found greater consistency in reporting (and a larger number of reporting plants) using TRI release data for odd-numbered years (1993, 1995, 1997, 1999, 2001, and 2003).

We considered two alternative approaches to provide a counterfactual baseline of HAP emissions. First, we compared the emissions behavior of regulated plants in two of the MACT-regulated industries with the organic HAP emissions of plants in similar but unregulated industry categories. We paired printing and publishing with the unregulated paper and other web (surface-coating) industry category (67 FR 72329). These two industry categories cover very similar coating activities and emit the same kinds of organic HAPs. ³⁴ We paired pulp and paper with another wood products manufacturing category, plywood and composite products (69 FR 45943). Plywood manufacturing is also part of the wood products industry; many of these plants are owned by firms that also operate pulp and paper mills. Plants in these two wood products industry categories emit similar organic HAPs and are generally located in the same geographic regions. EPA subsequently issued its final rules for the web surface- coating and plywood industries in a later round of MACT rulemaking (EPA, 2002a , 2004a ). ³⁵ For petroleum refining, pharmaceuticals, and wood furniture, we were not able to identify a closely related unregulated category.

We also compared emissions behavior for the plants in the five MACT industries in our study with the emissions behavior of plants in six additional unregulated industries – referred to as the “potpourri group” – that emit organic HAPs. The six selected industries are metal can, metal coil, metal furniture, miscellaneous coating manufacturing, miscellaneous metal parts and products, and plastic parts (EPA, 2002b , 2003a , b , c , 2004b , c ). EPA subsequently established MACT standards limiting the organic HAP emissions from the surface-coating operations of these six industries in a later round of MACT rules issued from 2002 to 2004, with compliance dates of 2005 to 2007. Since both the five MACT and the six unregulated industries emit VOC HAPs, any effects of EPA and state regulation to obtain reductions in VOCs and any residual effects of the TRI program would be captured by the potpourri control group.

Appendix B presents the average plant-level emissions (by year) for each of the five industries and for the plants in the control groups. ³⁶

2.1 Difference-in-difference model (Model 1)

For the difference-in-difference model (Model 1), we used the level of emissions as the dependent variable and the level of production (as measured by our TRI production index), the state’s League of Conservation Voters score, and the local ozone nonattainment status as control variables. The model also includes year dummies for both the control and the regulated industries:

(1) $$\begin{eqnarray}Z=\unicode[STIX]{x1D6FD}_{0}+\unicode[STIX]{x1D6FD}_{1}P+\unicode[STIX]{x1D6FD}_{2}NA+\unicode[STIX]{x1D6FD}_{3}LCV+\unicode[STIX]{x1D6FD}_{4}D+e\end{eqnarray}$$

where $Z$ is the level of emissions for each plant in the odd years from 1993 to 2003.

The independent variables are as follows:

$P=$ the TRI production index over each of the periods;

$\mathit{NA}=$ the 1-hour ozone nonattainment status of the area where the plant was located;

$\mathit{LCV}=$ the League of Conservation Voters score for the state where the plant was located; and

$D=$ a set of dummy variables for each of the years in the database, with separate dummy variables for plants in the unregulated control group versus plants in the treatment group subject to the MACT rule.

2.1.1 TRI reported production index (P)

Firms reporting to TRI also included information on the change in production activity for the reporting year relative to the previous year. We used this information to construct a production activity index over the relevant period. The production index controls for changes in production activity over several periods. ³⁷

2.1.2 Nonattainment area (NA)

Plants located in ozone nonattainment areas (NAs) may face continuing pressure to reduce their emissions of conventional pollutants (especially VOC emissions to meet ozone standards). In a study of toxic releases from petroleum refineries, Bui (2005, 1) finds that “TRI air releases are affected by being in more stringent regulatory regions for the criteria pollutants”. ³⁸ We used the 1992 nonattainment status for the 1-hour ozone standard for the ex ante period and the 1998 nonattainment status for the compliance and ex post periods.

2.1.3 State regulation (LCV)

Several studies have reported that state voluntary and regulatory programs achieved additional reductions in toxic releases beyond those required by EPA (see Bennear, 2007; Bui & Kapon, 2012). Gray and Shadbegian (2007) have also found that more stringent local regulatory requirements can result in lower emissions. They use an index for states based on environmental voting behavior as reported by the League of Conservation Voters (LCV). We would expect the sign to be negative if states with higher LCV scores required additional reductions in toxic emissions over the relevant period. On the other hand, if states with high LCV scores had already required substantial reductions in emissions, plants within their jurisdictions would not have needed to make substantial additional reductions, and we would expect a positive sign for the LCV coefficient. We used the LCV scores from 1992 for the ex ante period, 1996 for the early compliance period, 1999 for the compliance period, and 2001 for the ex post period.

2.2 First-difference estimate (Model 2)

We also considered an ordinary least squares (OLS) first-difference model to identify the effects of the air toxics rules, as follows:

(2) $$\begin{eqnarray}Y=\unicode[STIX]{x1D6FD}_{0}+\unicode[STIX]{x1D6FD}_{1}DP+\unicode[STIX]{x1D6FD}_{2}NA+\unicode[STIX]{x1D6FD}3LCV+\unicode[STIX]{x1D6FD}_{4}C+\unicode[STIX]{x1D6FD}_{4}C2+\unicode[STIX]{x1D6FD}_{5}XP+e\end{eqnarray}$$

where $Y$ equals the change in emissions over each 2-year period in the analysis.

The independent variables are as follows:

$\mathit{DP}=$ the change in the TRI production index over each of the periods;

$\mathit{NA}=$ the 1-hour ozone nonattainment status of the area where the plant was located;

$\mathit{LCV}=$ the League of Conservation Voters score for the state where the plant was located;

$C=$ dummy variable for the early compliance period, with a separate dummy variable for control versus treatment plants;

$C2=$ dummy variable for the last 2 years of the compliance period, with a separate dummy variable for control versus treatment plants; and

Table 2 Ex ante, ex post, and compliance periods for covered industries.

$\mathit{XP}=$ dummy variable for the ex post period, with separate dummy variable for control versus treatment plants.

2.2.1 Production, nonattainment, and state regulation variable

We used the same information described above for the production index, nonattainment, and state regulation variables to construct corresponding independent variables for Model 2.

2.2.2 Compliance periods for the first-difference estimates

We considered emissions behavior over four periods: a 2-year ex ante period before the rule was issued, an early compliance period including the first year in which the industry moved to comply with the rule (“Early transition” in Table 2), the last 2 years of the compliance period (Compliance in Table 2), and a 2-year ex post period after the required date for compliance. Table 2 presents the periods for each industry category.

3 Model results for covered industries

3.1 Model 1: Difference-in-difference results for level of emissions

The difference-in-difference model results for printing and publishing and pulp and paper suggest a sustained reduction in emissions from peak 1995 levels (Table 3). Overall, the difference-in-difference model explains 10% to 20% of the variation in emissions (see Appendix C, Tables C1 and C2). ³⁹ Using the potpourri group of industries as a control, the results are statistically significant for both industries for the years from promulgation of the rule to the MACT compliance date. In the cases using a paired industry as a control, the coefficients for the transition and compliance years are negative both for printing and publishing plants and for mills in the pulp and paper industry, but they are statistically significant only for pulp and paper mills and only for the transition period and the first ex post period after the effective date for compliance. For the other three industry categories, the difference-in-difference results are not statistically significant. ⁴⁰

Table 3 Difference-in-difference results for MACT-regulated plants.

$^{\ast }p<0.1$ , $^{\ast \ast }p<0.05$ , $^{\ast \ast \ast }p<0.01$ .

Looking at the control variables, the coefficient for the production index is positive and generally statistically significant; however, the magnitude of the effect of a change in production on emissions is relatively modest. The sign for the nonattainment variable is positive and statistically significant for the pulp and paper industry; the sign for this variable for the printing and publishing category is negative and statistically significant. The LCV variable carries a statistically significant negative sign for the pulp and paper industry. The nonattainment and LCV variables are generally not statistically significant for the other industry categories.

3.2 Model 2: First-difference results for changes in emissions

For the printing and publishing industry category, the results suggest a statistically significant reduction in emissions over the course of the compliance period as plants moved to comply with the MACT standards (Table 4; see Appendix C, Tables C3 and C4). For the pulp and paper industry, the compliance variables are negative – as expected – but they are not statistically significant. The results for the other three industry categories over the two compliance periods yield largely positive coefficients that are not statistically significant. Overall, the first-difference model explains less than 10% of the variation in the change in emissions over these several periods. ⁴¹

Table 4 First-difference results for MACT-regulated plants.

$^{\ast }p<0.1$ , $^{\ast \ast }p<0.05$ , $^{\ast \ast \ast }p<0.01$ .

Looking at the control variables, we find that the change in the coefficient for the production index is positive and generally statistically significant; however, the magnitude of the effect of a change in production on emissions is relatively modest. The sign for the nonattainment variable for NAs outside California and the Northeast is positive and statistically significant in the potpourri cases for printing and publishing and wood furniture. Our interpretation of this result is that the regulation of these plants – likely located in NAs in the Southeast and in Gulf Coast states – was relatively less stringent than for other NAs. The LCV variable generally carries a positive sign for the pulp and paper category and a negative sign for the other industry categories, but in all cases the coefficient is small and not statistically significant. ⁴²

3.3 Estimated emissions reductions for MACT rules

We developed estimates of the emissions reductions achieved (from baseline emissions) by the printing and publishing and pulp and paper plants in coming into compliance with the MACT rules. We focused on these two categories because the OLS regression results were reasonably robust.

For the other industry categories, the results were mixed. For the difference-in-difference results, the coefficients over the compliance periods were negative for pharmaceutical and wood furniture and positive for petroleum refining; however, they were generally not statistically significant. For the first-difference results, the coefficients were positive – a contrary outcome suggesting an increase in emissions – and not statistically significant. ⁴³

3.3.1 Printing and Publishing

The estimated reduction in organic HAP emissions for the average printing and publishing plant ranges from 100 to 150 tons per year, compared with the emissions behavior of the paired paper and web surface-coating category and the group of six potpourri industries. This represents a reduction of 60% to more than 90%. EPA estimated ex ante that the rule would reduce HAP emissions by 27% from the publication rotogravure printing industry (27 plants) and by 46% from the product and packaging and wide-web flexographic printing industry (1200 facilities). Because of the TRI reporting thresholds, these results likely reflect the emissions behavior of the publication rotogravure printing industry – 27 plants with the larger average baseline emissions – and suggest HAP emissions reductions that substantially exceed EPA’s ex ante projections.

3.3.2 EPA’s 1998 pulp and paper Cluster Rule

As noted above, EPA issued joint rules limiting air toxics emissions under the MACT provisions of the Clean Air Act and effluent guideline limits for toxics for two subcategories of the pulp and paper industry (63 FR 18504). This rulemaking followed almost a decade of substantial public and regulatory pressure, beginning in the late 1980s, to reduce releases of toxic pollutants – especially discharges of dioxin to water. In June 1989, EPA issued final guidance implementing Section 304(l); it spelled out a timetable and process for pulp and paper mill compliance with dioxin limits. ⁴⁴ The available evidence suggests that as a result, the industry made substantial reductions in its discharge of toxics to water in the years prior to 1995. ⁴⁵ To the extent that the selected dioxin control – typically, the replacement of chlorine as a bleaching agent – reduced other VOC HAPs in water discharges, the resulting control would also likely achieve reductions in air emissions (63 FR 18576). ⁴⁶ This continuing regulatory pressure on the industry through the 1990s complicates the identification of a suitable baseline for evaluating the effects of the MACT rule. ⁴⁷

Our results for the pulp and paper industry suggest that EPA overestimated the HAP air emissions reductions associated with the MACT rule. The difference-in-difference results for the pulp and paper industry suggest a reduction in organic HAP emissions ranging from 20% to 33% for the average plant, compared with the emissions behavior of the paired plywood and composite products category and the group of six potpourri industries. This is well below EPA’s ex ante projection of a HAP reduction of 60%. ⁴⁸ Similarly, our results in a separate paper suggests that the pulp and paper Cluster Rule yielded little or no additional reduction in toxic water discharges (Fraas & Egorenkov, 2015). It is our view that the Cluster Rule largely ratified changes the industry was already making in the early 1990s in response to EPA’s Section 304(l) initiative and served primarily to force the laggards to catch up. ⁴⁹

Several factors may have contributed to EPA’s overestimate of emissions reductions from the MACT rule. First, the baseline estimates for mills were based on calculations using model plants rather than monitoring data on actual discharges (63 FR 18545). Second, these estimates are relative to a declining business-as-usual baseline, since the control groups also realized a modest reduction in emissions over the 1998 to 2001 period. ⁵⁰ Finally, EPA made changes to its final rule by basing effluent limits on a less stringent set of control technologies and by allowing averaging across the various emissions points within a pulp and paper mill (rather than requiring each emissions point to meet a specific limit) (63 FR 18549). These changes – providing pulp and paper mills with less stringent discharge standards and greater flexibility in meeting air and water standards – may have operated to lower actual reductions in toxic releases.

This result – smaller reductions in toxic air emissions and water discharges – is consistent with and complements EPA’s recently completed retrospective cost study finding that the agency significantly overestimated the capital costs of the Cluster Rule (EPA, 2014b ; Morgan et al., 2014). EPA’s retrospective study offers several possible reasons for an ex ante overestimate of the capital costs, including the availability of flexible compliance options, the use of site-specific limits, and mill shutdowns and consolidations. In addition to these factors, we believe EPA may have overstated baseline emissions.

4 Discussion and recommendations

We set out with the objective of adding to our understanding of what technology-based standards actually accomplished. In addition, we believed that a retrospective study could offer lessons on ways to improve ex ante analysis of regulations and identify how retrospective analyses might be done – and done better – in the future.

The ambition of the project was to cover a large number of the air toxics rules issued during the 1990s in the first phase of the MACT program. However, we found that data on releases of toxics are surprisingly limited. We were able to conduct an analysis for only five of the air toxics rules issued under this program over the mid- to late 1990s. Data limitations and sample size precluded analyzing the effect of the MACT rules issued during this period for the eight other industries. ⁵¹

The results were mixed. The TRI data for the printing and publishing industry suggest a substantial reduction in HAP emissions – a reduction exceeding EPA’s ex ante projection – over the several years from promulgation of a final rule to the final compliance date. For pulp and paper, the TRI emissions data suggest some reduction in HAP emissions over the relevant period, but the reduction falls short of EPA’s ex ante projection. Gray and Shadbegian (2015) report similar results – that is, some reduction in air toxics but “smaller than the ex ante prediction and not always significant.” Our results and those of Gray and Shadbegian complement EPA’s conclusion in its recent retrospective cost study that it overestimated the capital cost of the pulp and paper Cluster Rule by 30% to 100% (EPA, 2014b ). Finally, our results suggest that the MACT rules for the other three industries yielded little or no additional reduction in air toxics emissions. However, these results should be viewed as preliminary and deserving of further study. The limited number of plants reporting consistently to TRI over the period of interest – not only in the early years of the program but also for some later years – underscores the difficulty of carrying out a retrospective study. Unfortunately, there is no other readily accessible source of data for toxic air emissions. We obtained emissions data covering roughly half of the plants subject to MACT in the petroleum refining and pulp and paper industries. Data problems restricted our analysis to roughly one-third of the plants subject to MACT in the pharmaceutical industry and less than one-fifth of the plants in the other two industry categories (Table 5). ⁵² Although there is no obvious bias, the absence of data for a substantial fraction of the plants in these three industries raises some concern about the representativeness of the results.

Table 5 Plants reported by EPA versus sample size in Model 1.

$^{a}$ EIA (1999) reports 159 operating refineries.

$^{b}$ EPA (2006) identifies 155 mills subject to the final Cluster Rule when it was issued.

In addition, the explanatory power of the two models is marginal. ⁵³ Other factors not accounted for in the models may, in fact, help explain the reductions in emissions during the compliance period. For example, additional, complementary information on these plants, such as firm profitability, may also be important in explaining plant emissions characteristics and improving model performance. It would be helpful to have a data set that combines emissions data with data on plant operations; for example, emissions data could be combined with plant-level data from the Census Bureau.

To facilitate future retrospective studies, EPA should include as a part of its final rule a specific plan for conducting a retrospective analysis. The agency should also provide for the collection of data – on emissions, plant production characteristics, and control measures adopted and costs incurred to comply with the rule – for at least a representative sample of plants plus a control group of unregulated plants. To some extent, EPA may be able to reduce the burden of retrospective studies by coordinating with existing data collection by the Census Bureau.

However, even if restricted to a representative sample of covered plants, the information required for these studies would be extensive and costly to collect. ⁵⁴ In addition, the cost of conducting retrospective studies and the competition from other EPA initiatives in a period of tight budgets will limit the number of studies EPA can undertake. As a result, EPA will need to be strategic in its selection of retrospective studies. ⁵⁵

In its review of agency rules under Executive Order 13563, the Office of Information and Regulatory Affairs (OIRA) should ensure that agencies establish in their rules a process for ex post evaluation of the effectiveness of important rulemakings. ⁵⁶ To implement this recommendation, OIRA could issue guidance identifying factors that agencies should consider in selecting rules for regulatory review, the kinds of measurable outcomes targeted in the analysis, the associated data requirements, the type of analysis that will be used, and the time period to be evaluated. OIRA has already provided some general guidance along these lines. In a memorandum titled “Executive Order 13563: Improving Regulation and Regulatory Review,” OIRA (2011a ) identified several areas that agencies should address in conducting retrospective reviews, including analysis of costs and benefits and coordination with other forms of mandated retrospective analysis and review. ⁵⁷ OIRA’s guidance should elaborate on these elements. ⁵⁸

In addition, EPA will need to obtain OIRA approval under the Paperwork Reduction Act for its data collection. The agency will have to show that the collection has “practical utility” and is the least burdensome way of obtaining the information. The act also requires the agency to go through a public comment process.

OIRA (2010b ) has moved to streamline its paperwork review process by establishing a generic clearance process for specific types of information collection focused on scientific research. In a memo titled “Facilitating Scientific Research by Streamlining the Paperwork Reduction Act Process,” OIRA (2010a ) has outlined options and strategies for agencies to use to streamline the process of getting Office of Management and Budget approval for information collections related to scientific research. We recommend that OIRA explicitly provide this streamlined process for research that collects data for retrospective studies.