Assessing Risk, Effectiveness, and Benefits in Transportation Regulation

Abstract

We review the practice of safety benefits analysis for federal transportation regulations in the USA. Using a case-study approach, we explore the linkages between risk assessment and benefits analysis, adding to previous work exploring these linkages for environmental health regulations. Challenges for calculating the benefits of transportation safety regulations arise because safety outcomes, like many noncancer health effects, typically do not have formal risk relationships like dose–response functions established for them. Analysts often rely on engineering or other expert judgments or resort to qualitative discussions to connect a regulatory intervention to its intended outcome. Challenges also arise when regulatory outcomes are intangible or do not have established metrics. Safety outcomes are not always measurable in concrete terms like mortality risk and may include difficult-to-operationalize concepts like “safety culture.” If the outcome is not measurable, then quantifying or monetizing the expected effects of a regulation is not possible, and the ability to conduct robust qualitative discussions also may be limited. Economists evaluating benefits for safety regulations encounter limitations analogous to difficulties found in health regulations. To inform policymaking effectively, economists and safety experts could look to the relationship developed in environmental economics between economists and health scientists.

1. Introduction

In the USA, the Department of Transportation (DOT) and the Environmental Protection Agency (EPA) issue most of the federal regulations aimed at reducing the risk of premature mortality for health and safety reasons. From fiscal years 2007 through 2016, DOT independently published 27 major rules with total estimated benefits of $22.3 billion to $40.8 billion in 2015 dollars. ¹ Most of the estimated benefits stem from safety requirements that aim to reduce injuries and deaths due to motor vehicle crashes. During the same period, the EPA published 39 major rules with total estimated benefits of $194.3 billion to $687.0 billion—a much larger amount than the benefits for DOT rules, even after adjusting for the number of rules. One reason for the difference is that EPA rules that reduce public exposure to fine particulate matter have some of the highest estimated benefits in the federal government. While the high benefits are due in part to the large population affected by the rules, EPA also uses established methods to assess the risk of cancer, cardiovascular mortality, and other adverse health outcomes from exposure to particulate matter (Environmental Protection Agency (EPA), 2010b). ² In this study, we investigate the role of risk assessment as a possible source of the differences in estimated benefits between the two agencies.

Given the significance of EPA and DOT regulations to the overall portfolio of risk-reducing regulation in the USA, it is worth considering the state of the practice for estimating benefits within the two agencies. For EPA health regulations, economists have assessed the strengths and weaknesses of current methods (McGartland et al., 2017). This paper similarly assesses the state of practice for DOT safety regulations. Describing the problems that commonly arise in risk assessment for safety problems is a first step in developing new methods to quantify or characterize benefits more robustly and assure that the benefits are appropriately considered in decision-making.

2. Risk assessment, effectiveness, and benefits

In regulatory impact analysis (RIA), the ability to conduct a thorough benefits analysis depends upon the availability of data on physical relationships between the outcome intended by the regulation and the action required by the rule. Understanding how to apply the physical relationships as part of a benefits analysis typically involves some amount of coordination between economists and experts from other disciplines. Quantifying the physical relationships, for example, may require expertise in toxicology, engineering, statistics, and other disciplines.

The scientific disciplines and source data applied to the risk assessments for federal regulations vary. ³ Benefits analysis for regulations that improve well-being through health-related benefits relies on scientific risk assessments. Scientific risk assessments include dose–response modeling to relate levels of exposure to a contaminant to the probability of experiencing negative health effects. For regulations affecting safety outcomes, the intervention tends to involve engineering solutions, and a risk assessment captures the physical relationship between the intervention and the expected reduction in the probability of death, injury, and other adverse consequences with a measure of effectiveness. Measuring effectiveness involves synthesizing a variety of data, including statistical or epidemiological data. We consider effectiveness analysis a specific form of risk assessment and analogous to dose–response modeling.

Previous research has discussed the methods applied in risk assessment for health-related outcomes and their limitations in estimating benefits for EPA regulations. Dockins et al. (2004) detail how risk assessments that evaluate outcomes that are not economically meaningful or fail to establish probabilities for health effects are of limited use in economic analysis of benefits. Axelrad et al. (2005) construct a case of a hypothetical chemical to illustrate to risk assessors the type of quantitative information most useful to economists when estimating benefits. McGartland et al. (2017) discuss how limitations in scientific risk assessments constrain benefits analysis and can cause important health effects to be excluded in the net benefits calculations that inform policy decisions. While the relationship between risk assessment and benefits analysis has been the subject of research in environmental regulation, the same is not true in safety regulation of the type typically considered by DOT.

The problem of determining the relationship between risk assessment and benefits analysis differs from the problem of determining appropriate values to assign to regulatory outcomes. This study considers the former problem—that is, how to apply physical relationships from risk assessment to quantify the changes in outcomes expected due to a safety regulation. Such outcomes need to be quantified prior to assigning monetary values, the economic ideal being willingness-to-pay (WTP) values (Office of Management and Budget, 2003, p. 18; U.S. Department of Health and Human Services (HHS), 2017, p. 11). For example, a common regulatory problem is to reduce the risk of premature fatality through the restrictions on certain economic activity. Risk assessment is the basis for quantifying the reduction in the risk expected, and economists value the risk reduction using the value of a statistical life (VSL).

To contrast the approaches to benefits analysis for safety and health regulations, we begin by outlining two example “ideal” cases in terms of data availability and methods development. For health regulations, we consider an environmental regulation where the primary health impact is cancer. For safety regulations, we consider a Federal Motor Vehicle Safety Standard (FMVSS) regulation where the primary impacts are injuries and deaths due to motor vehicle crashes. While significant analytical challenges remain for these types of regulations, the examples establish a frame of reference for other difficulties in safety regulatory analysis.

2.1. Health example: Cancer case study

A typical policy issue addressed through an environmental health regulation is human exposure to a contaminant with negative health effects. The regulatory goal is to reduce the presence of the contaminant or reduce exposure to it. We walk through an example analysis of benefits for reducing exposure to a carcinogen—an example loosely based on the RIA for EPA’s methylene chloride rule (Environmental Protection Agency (EPA), 2019)—because detailed methods for risk assessment and benefits analysis for carcinogens have been established. The risk assessment methods have been subject to criticism (Nichols & Zeckhauser, 1988; Abt et al., 2010) due to significant uncertainties that arise, for example, from needing to extrapolate information from animal studies to evaluate risk to humans. Nonetheless, the analytic difficulties are even greater for many noncancer health effects, where scientific information in a form amenable to benefits analysis is often unavailable.

In the simplest form, benefits for the regulation of a carcinogen are calculated as:

$$ Benefits=\Delta\;Exposure\hskip0.3em \ast \hskip0.3em Unit\ Risk\hskip0.3em \ast \hskip0.3em WTP $$

The key information needed to calculate benefits is a baseline measurement of population exposure to the contaminant and an estimate of exposure after regulation. For a carcinogen found in the workplace, an analyst might estimate the baseline population exposure using data on workers engaged in activities where the contaminant is present and sample measures of the contaminant in the relevant environments. A regulation banning the contaminant might effectively drive workplace exposure to zero. Alternatively, a regulation might require the use of personal protection equipment. Estimating the amount that the equipment reduces exposure is possible because the equipment can often be tested to determine the amount of protection offered. In some cases, data collected during a case study involving a specific population may be sufficient to support statistical analysis of the exposure change, which can then be applied to a more general setting. For example, advances in environmental modeling and data collection have allowed for more intricate measurement of exposure changes due to a regulation (Currie et al., 2021).

For regulation of a carcinogen, benefits analysis relies on a risk assessment to quantify the change in cancer risk expected due to changes in the levels of exposure to the contaminant. This change is “unit risk” in the equation above and is derived from a dose–response function. When the dose–response function is linear, its slope gives the excess risk of cancer to a population due to people’s exposure over their lifetimes. The risk is “excess” in that it is an increment beyond their expected risk without the contaminant in the environment.

Unit risk, or the slope factor, is the excess lifetime cancer risk estimated to result from continuous exposure to, or ingestion of, one unit of concentration of the contaminant. Unit risk multiplied by exposure yields the projected incidence of cancer. For example, an individual’s exposure for an airborne contaminant is calculated as:

$$ C\times EF\times ED $$

where C is the concentration of contaminant (e.g., μg/m³), EF is the exposure frequency (days per year), and ED is the exposure duration (years).

One can estimate the expected number of excess cancer cases from a contaminant by summing individual exposures to get the population exposure and multiplying that exposure by the unit risk. The reduction in cancer cases due to a regulation is obtained by conducting this calculation for the baseline exposure level without the regulation and for the exposure level expected once the regulation is in place.

The last step in estimating benefits is converting benefits in terms of the reduction in expected cancer risk to dollar values. For cases that would result in premature fatalities, the VSL is the appropriate weight. For nonfatal health effects, monetizing the effects using the avoided cost of illness is a common alternative to using WTP measures (Environmental Protection Agency (EPA), 2010a).

Health benefits due to regulating a carcinogen do not appear immediately because cancer generally has a latency period. Analysts account for this delay by discounting the estimates over the interval of time between reduced exposure to the contaminant and when the cancer would be expected to occur at the end of the latency period.

2.2. Safety example: FMVSS case study

The National Highway Traffic Safety Administration (NHTSA) regularly issues FMVSS rules to implement laws from Congress and has a well-established framework for conducting benefits analysis for these rules. For this example, based loosely on the RIA for National Highway Traffic Safety Administration’s (NHTSA’s) (2015) Electronic Stability Control Systems on Heavy Vehicles rule, we consider a type of FMVSS that targets crash avoidance and aims to reduce the probability of a crash altogether. Preventing a crash reduces the associated deaths and injuries, which is the primary source of benefits for this type of rule. Other types of FMVSS target crashworthiness and aim to reduce the probability of injury or death for vehicle occupants if a crash occurs. The analyses for both types of rules are similar in terms of how they translate into a reduction in the probability of deaths and injuries.

In benefits analysis, a key difference between FMVSS and carcinogen regulations is the use of actual deaths and injuries versus inferred deaths and injuries. The typical FMVSS regulation targeting crash reduction starts with a “body count” of the deaths or injuries due to the hazard. This regulatory problem differs from the case of an environmental regulation in that the impacts are directly observable in epidemiological data. The “body count” associated with carcinogen regulations, in contrast, is inferred or calculated indirectly (as discussed in Hammitt et al., 2019) from baseline exposure levels in the population and the dose–response relationship.

For an FMVSS rule, NHTSA calculates benefits using a formula with the general form:

$$ Benefits= Target\ Population\hskip0.3em \ast \hskip0.3em Effectiveness\hskip0.3em \ast \hskip0.3em WTP $$

The target population is measured as the actual fatalities and injuries to individuals involved in crashes in the preregulatory environment or baseline. Crash fatality data come from the Fatality Analysis Reporting System, a census of crashes involving fatalities on highways. ⁴ Injury estimates come from the National Automotive Sampling System/General Estimate System, a nationally representative sample of police-reported motor vehicle crashes in which trained data entry personnel interpret and code data. ⁵ Some of these fatalities and injuries would be eliminated, in theory, if the vehicles had the technology mandated by the regulation and had not crashed. Determining which crashes could have been prevented involves an engineering judgment. For a crashworthiness technology, identifying a target population may be more complicated because the crash is not assumed to be prevented; instead, the severity is presumed to be reduced. As with crash avoidance, determining reductions in fatalities and injuries involves an engineering or other professional judgment.

Some FMVSS analyses have a complication in that the required technology may have already been adopted by some percentage of the regulated vehicles. In this case, the target population will need to be adjusted to estimate the fatalities and injuries that would have occurred if no vehicle had installed the required technology. ⁶

The effectiveness rate for the technology is the key parameter in a safety benefits analysis and typically the most difficult parameter to estimate. For an FMVSS safety regulation, the sources of data and information for constructing an effectiveness measure are robust. Effectiveness rates might be based on computer simulations, expert panel assessments of crash data, and research experiments. For rules addressing crashworthiness, establishing an effectiveness measure often involves crashing vehicles, examining impacts to crash test dummies, and extrapolating those results to real-world crash scenarios. In the case of partial adoption of a technology, effectiveness could be derived from a statistical analysis of real-world crashes involving vehicles with and without the technology.

The effectiveness rate represents a synthesis or meta-analysis of many sources of data and is the key to quantifying the physical relationship between regulatory inputs. It serves the same purpose as the dose–response relationship in benefits analysis for the regulation of environmental contaminants. While methods for deriving an effectiveness rate measure for an FMVSS rule are well-established, this is not the case for other types of safety regulations, particularly in cases with limited epidemiological data on human injuries and deaths.

The final step in estimating benefits is converting the reduction in deaths and injuries to dollar values. For avoided crashes that would result in a premature fatality, the value equals the VSL. For avoided injuries, analysts approximate WTP values by applying a fraction of the VSL depending on injury severity (U.S. Department of Transportation (DOT), 2021).

Benefits due to reduced crashes accrue over the lifetime of vehicles once they are produced with the required technology. The benefits phase in, however, only as vehicles without the required technology are retired and replaced. Analysts account for the accrual and phase-in by discounting the monetary weight over the appropriate time intervals.

2.3. Comparing health and safety benefits analysis

The example regulations show that benefits analyses for the example health (carcinogen) and safety (FMVSS crash avoidance) regulations share many similarities. The goals for both types of regulation involve reducing the risk of fatal and nonfatal effects. For regulation of a carcinogen, the mechanism for achieving the goal is the change required by the rule to reduce exposure to a contaminant, which reduces cancer risk. The reduction in cancer risk provides an estimate of the number of cancer cases projected to be avoided due to the rule. This estimate can then be monetized according to existing values. For cases that would result in premature mortality, an analyst would use VSL; for cases that would result in morbidity (illness but not death), an analyst would use a cost of illness or WTP alternative if available. For an FMVSS crash avoidance rule, a requirement to install safety equipment is the mechanism to reduce the risk of a motor vehicle crash, which serves as the basis for estimating the number of fatalities and nonfatal injuries that would have been prevented if existing vehicles had been equipped with the equipment. Avoided fatalities are valued using VSL and nonfatal injuries are assigned fractions of the VSL, depending on injury severity.

The comparison shows one key difference in the basic data the two types of rules use for quantifying impacts, particularly for characterizing the baseline unregulated state. For a health regulation, the risk to be reduced is not measured directly in terms of number of cancer cases that are occurring and to be addressed within the context of the specific regulation. Rather, the level of exposure to the contaminant is measured, with the number of excess cancer cases (and associated morbidity or mortality) calculated using the dose–response relationship. A safety benefits analysis, in contrast, is an actuarial or forensic approach because it is grounded in past fatalities. The measurement of risk for a safety regulation like FMVSS begins with the historical record: the actual count of fatalities and injuries that are occurring and to be addressed by the regulation. This approach to estimating safety benefits can give the impression that impacts are more tangible for safety versus health regulations. However, it also creates a limitation for evaluating emerging safety risks that have yet to appear in the historical record.

While the actuarial approach offers a useful check for evaluating the plausibility of estimated outcomes, considerable subjective judgment is exercised in identifying injuries and deaths that would have been prevented with the technological intervention under consideration. In addition, the practice of grounding risk analysis in past fatalities creates another set of difficulties because not all safety problems in regulation can be reduced to fatalities. When a safety problem does not have a historical record, measuring economically meaningful outcomes is difficult, and the same problems described in health benefits analysis by Dockins et al. (2004) occur in safety benefits analysis.

Table 1 summarizes approaches to benefits analysis in the example health and safety regulations. If the underlying risk assessment yields a dose–response relationship or an effectiveness measure, then the benefits analysis for both is similar. If data are insufficient to support dose–response modeling or effectiveness measurement, then benefits analysis would have similar problems of unmeasurable and unquantifiable impacts.

Table 1. Comparison of regulatory analysis for rules involving carcinogens and FMVSS.

3. Methodology

To evaluate the methods used in safety benefits analysis, we reviewed DOT regulations and supporting RIAs from 2010 onward and selected RIAs for further review (Table 2). The primary criteria for selecting RIAs were whether they incorporated risk analysis into a quantitative benefits analysis in a novel fashion or whether they represented a recurring theme or problem in transportation safety regulation.

Table 2. RIAs selected for further review.

4. Challenges in safety benefits analysis

Two basic types of cases complicate safety benefits analysis: (i) cases where the effectiveness rate is unknown and perhaps inestimable and (ii) cases where the safety intervention affects outcomes that cannot be distilled to a fatality count or other measure for which an economic value has been established. These problems are similar to those encountered in health benefits analysis for regulations involving noncancer health effects. Specifically, without an effectiveness measure or grounding in a historical record, it is difficult to estimate economically meaningful outcomes to which common economic values such as VSL can be applied for monetization.

4.1. Effectiveness measurement for fatigue risk management and training

Accident studies usually identify multiple causes contributing to a single accident and often uncover behaviors or contributing factors that are common to a wide range of accidents. Lack of training and transportation sector employee fatigue are two of the most cited contributing factors across a wide range of accident types and modes of transportation.

Fatigue is a condition widely recognized to increase accident risk, and higher levels of fatigue increase that risk. Reduction of fatigue is a common objective of safety regulation. The linkage between fatigue and increased accident risk has been studied sufficiently to provide an empirical basis for estimating accident reductions expected from measures that address employee fatigue in the transportation sector.

The Federal Railroad Administration (FRA) examined historical fatigue-related accidents for the Hours of Service of Railroad Employees rule (Federal Railroad Administration (FRA), 2011). Accidents involving fatigue are human factor accidents that involve an error on the part of a train crew member, like not stopping at a red signal. FRA requires railroads to report rail accident and incidents involving damages above a certain monetary threshold ($11,200 as of January 2021). The reports include location, type of accident (e.g., derailment, collision), cargo type, damages incurred, and accident causes. FRA identified 616 human factor accidents involving 723 injuries and 8 fatalities during the 10-year period from 2000 to 2009 as potentially fatigue-related.

To estimate the number of human factor accidents that occurred under conditions of fatigue, FRA applied the results of a study of train crew work schedules and fatigue (Federal Railroad Administration (FRA), 2008). The study collected 30-day work histories of 2800 locomotive crew members involved in 1400 accidents and applied a biomathematical fatigue model to assess how work schedules contribute to increased fatigue and elevated accident risk. The modeling approach assigned fatigue scores through each worker’s shift including the time the worker was reported to have been involved in an accident. From all work histories combined, the approximate proportion of work time spent by workers at each fatigue level can be computed as well as the proportion of accidents occurring at any fatigue level. This information allows for calculating relative risk of an accident as a function of fatigue level and is like a dose–response function applied in modeling cancer risk.

Using the study’s analysis, FRA estimated that the measures to address fatigue considered in the rule would decrease the risk of a human factors accident by 67%. This reduction, when applied to the historical number of accidents (as well as the deaths and injuries that occurred due to those accidents), yields an estimate of the benefits of the rule. FRA made a further adjustment to account for its Positive Train Control (PTC) rule, which was expected to prevent 80% of PTC-preventable accidents (so that 20% of accidents remain), to get a net decrease of 13.3% (0.67 * 0.20 = 0.133).

Like employee fatigue, lack of training is often cited as a contributing factor to accidents. While training on certain tasks would logically seem to encourage adoption of safer practices that reduce the risk of an accident, demonstrating and quantifying the relationship between safety and training are difficult.

In the analysis for the Pilot Professional Development rule, the Federal Aviation Administration (FAA) cited several instances of accidents where experts considered unprofessional pilot behavior to be a contributing factor (Federal Aviation Administration (FAA), 2020). The purpose of the rule was to require behavioral training that the FAA determined was likely to reduce the risk of pilots engaging in problematic behaviors that increase the risk of an accident. However, it was not possible to separate the pilot mistakes due to unprofessional behavior from the technical mistakes that were also contributing factors in the examined accidents. Because it was not possible to measure the relationship between training and the probability of an accident, FAA limited the benefits analysis to a qualitative discussion.

FMCSA encountered similar difficulties quantifying the relationship between accident risk and training in the Entry Level Driver Training rule analysis (Federal Motor Carrier Safety Administration (FMCSA), 2016), despite widespread agreement among stakeholders that the training required by the rule would likely to lead to improvements in safety outcomes. The belief in training as a mechanism for improving safety was reflected by companies that would provide such training to their employees even without a requirement to so. While the companies may have had anecdotal evidence that employee training would increase safety, the information was insufficient to support quantitative analysis. The analytical problems encountered included lack of control groups; inability to control for driver-specific factors such as age, experience, and exposure; small sample sizes; and poor generalizability of results.

FMCSA could quantify the effect of training on fuel consumption and repair and maintenance costs. Expenses for fuel and repair and maintenance directly affect a company’s bottom line, probably more so than accidents, which remain relatively rare and are typically covered by insurance. Companies have greater incentive to track the effects of training on factors that directly affect profitability, especially if they believe overall safety performance is adequate. Quantifying these nonsafety benefits allowed FMCSA to calculate the reduction in crash rate needed for the rule to break even or for benefits to equal costs. This break-even analysis indicated that, if the rule resulted in a 3.61% reduction in crashes (central case), benefits would equal costs.

In sum, while it is reasonable to expect that measures to reduce fatigue and improve employee training would lead to improved safety outcomes, demonstrating a quantitative relationship is not always straightforward. For fatigue, it is necessary to measure employee fatigue levels throughout the work shift and the probability of an accident at each fatigue level. It is also necessary to measure quantitatively the reduction in fatigue expected due to the regulatory intervention. All of this is possible, as FRA has demonstrated, but methods for quantifying the effect of training on safety have yet to be developed.

4.2. Benefits of Safety Management Systems

Studies of major accidents show that they are the consequences of multiple, smaller failures. The “Swiss cheese” model of human error is a framework that describes the multiple, smaller failures that lead up to a major accident or catastrophic failure (Reason, 2000). The likelihood of the same circumstances repeating and causing the same accident is essentially zero, but the likelihood of preventing a similar set of circumstances leading to a different accident is probably much greater than zero. Taken to the extreme, failure to identify and treat these smaller failures can result in the same hazards becoming recurring “near-miss” events that cause people and organizations to under-estimate risks and reinforce risky decisions (Tinsley et al., 2012).

One way to reverse risky behavioral tendencies created by “near-miss” accidents is by establishing a Safety Management System (SMS). An SMS is a structured and documented set of procedures allowing company personnel to implement safety policies (Li & Guldenmund, 2018). The objective of an SMS is to proactively manage safety by identifying potential hazards and their risk and then by implementing risk mitigation measures before any accident occurs. An SMS is updated continuously and evolves based on observations of current work practices and recognizing the need for changes or additional safety protections. An SMS is thought to reduce safety hazards and incidents by creating a safety culture or an awareness of and commitment to safety shared by employees. The presence of an SMS alone could create benefits like increased employee satisfaction and operational efficiencies, but these are difficult to measure. In addition, a culture of safety culture can reduce human error and, in turn, reduce the probability of an accident and resulting damages. The latter category of benefits, which have concrete metrics like reduction in fatalities and injuries, typically is the focus of SMS benefits analysis.

In 2015, FAA required scheduled air carriers to establish and maintain an SMS in the Safety Management Systems for Domestic, Flag, and Supplemental Operations Certificate Holders rule (Federal Aviation Administration (FAA), 2015). The FAA based its benefits analysis on 123 human error accidents that the National Transportation Safety Board (NTSB), an independent federal agency with a congressional mandate to investigate major accidents, had investigated. In the investigations, NTSB identified probable causes for the accidents, such as flight crew error; the failure of industry to incorporate hazard data and follow procedures; and inadequate employee oversight. FAA’s Office of Accident Investigation and Prevention reviewed the NTSB reports and assigned a probability that each would have been averted if an SMS program had been in place. The assigned probability came from expert judgment rather than a statistical or other type of formal evaluation and for most accidents was 20%. The expert judgment of the reduction in probability of an accident serves the same purpose as an effectiveness rate and is applied to the total number of fatalities, injuries, and damaged aircraft associated with an accident to estimate benefits.

A few years later, the Federal Transit Administration (FTA) required transit agencies to implement an SMS in its Public Transportation Agency Safety Plans rule (Federal Transit Administration (FTA), 2018). The RIA cited several major accidents which resulted in multiple fatalities and injuries, as well as significant property damage. The NTSB identified multiple factors as probable accident causes, including operator fatigue; communication failures; lack of safety briefings for employees; inadequate safety procedures for carrying out work; and inadequate safety oversight by the transit agencies. Other probable accident causes included employee errors such as train operators not following rules and track inspectors not looking out for oncoming trains while repairing the tracks (U.S. Government Accountability Office (GAO), 2011). NTSB concluded that, had an SMS been in place, several of these accidents could have been avoided or reduced in severity. Thus, NTSB recommended that transit agencies adopt an SMS as required in the FTA rule. Unlike FAA, FTA could not assign a probability that an accident would not have occurred with an SMS in place; instead, FTA discussed benefits qualitatively.

The benefits analyses for SMS rules raise analytical issues. Disentangling the generalized impacts of SMS from the impacts of additional risk mitigation measures like employee training or improvements in operational procedures is difficult. Also, little attention has been given to the organizational benefits of SMS. Benefits like increased employee satisfaction or improved operational efficiencies might be useful for justifying SMS programs on their own merits, without needing to predict or speculate about the additional actions that translate into a reduction in accident probability. In addition to studies measuring the effectiveness of SMS on outcomes like employee satisfaction and efficiency, economic studies to value the effectiveness would be necessary.

The effects of SMS on accident probabilities depend upon the mitigations or actions that a business would need to undertake to prevent the accidents from happening. A reduction in probability of an accident does not occur due to the presence of SMS alone; the business must also identify risks and undertake further actions to mitigate the risks through the SMS process. In addition, without knowing the specific risk mitigation measures, it is not possible to estimate the full cost of an SMS before implementing it. The uncertainty in measuring SMS costs limits the applicability of other tools that could provide insights into benefits when the lack of an effectiveness measure prevents quantitative analysis. For example, break-even analysis can identify the number of accidents that would need to be prevented to balance benefits against costs, but it does not provide much insight for SMS programs because of uncertainty in measuring costs. When costs are uncertain or cannot be fully estimated, it is not possible to calculate the break-even value.

4.3. Low-probability, high-consequence events: multiple causes and interventions

Some types of safety-related accidents occur infrequently but have catastrophic consequences in terms of death and injury, generating considerable regulatory interest. While estimating the probability that the accidents will occur is difficult when they do not occur in most years, benefits analyses for regulations that aim to prevent these low-probability, high-consequence events often try to follow the historical record framework used for more common events. Accident investigations often identify specific causal and contributing factors for the events, and experts use that information to judge the degree to which alternative procedures could have prevented the accident. If possible, this judgment is expressed as a direct estimate of the reduction in the probability that the accident would have occurred if the alternative procedures had been followed. Adoption of the probability-reducing procedures may then be recommended as new regulations. Depending on the severity of the instigating events, the recommendations can become Congressional mandates for regulation.

Because safety benefits analysis tends to be anchored to a historical record, a risk assessment often begins with identifying causes of specific accidents and considering whether a specific remedy could have prevented the accident. The baseline scenario, or “world without regulation,” is the world as it already happened, and the hypothetical policy scenario asks how different that world would have looked if the regulation had been in place. Forming the hypothetical policy scenario is complicated because a specific remedy will rarely be completely effective at addressing an accident cause. For example, inadequate rest may be identified as one of several causes contributing to an accident, but a rest requirement might not have necessarily prevented the accident from occurring. No measure is guaranteed to eliminate the problem of human fatigue; even if it could, other contributing factors may remain untreated, and the accident might still have happened.

In its Flight Duty and Rest rule, FAA relied on experts to estimate the probability that each of a set of historical crashes would have been prevented if the rule’s requirements had been in place at the time of the crash (Federal Aviation Administration (FAA), 2011). FAA calculated expected benefits by multiplying these expert-based probability estimates by the economic costs of the actual safety consequences and adding the results across all relevant historical events (Table 3).

Table 3. Benefits analysis in the FAA flight and duty rule.

Airline crashes are rare and typically occur due to multiple contributing factors. A single crash or set of crashes might result in several separate regulations independently aimed at addressing the identified causes of the crash. For example, FAA’s rule affecting pilot training requirements (Federal Aviation Administration (FAA), 2020) drew upon a similar pool of accidents as its duty and rest rule. FAA accounts for the use of the same crashes for multiple rules independently addressing the causal factors by ensuring that the sum of the crash reduction probabilities does not exceed one. For example, Colgan Air Flight 3407 crashed in Buffalo, NY in 2009, and the NTSB accident investigation identified several probable causes of the crash. FAA used the crash as the basis for benefits estimation for four rules, assigning crash reduction probabilities of 0.5, 0.2, 0.2, and 0.05. ⁷ With all four rules in place, the expected reduction in probability that the crash would have occurred is 0.95.

The FAA approach to estimating risk reductions and benefits could be applied in other settings where one or more large accidents give rise to a suite of regulations. Analysts will need foresight regarding the actions that will be undertaken to address individual contributing factors as well as estimates of their relative importance.

4.4. Emerging risks and lack of a historical record

When accidents are rare or due to multiple causes that cannot be readily disentangled, focusing on past events limits the degree to which risk assessments can yield effectiveness measures that analysts can use in a quantitative benefits analysis. When the regulatory problem involves a new and emerging technology for which no historical record exists, the problem is compounded, and safety benefits analysis will need to shift from its forensic nature to a more forward-looking or prospective approach. This is the case for “enabling regulations,” which expand production or consumption opportunities by revising existing regulations to allow new or previously prohibited or limited economic activity.

Enabling regulations may remove barriers to adopting emerging technologies but do not preclude establishing some restrictions to manage emerging and mostly unknown risks. FAA’s Part 107 rule, for example, expanded options to allow individuals to obtain a pilot certificate to operate unmanned aircraft (Federal Aviation Administration (FAA), 2021). The RIA focused on the new economic opportunities created by the rule: the rule lowered costs to pilots already operating small UA and considerably reduced costs to entry, which would attract new pilots to the industry and expand business opportunities. Similarly, FRA treated the creation of a new high-speed rail equipment tier (Tier III) as a business opportunity that would facilitate implementation of high-speed rail and increase the probability that a system would be completed in the USA in the next 30 years by 10–25% (Federal Railroad Administration (FRA), 2018). The RIAs for these rules did not formally consider the safety impacts of allowing these emerging technologies.

Safety benefits analyses for enabling rules are more speculative than analyses for known and experienced hazards. With no historical basis for calculating event probabilities and perhaps only a vague notion of potential consequences, quantitative safety benefits analysis may not even be possible. In this case, qualitative analysis may be the best approach, as in FAA’s analysis of its Operations over People rule, which extended the Part 107 rule by allowing routine operations of unmanned aircraft over people and routine operations at night.

5. Discussion and conclusion

In examining example cases of environmental health and transportation safety regulations that are “ideal” in terms of data availability and methods development, we see only minor differences in the approaches to benefits analysis. Effectiveness measurement serves the same purpose as dose–response modeling, with the major distinction stemming from the underlying disciplines from which the relationships are obtained. Dose–response modeling is an exercise in risk assessment, which has foundations in the health sciences. Effectiveness measurement, in contrast, often involves significant engineering judgment, though statistical methods can be applied when sufficient data exist to support such analysis.

When an effectiveness estimate is unavailable in safety benefits analysis, the problem is like the lack of a dose–response relationship in health benefits analysis. Analysts lacking effectiveness estimates often resort to discussing benefits qualitatively rather than attempting work-around forms of quantitative analysis. While analysts have sometimes employed creative approaches to quantifying benefits when effectiveness measures do not exist, as shown in FAA’s approach to analyzing multiple interventions to address a fixed set of historical airplane crashes and incidents, these approaches are not generalizable to all cases.

Research on risk assessment and benefits analysis for health regulations indicates that risk assessors often focus on intermediate effects that are not necessarily amenable to economic valuation. Economists encounter similar problems in safety benefits analysis, where unmeasured and unquantified benefits may not be adequately considered in policy decisions. For example, interventions like SMS aim to reduce safety risks through improving safety culture within an organization. While measuring safety culture is challenging, valuing it is even more so. Intermediate safety outcomes need to be translated to other outcomes such as expected reductions in the probability of an accident, which in turn can be monetized using existing economic values.

The most significant difference is that the measurement of risk for safety benefits analysis is actuarial or forensic in that it is based on past outcomes like accidents and fatalities. In a sense, the evaluation of a regulation involves revisiting the past and asking how the action could have undone outcomes that have occurred. The grounding in a historical record can give the impression that impacts are more tangible for safety regulations than for health regulations. The historical record also serves as a validity check against which to compare estimated benefits. If a regulatory intervention has high estimated benefits by preventing more accidents than had occurred historically, such a finding might call into question the reliability of those estimates. Benchmarking against known outcomes also can facilitate retrospective analysis: NHTSA, for example, evaluates many of its past regulatory policies using the same accident data it uses for prospective analyses (National Highway Traffic Safety Administration (NHTSA), 2023).

The methods applied in health benefits analysis do not appear to be constrained by the historical record in the same way. It is not possible to measure directly the “body count” for EPA-type of health regulations; that number must be inferred from exposure levels in the population and the dose–response relationship. The steps involved in conducting this inference have their own uncertainties, challenges, and controversies, but they provide a potential structure for framing a benefits analysis for a safety regulation when the historical record is limited. While the comparison between EPA and DOT analysis did not yield an easy fix that DOT analysts can apply when an effectiveness measure is unavailable, the practice of grounding in past fatalities and injuries may be worth reconsidering.

While not a central focus, our study provides some insight into the limitations of using agency RIAs to judge the overall quality of regulations within and across agencies. Scholars (Sunstein 2000; Hahn, 2004; Parker, 2006; and others) have debated the merits of using metrics like net benefits and cost per life saved to rank regulations. A low benefits value or the lack of quantitative benefit analysis may serve more as an indication of the state of the scientific information and its availability in a form amenable to economic analysis than as an indication of the inherent quality of a regulation. In these cases, making the information gaps and analytical difficulties transparent can spur the development of new research (Hahn, 2004).

The issue of incomplete information is especially salient for addressing emerging safety problems where a historical record for conducting risk assessment does not exist. The challenge of performing safety benefits analysis without accident records is likely to persist, particularly for autonomous vehicles and associated emerging risks as technologies develop. Benefits analysis for emerging safety risks will likely require adopting a new, more forward-looking framework. Without historical records, it is hard to evaluate the emerging risks accurately, and benefits analysis will probably always involve some amount of speculation. Nonetheless, policymakers may find it preferable to accept some inaccuracy in the analysis instead of waiting for a record of accidents, injuries, and fatalities to accumulate.