Risk-Informed Benefit–Cost Analysis for Homeland Security R&D: Methodology and an Application to Evaluating the Advanced Personal Protection System for Wildland Firefighters

Abstract

This article describes a methodology for a risk-informed benefit–cost analysis that includes (i) risk analysis to quantify risk reduction benefits and (ii) uncertainty analyses to quantify probability distributions over costs and benefits. It also summarizes the lessons from 25 applications of this methodology to evaluate R&D projects of the Science and Technology Directorate of the Department of Homeland Security. The article then illustrates the methodology with a specific application to evaluate the benefits and costs of the Advanced Personal Protection System (APPS), a new garment system developed to protect wildland firefighters. The goals of the APPS project were to reduce risk and to improve comfort. The cost analysis revealed that the APPS garments are more expensive by about $279 per garment system. Total costs were roughly $7.3 million, including the upfront project cost and the increased 5 year cost of purchasing the APPS. Benefits from reduced injuries and fatalities resulted in 5 year benefits of about $19.3 million, with an NPV of $13.6 million in 2019 dollars. In the base case, the benefit–cost ratio was 2.87 and the return on investment was 187 % over 5 years. Taking the perspective of a decision-maker when the project was first funded in 2011, NPVs are $11,993,728, $10,025,519, and $7,967,479 in 2011 dollars for discount rates of 0, 3, and 7 % respectively. An uncertainty analysis of the NPV showed a large variability, ranging from the 5th percentile of $6.4 million to a median of $19.3 million to the 95th percentile of $43.7 million in 2019 dollars. This large range was primarily due to the uncertainty about the reduction of fatality and injury risks and the market penetration rates of the new garments.

1. Introduction

To improve government research and development (R&D) planning and resource allocation, it can be useful to estimate the costs and benefits of the R&D projects. The basic idea is to estimate how government operations can be improved by implementing the results of R&D projects and to compare the costs of the investment with the anticipated benefits from the change in operations. An example is the development of an improved hurricane prediction model that can improve government decisions to evacuate people prior to the arrival of a hurricane, thereby saving lives while reducing unnecessary evacuations.

Since 2016, a team of economists and risk analysts at the National Center for Risk and Economic Analysis of Terrorism Events (CREATE) has explored using benefit–cost analyses (BCAs) to inform managers about R&D projects and products funded by the Science and Technology (S&T) Directorate of the Department of Homeland Security (DHS) (see von Winterfeldt et al., 2019; CREATE, 2018, 2019a, 2019b). Initially focused on projects funded by the Office of University Programs (OUP) to estimate future benefits, the team later applied retrospective BCAs of research and development products funded by the First Responders Group (FRG) of S&T. The FRG group is tasked with, among other things, developing innovative solutions for use by the emergency preparedness and response community for day-to-day operations and large-scale emergencies.

Based on this experience, we adapted the basic steps for a BCA (Graham, 2008; Robinson & Hammitt, 2013; Boardman et al., 2018) to include improvements in safety and security. This led to what might be called a risk-informed BCA (Farrow, 2017). Risk analysis is used in two different ways in this modified BCA. First, for many security-related R&D projects, one of the important benefits is risk reduction (or its counterpart, security increase). Risk analysis therefore becomes an important part of benefits assessment. Second, there often is substantial uncertainty about input parameters of the cost and benefit models used in security-related BCAs. To quantify these uncertainties, sensitivity analysis, and probabilistic simulation are useful. Both aspects of risk are incorporated in this risk-informed BCA methodology.

The details of the methodology are described in the next section. Following this, we describe an application of the methodology to a BCA of a research project that developed and evaluated a wildland firefighter Advanced Personal Protection System (APPS). We conclude with a discussion of the limitations and opportunities for conducting BCAs in the homeland security R&D context.

2. Methodology

The risk-informed BCA methodology used to evaluate homeland security R&D projects and products consists of the following steps:

1. (i) Selection of R&D projects with potential risk reduction benefits

2. (ii) Baseline analysis: Cost, risk, and performance under the status quo

3. (iii) Cost analysis: R&D and implementation costs

4. (iv) Benefit analysis: Cost savings, risk reduction, and performance improvement

5. (v) Analysis of net benefits, benefit–cost ratios (BCRs), and return on investment (ROI)

6. (vi) Sensitivity and uncertainty analysis

7. (vii) Recommendations

Each of these steps is described in the following subsections.

2.1 Selection of R&D projects

An R&D manager pursuing a simple budget-constrained maximization would like to invest in projects starting with the highest net present value (NPV) (Boardman et al., 2018) or the highest BCR (Weinstein & Zeckhauser, 1973; Bellinger, 2018) and working down until the budget constraint is reached. An important task is to prioritize and select proposed R&D projects that have not been implemented or used yet. Some R&D managers in DHS also found it useful to evaluate projects retrospectively to demonstrate the value of past investments. Earlier BCAs (von Winterfeldt et al., 2019) were conducted on both prospective and retrospective cases, and, in some cases, a mix of both. In addition to selecting a mix of projects, it was also important to the R&D managers that the past or future users of the research products resulting from the R&D projects considered them both of high value and, in case of prospective analysis, had a high likelihood of being implemented and used.

Our approach to selecting the FRG cases was to identify R&D projects that were close to implementation or had been implemented, which necessarily passed some usefulness or bureaucratic criteria although not necessarily a benefit–cost criteria. While not a random sample, the analyses are useful to managers to investigate causes of success and to assess retrospectively whether expenditures generated positive net benefits. In this study, R&D managers selected nine R&D projects based on the current commercial or governmental use and the availability of data.

2.2 Baseline analysis: cost, risks, and performance under the status quo

The baseline analysis begins with an assessment of the costs of current operations, without the use of the R&D product that is being evaluated. For example, in the BCA of the Wildland Firefighter APPS garments described in this paper, there is the cost of the legacy garments which would need to be purchased if the APPS garments were not available. These costs have to be estimated for a given base year and assessed using appropriate cost adjustments to reflect the value of money.

The baseline performance and risk metrics varied between R&D projects and, in some cases, there were multiple metrics to be aggregated into a monetary present value. In some of the past case studies, baseline performance was measured by a commonly used metric, such as the number of nuisance alarms of a current detection device or the expected number of fatalities with current inspection operations. For example, in one study concerned with reducing hoax calls received by the U.S. Coast Guard (USCG), we established a base line of the number of hoax calls in a given year. These calls are costly because they trigger expensive search and rescue missions. The intent was to evaluate a new computerized voice recognition and analysis tool that had the potential to reduce hoax calls.

In other security-related projects, the baseline performance was the current risk level, usually expressed as a function of threat, vulnerability, and consequences. Both the threat and vulnerability aspects were quantified as probabilities, using data and expert judgment. For example, when estimating the risk of attacks on passenger ferries in the New York Harbor, we used worldwide data of terrorism attacks on ferries to scale the attack probabilities (threat probabilities) to the ferries in New York. Vulnerability was quantified as the probability of success of an attack, given an attempt, using expert judgments and models. Consequences were modeled using data of past successful attacks worldwide, in combination with passenger volumes and likely lives lost. The intent was to determine how a better allocation system for protecting ferries with USCG patrol boats would reduce the risk of ferry attacks in New York Harbor.

The Wildland Firefighter APPS described in this paper produced and evaluated an improved design for firefighter garments. These had several potential benefits, including a reduction of fatality and injury risks and improved comfort. As a baseline risk, we estimated the fatality and injury risks of firefighters prior to the introduction of the new APPS garment. Comfort was estimated using a real-life trial of wearing both the legacy garments and the APPS garments.

As in these examples, risk analysis is a major part of both the baseline analysis and the benefits analysis. When conducting these risk analyses, the estimation of probabilities of threat and vulnerabilities turned out to be the most challenging tasks, which required the use of often sparse data, models, and expert judgments. In contrast, consequence estimates (assuming a successful attack) were relatively straightforward.

2.3 Cost analysis: R&D and implementation costs

Having selected the R&D projects and identified baseline costs and impacts, the next step in the methodology was to assess their development and implementation costs. For this purpose, a cost accounting template was developed that assured that all initial investments were counted, in addition to transition, implementation, maintenance, and upgrade costs to the extent possible.

In some cases, all costs were upfront (i.e., project costs prior to implementation), with benefits occurring later. In other cases, costs continue into the future, both as baseline costs (i.e., cost without the implementation of the project) and the costs of operating and maintaining the new research product.

Costs and benefits occurred in various past years, with a cutoff point of 5 or 10 past or future years of use for each project. Future values were discounted at a rate informed by professional practice and government guidance (see e.g., Moore et al., 2013; U.S. Council of Economic Advisers, 2017). While there remains a large debate about the appropriate rate of future discounting (e.g., Burgess & Zerbe, 2013; Moore et al., 2013), the government audience for this work was most comfortable with approaches based on the real rate of return as suggested in U.S. government benefit–cost guidance (OMB, 2003, 2017).

While it is standard to apply a common future discount rate across projects, the appropriate rate for retrospective analysis is less clear. Presumably, a retrospective analysis would apply the rate of return relevant during that time period, either with or without inflation as appropriate. An investigation of the real interest rates between 2000 and 2019 indicated that in many of these years the real rate was close to zero (U.S. Council of Economic Advisers, 2017). Consequently, the approach used for each of the 25 cases, and for the APPS case, in particular, was to increase past values for inflation to the target year (here 2019), but to use a zero-retrospective real (net of inflation) rate of interest across projects.

In addition to this approach, we also analyzed from the perspective of decision-makers at the time of funding the project (in the APPS case, 2011) and conduct a BCA as if they were looking into the future to 2019. Using this perspective, we used the inflation rates between 2011 and 2019 to adjust the benefits and costs to 2011 dollars, then discount them, at the OMB approved rates of 3 and 7 %.

2.4 Benefit analysis: cost savings, risk reduction, and performance improvement

Assessing the benefits of R&D projects aimed at improving homeland security decisions and operations is much more difficult than assessing their costs. The largest difficulty is that there are several types of benefits which require different assessment models. Benefit–cost textbooks are replete with models for various situations (e.g., Boardman et al., 2018), but few are specific to security. Based on previous research, we identified several benefit categories and models from the decision analysis literature and from BCAs that seemed to span the case contexts (CREATE, 2019a, b; von Winterfeldt et al., 2019).

For many security-related R&D projects risk reduction is the major benefit, that is, the reduction of threat, vulnerability, and negative consequences. Like the baseline risk analysis, which can often be based on statistics, the estimates of risk reductions due to the implementation of an R&D project is very complex and involves many uncertainties. Fortunately, in many cases, the “relative” improvement in risk reduction (e.g., the percentage of reduction of threat or vulnerability probabilities) was usually sufficient to arrive at benefit estimates (Farrow & von Winterfeldt, 2020).

In prospective analyses, we used a version of decision analysis models to conduct the BCAs. The standard decision analysis models are based on decision tree analysis and value-of-information analysis (Clemen & Reilly, 2014). These models consider future project funding and implementation decisions, depending on the yet unknown results of a project, similar to real options analyses (Black & Scholes, 1973; Merton, 1973; Smith & Nau, 1995).

2.5 NPV calculations, BCR, and ROI

Cost and benefit information was presented in several ways: by the net benefits, the BCR, and the ROI. Since costs and benefits are usually distributed over time, often with upfront costs and delayed benefits, a proper calculation has to consider the time value of money. In our methodology, as noted above, we inflate costs incurred prior to 2019 by the Consumer Price Index (Bureau of Labor Statistics, 2020). When projects had future benefits projections, we discounted costs and benefits by a social discount rate of 3 %. In the APPS case study, all benefits were in the past (2015–2019), and therefore we used the CPI to adjust past costs and benefits.

The BCR is defined as usual as the ratio of the present value (PV) of the benefits divided by the PV of the costs, and the ROI is the ratio (as a percentage) of the NPV of the benefits divided by the PV of the cost.

2.6 Sensitivity and uncertainty analysis

We paid particular attention to issues of uncertainty regarding key parameter values of the cost and benefit calculations. While cost estimates are usually fairly well-established with little or no uncertainty, many of the inputs to the benefits models are highly uncertain, especially if the research product has not been implemented or used yet.

The first sensitivity analysis determines the break-even values, by calculating how much an input parameter of the costs or benefits calculations would need to be changed in order to produce an NPV that is above 0 or an NPV that exceeds the initial investment in an R&D project.

Next is a “tornado analysis” (Clemen & Reilly, 2014; SensIt, 2017). After defining a base case, we determine lower and upper estimates for each uncertain parameter. Then we calculate the net benefits for each parameter at its low and high levels. The tornado diagram shows the ranges of the net benefits as horizontal bars, with the largest bar at the top and successively shorter bars below (thus the name “tornado”). This diagram provides us with information about which parameters matter most to the net benefit calculations and which matter the least.

Following the tornado analysis, we conduct a complete uncertainty analysis for the parameters that matter most. Since we have to deal with multiple research products and hundreds of uncertain parameters, a detailed assessment of the uncertainties using expert judgment was not feasible. Instead, we use triangular distributions throughout, with the low, base case, and high estimates defining the triangular distribution for each parameter. Given that the base case parameters are defined as a local tendency measure (mean, mode, or median) and that the lower and upper values are selected to cover a wide uncertainty range, using a triangular distribution preserves both the central tendency and the range of the uncertain parameters. Triangular distributions often provide higher densities in the tails of the distribution than smooth distributions (e.g., the normal, log-normal, or beta distributions) and are therefore likely to exaggerate the tail uncertainties somewhat. This is, however, counteracted by the fact that the lower and upper end of the triangular distribution seldom are absolute values, thus leaving some chance of parameters outside the triangular bounds.

Using these triangular distributions, we employ a probabilistic simulation software called SimVoi (SimVoi, 2017) to create the distribution over the NPV for each research product. As summary measures, we use the 5th percentile, the median, and the 95th percentile of this distribution over net benefits.

2.7 Lessons learned from past applications of the risk-informed BCA methodology

We learned many lessons in the process of conducting 25 BCAs, including:

1. (i) Reasonable “first order of magnitude” BCAs could be conducted for most homeland security R&D projects, with some exceptions that were due to lack of data or an inability to link the R&D project to DHS decision-making and operations.

2. (ii) While the cost part of the assessment was usually straightforward, the benefits assessment, especially the reduction of risks, was complex and difficult.

3. (iii) The use of risk analysis helped to deal with the problem of assessing the benefits of risk reduction, especially with assessing the value of reductions of threats, vulnerabilities, and consequences.

4. (iv) Using sensitivity analysis to identify the important benefits parameters and uncertainty analysis (Monte Carlo simulation) to quantify the uncertainty ranges of important input parameters made it possible to conduct the BCAs without getting overly fixated on precise estimates.

We also learned that it is useful to distinguish several impact categories and associated benefits models in the homeland security area:

1. (i) Improved performance relative to cost

   1. (a) Reduced cost at the same performance level

   2. (b) Increased performance at the same cost level

   3. (c) Cost-effective improvement of performance

2. (ii) Reduction of risks

   1. (a) Reduction of threats

   2. (b) Reduction of vulnerabilities

   3. (c) Reduction of consequences

3. (iii) Improved signal detection capabilities

   1. (a) Increased detection rates

   2. (b) Reduced false alarm rates

4. (iv) Value of information for improved operations and decision-making

   1. (a) Improved operations through training

   2. (b) Improved decision-making through better information and communication

These impact categories are not mutually exclusive, and in many of the 25 past applications, several impact categories were used to estimate the overall benefits of a research project.

3. Application to the APPS for Wildland Firefighters

3.1 Description of the project

The APPS for Wildland Firefighters project was aimed at developing and evaluating a new protective garment system for wildland firefighters. The U.S. Army Natick Soldier Research, Development, and Engineering Center (NSRDEC) in Natick, Massachusetts, with sponsorship from the Responder Technologies (R-Tech) Program of the S&T Directorate at DHS and the Department of Agriculture’s U.S. Forest Service (USFS), developed and tested the improved garment system between April 2011 and December 2013. The project and its results are described in detail in DHS S&T (2014a, b).

This was one of six projects that CREATE evaluated for the First Responder Group of S&T. The five other projects were a mix of products (e.g., readers to detect laundered money on prepaid cards, radio-communication software, and hardware to synchronize communications in emergencies) and flood emergency aids (e.g., flood sensors, flood resilience standards, mapping the extent of observed floods; see CREATE, 2019b).

The goals of the APPS project were to (i) improve heat absorption qualities and overall thermal protection, (ii) generate a better heat loss rating to minimize heat stress, and (iii) improving the weight, form, and fit of the Personal Protective Equipment (PPE) system. By achieving these goals, the project sought to decrease the rates of heat injuries and fatalities and increase the comfort of firefighters. This would ultimately promote operational effectiveness of wildland firefighters, thereby making the success of wildland firefighting missions more likely (DHS S&T, 2014a, b).

Both objective and subjective evaluations were conducted to evaluate the APPS. Objective testing was strictly technical and quantitatively assessed fabric performance in terms of heat absorption coefficients. Subjective testing consisted of a “wear trial” in close cooperation with CAL FIRE, which provided personnel and equipment for testing the new garments in real firefighting situations. Both forms of testing the APPS system confirmed that the project did achieve the aforementioned goals and therefore validated the project’s application (DHS S&T, 2014a, b).

In this BCA, we focus on the benefits of reducing injuries and fatalities due to the improved heat absorption of the advanced garment system. We compare these benefits with the cost of the R&D project and the increased costs of the new firefighter garments.

3.2 Baseline analysis

The baseline analysis consisted of assessing the wildland firefighters’ risks of fatalities and injuries, with the intent to compare the performance in the baseline (with legacy garments) and the performance of the new advanced garments. We examined fatality data between 2007 and 2017 provided by the Federal Emergency Management Agency (FEMA, 2018). Injury data were provided in NFPA (2018). The baseline analysis also included an estimation of the costs of the legacy garments.

We were unable to find the exact number of wildland firefighters, which is needed to determine the annual fatality and injury risk (see also Britton et al., 2013, who had a similar problem). To make an estimate of the number of wildland firefighters, we used the percentage of the number of wildland firefighter fatalities and injuries to the number of fatalities and injuries of all firefighters reported in FEMA (2018) and NFPA (2018). The fatality percentage is 9.5 % and the injury percentage is 15.8 %. In the base case, we used the lower of these two percentages, rounded up to 10 %. Applying this percentage to the 1,065,433 firefighters in the United States resulted in an estimate of 106,543 wildland firefighters. This number is used throughout this BCA.

3.2.1 Baseline fatalities and associated costs

FEMA (2018) lists nine distinct fatality types among wildland firefighters (see Table 1). These included aircraft accidents, burn-related deaths, falls, heart attacks and strokes, heat exhaustion, motor vehicle accidents, respiratory-related deaths, being struck by trees/debris, and other/unclear deaths. Five of these fatality types are potentially related to the garments that the firefighters wore at the time of their deaths. These included burn-related deaths, heart attacks and strokes, heat exhaustion, respiratory-related deaths, and other/unclear deaths.

Table 1 Firefighter fatality data.

Table 2 provides the resulting statistics analyzed from the fatality data in Table 1. Prior to 2015 (the year the new garments were introduced), there were, on average, 9.25 garment-related deaths per year. The individual wildland firefighter’s risk of dying from garment-related causes is thus calculated to be 8.68 × 10⁻⁰⁵ (9.25 divided by 106,543).

Table 2 Analysis of firefighter fatality data.

Valuing of unidentified, statistical lives has a long history in BCA, with Viscusi (2009), Merrill (2017), and Knieser and Viscusi (2019) estimating the current value of a statistical life (VSL) at about $10 million; this is also roughly consistent with U.S. government guidance. Multiplying the annual number of expected fatalities (9.25) with the $10 million VSL results in the annual expected costs due to wildland firefighter fatalities of $92,500,000.

3.2.2 Baseline injuries and associated costs

Table 3 shows the annual frequencies of injuries for all firefighters and for wildland firefighters between 2007 and 2017. Prior to the availability of the APPS garment in 2015, wildland firefighters used either the legacy garments described below or similar garments. For the baseline, we can therefore use the death and injury statistics involving the legacy garments. Between 2007 and 2014, there were, on average, 11,578 injuries per year related to wildland firefighters (see Table 3; FEMA, 2018; NFPA, 2018). Of these annual wildland firefighter deaths and injuries, 4891 were related to garments. Therefore, the individual annual risk of injury of a wildland firefighter is 4.59 %.

Table 3 Frequency of injuries of firefighters.

NFPA (2018) lists 10 distinct injury types among firefighters (both general and wildland, see Table 4). These included burns (fire/chemical), smoke/gas inhalation, other respiratory distress, burns and smoke inhalation, wounds/cuts/bleeding/bruises, dislocations or fractures, heart attacks or strokes, strains/sprains/muscular pain, thermal stress, and other. Of these injury types, we excluded two that clearly did not relate to the garments that firefighters wore at the time of their injury: dislocations or fractures, and strains/sprains/muscular pain.

Table 4 Types of firefighter injuries and associated costs

Like VSL, valuing injuries has a long history, recently summarized by Robinson and Hammitt (2013). A variety of methods including direct and indirect cost of injury, stated willingness to pay, and quality-adjusted life years have been used. We used as a primary source the catalogs of injury costs provided by Finkelstein et al. (2006), who separate out costs associated with injuries requiring hospitalization and costs associated with injuries that did not require hospitalization. For the high end of the range of costs in Table 4, we used the hospitalization costs, and for the low end of the range, we used the costs of ambulatory or emergency treatment. For the range of costs for heart attacks, we used a different set of sources (Vernon, 2010; Pierce, 2014; American Heart Association News, 2017; UT Southwestern Medical Center, 2017).

Table 5 shows the average annual incidence of wildland firefighter injuries together with the Finkelstein et al. (2006) costs of injuries. FEMA (2019) estimates that 46 % of firefighter injuries involve lost workdays. We applied this percentage to the cost of hospitalization vs. 54 % to the cost of an emergency room visit to arrive at a weighted average cost of injuries (the weighted average cost column in Table 5). We then calculated the total annual cost using this weighted average cost and the annual frequencies of incidence (2nd column in Table 5). As the last step, we inflated the costs reported by Finkelstein et al. (2006) from the year 2000 to the base year of our analysis, 2019, using the CPI inflator of 1.52 (Bureau of Labor Statistics, 2020).

Table 5 Baseline costs associated with firefighter injuries.

The total inflation-adjusted annual cost of injuries per firefighter were calculated as follows:

(1) $$ T=\left({w}_k/{\sum}_{k=1}^n{w}_k\right)\ {c}_kd, $$

where T is the average annual cost of injuries per firefighter ($13,593), w_k is the number of injuries per year of type k, n is the number of injury types, c_k is the weighted average cost of injury of type k and d is the CPI multiplier escalating costs from 2000 to 2019 (d = 1.52).

The average annual medical cost of injuries per wildland firefighter ($13,593) is a key cost estimate that will be used for benefit estimation in the next section. In addition to medical costs, there are also productivity losses associated with injuries. Finkelstein et al. (2006) do not provide productivity losses for all types of injury, but they show a range of productivity losses from $1255 (emergency room) to $5983 (hospitalization) for all types of injuries (year 2000, males only, inflated to $1908 and $9094, respectively, in 2019). Using the same weighted average (46 % hospitalization), we will use the $5213 as the base cost due to loss of productivity per firefighter injury.

3.2.3 Baseline cost of firefighter garments

We obtained cost estimates for the legacy systems by evaluating the costs of their components (pants, jackets, etc.). The Wildland Firefighter Personal Protective Equipment (WLFF PPE) Selection Guide (DHS S&T, 2014a, b) was used to estimate the cost of the legacy system, and the APPS, WLFF PPE Clothing System Program, Final Report was used to estimate the cost of the advanced system (DHS S&T, 2014a, b).

After reviewing vendor sites, we estimated that the legacy system cost $947 (DHS S&T, 2014a, b; Swafford, 2017; CX Urban Interface Fire Coat, 2019; CX Urban Interface Vent Pants, 2019; Ethos Wildland Fire Pant, 2019; Flamestretch® Sock System, 2019; FR Phase 1 Ribbed Boxer Brief, 2019; CALPIA Store. 2019a; Kenyon Consumer Products, 2019; Parrish, 2019; Vector Wildland Fire Pant, 2019). Table 6 displays the component cost breakdown of the legacy system from various vendors.

Table 6 Legacy system cost data.

3.3 APPS cost analysis

We obtained cost estimates for the advanced system from several vendors and estimated a low-end average of $1259 and a high-end of $1349 (DHS S&T, 2014a, b; Swafford, 2017; Coats, 2019; Flamestretch^® Sock System, 2019; FR Phase 1 Advanced Cooling T-Shirt, 2019; GEN II Pants, 2019; Interface Pant, 2019; Men’s Station Wear Base Layer, 2019; 2 Pack of GI Sand 50/50 Boxers, 2019). Table 7 displays the component cost breakdown of the APPS system from various vendors.

Table 7 APPS cost data.

We are primarily interested in the difference in price between the legacy garments and the advanced garments, which ranges from $234.57 to $324.57 (see Table 8). For the base case analysis, we used the average of these two differences, or $279.

Table 8 Legacy vs. APPS garment cost differences.

To determine the baseline cost of the legacy garment and the cost difference between the legacy garments and the APPS, we need the annual sales between 2015 and 2019. Unfortunately, the vendors did not provide us with access to actual sales numbers, which they considered this proprietary information. Instead, we used a market penetration model with a sales estimate of 5 % of the wildland firefighter population per year to determine how many firefighters would use the APPS garments. Considering the increased comfort and reduced risks that the APPS provided, this may be considered a low number. However, this is counterbalanced by the increased cost of the APPS (about 30 %). To account for the uncertainty in market share, we used a relatively large range from 2.5 to 10 % in the sensitivity analysis.

The market penetration analysis assumed a 5 % per year market penetration for 106,543 wildland firefighters for the first 5 years (2015–2019). After this, the garments purchased in 2015 would need to be replaced and the cycle begins anew. We ignored additional garment purchases after the first 5 years because there likely will be new products and developments after this initial cycle. With these assumptions, there will be a total of 24,107 purchases in the first 5 years, thus costing $6,724,474 more (in 2019 dollars) than the legacy PPE system (see Table 9).

Table 9 Total 5 year cost differences between legacy and APPS garments.

The formula to calculate the total cost of $6,724,474 is based on the market penetration of the APPS, which determines the sales in year t. If the potential market S is the number of wildland firefighters at the beginning of 2015 (year 1 of sales) and m is the market penetration rate, then in year 1, the sales of the new garment will be mS. In subsequent years the sales will be

(2) $$ {S}_t=\left(S-{\sum}_{j=1}^{t-1}{S}_j\right)\hskip0.5em m, $$

where S_t is the sales in year t > 1, S_j is the share of the market already supplied with the APPS in prior years j < t, and m is the per cent of market share applied to the remaining sales volume each year. The cost difference in Table 9 is based on deflating the 2019 cost difference ($279) to the years prior to 2019 using the CPI deflator.

In addition to the increased purchase cost of the APPS garment, DHS S&T also had to cover the costs of the project (i.e., the development of the advanced garments), which lasted from April 2011 to December 2013. We were unable to ascertain the precise costs of this project, so we used the median cost of several projects funded by S&T during the same time period, which was about $500,000. Inflating past values using the CPI as discussed earlier, we estimated the equivalent 2019 cost to be $557,097 (see Table 10). Therefore, the total cost of the project (including implementation for 5 years) is estimated at $7,281,571.

Table 10 S&T project cost.

3.4 Benefit analysis

The main benefit of the advanced garments considered in this BCA is the reduction of fatalities and injuries. According to the U.S. Army Natick Center (2014), the new garments lower the heat absorption coefficients by about 10 %. As the first cut, we apply this 10 % heat reduction coefficient to a 10 % reduction in fatalities and injuries. It is possible that a marginal reduction in the heat absorption coefficient may not affect fatalities if those are driven by heat factors significantly beyond a threshold, nor might the heat reduction factor affect respiratory or other specific causes of mortality. Consequently, we also carry out sensitivity analysis where only injuries but not fatalities are avoided.

Table 11 shows the sales estimates for the first 5 years after the APPS were introduced. APPS sales in a given year t were calculated as in Equation (2). The rest of the calculation can be expressed by

(3) $$ {F}_t=5{S}_t{f}_t\left( VSL/{d}_t\right)p, $$

Table 11 Market penetration, deaths, costs, and cost reduction with APPS.

where F_t is the PV of fatality cost reduction in year t, S_t is the number of sales in year t (from Equation (2) and Table 9), f_t is the fatality rate in year t, d_t is the CPI deflator from 2019 to year 2019–t (t = 0–4) and p is the percentage of fatality reductions.

Over the 5 years, the total number of firefighters who will wear the APPS garments is 24,102, or 23 % of all wildland firefighters (second column of Table 11). Applying the 8.68 × 10⁻⁰⁵ annual individual fatality rate results in the total number of expected fatalities for each year (the third column of Table 11). The fourth column of Table 11 shows the VSL estimates, based on a VSL of $10,000,000 in 2019 and deflating by the CPI back to 2015. Column five shows the total nominal cost of the fatalities by year.

Due to the 10 % reduction of fatality risks when wearing the APPS, we can determine the reduced expected costs, which are shown in the sixth (nominal) and seventh (PV) columns of Table 11. Thus, the NPV of the benefits of the APPS garments in reducing fatality risks is $10,462,631 million.

For injuries, we use the baseline injury cost per person per year of $13,593. Using the total annual costs of injuries, a 5 % market penetration rate of the APPS garments, a useful life of 5 years for the APPS, and a 10 % reduction in injury risks, we arrived at a total benefit (reduction of injury cost) of $7,518,696 (see Table 12). The calculations are the same as in Equation (3).

Table 12 Market penetration, injuries, injury cost, and cost reduction with APPS.

Table 13 shows a similar analysis for the loss of productivity, with a base case loss of productivity of $5213 (see baseline analysis). The calculations are the same as in Equation (3).

Table 13 Market penetration, loss of productivity, and cost reduction with APPS.

Combining the fatality risk reduction, the injury risk reduction, and the productivity loss reduction, we arrive at a total PV of the benefits of $20,864,858. Considering the reduction of medical and productivity losses of injuries only, the NPV is $13,583,287.

Table 14 shows the yearly costs and benefits from 2011 to 2019. Note that there is a gap year in 2014. All S&T costs ended on 31 December 2013, but the final product was not delivered until July 2014. Therefore, we assume that production and sales did not start until January of 2015.

Table 14 Yearly costs and benefits.

Table 15 shows both the inputs (upper part) and the outputs of the benefit–cost calculations.

Table 15 Inputs and outputs of the BCA.

In the analysis to this point, we used a de-facto zero discount rate, since during the time period of consideration (2011–2019) the opportunity cost of government investments was close to zero. Since this is a retrospective BCA, we knew the relevant inflation rates and cost of government borrowing (see also Farrow & von Winterfeldt, 2020). Reviewers of this paper suggested that we take the perspective of a decision-maker in 2011, who does not know these numbers and thus uses a discount rate to capture the uncertain future value of money. We conducted this analysis using a 3 and 7 % discount rate. The results are shown in the last two columns of Table 16, with an NPV (in 2011 Dollars) of $10,025,519 for a 3 % discount rate and an NPV (in 2011 Dollars) of $9,357,769 for a 7 % discount rate. These NPV are 84 and 68 % of the undiscounted NPV, thus still showing a significant BCR and ROI.

Table 16 NPV calculations from a 2011 perspective with discounting.

3.5 Net benefits, BCR, and return on investment

Using the estimates of costs ($7,281,571) and benefits ($20,864,858) in Table 15, we determine that the net benefits (NPV of Benefits-NPV of Costs) are $13,583,287 in the base case. The BCR is 2.87 and the return on investment is 187 % over 5 years. If we ignore the reduction of fatality risks and calculate the benefits only for the reduction of injury risks, the total benefits are $10,839,654 the net benefits are $3,749,881 the BCR is 1.51, and the ROI is 51 %. Although we here report individual estimates to the dollar, the following sensitivity and uncertainty analyses make clear that our estimates can vary by tens of millions of dollars depending on the uncertain parameter values.

3.6 Sensitivity and uncertainty analysis

3.6.1 Break-even analysis

Two key uncertain parameters are the reduction of injury and fatality risks due to the APPS and the market penetration rate. The break-even point (NPV = $0) is reached when the reduction of fatality and injury risks is about 3.42 % or when the market penetration rate is 0.18 %, which corresponds to the sale of 985 APPD garments. To make up the initial R&D investment of $500,000 ($565,000 in PV), 1917 APPS garments would have to be sold. These break-even values are at the very low end of plausible estimates and thus, the break-even analysis suggests that there is a high likelihood of a positive NPV.

3.6.2 Sensitivity analysis

A tornado analysis (see Clemen & Reilly, 2014) of the input variables that influence the NPV, BCR, and ROI outputs are:

1. (i) S&T project cost

2. (ii) Cost difference between APPS and legacy garments

3. (iii) Annual fatality risk (garment-related)

4. (iv) Annual injury risk (garment-related)

5. (v) Reduction of fatality and injury risks

6. (vi) Cost of a fatality

7. (vii) Cost of an injury

8. (viii) Market penetration rate/year

The ranges for these input variables are shown in Table 17 for each variable. WE used the low-end and high-end data for the APPS cost difference and cost of injury (medical and productivity) as described in previous sections. For other variables, we used reasonable bounds.

Table 17 Ranges of input variables for the BCA.

The resulting tornado diagram is shown in Figure 1, suggesting that the most important input variables are (in order): the reduction of injury and fatality risks when using the APPS, the APPS market penetration rate, and the annual fatality and injury risks. Interestingly, the S&T project costs do not have a large impact within the range of plausible costs because the total costs are dominated by the cost differential between the APPS garments and the legacy garments.

Figure 1 Tornado analysis of input variables to the benefits model.

Since the reduction of the fatality and injury risk and the market penetration variables are the most important ones, we conducted a two-way sensitivity analysis of the NPV as a function of these variables (see Table 18). The NPV at the base case values of 5 % market penetration and 10 % risk reduction is 13,583,452. For a very low value of risk reduction of 1 % the NPV is negative, but for most reasonable values it is positive at 5 % or above.

Table 18 Sensitivity analysis of NPV to market penetration and risk reduction.

3.6.3 Probabilistic simulation

For reasons described in the methodology section of this paper, we assigned triangular distributions to the input parameters shown in Table 16. The triangular distribution is defined by a minimum, modal, and maximum value. We assigned the minimum of the triangular distribution to the low case of the tornado analysis, the mode to the base case, and the maximum to the high case. Using these input distributions, a probabilistic simulation showed a large variability in NPVs, ranging from the 5th percentile of $6,410,421 to a median of $19,266,518 to the 95th percentile of $43,662,591 (see Table 19 and Figure 2).

Table 19 Simulation statistics of the NPV.

Figure 2 Probability distribution over NPV.

This is a very large range, primarily due to the uncertainty about the fatality and injury risks, the reduction of these risks, and the market penetration of the APPS.

3.7 Assumptions and limitations

This BCA showed a positive NPV, as well as a reasonably high BCR and ROI, as a result of the APPS project and the resulting increased use of APPS. When using both fatality and injury risk reductions, the NPV is positive for all plausible input parameters. When using only injuries, the NPV, BCR, and ROI estimates are reduced substantially, and, with some parameters, the NPV is negative due to the cost difference between the APPS and the legacy garments.

The reduction of fatality and injury risks (10 % in the base case, ranging from 5 to 20 %) is the most important variable according to the tornado analysis. Estimating the reduction of these risks due to the APPS is a very challenging task. In this analysis, we assumed that the reduction in risk is proportional to the reduction of the heat absorption coefficient (10 %). To obtain better estimates of the reduction of fatality and injury risks, additional data and further statistical analysis of these risks with and without APPS would be useful, but the data are currently not available. In addition, an expert elicitation workshop that focuses on the relationship between the heat absorption improvement and the reduction in fatality and injury risks could be conducted.

The second most important variable is the market penetration rate. We were unable to obtain estimates of actual sales due to proprietary data concerns by the vendors. Instead, we used a market penetration model with a fairly low market penetration rate. This low rate was due to balancing the perceived benefits of the APPS (reduced risk, increased comfort) against a 30 % increase in cost.

One cost estimate that is usually readily available in the project cost, which was incurred by the S&T Directorate of DHS. Due to changing leadership at the project management level, we were unable to ascertain these costs and had to make rough estimates based on the costs of similar FRG projects. Fortunately, even a large range of costs (from $250,000 to $1 million) played only a minor role in determining the NPV, BCR, and ROI.

An interesting by-product of this analysis was that the BCR and the ROI estimates were quite stable, even with large swings of the NPV. This occurred because with increased benefits, there also is an increase in cost, particularly when more APPS are sold.

4. Conclusions

This BCA, like the other 24 conducted by the CREATE team, demonstrated that a BCA for homeland security R&D projects can be conducted and produce meaningful results. Key to conducting these BCAs was the inclusion of risk reduction as a measure of benefits, and a sensitivity and uncertainty analysis of the inputs into the BCA to observe changes in BCA metrics. Without the sensitivity and uncertainty analyses, these BCAs would likely be criticized for making precise assumptions that cannot be defended.

We learned several lessons carrying out the full set of analyses:

1. (i) BCAs can be performed both ex ante and ex post, as long as the uncertainty about the inputs of the analyses can be taken into consideration to properly reflect the uncertainty in the outputs.

2. (ii) Projects and products which have not been transitioned yet are the hardest to analyze, and they leave the largest uncertainties in BCA metrics. In contrast, projects which have been transitioned are easier to evaluate, and those that have resulted in commercial products are the easiest and least uncertain.

3. (iii) Costs are usually fairly easy to determine, but benefits often require the development of specific risk models, for example, the benefit of reducing risks of fatalities or injuries, or providing deterrence and increasing the likelihood of interruptions of attacks. These risk models vary from case to case.

4. (iv) Discounting in retrospective BCAs is a controversial issue. Government recommendation for discounting future costs and benefits are for a range from 3 to 7 %. In retrospective BCAs we know the cost of government borrowing, which we used as a lower bound for discounting in this case analysis. As an alternative, we took the perspective of a past decision maker, made adjustments based on known inflation rates and discounted at 3 and 7 %. Compared to the 2011 NPV without discounting ($11,993,581) the retrospective analysis yielded lower NPVs by 84 and 78 % respectively. While lower, these NPVS are still substantial.

The BCA for the Wildland Firefighter APPS was an atypical case where project cost estimates were hard to estimate because of lost project data combined with changing project managers. While this was a rare occasion, it points to the importance of recordkeeping of project costs. In addition, this BCA showed pathways of improving benefit–cost estimates by obtaining data from vendors (to replace the market penetration model) and expert judgments from risk modeling specialists (to improve the estimates of fatality and risk reduction). A better involvement of market experts and of subject matter experts would have improved the BCA substantially.