Some Pitfalls of Practical Benefit-Cost Analysis

Abstract

Practitioners of benefit-cost analysis face many difficulties. Despite the best training, guidance, and intentions, practitioners can stumble: actual benefit-cost analysis is hard and mistakes get made. Over the years, I have collected the mistakes I have seen in actual benefit-cost analyses. Many of the same mistakes occur over and over: the pitfalls of practical benefit-cost analysis. In this paper, I describe common pitfalls and suggest ways to avoid them.

1 Introduction

I have spent more than 23 years doing and reviewing benefit-cost analyses of public policies. During that time I have worked on hundreds of analyses and seen just about everything that can happen to an analysis. I have learned – the hard way – that despite the best training, guidance, and intentions, practitioners can still stumble: practical benefit-cost analysis is hard and mistakes get made. Over the years I have collected mistakes made in actual benefit-cost analyses and kept them in an informal file. Many of the same mistakes occur over and over, which has led me to sort them into buckets by the type of mistake; these buckets are the pitfalls that practitioners fall into, even as they attempt to apply the best methods and practices to estimate benefits and costs. And I have learned to look for these pitfalls when reviewing a benefit-cost analysis, because they can slide unnoticed into an analysis.

The problems embodied in the pitfalls are not obscure; following best practices in benefit-cost analysis should in principle prevent these problems. ¹ In actual benefit-cost analysis, however, difficulties arise and analysts often trip up, succumbing to pitfalls that bollix up an analysis. In this paper, I describe some common pitfalls and suggest ways to avoid them in practical work. ²

2 Pitfalls

2.1 Kansas is not Oz: the not-so-obvious problem of identifying market failure

Policy analysis is economics applied to the real world, not to an idealized or stylized world: Kansas, not Oz. Forgetting the difference between real and ideal can confuse analysts who have to match theoretical constructs with real-world counterparts. The benefit-cost version of this widespread problem occurs when the economist identifies the market, government, or behavioral failure that makes a policy potentially effective. If a failure – something broken – is not found or is incorrectly identified, the estimated benefits and costs of fixing it could well be incorrect. Market failures fit into the traditional categories: imperfect markets; public goods; externalities; and information failures, especially information asymmetry. Government failures include rent seeking, bureaucratic inefficiency, and rational ignorance. Another class of failure that economists now include is intrapersonal or behavioral failure, including present bias, lack of self-control, projection bias, time inconsistency, addiction, and other forms of inconsistent behavior. Behavioral failures establish the potential for welfare-increasing public policies in much the same way as traditional sources of market failure.

The Kansas–Oz pitfall arises not because we cannot find a failure but because we can always find a market, government or behavioral failure. In L. Frank Baum’s Wizard of Oz, a twister carries Dorothy to Oz, a land where nothing works or looks like Kansas; she eventually returns to Kansas and the real world, knowing that Kansas will not be like Oz. In contrast to Dorothy, economists who leave the Oz of theory for the Kansas of reality often forget that things are different in Kansas, where monkeys do not fly, scarecrows do not talk, and consumers’ decision functions may not fully and instantly incorporate all relevant information. Economists tend to assess Kansas by the standards of a theoretical Oz, and because Kansas is not like Oz, they conclude there is something wrong with Kansas.

Market, government, and behavioral failures are easy to find because markets and people are imperfect; public goods are everywhere, with many ostensibly private goods having some degree of publicness; negative and positive externalities are ubiquitous; and information is always and everywhere imperfect. Rent seeking and rational ignorance are also widespread. Real human beings are seldom perfectly rational; most go through life making one behavioral mistake after another.

The analyst should not assume a world of perfect information, perfect competition, selfless government, and perfect behavior and then be shocked to find that Kansas falls well short of all these perfections. Yet, many policy analyses make this mistake. A recent (and fairly typical) benefit-cost analysis that I reviewed stated:

“If all market participants had perfect information and maximized utility, welfare would be maximized but in this case consumers do not have perfect information. Because consumers are unaware of the risk associated with this product, mandatory quality standards will enhance welfare”.

But virtually all information is imperfect and asymmetric. Asymmetric information is not necessarily evidence of market failure; it is a basic characteristic of markets. If we find incomplete information with market participants and asymmetric information, all we have shown is that we are in Kansas, not Oz. ³

The Kansas–Oz pitfall can be avoided. Asking several questions can help: What is the problem the policy is trying to fix and how large is it? Size matters; we know that the perfection of Oz cannot be found but is the inefficiency creating problems large enough to justify the costs of intervention? Are we starting with an anecdote based on a one-off event or is the problem large and continual? In some cases it may not be necessary to identify a specific market failure; it may be sufficient to determine that real-world transactions costs prevent actions that would maximize welfare. Zerbe and McCurdy (1999) suggest a transaction costs approach that asks what costs prevent a private solution and how the policy intervention reduces those costs.

For most benefit-cost analyses, then, the way to avoid the Kansas–Oz pitfall is to stay in Kansas. The economist should find a real failure with substantial consequences: a specific market, government, behavioral failure, or transaction cost that prevents fixing what is broken.

2.2 Analysis interruptus: the perils of incomplete analyses

When we do a benefit-cost analysis of a policy intervention, we add up all the costs and add up all the benefits and then compare the two. For a proposed action or forthcoming policy, both the benefits and costs must be forecast. Forecasts are always a challenge but the economics profession has a long history of turning current and past data into estimates of future effects. With good data readily available, we can make a decent forecast of the benefits and costs of the proposed action.

Much of the time, however, we lack data or other evidence to estimate all of the likely effects. In those situations, we may use partial measures. In the best of times, what the partial measures exclude (because we cannot estimate them) will be small relative to what we include. If the exclusions are larger and more significant, they may nonetheless produce a good estimate of benefits and costs if the missing data likely introduce a small bias. We can also use partial estimates when, although the missing data and estimates might be large relative what we can estimate, we know the direction of the bias.

These instances of partial analyses present difficulties but not serious problems. A pitfall arises with partial measures that cannot necessarily be used to infer the whole. In estimating the effects of public health measures, for example, medical costs can be a dangerous and ambiguous measure. Intuition tells us that a decline in medical costs is a good thing (a benefit) and a rise in medical costs is a bad thing (a cost), but that is not necessarily so. ⁴

Laws requiring motorcycle and bicycle riders to wear helmets can serve as an example of a policy that generates an ambiguous measure. Helmet laws reduce the likelihood of death from accidents but because nonfatal accidents lead to higher medical costs than fatalities, mandatory helmet laws increase medical costs. The increased medical costs alone, if interpreted as a bad thing, give us a partial measure that conveys misleading information about the net benefits of the law.

Avoiding the pitfall of analysis interruptus can be challenging. For a potentially ambiguous partial measure, such as a change in medical costs, we should identify and describe the market failure as a way to show that the partial measure is consistent with fixing the market failure. We use a qualitative analysis to correctly interpret the partial measure. The best way to avoid this pitfall, then, is to find something that can be measured and interpreted within the framework of the identified market failure and associated qualitative analysis.

One fix to partial analyses that should be avoided is filling in all missing parts of the analysis with stories, the next pitfall.

2.3 Twice-told tales: a wholly qualitative analysis (almost) always ends badly

When unable to find or generate any evidence on the effects of a policy, analysts often substitute a wholly qualitative analysis, twice-told tales. Qualitative descriptions of the benefits and costs belong in an analysis: a story should go with the numbers; the story explains why the evidence represents the benefits and costs of the policy. But when all of the benefits (and perhaps costs as well) are qualitative, we encounter a major pitfall.

One reason for the pitfall is that as much as we try to keep the description of costs or benefits qualitative, an implicit quantitative conclusion often takes control of the story. A qualitative analysis, for example, may imply that although we cannot estimate benefits, we are sure that they are larger than costs, or vice versa. Stating or implying that benefits exceed costs or that costs exceed benefits becomes a crude quantitative estimate of benefits and costs because any comparison of two magnitudes is inherently quantitative, even if we do not actually measure the magnitudes. If we say that benefits are larger than costs, we are making a quantitative assessment. I saw an example of this practice in a recent analysis. The analyst estimated costs to be $75 million. In addition, the analyst concluded that the rule was not economically significant (that is, the effects were under $100 million) and that benefits exceeded costs. Combining these conclusions meant that the analysis in effect stated that the rule’s benefits were somewhere between $75 million and $100 million, a fairly precise estimate as these things go.

Break-even calculations are a particularly risky variant of qualitative analysis because they incorporate some quantification. If we have estimated costs but not benefits, we can nonetheless calculate how large the effects of the policy would have to be for benefits to equal costs. When the break-even calculation is coupled with a qualitative description of benefits, the qualitative description may give the impression that we know whether net benefits will be positive or negative. Unless we keep reminding readers that, despite our story, all we can really say is that net benefits will be positive or negative, they will reach conclusions about the relative sizes of benefits and costs. Indeed, if their prior beliefs are strong enough, readers will reach conclusions about the relative sizes of benefits and costs even with continual reminders that we have no estimates.

The best defense against telling tales is to use methods, partial measures, models, or proxies that will allow estimates of costs and benefits. Suppose, for example, a regulation reduces exposure to a contaminant known to be dangerous but with no validated estimates of the relationship between exposure and harm (that is, no dose–response estimates at baseline exposures). The estimated reduction in exposures can serve as a measure of the effectiveness of the regulation, allowing the substitution of a cost-effectiveness measure for a cost-benefit measure. ⁵

If no estimates can be generated, the qualitative assessment should make the market failure the centerpiece and not go beyond describing in neutral terms what the benefits or costs would be, with the descriptions closely tied to the market or other failure that motivates the policy.

2.4 Can we get there from here? Causation present and accounted for – or not

Identifying causation causes much grief for practical benefit-cost analysts. In quantifying benefits and costs, we have to ensure that links in the analysis are links in the real world. The failure to link the causes of the problem to the effects of the regulation is a major pitfall. Real-world problems often have multiple causes, so the proposed fixes may sound appropriate and effective. If the problem involves health, causation is particularly problematic. How much of a given cancer is indeed caused by the failure we have identified?

Why would a policy not work? The policy might be aimed at a problem that is not actually addressed by the policy. For example, if adverse health events arise because a particular product is misused by consumers, testing for product quality will not help. Conversely, a warning label will not prevent harms caused by design or production failures inherent in the product. For harms associated with product use, the question must be whether the harms are generated by a defective product or the behavior of users. Bad outcomes can arise through behavior or other factors not addressed by the policy lever.

In one benefit-cost analysis I worked on, this pitfall arose with a regulation aimed at reducing the harm from a product whose consumption was associated with serious adverse health events. The regulation established testing and production controls for manufacturing the product; the analysis estimated benefits as the value of reduced adverse events. The estimated benefits, however, did not come from evidence linking poor production practices to the adverse events. Subsequent analyses revealed that accidental and deliberate consumer misuse of the product – not manufacturing problems – caused the adverse health events.

To avoid this pitfall, the analyst should ask: How will the policy change behavior and change the environmental, health or safety problem? And how do we know? The benefits and costs of a policy response are the reaction to a single event and are inextricably linked. If the causal mechanisms are specifically laid out, the relationship between costs and benefits will be clear.

2.5 Missed opportunities: social costs can hide in plain sight and transfers are not social costs

Economists grow up with the notion that social costs are opportunity costs, the value foregone when resources are allocated to one use rather than another. In practice, however, most data on costs come from sources that either do not make the distinction between opportunity costs and transfers or fail to include opportunity costs that do not involve monetary transfers. Some of the biggest mistakes in benefit-cost analysis arise from missing these opportunity costs or failing to recognize transfers.

A few years ago, I worked on an analysis of a ban on an ingredient used to produce a common consumer product. The ingredient had been identified as a possible carcinogen: a ban would reduce exposure but would require substitute ingredients and changes in production processes. I estimated the benefits as the value of the risk reduction and compared them with the increased costs of producing the good and found them to be about equal. In that analysis, however, I neglected to include all costs because I failed to account for product quality. The products made with the alternative ingredients and methods were substantially lower in quality, which would have imposed additional costs on consumers. If the costs of the reduction in quality had been included, the measure’s costs would have substantially exceeded benefits.

In other cases, dollars are not opportunity costs. The pitfall here is the confusion of transfers with opportunity costs. For example, business estimates of the costs of a statute or regulation often include expected revenue loss as a cost; indeed, sometimes lost revenue is the largest cost from the standpoint of the business but it is a transfer, not a social opportunity cost.

In an analysis of a regulation that led to changes in product regulatory categories in an industry, we found that because firms in the industry paid license fees based on product categories, the principal effect of the regulation was to change fees. Moreover, the change in fees accounted for the largest effects of the policy. Our cost estimates mistakenly included the increased fees, which were transfers, as social costs.

To avoid this pitfall, we should identify the new activities that will be undertaken or the existing activities that will cease as a result of the policy and estimate the resource costs of those activities; the resource burden must be assessed without regard to nominal monetary effects or finance.

2.6 Who’s on first and what’s on second? Baselines can be slippery

We estimate benefits and costs relative to a baseline. The estimated benefits and costs of a policy intervention compare the world without the policy intervention to the world with the intervention. The comparison is not before-and-after but a counterfactual with-and-without. The need to use the counterfactual approach to baselines creates difficulties that a simple before-and-after calculation would not face.

One long-running controversy in benefit-cost analysis involves whether regulatory agencies typically overestimate or underestimate costs in regulatory cost-benefit analyses. ⁶ One potential source of over- or underestimation results from changes that affect the baseline, which will undercut an analysis that assumes current production and cost functions are fixed. In fact, production and cost functions change continuously and not accounting for those movements can lead the analyst to overestimate or underestimate costs.

Moving baselines can affect the benefits as well as costs. If a regulation reduces the risk of an adverse health or safety event but that risk is already declining, the benefits of the intervention could be overstated. Conversely, if risk is growing, the benefits of the intervention could be understated.

In estimating costs over time, we should look at likely changes in overall costs of production. For some procedures, particularly those with a large service component, costs could rise over time. Far more common, however, are situations where we should expect costs to fall over time. Costs of production, holding quality constant, tend to fall over time in progressive manufacturing and service industries. If we include a cost for electronic computing that lasts for several years, we should not assume that the real costs will stay the same.

Another common baseline problem is difficulty separating the stationary and moving parts. Cumulative effects should be considered in the baseline. The incremental effect of another regulation on a highly regulated sector may be quite different than an incremental regulation on a lightly regulated sector.

Some years ago I reviewed an analysis that can serve as an example of the perils of a moving baseline: a benefit-cost analysis of a policy to require production controls to reduce a potentially fatal risk associated with certain processed products. Although the analysts estimated that the processing controls would lead to a substantial reduction in risk, they neglected to account for recent changes at the raw material (preprocessing) stage that had largely eliminated the risk. The policy-generated additional risk reduction under the new baseline proved to be negligible, with a value far less than the cost. The baseline changed because private actions and other regulations mostly eliminated the contaminant before processing.

Baselines, then, should in most cases contain dynamic elements. The best way to avoid the pitfall is to consider how the baseline has been changing, and how it is likely to change.

2.7 The forgotten little people: the dangerous narrows between neglecting behavioral failures and paternalistic analysis

The welfare implications of benefit-cost analysis flow from the belief that the choices people and firms make follow from optimizing behavior subject to constraints. Analysts can therefore use those choices as the basis for measures of subjective well-being. Behavioral economics (or psychology and economics) has shown that behavioral anomalies may mean that people’s choices may not maximize well-being. As Thaler (2000) puts it, Homo sapiens may not behave like Homo economicus. In assessing benefits and costs of policy interventions, the analyst needs to remember that policies affect real people, who may or may not be optimizing.

Behavioral economics has identified many behavioral anomalies with implications for practical benefit-cost analysis (Weimer, 2017). As described under the first pitfall, behavioral failures must be considered along with traditional market failures in order to identify the problem a policy aims to fix. ⁷ Indeed, much of the behavioral literature has been about policy nudges, which use behavioral anomalies to influence people to make better decisions in the sense that their welfare improves. The challenge for benefit-cost analyses is to estimate the gains and losses from nudges and other policies aimed to correct behavioral failures.

The pitfall here is two-sided: (1) not seeing a behavioral failure, and (2) seeing a behavioral failure that is not really there. ⁸ In the first case, the analyst fails to recognize that net welfare can be improved by a policy that does not accept preferences revealed through individual choices as necessarily optimal. In the second case, the analyst fails to see that apparently anomalous behavior is in fact optimal.

Benefit-cost assessments of environmental, health, and safety policies are prone to this two-sided pitfall. Many behavioral anomalies lead to outcomes that reduce health and well-being. Failure to recognize that observed negative health effects are caused by behavioral anomalies would lead the analyst to not include health gains in a benefit-cost analysis of a regulation reducing the consumption of some product. For example, a person might have a revealed preference for recreational drugs and yet would prefer not to use them. The person may use drugs because of a behavioral failure leading to a lack of self-control when an opportunity arises to purchase and use drugs. A policy to make the recreational drugs difficult to purchase would increase the person’s welfare, despite the revealed preference for the drugs. ⁹

The other side of the pitfall is for the analyst to assume that any behavior that reduces health or safety must reduce welfare, so that any improvement in health or safety is a benefit with no offsets. But some behaviors that increase individual and social well-being do not increase health (and may even reduce health). As Cawley (2011, p. 123) puts it: in many instances “people are sometimes willing to accept worse health in exchange for other things they value (such as money or pleasure)”. Many years ago, I reviewed a benefit-cost analysis of a policy to discourage consumption of certain unhealthy foods. The analysis measured the benefits of the policy as the value of the health improvements associated with reduced consumption of the “bad” foods. The analysis assumed that behavioral failures alone explained the consumption of those goods, ignoring that people probably chose those foods because they liked them.

The two faces of this pitfall complicate practical benefit-cost analysis. The best one can do is to be humble, be aware of behavioral economics and its implications for a behavioral welfare economics, and remember that the analysis is about real people.

The final pitfall in a sense encompasses all of the others and more: ancestor worship.

2.8 Ancestor worship: mistakes were made

Ancestor worship means the uncritical reliance on previous benefit-cost analyses. An existing analysis that seems ready-made for re-use is always tempting and appallingly easy in this digital era; an economist doing a benefit-cost analysis need only go through the files and find an earlier assessment of a similar problem. If a previous analysis dealt with the same (or a similar) policy problem, why not (with a few appropriate modifications) use it again? For one reason, uncritically using a preceding analysis can lead to problems if the old analysis embodies one or more of the pitfalls already described. The previous pitfalls, as well as many others lurking out there, tell against re-using a benefit-cost analysis or extracting relevant parts of it for current use. Moreover, the fact that a benefit-cost analysis survived critical scrutiny the first time around does not mean that it was perfect or even good. The reviews of the analyses by insiders and outsiders may have missed flaws or even failed to spot a critical mistake. ¹⁰ Indeed, the mistakes may have cascaded throughout the review chain and public comments. My pitfalls files contain many examples from analyses that successfully made it through all stages of review.

In addition to potential flaws in old analyses, another reason to avoid ancestor worship is that methods and measures change, sometimes as part of the normal course of events and sometimes because we get better at benefit-cost analysis. Ancestor worship can lead to mistakes as simple as neglecting to update costs to take account of changes in technology or industry structure. Missed changes in baseline are another pitfall often caused by ancestor worship. If the analyst fails to perceive that benefit-cost times are changing, he or she might well continue to use an analysis based on an industrial structure that no longer exists or a labeling convention that has disappeared.

Having an existing benefit-cost analysis for a policy can be a blessing but the older it is, the more likely it is not good enough. If we cannot resist the temptation to re-use old analyses, we can at least impose a personal statute of limitations. Economists should not use any analysis older than seven years, except as a starting point for a wholly new analysis. Older analyses are always wrong because we know more now and because the world has changed. Let the dead analyses rest in peace.

Final thoughts

This essay considers many difficulties of practical benefit-cost analyses, mostly on-the-ground difficulties, not elevated philosophical and theoretical questions. The nature of applied welfare economics, with its combination of normative and positive economics, makes all its practical measures compromises, with considerable latitude for the judgement of the analyst. The conventions that make applied welfare economics possible embody practical compromises with the dictates of welfare theory.

If we accept benefit-cost analysis as a valid, if imperfect, measure of changes in social welfare, we must ensure it is done correctly and done well. Freeing applied welfare economics of the straightjacket of pure theory makes practical measures possible but also brings the pitfalls I have identified into play. Avoiding the pitfalls requires constant vigilance. But forewarned is forearmed so here’s hoping we can all escape these pitfalls and do benefit-cost analysis correctly and well.

Disclaimer

The findings, interpretations, and conclusions expressed in this article are those of the author in his private capacity, and they do not represent in any way the views of the Food and Drug Administration.