Limits of the Numerical: The Abuses and Uses of Quantification, ed. C. Newfield, A. Alexandrova and S. John. University of Chicago Press, 2022, 317 pages.

This edited volume is a welcome and timely addition to scientific and normative debates over quantification. Quantification, or the numerical representation of the world, is central not just to science but also to politics, the economy and culture. Quantification, for example, enables standardization, which is central to the proper functioning of markets and bureaucracies. And indeed, one of the driving commitments of the volume is a view of quantification as embedded in and shaped by socio-political institutions.

While this claim may sound obvious, it comes with certain substantive commitments about what quantification is, and accompanying methodological commitments about how best to investigate it. The Limits of the Numerical conceptualizes quantification as both a value-laden process and a product. It is, on the one hand, a set of institutionalized processes constituted by actors with certain kinds of role-related powers. And it is also a product that embodies the values of its designers, is interpreted against particular institutional and cultural systems, and influences the world in line with those values. This view of quantification supports a methodology of case-based analysis. The analysis of cases allows the author’s conclusions to be sensitive to the empirically and normatively significant particulars of socio-political institutions. Such analysis is crucial to build a so-called mid-level theory of quantification, or a theory that is sensitive to certain types of contexts.

This volume is particularly timely given recent shifts in those socio-political institutions. Two important changes are the decline of trust in experts and the rise of the scored society. As Chatterjee (Chapter 1) discusses, previous accounts of quantification took it to have a depoliticizing effect, successfully removing issues from the reach of public debate into the hands of trusted experts. But, this depoliticization proved more historically contingent than previous scholars realized. Politicians in countries such as the USA and the UK have deliberately and systematically undermined public trust in bureaucratic expertise. Based on previous theories, one might expect that the use of numbers in politics would decline as issues get put back on the public agenda. But, declining trust in experts has not led to a decline in quantification in politics, as Chatterjee argues; instead, it has led, for example, to a populist rise of crowdsourced numbers.

Quantification, however, has not only changed in politics. Most of us have to navigate what Citron and Pasquale (2014) term the scored society, in which pervasive surveillance technologies and algorithms are used to score individuals, thereby determining their access to important social goods or liability to punishment. One area in which I would have liked to see more discussion in the volume is the impact of surveillance technologies and AI on quantification. For example, considering the impact of modern computing on quantification also helps to bring out the ways in which quantification re-shapes our social relations, another topic that I would like to see covered in this volume (Fourcade and Healey 2013).

Limits of the Numerical answers a pressing need for a deeper investigation into scientific, moral and socio-political questions about quantification raised by our historical moment. The individual contributions have been grouped into three overarching themes. The first theme, discussed above, is the decline of trust in experts and the rise of new political practices of quantification. The second theme is narrative. Narrative is often held up as a contrast or corrective to the abuses of quantification; but, as the authors chart in Chapters 3–5, narrative can introduce further biases and other distortions. The third theme is the relationship between accuracy and the political utility of numbers. The different case studies of Chapters 6–10 provide a nuanced picture of when and why accuracy is morally beneficial, and when accuracy and other values can come apart.

This volume offers rich and thoughtful analyses to learn from and to disagree with. Below, I will draw out important lessons and gaps from the remaining two themes: values in quantification and narrative. The major upshot of the comments below is a call for more explicit theorizing about foundational moral questions in political philosophy about justice and power, as well as more applied questions about how to design organizations and institutions to promote useful quantification.

I’ll start with the relationship between epistemic and non-epistemic values in quantification. Questions about the role of values in quantification are raised early in the volume by the definition of quantification as ‘numerical representation where it did not exist before to describe reality or to affect change’ (11). From the start, the volume adopts a value-laden view of quantification, according to which non-epistemic values are relevant in the building and assessment of numerical models and measures. This value-laden commitment raises two further sets of questions that must be answered in order to build a mid-level theory of quantification.

The first set of questions centres around whether the aim of describing reality and of affecting change each produce different types of numerical objects. Social scientists do not merely study the social world; they also change it, by acting as teachers, policy advisors, public intellectuals and paid consultants. Social science, one might argue, can be used in a way similar to how an engineer uses physics to build bridges, namely, using scientific knowledge to change the world to achieve certain aims (Guala 2007). But here, the authors seem to make a stronger claim, namely, that representation and affecting change can come apart very drastically, perhaps completely. Such a view is in line with work in the philosophy of economics, for example, that distinguishes between representative and performative models. For example, so-called Barnesian performative models are those that cause the world to be more like the model, and may not be evaluable at all, or in large part, in terms of epistemic standards (MacKenzie 2006).

Here, it would be helpful to hear more about whether the authors are committed to, say, a sharp difference between performative and representative models. One reason to doubt this neat divide is given by Alexandrova and Singh’s (Chapter 8) reflection on wellbeing. There, they contrast the aims of the UK’s Office of National Statistics (ONS) to represent the UK’s wellbeing over time with the Origins of Happiness’ proposal to use a single wellbeing metric to assess policy. However, there is not a sharp divide between representation and use: even the ONS, for example, aims to represent wellbeing using a multiplicity of ‘standards that this community itself endorses’ (192). Given the agency’s role and the political context, accurate representation and fidelity to residents’ self-understanding are both important values for the assessment of the goodness of the measure.

Another set of questions is around the relationship between epistemic and non-epistemic values in the creation and use of models. Here, different authors seem to take different views on this question. On my reading, there is a divide between John (Chapter 6) and Alexandrova and Singh (Chapter 8) on the one hand and Badano (Chapter 7) and Lusk (Chapter 9) on the other. John and Alexandrova and Singh seem to adopt positions more aligned with Helen Longino’s (1995, 1996) view that a plurality of values play a variety of roles throughout the scientific process. John (6.1), for example, argues that values help to resolve underdetermination in nutrition science, which is plagued by methodological difficulties in isolating the causal effect of particular changes on individual health. By contrast, Badano and Lusk seem to adopt positions more in line with Heather Douglas’ (2016) view that non-epistemic values play limited roles in science, confined to resolving epistemic uncertainty. Badano (Chapter 7), for example, argues that the National Institute for Health and Care Excellence (NICE), which is responsible for appraising health technologies in the UK, did not have enough evidence to set its cost-effectiveness threshold very precisely. But, Badano argues, moral values, in this case the value of transparency, can determine whether and how to resolve uncertainty by setting a precise threshold. In order to develop a mid-level theory of quantification, one will have to commit to a particular view of the role of values in science.

For any of the authors in the value in science debate, however, moral and political values are central to the scientific process. Thus, the inclusion of these values in a mid-level theory of quantification elicits a need to explicitly theorize about the background political, economic and social institutions in which quantification is used. This point is further strengthened by considering the discussion of narrative in Chapters 3 and 4 and of transparency in Chapter 7.

The case studies of Chapters 3 and 4 reveal two different roles that narrative plays in the maintenance or erosion of quantification practices. The first is a justifying role. Steffan (Chapter 3) charts how audit narratives, or stories that explain how audit leads to improved outcomes, function in higher education assessment. There, narrative frames problems as suitable to be solved by audit, changes and cements power structures, and communicates the value of audit to different stakeholders. The second is a challenging role. Junghans (Chapter 4) charts how pharmaceutical companies use the narratives of patients with rare diseases – for which it is difficult to gather sufficient statistical evidence about the effectiveness of treatments – to undermine ‘quantitatively grounded modes of regulation’ (109) for their own benefit.

Chapters 3 and 4 point to two further lines of investigation that are essential to understanding the institutional interplay between narrative and numbers. The first is the differing roles narrative and numbers can play in the situated deliberation of an agent. Steffen (Chapter 3), for example, discusses the role of narrative in creating relations of accountability through audit. Once numerically driven audit processes are established, one might imagine, numbers would take over from narrative in the generation of solutions to audit-defined problems, through the scoring and rating of individuals (Fourcade and Healey 2013). Research from sociology, however, suggests that agents tasked with using numerical models or bureaucratic rules to make decisions often solicit narratives to establish facts that they take to be relevant to their decision-making, such as facts about moral responsibility (Moss and Tilly 2001; Moulton 2007; Kiviat 2019). Kiviat (2019), for example, has found that hiring professionals that are tasked with using credit reports in hiring solicit narratives from job applicants to infer whether applicants were morally responsible for unpaid debt.

Thus, we seem to have a conflict in the use of narrative: narrative about the relationship of audits to improved outcomes establishes and maintains the practice of audit, but the demand for individualized narratives can undermine the culture and institutional practice of audit. One question here is whether this demand for narrative from individual decision-makers marks a failure to establish a full practice of audit, or whether narrative and numbers will always be necessary complements in decision-makers’ reasoning.

This last point brings us to the second line of investigation, which centres around questions of power, justice and organizational design. Decision-makers often use discretion to solicit narratives, which grounds some of the uses and abuses of narrative touched on in Chapters 2, 3 and 4. Discretionary powers are role-related institutional powers to, for example, select relevant rules, interpret rules, or make decisions based on facts not covered by the rules (Zacka 2017). It can enable abuses of power or biased decision-making, even when decision-makers are responsive to individual narratives. For example, narratives can introduce morally problematic biases because decision-makers assign moral responsibility to similar individuals differently, in light of their own socially inflected beliefs and experiences (Kiviat 2019). Or, as Chapter 4 shows, decision makers can use narrative to abuse power, as decision-makers push for more discretion in order to circumvent rules for their own gain.

Unless decisions are fully automated, discretion seems ineliminable (Zacka 2017). Indeed, this point sits well with the volume’s commitment to the primacy of interpretative frames over numbers. However, what Junghans’ analysis in Chapter 4 points us to is that any theory of quantification needs to explicitly theorize about institutional relations of power and questions of justice. There are a few literatures in philosophy that one might draw from to fill this gap. Work on hermeneutic injustice is helpful to diagnose asymmetric power to shape narratives (Fricker 2007; Kidd et al. 2017). For example, one of the problems noted by Junghans is that pharmaceutical companies are the ones with the power to elicit, label and construct narratives of patient experiences, instead of patients. There are also important moral questions about the aims of decision-makers. One might criticize pharmaceutical companies for pursuing profit alongside or at the expense of justice; to do so, however, would be to take a stance on whether individuals ought to be motivated by reasons of justice in the economy or civil society (see here the debate between Rawls 1971 and Cohen 1997, or work in business ethics such as Heath 2014), or on questions of whether and why the privatization of social goods such as healthcare is problematic (Cordelli 2020). Finally, more needs to be said about how organizations can be designed to promote morally and epistemically desirable exercises of discretion (Zacka 2017).

To reiterate: Theorizing about the use of numbers and narrative to affect change must address questions in political philosophy about power and justice. To close, I will hone in on one such question, about the relation between transparency and quantification. Badano (Chapter 7) argues that the requirement of public justification supports the greater use of quantification, as quantification tends to be more transparent, and thus more justifiable. While I am sceptical that either transparency or the public reasons framework are the best way to cash out decision-makers’ epistemic obligations, the chapter is a valuable addition to a burgeoning literature in political philosophy on transparency and quantification (Maclure 2021; Nguyen 2022; Vredenburgh 2022; Babic and Cohen 2023).

Democracies tend to be more transparent (Meijer 2014), and there is a compelling philosophical tradition that connects democracy and transparency (Waldron 2016). But, I disagree that transparency is the relevant epistemic good that individuals are owed from public decision-makers at the level of particular decisions, policies or processes. And indeed, Badano’s own discussion of NICE shows us why. As he discusses, transparency can lead to a surfeit of information; thus, for Badano, justifications ought to be ‘manageable’ (168). But, transparent justifications can be arbitrary or deliberately misleading, such that they undermine the aim of public justification. For example, because most decisions are justifiable by many reasons, decision-makers can deceive individuals about the motivating reasons for a decision or policy (Nguyen 2022; Babic and Cohen 2023). Thus, we ought to look beyond transparency to more robust requirements such as explanation (Vredenburgh 2022).

This mis-identification of transparency as the epistemic goal arises from Badano’s commitment to public justification as the source of the relevant requirements. Public justification, however, is a fairly minimal constraint on decision-makers, requiring that institutional arrangements be endorsable. Because the account focuses on whether the reasons can be provided, it misses out on the instrumental value of actually being provided with information for individual and collective agency (Vredenburgh 2022).

I want to close by reflecting on where the volume leaves us in terms of its ambition to develop a mid-level theory of quantification. Three core elements of such a theory come out of the discussion above. First, there is the question of what quantification is. Quantification, according to this volume, is both an institutional process and a product. Second, there is the question of how it is evaluated. This volume develops the view that both epistemic and non-epistemic values play a role in quantification. And, cross cutting both of those is the importance of political morality for any theory of quantification.