Mapping a causal scheme of indicators in the COVID-19 crisis

Abstract

The COVID-19 crisis has been accompanied by an extensive use of indicators, such as those related to COVID infections and deaths, but also a good number of COVID policy indicators. This paper discusses these indicators from the perspective of a legal scholar with an interest (and some expertise) in comparative law and empirical legal studies. This means that this paper does not engage in the details concerning epidemiological and medical issues of COVID infections and deaths. Rather, it focuses on two main issues: first, it develops and maps a general causal scheme of indicators and their underlying real-world phenomena in the COVID crisis; second, it shows how such a causal scheme has been, and can be, applied in comparative empirical legal research. Yet, it also notes the difficulties of proving causal relationships and some attempts to overcome them.

1 Introduction

Many academic disciplines aim to identify causal relationships. This is the case for the natural sciences, but frequently also the social sciences. It has, for example, been said that ethnography has the ‘ability to uncover causal mechanisms and trace processes’ (Abend et al., 2013, p. 606). Yet, more commonly, it is quantitative research using inferential statistics that intends to provide proof of causal relationships. In this regard, indicators of social phenomena play a key role in providing researchers with the necessary data to be used in econometric research. In addition, indicators can have more direct causal ambition, as some of them have the explicit aim to influence behaviour, which has been called the ‘governance effect of indicators’ (Davis et al., 2012). For example, this is the case for various governance and law-related indicators issued by international organisations such as the World Bank.¹

The COVID-19 crisis has been accompanied by an extensive use of indicators, as also discussed in the other contributions to this Special Issue. The most obvious are those related to the spread of the virus and the corresponding health impact of the pandemic, such as the COVID-19 dashboards published on the websites of the World Health Organization (WHO) and Johns Hopkins University.² Some attempts have also been made to create a ‘pandemic misery index’ that combines both the health effects and the economics effects of COVID-19.³ Furthermore, there is a rich set of indicators on the policies that governments have pursued in the wake of the COVID-19 crisis. General indicators can be found in the Oxford COVID-19 Government Response Tracker (notably, its Stringency Index recording the strictness of lockdown policies),⁴ the COVID-19 Government Measures Dataset by the think-tank ACAPS⁵ and a Public Health and Social Measures (PHSM) Severity Index available on the European dashboard of the WHO.⁶ There are also further sets of specific policy indicators, such as on face-mask requirements,⁷ travel restrictions⁸ and import–export policies.⁹

However, despite the relevance of indicators in the COVID-19 crisis, there has been little discussion about the way in which these indicators relate to each other and to other phenomena in a causal way. This paper will address this topic in two main parts: first, it outlines a proposal for a general causal scheme of indicators in the COVID-19 crisis; second, it discusses how this causal scheme can be applied in comparative empirical legal research, followed by a conclusion.

2 Developing a causal scheme of indicators in the COVID-19 crisis

A good starting point for thinking about causal relationships and indicators in the COVID-19 crisis is a paper by George et al. (2020) entitled ‘A guide to benchmarking COVID-19 performance data’. The main aim of this paper is the identification of performance data related to COVID-19. Yet, it also indicates a seemingly straightforward causal scheme. Specifically, it suggests that we can simply distinguish between two sets of indicators. On the one hand, there are ‘policy and strategy indicators’, ‘capacity indicators’ and ‘environment indicators’. These impact ‘output and outcome indicators’ on the other, specifically including ‘testing for COVID-19’ and ‘COVID-19 deaths’. Figure 1 illustrates this position in a causal diagram.

Figure 1. George et al.'s position on indicators in the COVID-19 crisis

It is helpful that George et al. include the categories of ‘capacity indicators’ and ‘environmental indicators’, which are not necessarily specific to the COVID crisis. They are relevant here as they relate to both COVID-19 infections and deaths. Specifically, for ‘capacity indicators’, George et al. refer to WHO data on nurses and medical doctors, hospital beds and health spending. However, more general measurements are also available: the WHO scores countries according to their application of the International Health Regulations (IHR)¹⁰ and there are there are also two private indices on the ability of countries to prevent health threats.¹¹ For environmental factors, George et al. indicate the examples of population density and age of population, while one could also think about other factors such as the existing health conditions of the population (e.g. obesity) and cultural characteristics (e.g. frequency of interactions with other persons; prevalence of multigenerational homes).

Yet, the causal narrative presented by George et al. is also quite simplistic. Thus, the remainder of this section aims to challenge it and suggests a more complex causal scheme of indicators in the COVID-19 crisis. It will do so by way of presenting a causal diagram that incorporates the possibility of feedback mechanisms. Naturally, such a diagram cannot consider all possible considerations that can play a role in reality. Therefore, while the following aims to go beyond the causal scheme suggested by George et al., it does accept their idea that it is fruitful to think conceptually about causal relationships and indicators in the COVID crisis.

The causal scheme of Figure 2 incorporates the position by George et al., most notably the causal link between COVID policies and infections/deaths. With respect to indicators on health capacities and environmental factors, however, the diagram distinguishes between the impact on COVID infections and on COVID deaths.¹² Stating that there are such links is not meant to imply that there is always such a causal relationship. For example, there was no causality between COVID policies and infections for the very first COVID cases. It is also possible that certain factors, such as investment in health capacity, are ineffective and therefore do not have an impact on the number of COVID deaths.

Figure 2. Possible causal scheme of indicators in the COVID-19 crisis

Going beyond George et al., the following suggests further key issues that are of relevance for a causal scheme of indicators in the COVID-19 crisis: law-making procedures, prior legal rules and idiosyncratic factors, the distinction between the indicator and the underlying real-life phenomenon, as well as possible feedback mechanisms.

2.1 The role of variations in law-making, legal models and idiosyncratic factors

The causal diagram of Figure 2 suggests that the substance of COVID policies is dependent on the law-making procedures. This should be understood widely. For example, it refers to the topic of whether governments can make laws in emergency situations without involvement of the parliament. While imposing high requirements on law-makers may be counter-productive in the fight against a pandemic, law-making procedures also play a role in how far law-making institutions are accountable to the public by standards such as the rule of law (all to be further discussed in section 3.1, below). In addition, prior legal rules are bound to be a determinant for COVID policies. There may be a path dependence to legal models used previously, such as whether to use tools like administrative or criminal law to regulate behaviour. Specifically, it also seems likely that recent prior experience with another pandemic may, ideally, enable a law-maker to formulate a targeted and effective response to the COVID pandemic.¹³

Beyond the aspired reduction of COVID infections (e.g. through a measurable decline in de facto mobility¹⁴), COVID policies have further implications on society. For example, lockdown policies have an effect on economic activity (Deb et al., 2020),¹⁵ which, in turn, have led law-makers to provide financial support and relief to business and citizens (Capano et al., 2020) and to adjust certain rules, for example, in labour law restricting the ability to lay off workers during the pandemic.¹⁶ Lockdown policies have also been observed to have had an effect on the types of crimes committed in this period (Mohler et al., 2020). With respect to the effect of COVID policies on health, it is not only COVID infections that should be considered. For example, some of these policies are likely to have the negative effect of people not seeking medical help for other health problems (or even having a more general effect on ‘health behaviour’¹⁷). Yet, some effects may also be of a positive nature: for example, face-mask requirements reduce all viral infections; lockdown policies lower air pollution and they may also have reduced the death rate of under-18-year-olds (by limiting their ability to engage in risky activities).¹⁸

The causal diagram also suggests that idiosyncratic factors can play a role for both COVID policies and COVID infections/deaths. With respect to COVID policies, for example, a newspaper paper states that a single local council official who imposed the first lockdown may have prevented a major COVID outbreak at the beginning of the pandemic in Germany.¹⁹ As regards COVID infections and deaths, idiosyncratic factors mean that these numbers should not simply be seen as a result of governments having ‘failed’ or ‘succeeded’ in their COVID policies and provision of health capacities. For example, in the Italian region of Lombardy, it was perhaps simply bad luck that a football game of Atalanta Bergamo coincided with the first COVID cases and thus led to a fast and wide spread of the virus in this region (and in Bergamo in particular).²⁰ While an indicator of major sports events could fall under the heading of ‘environment factors’, this would not capture the fact that this particular event happened at this particular date. Moreover, it is the nature of the virus that even the behaviour of one single person can matter if this person sets in course a chain of infections that leads to its spread in the population.

2.2 The distinction between the indicator and the underlying real-life phenomenon

Up this point, this text has not yet distinguished between the indicator²¹ and the real-life phenomenon that the indicator is meant to represent. Nonetheless, this distinction is crucial in order to fully understand the role of indicators in the COVID crisis. Therefore, Figure 2 always distinguishes between both categories, indicating with ‘⥲’ that the relationship is only an approximate one. How far it is ‘close’ depends on the quality of the indicator and the context in which it is applied.

For example, indicators that aim to measure COVID policies, such as the Oxford Stringency Index (see section 1, above), can meaningfully compare countries that have enacted conventional measures (lockdowns, face-mask requirements, etc.) that are well enforced. However, this may not be the case where law-makers have adopted more idiosyncratic rules, or where the law in the books and the law in practice diverge.²² Additionally, there are many different ways in which COVID-infection data can be measured (e.g. as absolute numbers, per-capita numbers, reproduction number, test-positivity rate, infections with symptoms, infections requiring hospitalisation). And, even with respect to COVID deaths, it has been controversial, for example, whether to use fatality data or to calculate lost years of life expectancy, how persons with multiple health conditions are accounted for and whether ‘excess-deaths’ data can be a more objective measurement (Hantrais and Letablier, 2021, pp. 16–31; Colombi Ciacchi, 2020).²³

Given the degree of subjectivity that is involved in any construction of indicators, it is important that COVID indicators (too) are as clear and transparent as possible. For example, this raises concerns about the COVID ‘Safety Assessments’ published by the private venture capital company Deep Knowledge Ventures (DKV), as it includes undisclosed ‘proprietary metrics’.²⁴ As far as it can be determined, their ranking also combines diverse elements related to the number of infections, government policy, health capacity and so forth, making it difficult to see what such ranking of countries really tells us.

2.3 Possible feedback mechanisms

The distinction between indicator and real-life phenomenon is also important because it enables us to understand the feedback mechanism of the causal diagram (dashed lines in Figure 2). Notably, it is possible that there is also a reverse causal relationship, given that COVID infections and deaths can also affect COVID policies through the indicators of those infections and deaths. Governments may have better access to information than newspapers that report on the actual numbers. Yet, it is clear that, given the many asymptomatic COVID infections, it is impossible to know the true total number of infections. Thus, governments are bound to act on the imperfect numbers that exist, and the same also applies to the feedback mechanism to health capacities. How exactly this is done depends on the country in question and it will thus be topic of the next section (see section 3.1, below).

Finally, the diagram indicates that there can be a direct feedback mechanism between the indicators of COVID infections and the true number of infections. As the former numbers are published in newspapers, on websites, by governments themselves or through other means, the public are aware of them. In fact, data from Germany show that citizens estimate the risk of becoming severely ill as even higher than the actual risk (Hertwig et al., 2020).²⁵ Thus, it also seems likely that the public will take this information into account when considering their own behaviour, for instance, in terms of applying forms of ‘social distancing’. This governance effect of indicators (see section 1, above) can also be used to inform government policy, for example, to improve acceptance and compliance with restrictions on mobility (or even to decide whether softer forms of restrictions may be sufficient).

3 Applying the causal scheme in comparative empirical legal research

The main legal element of the causal scheme presented in the previous section are the COVID policies. Since the start of the pandemic, international organisations, think-tanks and academic research have taken an interest in comparing such policies.²⁶ Some of these comparisons employ indicators coding the policies of different countries. Methodologically, these indicators typically follow a ‘functionalist black-letter’ approach of coding the law. For example, a variable of the Oxford's Stringency Index codes whether internal movement between different parts of the same country is restricted.²⁷ This variable is ‘functionalist’ in the sense that it is not interested in the precise wording of these rules, but their aspired outcome, namely the restriction of internal movement; yet, if there is such a restriction in the ‘black-letter rules’, the compliance and enforcement of these rules are not examined.

Specifically, this section will relate the causal scheme to comparative empirical legal research. The comparative element usually refers to the state/country level, as states that determine many COVID-related policies and indicators often compare countries. However, decentralised responses to COVID-19 are also prevalent (Aubrecht et al., 2020; Goolsbee et al., 2020) and can therefore also be compared (perhaps to see which type of response is preferable; cf. Büthe et al., 2020). The empirical element of the following discussion provides a link to the growing field of ‘empirical comparative law’, which discusses, amongst others, some of the methodological problems of research that uses comparative legal information in order to establish causal regularities (Spamann, 2015).

The studies discussed in this section have tried to establish the role of COVID policies on both sides of a possible causal equation – that is, the reasons for and the effects of different COVID policies. Most of these studies are, so far, published in working papers. Thus, they have not yet been peer-reviewed, and the authors of the papers may still revise them given the ongoing nature of the pandemic. Consequently, while the following will highlight some of the methodological challenges of such empirical research, it is also cautious in its critique given the preliminary nature of their findings.

3.1 The reasons for different COVID policies

Since the start of the pandemic, many comments and some empirical studies have explored why countries differ in their COVID policies. To start with, politics seems an obvious explanation. For example, it has been found that local COVID policies in the US reflect that counties with a ‘lower GOP vote shares were more likely to enact early sheltering policies’ (Goolsbee et al., 2020, p. 2). It has also been suggested that autocratic states may be able to impose and implement harsher COVID lockdown measures (Mattei et al., 2021). In other words, according to an empirical paper, ‘policy responses in democracies were less effective in reducing deaths’ which is said to ‘imply that democratic political institutions may have a disadvantage in responding quickly to pandemics’ (Cepaluni et al., 2020, p. 1). Yet, according to other empirical research, liberal democracies have the advantage that the availability of free media leads to more accurate data on COVID-19 deaths and thus more adequate policy responses (Besley and Dray, 2020).

It also needs to be noted, however, that, in democratic countries, the responses to COVID-19 have not necessarily been in the hands of their parliaments. On the one hand, this refers to the use of government emergency powers and their potential threat to ‘democracy, human rights, and the rule of law’.²⁸ Despite this, a recent empirical paper, drawing on a global survey of over 100 countries, ‘finds that, contrary to this conventional wisdom, courts, legislatures and subnational governments have played important roles in constraining national executives’ (Ginsburg and Versteeg, 2020, p. 1). On the other hand, some of the powers to deal with the pandemic have been allocated to scientific experts. While scientific advisory groups play a role in many countries,²⁹ particular attention has been paid to the case of Sweden, given the high degree of autonomy of the Public Health Agency of Sweden (Folkhälsomyndigheten). In this regard, it noteworthy that Sweden did not introduce a full national lockdown and thus had lower scores in the ‘stringency index’ than other European countries (Petridou, 2020). Yet, its mere use of recommendations also seems to have increased ‘social distancing’ and reduced travel.³⁰ It can also be argued that COVID policies can be ‘softer’ in countries where the population is in, any case, preferring a greater degree of interpersonal distances, as established in cross-cultural psychological research (e.g. Sorokowska et al., 2017; Kreuz and Roberts, 2019).

A core question is how far differences in COVID infections and deaths can explain differences in COVID policies (as also illustrated in Figure 2). At a general level, this seems to be the case. According to research using the Oxford COVID-19 Government Response Tracker, ‘government responses have become stronger over the course of the outbreak’ whereby ‘some of them immediately ratchet up measures as an outbreak spreads, while in other countries the increase in the stringency of responses lags the growth in new cases’ (Hale et al., 2020b, p. 11). Similarly, according to research based on the ACAPS data (see section 1, above), the rigidity of government responses to COVID is related to the number of days after the first death and the number of accumulated cases (Porcher, 2020).

Nonetheless, further details complicate the picture. Often, it will be the case that governments use data on COVID infections and deaths in a strategic way. For instance, the UK government has been accused of using coronavirus graphs and testing targets as a ‘number theatre’,³¹ and the Serbian government has been charged of underreporting cases prior to the elections.³² Governments can also be interested in giving emphasis to numbers about the rise in COVID infections and deaths in order to influence citizens’ behaviour (namely to stay at home, to reduce social contact, etc.), as happened in Austria.³³ Moreover, numbers can be included in government measure themselves. For example, in late 2020, South Korea adopted a ‘five-stage social-distancing scheme’ with differentiated stringency of rules according to areas that have (1) fewer than 100, (2) between 100 and 300, (3) between 300 and 400, (4) between 400 and 800 and (5) more than 800 cases a day,³⁴ and Italy distinguished between ‘yellow’, ‘orange’ and ‘red’ regions based on a list of twenty-one indicators.³⁵

With respect to the use of comparative empirical methods, the main problem is that the number of COVID infections and deaths may not only influence the COVID policies, but – except for the very first COVID cases – COVID policies also influence COVID infections and deaths (as further discussed in section 3.2, below). This problem of ‘law's endogeneity’³⁶ is a frequent topic of empirical comparative law given that it is often plausible to assume that there is a mutual relationship between law and society (e.g. Chong and Calderon, 2000). Econometrics has developed some tools to deal with complex causal relationships, such as system dynamics and structural equation modelling; yet, there are few examples, and all of them from other disciplines, that apply those tools to questions that involve legal variables (e.g. Ayyagari et al., 2013; Rindermann and Carl, 2018).

A more frequently used approach is to search for an instrumental variable (IV). In order to address the problem of an endogenous independent variable, such an IV needs to be highly correlated with this endogenous variable but uncorrelated with the error term of the equation (i.e. it needs to be exogenous to the dependent variable). A prominent line of research has used the ‘legal origin’ of countries as an IV (e.g. La Porta et al., 2006; Djankov et al., 2008). The rationale is that being a common- or civil-law country influences the country's specific rules on a particular matter and that, for most countries of the world, it was the exogenous colonial impact that made a country a member of the common- or civil-law family. However, regarding the COVID pandemic, it cannot be assumed that responses follow the legal-origin divide.

An alternative is to use lagged independent variables for data that have a time dimension (i.e. panel data). This is based on the intuitive motivation that the past can explain the future, but not vice versa. For instance, this approach has been used for studies dealing with the determinants of corporate-tax rates, investors and employment protection across countries (e.g. Wang, 2021; Pagano and Volpin, 2005). In the present case, it may be feasible to conduct such a panel analysis, as data on both COVID policies and COVID infections/deaths are available across time. For example, such an analysis could use the Oxford COVID-19 Government Response Tracker for COVID policies and the data on COVID infections/deaths as (imperfect) indicators for their true numbers.

However, even in this case, a further problem remains for comparative empirical research, namely the cross-border nature of both COVID infections and policies. As the diffusion of the pandemic can be related to social contacts and mobility (Solivetti, 2020), it is clear that infections easily cross borders. Yet, it has also been shown that COVID policies too are influenced by developments in other countries (Cheng et al., 2020; Lundgren et al., 2020). Thus, this is a case of the general problem that countries are not independent units of analysis called ‘Galton's problem’. It derives from a disagreement between Sir Edward Tylor and Francis Galton at an event in 1889: Tylor presented his anthropological research in order to show deep commonalities between cultures, but Galton objected that these similarities could equally be due to cross-cultural borrowing (Naroll, 1965). Econometrically, this creates the problem of spatial autocorrelation and, while there are some tools to account for this problem, it has been noted that the lack of fully independent units has not received much attention in empirical comparative law (Spamann, 2015, p. 146, fn. 27). It may also be said that, considering the information in Figure 2, it may be more fruitful to conduct qualitative work (rather than econometrics) in order to find out whether similar countries – namely countries with similar infection and fatality rates, similar environmental factors, similar health capacities, etc. – are likely to transplant rules from each other related to COVID policies.

3.2 The effects of different COVID policies

Many COVID policies aim to reduce COVID infections and deaths. Nevertheless, as with any legal rules,³⁷ it is not a matter of course that these policies are really effective. On the contrary, it is possible that some of them have the opposite effect of their intentions. For instance, closing universities can mean that students, who may be asymptomatic carriers of the virus, return to their parents’ home and infect more vulnerable family members. Reduced opening times of shops and restaurants as well as curfews may mean that cities and towns are more crowded at the times when everything is open. Face-mask requirements may give people a false sense of security and thus make them act less responsibly.³⁸ And, more generally, it may be argued that most COVID policies seem to apply a one-size-fits-all solution, while it could be better if the population understands and acts according to the specific risks that certain activities entail.

It is thus helpful that studies have explored this topic empirically. Some of them find a strong relationship between COVID policies and COVID infections/deaths. For example, in one study, researchers estimated ‘the effects of 1,700 local, regional and national policies on the growth rate of infections across localities within China, South Korea, Italy, Iran, France and the United States’, finding that anti-contagion policies have indeed substantially slowed this growth (Hsiang et al., 2020). Another study of data from eleven European countries estimates that, in spring 2020, non-pharmaceutical interventions such as lockdowns averted around 3.1 million deaths (Flaxman et al., 2020). A similar result is reached in a paper of the Oxford COVID-19 Government Response Tracker project, finding that a ‘lower degree of government stringency [as measured by their index] and slower response times were associated with more deaths from COVID-19’ (Hale et al., 2020a, p. 3).³⁹ Yet, it should be noted that the analysis of these studies is rather limited as far as the law is concerned, particularly due to their focus on black-letter rules. The general debate in ‘numerical comparative law’ has also shown that the construction of legal indices, in particular when made by non-lawyers (as here), may be biased in its selection of variables and coding of legal information (e.g. Siems, 2018, pp. 208–212).

Other research has found that environmental factors (using the terminology of Figure 2) play a role in a complex manner. According to one study, the interaction of countries with a ‘tight culture’ and ‘effective’ governments is associated with lower COVID-19 growth and mortality rates (Gelfand et al., 2021), while another study found that cultural variations in individualism and tightness affected the containment of COVID-19 regardless of the stringency of government responses (Cao et al., 2020). Specifically exploring variations in compliance, studies have found that: higher trust in policy-makers leads to better compliance (Bargain and Aminjonov, 2020), lockdowns are less effective in more individualist countries as their populations comply less with social-distancing rules (Bian et al., 2020) and laws mandating physical distancing are more likely to be violated in places with a low belief in science (proxied by a variable about the proportion of climate-change sceptics) (Brzezinski et al., 2020).⁴⁰ In addition, the prevalence of idiosyncratic factors (see Figure 2 and section 2.1, above) means that it is difficult to use comparative data in order to prove the effect of COVID policies (in other words, to be sure that findings have a high degree of external validity), including the relevance of many within-country variations as regards the spread of the virus. It is also the noteworthy that even the most extensive attempts of contact-tracing are not always successful⁴¹ and thus the spread of the pandemic remains unpredictable.

It follows that, econometrically, any comparative empirical study on the effect of COVID policies is prone to the problem of omitted variables. In this regard, it is also important to consider that there are limitations on the number of variables that can be included in country studies. The general econometric literature suggests that one needs to have at least ten to twenty observations for each independent variable (Harrell, 2015, p. 72). The use of country-level data, however, means that the number of observations is limited to the number of countries in the world. What is more, leaving out variables that are potentially relevant not only reduces the explanatory power (R ²) of a study, but also can make the entirety of the results unreliable due to an ‘omitted variable bias’, namely when an omitted variable is a confounding factor to the equation – that is, it is correlated with the dependent variable and at least one of the independent variables.

As noted in the previous section (see section 3.1, above), there is also the issue of endogeneity given that not only do COVID policies influence COVID infections and deaths, but COVID infections and deaths also influence COVID policies. In empirical research, apart from the tools mentioned in the previous section, quasi-experimental methods can be a possible solution. Their main advantage is that, as experiments, they distinguish between treatment and control groups, and doing so may reduce the problems of omitted variables and endogeneity. Specifically for COVID research, a recent paper discusses the possibility of one type of quasi-experiment, namely a difference-in-differences research design. Even so, it then notes that ‘the dynamics of COVID, the way people respond to it, and the flood of policy responses’ make it difficult to guarantee ‘assumptions about the comparability of treatment and control areas’ (Goodman-Bacon and Marcus, 2020, p. 1).

A further fundamental conceptual as well as empirical problem relates to ‘law's normativity’. In the present case, is it really beyond doubt what the ultimate aim of COVID policies should be? Such scepticism contrasts with attempts to rank countries such as the DKV ranking (see section 2.2, above) and statements in the media such as the one that ‘as governments fumbled their coronavirus response, these four got it right’.⁴² The main problem is that many of the effects of the COVID pandemic are not easily comparable with each other, such as (1) losing one's life, (2) being ill, (3) being in lockdown for an extended period – and thus, for example, being separated from close family members, not being able to attend school or university, or suffering from mental health problems,⁴³ (4) being prohibited to pursue certain hobbies and (5) suffering economically. Some attempts have been made to address this issue, for example, to present lost lives in monetary terms (Miles et al., 2020 on UK guidelines that a year of life lost equals £30,000), to use external benchmarks such as the effect of the pandemic on the UN's Sustainable Development Goals (Alibegovic et al., 2020) or even to aggregate multiple effects in a form of ‘misery index’.⁴⁴ Yet, it seems doubtful whether this can solve the problem of incommensurability.

It can thus be argued that the question about the ‘right’ aim of any COVID policy is simply a subjective one. Subjectivity also comes into play, as policy responses are based on a risk assessment. This means that the decision is often between avoiding either false positives or false negatives. For example, if there is the possibility of a COVID case in a particular factory (or university, company, etc.), is it always preferable to shut down this factory as a precautionary measure or should there be a probability threshold to justify such a measure? In other words, policy-makers may desire highly accurate predictions from experts in order to implement measures that can contain the virus; yet, it is clear that any such estimates also contain many sociological and normative assumptions (Brandmayr, 2020).

Survey-based research has made some attempts to uncover the views and preferences of citizens in the COVID crisis. For example, a study by the Pew Research Center asked citizens questions such as whether they believed that their government did a ‘good job’ in handling the pandemic.⁴⁵ More specifically, another study asked respondents ‘whether and the extent to which citizens are willing to trade off civil liberties during the COVID-19 pandemic’, amongst others, finding that people in the US are less willing to sacrifice rights than those in China (Alsan et al., 2020). This latter example also points towards a limitation of such surveys, namely that they only work well if the phenomenon under investigation is comparable across countries (which cannot be said to be the case about civil liberties in the US and China). The dependency on the specific point in time at which a survey is conducted can be seen as a further limitation. For example, a study from Germany found that ‘the widespread support for the containment and delay policy measures steadily decreased over time as did feelings of threat and subjective risk perceptions’ (Naumann et al., 2020, p. 199).⁴⁶

The relationship between empirical research and ‘law's normativity’ can also be reassessed from the perspective of comparative law. According to statements by Nelken, there are ‘different popular ideas in different countries about the purposes of law and what is to be expected from it’ (Nelken, 2007, pp. 124–125) and it may be that ‘in Anglo-American countries something is right because it works; in other countries a response works because it is right’ (Nelken, 2010, p. 26). From the perspective of empirical legal research, it has also been said that ‘it depends on the normative purpose whether avoiding false positive decisions is indeed paramount, or whether false positives and false negatives have to be balanced out differently’ (Engel, 2018, p. 18).

As regards the COVID pandemic, it also follows that it is, of course, useful to conduct empirical research on the effects of COVID policies. Yet, the limitation remains that any comparative facts about a particular causal (or even just correlational) relationship do not answer the ultimate decision of what this means for the right policies in a particular place.

4 Conclusion

Indicators are a core feature of the COVID crisis. They are relevant for all citizens, as the information about COVID infections and deaths is bound to influence their daily decisions. They are also an opportunity for different lines of research.⁴⁷ It was the aim of this paper to discuss indicators in the COVID crisis from the perspective of a legal scholar with an interest (and some expertise) in comparative law and empirical legal studies. This meant that this paper did not engage in the details concerning epidemiological and medical issues of COVID infections and deaths. Rather, it focused on two main issues.

First, it developed a general causal scheme of indicators in the COVID crisis. This part mainly centred on a causal diagram (Figure 2). As for any presentation on complex issues, this diagram was not meant to include all details that could potentially be relevant in this field. Yet, it is suggested that such a scheme is helpful in mapping the main causal relationships between indicators and real-world phenomena in the COVID crisis. Notably, it can show that there are connections at three levels: between indicators and their underlying real-world phenomena, between indicators and other real-world phenomena, and between real-world phenomena themselves.

Second, this paper discussed how such a causal scheme has been, and can be, applied in comparative empirical legal research. It mainly focused on the COVID policies and, thus, in the spirit of a causal scheme, it analysed research on the reasons for different policies on the one hand and the effect of different policies on the other. In its assessment, this paper endorses the general ambition to engage in research that tries to show such causal relationships. However, it also noted that the current empirical studies related to COVID policies are rather limited, as they do not test complex causal schemes whereby many of the elements would be dependent on each other (such as Figure 2). This paper also addressed the fact that these studies face difficulties in proving causality akin to much of the research of empirical comparative law. Thus, it is suggested to be cautious about alleged proven claims of causal connections.

Finally, in discussing these topics, this paper aims to advance the view that it is worth researching indicators not only individually, but also in relational terms. The causal scheme presented here referred to some indicators that are not specifically about the COVID crisis, such as rule-of-law and environmental indicators. Future research could thus expand the causal scheme considering the interconnected ecologies of indicators and incorporating the findings of this paper.