A General Approach to Measuring Electoral Competitiveness for Parties and Governments

Abstract

We develop a general approach to measuring electoral competitiveness for parties and governments, which is distinct from existing approaches in two ways. First, it allows us to estimate the actual probability of re-electing the incumbent into office, which lies closer to the theoretical concept of interest than most widely used proxies. Second, it incorporates both pre-electoral competitiveness—that is, the uncertainty about the outcome of the upcoming election—and post-electoral competitiveness—that is, the uncertainty concerning who will form the government given a certain election result. The approach can be applied to, and compared across, a multitude of institutional settings and is particularly advantageous in analyses of multiparty democracies. To demonstrate its full potential, we first apply the approach on 1,700 local government elections in Sweden. Three advantages over existing approaches are documented: Our election probability measure shows substantial variation over the election cycle, it can be accurately measured for a single party as well as a government, and it is more capable of predicting re-election into office than any previous measure of electoral competitiveness. A second application on 400 national elections in 34 democracies shows that the approach also works well in a more challenging cross-national setting.

1 Introduction

Electoral competitiveness¹ has been recognized as “simultaneously one of the most central and mismeasured constructs in the study of democratic politics” (Kayser and Lindstädt 2015, 242). At the most basic level, the essence of this elusive concept lies in the certainty with which it can be foreseen what coalition, party, or candidate that will execute political power after the next election. The more uncertain is the outcome, the more competitive the election is said to be. In studies of the policy consequences of electoral competitiveness, the obvious focal point is the likelihood that the incumbent executive will be ousted from office at the next election (Boyne 1998; Pettersson-Lidbom 2001; Immergut and Abou-Chadi 2014; Kayser and Lindstädt 2015).

Accordingly, an ideal measure of electoral competitiveness in policy analysis would be based on truthful responses by incumbents to questions about the probability that they will remain in power after the next election, posed at the time of policy-making (cf. Boyne 1998). Although most policy analysts would probably subscribe to this ideal, the difficulties involved in collecting data on such responses have led scholars to resort to a wide variety of proxies of electoral competitiveness.

A common trait of most of these measures is that they only capture what we may call the pre-electoral competitiveness, as exemplified by factors such as the vote margin of the ruling coalition (Canes-Wrone and Park 2012), the historical degree of electoral volatility (Boyne 1998), or the plurality party’s likelihood of losing plurality status (Kayser and Lindstädt 2015). As such, these measures are blind to the post-electoral competitiveness associated with how the government formation process plays out once the election results are in. In studies of electoral competitiveness in multiparty systems, this omission is problematic because, as we know from the vast literature on government formation, parties differ systematically in their capability of translating parliamentary seats into cabinet seats (e.g., Martin and Stevenson 2001, 2010; Glasgow and Golder 2015). Although steps have recently been taken to incorporate some aspects of the government formation process into measures of electoral competitiveness (e.g., Immergut and Abou-Chadi 2014), this is yet to be done in a systematic and comprehensive manner.

In this paper, we propose a general approach to measuring electoral competitiveness for parties and governments, which combines the pre- and post-electoral competitiveness into one joint measure. The measure is conceptualized in terms of the probability that a given actor will be elected into the government office. In most real-world applications, the actor of interest will probably be an incumbent party or coalition, in which case it is a probability of re-election into office, but nothing prevents us from estimating office probabilities for noncabinet actors. Because the measure is constructed in a way that allows for considering various behavioral and institutional factors—such as voter volatility that increases uncertainty of elections and investiture rules that make minority governments less likely to form—it should be flexible enough to be applied to and compared across any party constellation in any democratic system.

Our measure is designed so as to satisfy the six criteria helpfully proposed by Kayser and Lindstädt (2015, 243) for a useful measure of electoral competitiveness. It is (a) conceptually clear, as it measures the actual concept of interest—the expected probability of (re-)election—rather than some proxy for electoral security. It (b) applies a unit of analysis matching the actors—that is, a party or a constellation of parties—rather than referring to system-level characteristics such as electoral institutions or measures of democracy. It (c) provides consideration for electoral volatility, (d) has a direct connection to the loss of power, (e) is measured at an interval-level scale, and (f) is in congruence with the executive’s own perspective on his or her electoral security. Some recently proposed measures have made significant advancements with respect to several of these criteria, including Kayser and Lindstädt’s (2015) “loss probability of the plurality party” and Abou-Chadi and Orlowski’s (2016) “likeliness of a vote-swing sufficiently large to alter a party’s bargaining position in the legislature.” Still, to the best of our knowledge, the measure introduced here is the first to satisfy all six criteria—at least for applications where it cannot be safely assumed that the plurality party is also the party of the chief executive (cf. Kayser and Lindstädt 2015).

Whereas in principle our approach has a general applicability, it is complex and computationally demanding. Although, at a very minimum, it requires only party-specific information on vote shares and incumbent government membership, the measure is designed for fairly information-rich settings and it performs better the more data are utilized. In particular, we recommend the use of recurrent vote intention polls (e.g., Canes-Wrone and Park 2012). We document two advantages of doing so. First, unlike measures based on vote shares from the previous election, predicted probabilities that incorporate vote intention polls vary over the election cycle and should better approximate politicians’ perceptions of their re-election chances at the time of policy-making. Second, the use of polling data increases the predictive capability of the measure.

In Section 2 of the paper, we present an overview of the four-step procedure that we use to construct our re-election probability measure. We then report two demonstrations of how our approach can be used to estimate re-election probabilities for the incumbent. To demonstrate the full potential of the approach, we first apply it on rich data from approximately 1,700 local government elections in Sweden. This application is described in Section 3. Validity tests reported in Section 4 show that the resulting probability measures are accurately estimated and more capable of predicting re-elections than any measure of electoral competitiveness found in the existing literature, and that this holds true even when our approach is simplified in various ways. Second, we test the performance of our approach in a more challenging setting, using data from approximately 400 national elections in 34 democracies. As summarized in Section5, the cross-national measures also perform well, albeit not on par with those from the more favorable local government setting. Section 6 concludes with some remarks about the importance of considering both pre-electoral and post-electoral uncertainty when constructing measures of electoral competitiveness.

2 A General Approach to Measuring Electoral Competitiveness

There are two sources of uncertainty that need to be considered when estimating the probability that a certain party or coalition of parties will enter office after the next election. The first source is the pre-electoral uncertainty, which captures the fact that there is an unlimited number of possible election results and that no one knows which one of them will be realized. While polls may provide a good idea of the most probable election outcome, it is—as we shall see—just as important to model the uncertainty surrounding this forecast.

The second source of uncertainty, which we may refer to as post-electoral uncertainty, concerns the government formation process that takes place after the election. That a party performs well in an election does not necessarily imply that it is more likely to enter the government, and vice versa. For example, a party located close to the center of the ideological space, which has participated in government before, is more likely to succeed in the post-electoral bargaining and join the ruling coalition than a peripheral and inexperienced party that has won a similar seat share (Bäck and Lindvall 2015). Measures that do not consider the type of competition that plays out in the postelection coalition bargaining—including standard measures of electoral closeness—are therefore not suitable in multiparty systems, where coalition governments are the norm (Strøm 1989).

Combining these two uncertainties is quite straight-forward. In general terms, we can think of the pre-electoral probability $P(O)_p$ that party p enters office as the sum of the probabilities that the party enters office given a certain election outcome $E_v$, that is $P(O_p|E_v)$, over all possible election outcomes weighted by their respective probabilities $P(E_v)$ as follows:²

(1)$$ \begin{align} P(O)_p = \sum\limits_{v=1}^n P(O_p|E_v) \times P(E_v). \end{align} $$

This simple structure is intuitive and can easily be applied to virtually any institutional setting. For the reasons stated above, in a multiparty system, both these components are also necessary to create accurate election probability estimates. Hence, we believe that most attempts to estimate electoral competitiveness would benefit from using this two-component structure as their starting point. Yet, we know of no other study that attempts to capture both pre- and post-electoral uncertainty in the same measure.

2.1 A Four-Step Procedure

How then should one proceed to estimate these probabilities? There is a myriad of ways to model the two sources of uncertainty, and the best choice will depend on the institutional setting, what data are available, and what the measure is going to be used for. Generally, however, we believe that any approach would in one way or another involve each of the four steps outlined in Figure 1.

Figure 1 Schematic overview of the four-step approach used to estimate electoral competitiveness.

Step I consists of modeling the pre-electoral uncertainty. Here, we develop a forecasting model that uses data on prior election results—optionally supplemented with polling data—to predict the most likely election outcomes and to quantify the uncertainty surrounding these estimates. In the applications in this paper, we create our vote share predictions using standard ordinary least squares (OLS) regressions,but in other applications it may be preferred to use other data or substitute these regressions with Bayesian forecasting models. To model the uncertainty around these predictions, we choose to simulate a dataset of potential election results, by re-sampling residuals from other observations and adding them to the predicted election outcome.³ The 500 simulated election outcomes ($v_{pits}$) generated in this step vary between parties (p), simulations (s), political entities (i), and years for which the forecast was made (t).

In Step II, we estimate a model of post-electoral uncertainty, which for any given election outcome (i.e., a set of party vote shares) can predict the probability of a certain party or coalition of parties entering into power. We do so using the “potential coalition” framework, which has dominated the empirical literature on government formation over the past 15 years (Martin and Stevenson 2001, 2010; Bäck 2003; Debus and Gross 2016). In this framework, the unit of analysis is a government formation opportunity, occurring after an election or when, for any other reason, the incumbent government resigns. The government formation process is modeled as a discrete choice problem in which the parliament selects one government from a choice set consisting of all potential governing coalitions that are, in theory, available for consideration given the number of parties in the parliament.⁴

The outcome of this exercise is a model that may predict, for each potential government coalition j in each political entity i after each election y, a probability of realization based on a number of characteristics of that coalition $\boldsymbol {x}_{jiy}$. These are computed based on the realized characteristics of the parties in the parliament, including, importantly, the seat share distribution. The potential coalition framework also allows for modeling the impact of institutional factors that vary across government formation opportunities and that may favor particular types of coalitions.

In Step III, we take the model estimated on realized data in Step II and plug in the simulated election outcomes generated in Step I. More specifically, we first identify all possible combinations of parties that are represented in the parliament, for all simulated election outcomes. These combinations of parties comprise the potential governments. For each potential government $jits$, we calculate the set of characteristics that is used to estimate the coalition’s probability of entering office, $\boldsymbol {x}_{jits}$. We then use the coefficients $\boldsymbol {\beta }$ generated by the model in Step II to calculate, for each simulated potential government in each year, the predicted probability that it will enter office, based on its vote share as forecast in Step I, $p(O)_{jits}$.

In Step IV, to arrive at a measure of the electoral prospects of the actor of interest, we just need to aggregate the data in the way we desire. For example, to calculate for a given point in time, the estimated probability that party p will be part of the government that enters office after the next election, we would sum up, for that point in time, the probabilities for all simulated potential governments that include party p, and divide that sum by the number of simulated election outcomes. For our main measure of the re-election probability of the incumbent, which we produce in our present demonstrations, we calculate the average election probability for the incumbent parties, weighted by their respective seat share.

The flexibility of the proposed approach makes it possible to generate comparable election probabilities for parties and governments in any democratic system in which the government is responsible to the legislature, and for which the minimally required data are available. Technically, the approach may also generate probabilities for parties in systems where the head of government is popularly elected, such as the United States, in which case Step II is redundant. Yet, for someone whose only interest lies in presidential systems, there would obviously be little to gain from modeling the post-electoral uncertainty.

3 Application on Swedish Local Governments

3.1 Description of the Case and the Data

For a first demonstration of our approach, we find it useful to apply it in a setting where we believe that it may come close to realizing its full potential. For such an application, focusing on subnational elections is advantageous because it allows us to study a large number of elections with a common party system and institutional environment. We use the case of Swedish local governments, which provides rich and consistent data on election results, government composition, and vote intentions, for 290 political entities over more than 20 years, with a configuration of competing parties that is relatively stable across both space and time. The richness of the data also enables us to evaluate how sensitive the approach is to the exclusion of data which may not be available in other applications.

Another advantage of using this case is that elections as well as government formations in Swedish municipalities follow a similar logic to many other proportional representation (PR) systems (Bäck 2003). Municipalities are governed by a local council consisting of 21–101 seats, to which members are elected from multimember electoral districts in September every fourth year. Swedish municipalities have a “quasi-parliamentary” system, in which a majority coalition (or party) appoints the Mayor (kommunstyrelseordförande, KSO) and the committee chairs and vice chairs. This coalition (or party) is commonly regarded as the equivalent of a national government and tends to exert a particularly strong influence on policy (Bäck 2003; Gilljam, Karlsson, and Sundell 2015; SKL 2018). The local councils typically contain between five and nine parties. Although local parties exist, most parties are local-level branches of national parties, among which policy positions mostly vary according to traditional patterns in a two-dimensional policy space (Bäck 2003).

Moreover, Swedish municipalities have experienced similar trends as many other PR systems in terms of an increasing number of parties both in the legislature and in the governing coalition, which makes bargaining processes increasingly complicated and difficult to predict (SKL 2018). As such, it represents a case where bargaining complexity is on a similarly high level as in most other European PR systems.

Each step of our estimation procedure makes use of the same core dataset, in which each row represents one of the 290 Swedish municipalities observed at 1 year between 1994 and 2018 (available at Cronert and Nyman 2020a,b). Political variables at the local level include vote shares and seat shares in the local council, for each of the eight dominant parties in Swedish politics⁵ plus a residual category for any other party (mostly local parties). These data are derived from Statistics Sweden (2018a). The dataset also includes data on which parties are members of the local government, as well as which party holds the position of Mayor, retrieved from the Swedish Association of Local Authorities and Regions (SKL 2018) and Johansson (2010).

In addition, the dataset includes a number of covariates at the national level: annual vote intention poll data for each of the eight aforementioned national parties from Statistics Sweden’s Party Preference Survey (PSU) carried out in May of each year (Statistics Sweden 2018b), and a measure of the ideological (left–right) position of these parties retrieved from the Chapel Hill expert survey (Polk 2017). For the sake of simplicity, this measure is used as a proxy for the ideological position of the local branch of the respective party. At the cost of some accuracy, we ascribe to each local party the period-specific median position among the national parties.

3.2 Step I: Forecasting Election Results

The very first step of our approach is to forecast the expected vote share v in the upcoming election, for each party (p) in each municipality (i) and for each election year (y). In the current demonstration, we create predictions for each year during the election period (not only election years), so our predicted election outcomes vary between calendar years (t).

Forecasting is both about making predictions and estimating their accuracy. For many applications, prediction is the primary objective, but for our purposes, correctly estimating the uncertainty surrounding these predictions is just as important. Knowing the most likely outcome of an election says little about the incumbent’s re-election prospects unless we also know how certain we are about this outcome and what the possible alternatives look like.

In order to identify the relevant alternatives and their respective likeliness, we need to create a distribution of possible election outcomes. Our forecasting model therefore consists of two components. The first component is a regression model that predicts the expected outcome in the next election. The second component is a simulation exercise where we re-sample residuals to approximate random draws of election results from an imagined probability distribution of outcomes.

Making the Prediction

Our election predictions primarily rely on the party’s previous election result in the same municipality as well as how the support for the same party on the national level has changed in the yearly nation-wide PSU polls. In our main specification, we also include incumbent dummies to capture the electoral cost of ruling as well as a set of interaction variables that allow the coefficient for the national polls to differ between parties as well as depending on the time remaining to the next election. However, as reported in Section 4, our measure performs well also if the prediction model is simplified so that the party’s previous election result is the only predictor.

The full equation, which is estimated on the period $t \in \{1995 \ldots 2018\}$ using OLS, can be written as:

(2)$$ \begin{align} v_{piy} = \alpha_0 + {\alpha_1}_p v_{piy-1} + \alpha_2 i_{pit} + \alpha_3 \Delta q_{pt} + {\alpha_4}_p \Delta q^*_{pt} + \boldsymbol{\beta} \prime(\boldsymbol{\phi}_{pit}\Delta q_{pt}) + \boldsymbol{\gamma}_{it} + \boldsymbol{\psi}_{pit} + e_{pit.} \end{align} $$

Here, $v_{piy}$ is the realized vote share for party p in municipality i at the upcoming election year y, where $y \in \{1998,2002,2006,2010,2014,2018\}$, $v_{piy-1}$ is the party’s realized vote share in the previous election year, $i_{pit}$ is a binary indicator for whether the party is included in the incumbent coalition, and $\Delta q_{pt}$ is the difference between the party’s support in the national PSU poll in year t and its support in the poll in the previous election year. $\Delta q^*_{pt}$ measures the change in the polls for other parties within the same side of the left–right divide, which is included because levels of support for ideologically adjacent parties tend to be negatively correlated. $\boldsymbol {\phi }_{pit}\Delta q_{pt}$ is a vector of interaction variables, which allow the effect of the national polls to differ (1) between parties, (2) depending on the previous election result, and (3) depending on the time left to next election. To ensure that the predicted vote shares sum to unity, we include fixed effects at the municipality–year level ($\boldsymbol {\gamma _{it}}$). Lastly, $\boldsymbol {\psi }_{pit}$ is a vector of binary indicators for whenever there is missing data on any other variable (in which the case the missing values are replaced by zeroes).

All parties are not represented in every municipality and it is not obvious when to include a party in the forecasts. We choose to exclude parties that did not receive a single seat in any municipality during the previous election (i.e., we give them zero votes in all simulations), but include them as soon as they have received at least one seat somewhere in Sweden—effectively giving them a small but positive probability of receiving a seat.⁶

Modeling the Uncertainty

Even if the model described above provides us with reasonably precise predictions of the local parties’ vote shares, these predictions will still deviate from the actual outcomes. Because the expected size of these deviations has an effect on the probability of re-election into office, it is important that we do our best to model this uncertainty correctly.

We do so using a simulation approach with re-sampled prediction errors, in which we use the empirical distribution of residuals as our estimate of the true distribution of our model’s uncertainty.⁷ By re-sampling residuals from this distribution, and adding them to our fitted values, we get a set of predictions which approximate the entire probability distribution of election outcomes. Unlike most bootstrapping techniques, we do not reduce this distribution to some parameter that we are interested in. Instead, as described in Section 3.4, we feed this sample of possible election outcomes to a coalition formation model in order to estimate a separate probability of entering office for each party in each simulated election.

Because the outcome variable is fractional (vote shares that sum to 100%) the parties’ residuals will be negatively correlated within each election. If voters are more likely to switch between ideologically adjacent parties, this negative correlation will be stronger for certain pairs of parties. To replicate these correlations, we use a specific block bootstrap where we re-sample individual local elections (including 7–9 party-specific residuals) instead of individual residuals, and assign the residuals from the sampled block so that they always belong to the same party.

Because the absolute size of the residuals is correlated with the party’s predicted vote share, we divide the municipal elections into 10 clusters with similarly sized parties. When re-sampling the blocks of residuals, we only draw blocks from within each of these 10 clusters of elections.

We have also evaluated two simplifications of this uncertainty modeling procedure. The first simplification is to skip the aforementioned blocks and clusters and instead re-sample each party’s residuals as if they were independent of the other parties’ residuals as well as of the size of the party. The more far-reaching simplification is to skip the simulation altogether and plug the predicted election outcomes directly into our coalition formation model as if there were no uncertainty around these forecasts. As reported in Section 4, evaluations show that simulation is important for the performance of the measure, whereas the added value of blocks and clusters is small.

After the simulation is completed, we calculate the number of seats distributed to each party using the modified Sainte–Laguë method with quotient 1.4 prescribed by the Swedish election law.⁸

3.3 Step II: Modeling Government Formation

In parallel with Step I, we need to develop a model to account for how parties’ election results affect their likelihood of entering the government after the upcoming election. We do so using the aforementioned potential coalition framework, which has come to dominate the literature on government formation, in studies both at the national level (Martin and Stevenson 2001, 2010; Glasgow, Golder, and Golder 2012; Glasgow and Golder 2015), and at subnational levels (e.g., Bäck 2003; Bäck et al. 2013; Debus and Gross 2016). In this approach, the government formation process is perceived as a discrete choice problem in which the parliament chooses one and no more than one of all the potential governing coalitions that the parties in the parliament may form.

The outcome of this exercise is a model that may predict a probability of realization for each potential government, based on a number of characteristics of the parties in the legislature and of the institutional context in which the formation process takes place. Having surveyed the government formation literature, we identify 32 factors that have been claimed to be important for government formation and that are applicable to Swedish local government. These factors include whether the potential government is a minority cabinet, whether it consists of the incumbent parties, the width of the ideological range among the included parties, their combined parliamentary seat share, the extent to which they have previously governed together, among many others (for the complete list, see the Online Appendix, Section A1). Because our purpose is not to test any hypotheses but simply to predict the outcomes of government formation processes, and because we have enough data not to be overly worried about over-fitting, we include all 32 variables in our model.

In some applications, it may not be possible to calculate all these variables or include all of them in the estimation. As reported in Section 4, we have therefore evaluated a simplified version of our measure, for which the coalition formation model only includes ten key variables that are easy to calculate. Although we find that the measure performs better when more variables are used in this step, the simplified measure also performs reasonably well.

Following the bulk of the existing literature, we model the government formation process using a conditional logit model.⁹ In this model, the probability $p(O)$ that the potential government j is chosen out of the set of J potential governments in the formation opportunity occurring in municipality i after election y is:

(3)$$ \begin{align} p(O)_{jiy} = \frac{e^{\boldsymbol{\beta}\prime\boldsymbol{x}_{jiy}}}{\sum_{j=1}^J e^{\boldsymbol{\beta}\prime\boldsymbol{x}_{jiy}}}, \end{align} $$

where $\boldsymbol {\beta }$ is a vector of coefficients and $\boldsymbol {x}_{jiy}$ is a vector of characteristics associated with potential government j in formation opportunity $iy$. The output of such a model, run on the realized governing coalition outcomes of approximately 1,700 government formation opportunities in Swedish municipalities between 1998 and 2018, is reported in full in the Online Appendix(Table A1). Suffice it to say here that our model has a satisfactory fit compared to existing studies, as the Pseudo $R^2$ parameter of $0.58$ reported in our model is higher than those reported in previous work, ranging from $0.33$ to $0.57$.¹⁰ For our current exercise, the output of primary interest is the vector of 32 coefficients ($\boldsymbol {\beta }$), which is saved to be used in Step III.

3.4 Step III: Predicting the Potential Governments’ Office Probability

We begin the third step by identifying—for every election simulation s generated in Step I—all possible combinations of parties j that are predicted to receive at least one seat in the upcoming election in municipality-year $it$. For all these potential governments, we then calculate exactly those 32 government characteristics $\boldsymbol {x}_{jits}$ that are used in the government formation model in Step II. Many of these characteristics are a function of the simulated seat shares and therefore vary between the simulations. To predict each potential government’s probability of entering office, we then use the vector of coefficients ($\boldsymbol {\beta }$) estimated in Step II and apply them to the government characteristics calculated on the simulated data.

3.5 Step IV: Aggregation

For each municipality-year, we now have 500 simulations of the outcome of the next election, and for each simulated election outcome, we have predicted the probability of entering office for each potential government. Depending on how we aggregate these probabilities, we can now calculate the predicted election probability for any possible party or set of parties.¹¹

The first step in this aggregation procedure is to calculate the office probability of party p in municipality i at year t. Formally, for any single simulation, this can be written as the sum of the product of the office probability of each potential coalition $p(O)_{jits}$ and a binary indicator $m_{pjits}$ for whether party p is a member of that coalition. By averaging over all simulations $s \in \{1 \ldots S\}$, we get the party’s estimated office probability $p(O)_{pit}$ as follows:

(4)$$ \begin{align} p(O)_{pit} = \frac{1}{S}\sum_{s=1}^{S}\sum_{j=1}^{J} p(O)_{jits} m_{pjits.} \end{align} $$

We may then use these party-specific probabilities to calculate re-election probabilities for, say, the largest party in the incumbent coalition or an average for all incumbent parties, weighted by their respective seat share.

Because the national vote intention polls are carried out in May of each year, the expected election outcomes vary between years. With the Swedish general election being fixed to the third Sunday in September every fourth year, we may thus produce one predicted probability for May of the election year ($t = y$), and another one for May of each of the 3 years before the election ($t-1 \ldots t-3$).

To give the reader an idea about what the estimated probabilities look like, Figure 2 shows the distribution of the re-election probabilities for all incumbent parties (left-hand panel), as well as for the incumbent government as a whole when each participating party is weighted by its seat share (right-hand panel). The reason why the two distributions look so different is that large parties as well as single-party governments are up-weighted in the right-hand panel, and they tend to have a relatively high probability of staying in office.

Figure 2 Distribution of estimated re-election probabilities for incumbent parties in Swedish municipalities.

So how should these probabilities be understood in terms of competitiveness? All conceptualizations of electoral competitiveness have in common that if there is a high probability that the incumbent will be re-elected (p is close to 1), competitiveness is considered to be low. However, for probabilities in the bottom half of the distribution ($p<0.5$), a common practice is lacking. Many scholars do not make a clear distinction between situations where p is close to 0.5 and situations where the incumbent is likely to be ousted ($p\ll 0.5$). Some, however, use electoral competitiveness more specifically with reference to close elections ($p\sim 0.5$) and see competitiveness as decreasing with the absolute distance of p from 0.5 (e.g., Strøm 1990; Canes-Wrone and Park 2012; Seiferling 2020). In our view, the latter usage is advantageous in terms of conceptual clarity, and it corresponds to how the concept was presented in the introduction of this paper—that is, that the more uncertain the outcome, the more competitive the election. However, this is not to suggest that high and low election probabilities are equivalent—on the contrary, these are situations where we may expect rather different behavior among incumbents (Seiferling 2020). To clarify this point, we find it useful to conceptually distinguish between, on the one hand, the electoral competitiveness facing an actor ($1-|p-0.5|$) and, one the other hand, the electoral safety (p)—or, conversely, the vulnerability ($1-p$)—of that actor.

4 Evaluation of Measurement Validity

Our primary claim to measurement validity is based on (1) the theoretical closeness of our measure to the actual concept of interest in most studies of electoral competitiveness—the probability of being (re-)elected into office—and (2) our estimation procedure which takes into account both pre- and post-electoral uncertainty and allows for modeling the effects of behavioral and institutional factors. Nevertheless, the validity of our measure can and should also be evaluated empirically.

In this section, we evaluate our measure according to two empirical criteria: (1) accuracy of uncertainty estimation, and (2) predictive capability. The first criterion concerns whether our measure of uncertainty corresponds to the actual level of uncertainty in the sample for which we make predictions. The second criterion refers to how well our measure performs in terms of predicting when an incumbent government will be replaced or re-elected into office. This test will be done in comparison with other measures that have been used as indicators of electoral competitiveness in previous research.

Considering that our procedure is rather computationally demanding, it is also worthwhile to compare our main measure to the four simplified versions of the measure described in the previous section.¹² To recollect, we have, first, used a reduced model for the election forecast in Step I, with previous election results being the only predictor. Second, we have simplified the simulation procedure in Step I by simply drawing party-specific residuals without the blocs and clusters described in Section 3.2. Third, we have skipped the re-sampling of residuals altogether, effectively assuming that the election forecast will be realized. Fourth, we have rerun Step II using a government formation model which only includes ten key potential government variables that are central in the literature and easy to calculate.

4.1 Accuracy of Uncertainty Estimation

If we have estimated the uncertainty correctly, there should be a 1:1 relationship between our prediction of election probability and the outcome—that is, successful election into office—such that, for any given set of predictions, the share of actual successes should equal the average estimated probability. To test this, Figure 3 reports the share of successful elections into office over the estimated probability of success, with data being “binned” into 20 ventiles based on the estimated probabilities. The theoretical 1:1 relationship is illustrated by the diagonal line.

Figure 3 Comparison between average estimated election probabilities and average outcomes in Sweden.

Figure 3 contains three panels. As regards the left-hand panel, the y-axis reports an indicator on the rate of actual entry into office after the next election of all individual parties in our dataset, for each observed year between 1995 and 2018 (in total 62,532 party-year observations). The bins on the x-axis divide these observations into 20 subsamples based on our measure of election probability in that year. The center panel instead reports an indicator on the rate of actual re-election into office of the incumbent government, where each incumbent party is weighted by its seat share.¹³ The bins on the x-axis divide the 6,820 observed municipality-years into 20 subsamples based on our measure of the seat share weighted re-election probability of the incumbent, as described in Step IV above. In the right-hand panel, the outcome indicator instead reports the rate of actual re-election of the largest party among the incumbent parties, and the bins are now based on the specific election probability for the largest incumbent party.

In all three panels of Figure 3, all the bins lie reasonably close to the diagonal line. These tests suggest that our probabilities are correctly estimated and that there is no systematic over- or underestimation in our measure. To evaluate the importance of modeling both pre- and post-electoral uncertainty for estimation accuracy we have produced equivalent plots for the re-election of the incumbent government, for cases where our probability estimates have been constructed with the aforementioned simplifications imposed on Step I or II (reported in the Online Appendix, Figure A2, to save space). Whereas three out of the four simplifications do not markedly reduce estimation accuracy, the evaluation reveals that the electoral competitiveness would be under-estimated if we were to omit re-sampling of residuals in Step I and instead take parties’ forecast election results for granted.

4.2 Capability to Predict Re-Election into Office

The second part of the validity evaluation consists of testing how well our probability measure performs in terms of predicting when an incumbent government will be ousted or re-elected into office. In fields where probabilistic forecasts are commonplace, such as epidemiology and meteorology, predictions of probabilities are often evaluated using Brier scores or other types of scoring rules (Brier 1950). That would have been a good practice also within this field, if most measures of electoral competitiveness could be interpreted as probabilities. Because this is not the case, we can only calculate Brier scores to compare the performance of different versions of our own measure. These comparisons are reported in the Online Appendix (Section A3).

Instead, to enable a meaningful comparison with previous measures, we here use a regression approach. Specifically, we evaluate our re-election probability of the incumbent as measured at four time points (t, $t-1$, $t-2$, and $t-3$) as well as the four simplified versions as measured at t, and we compare their predictive capability with that of previous measures of electoral competitiveness. The comparisons include 18 existing measures, which are previewed in Figure 4 and described more closely in the Online Appendix (Section A1.2). For some of the measures, we have made minor adaptations to make them applicable to the Swedish case.

Figure 4 Comparison of measures’ capability to predict re-election into office in Sweden (adjusted $R^2$).

To be clear, most of these measures are not devised specifically to predict re-election into office, but are based on other—mostly pre-electoral—conceptions of electoral competitiveness. Nevertheless, to the extent that one conceives of electoral competitiveness as a function of the probability that the incumbent executive will remain in power, it should follow that some predictive power with respect to that probability is a desirable feature of any measure of electoral competitiveness.

The bar chart in Figure 4 confirms that our measure represents a substantial improvement over all existing measures in terms of predicting re-election into office—a reassuring finding, considering how computationally demanding our measure is compared to the alternatives. For each measure, the bar in the darkest shade of gray represents the adjusted $R^2$ coefficient from an OLS regression of the seat share weighted re-election of the incumbent parties (see footnote 13) with the measure in question as a single predictor. Analogously, the two bars in lighter shades of gray represent the adjusted $R^2$ coefficients from OLS regressions of the re-election of the largest incumbent party into the governing coalition and—as a harder test—of the Mayor’s party into the Mayor’s office, respectively, with the measure in question as a single predictor. In these three models, our incumbent re-election probability (IRP) measure refers to that of (1) all incumbent parties weighted by their seat share, (2) the largest incumbent party, and (3) the Mayor’s party, respectively. The results are unambiguous: At each of the four forecasting horizons, our main measure represents a substantial improvement over each existing measure in each of the three models.

Another encouraging result from the top segment of Figure 4 is that, in line with expectations, our IRP measure performs better the shorter the forecasting horizon to the upcoming election. For the model of all incumbent parties, the adjusted $R^2$ score of the measure in May of the election year (t), at 0.231, represents a 20% improvement over the same measure 3 years prior to the election year ($t-3$), at 0.192. This confirms the added value of using vote intention polls—even on the national level—to update parties’ popularity perceptions.

The four simplified versions of the IRP measure all end up in the middle segment of the figure. This result implies that efforts to improve the quality of the models—in both steps of our approach—have the potential to pay off. However, it also shows that using even a very simple election forecasting model in Step I, or a fairly limited government formation model in Step II, generates an IRP measure with stronger predictive capability than any of the existing measures.

In conclusion, the validity evaluations have documented a number of attractive features of our election probability measure. First, the probabilities appear to be accurately estimated; there is no evidence of systematic over- or underestimation in relation to the actual probability of re-election at the point of measurement. Second, the measure shows substantial variation over the election cycle and its predictive capacity improves the as the forecasting horizon before the election shrinks. Third, irrespective of which forecasting horizon or which simplification we consider, the capability to predict re-elections is higher for our measure than for any other evaluated measure.

5 Cross-National Application

Whereas the Swedish case analyzed above is suitable for assessing the full potential of our approach, it leaves us wondering how well the approach works in less favorable settings. Our Swedish dataset contains more than a thousand election outcomes for each party, but in other cases, such as national elections in a cross-country setting, scholars may have to get by with just a handful. In such instances, it is important that the statistical relationships travel relatively well across institutional contexts, or else our re-election probability measure would quickly lose its predictive power.

To begin exploring the various challenges facing our approach in cross-national settings, we next apply it to 420 national elections in 34 parliamentary or semi-presidential democracies across Europe and the Anglo-Saxon world. This application is naturally more challenging than the one on Sweden. To begin with, the data are less comprehensive, both with respect to the number of observations and the availability of some variables. Moreover, the cases accommodate many more instances of new parties, party mergers and splits, and pre-electoral alliances, about which we do not always have full information.¹⁴ Furthermore, there is a much greater diversity in terms of the configuration of competing parties as well as the institutions surrounding elections and government formation. Thus, it is crucial to consider that, due to these institutional differences, electoral dynamics as well as government formation dynamics differ between countries.

In the present application, we limit ourselves to considering the perhaps most fundamental distinction in this regard among the observed countries; that between the 23 “old” Western democracies and the 11 “new” postcommunist democracies in Central and Eastern Europe (Döring and Hellström 2013; Savage 2016). In future work, our measures could surely be further refined by considering additional institutional variations, and by incorporating more case-specific sources of information. However, for the sake of simplicity, we base this application on a single data source—the ParlGov database (Döring and Manow 2019)—which includes the necessary data on election results and government composition, as well as some key party characteristics. Our only supplement is vote intention poll data for a subset of the elections, compiled by Jennings and Wlezien (2018).

To create a challenging setting, we strive to include as many of the observed elections as possible, yet some exemptions are necessary. First, we exclude Cyprus, Switzerland, and Turkey because of their presidential or hybrid systems of government. We also omit all elections prior to 1945 as well as each country’s two first elections after the Second World War, where we could expect extreme levels of electoral volatility, and 71 elections where there has been a party split or merger involving a party that received more than 25% of the votes in the previous election.

5.1 Key Modeling Considerations

We apply the same four-step procedure described above. To conserve space, the detailed modeling decisions are reported in the Online Appendix (Section A2), but some key differences between this and the Swedish application are worth mentioning here. In Step I, the largest difference is that our main specification does not include polling data when forecasting election outcomes (cf. Equation (2)). Instead, the party’s vote share is regressed only on variables derived from the ParlGov data, including, among others, a cubic spline of its previous vote share. Particularly, to account for the larger electoral volatility and the less institutionalized party system in postcommunist countries, we add interactions between an indicator for the new democracies and two indicators for previous vote share and for being a new party.

An advantage of this specification is that it relies only on the ParlGov dataset and requires no poll data. However, to improve forecast performance, we also run a second version of this model where, whenever possible, we simply replace the party’s previous vote share with its average vote share in all vote intention polls collected during the 365 days preceding the election date. For both versions, we use a simpler method for re-sampling residuals.

In Step II, there are four main differences regarding our government formation modeling. First, unlike in the Swedish case, we do not have the institutional knowledge regarding pre-electoral alliances, the existence of cordon sanitaires around particular parties, etc., for all 34 countries. Instead, to capture similar dynamics, we now introduce two new variables based on the individual parties’ history of incumbency and on whether combinations of parties have governed together before. Second, we need to consider the differences between old and new democracies, for instance that incumbent parties and established parties do not have the same bargaining advantage in new democracies (Döring and Hellström 2013; Savage 2016). Therefore, we include two interaction terms between an indicator for new democracies and the two aforementioned new variables. Third, we omit a number of variables that are either redundant or not possible to compute with the data at hand. Taken together, these changes result in a set of 18 potential government variables, which, when included in the coalition formation model produce in a model fit very similar to that of the Swedish application. Fourth, because of the relatively small number of elections in this dataset and the large heterogeneity among countries, we found it important to model the uncertainty surrounding the coefficients in the government formation model. To do that, we estimate a separate model for each simulation, run on a bootstrapped sample of government formation opportunities.

Step III and Step IV closely follow the Swedish application. As a result, we end up with three familiar sets of election probabilities to evaluate—for all parties, for the incumbent government where the incumbent parties are weighted by their seat shares, and for the largest incumbent party—measured with and without supplementary vote intention poll data.

5.2 Evaluation of Cross-National Measures

Our evaluation of the cross-national measures indicates that they perform well with regards to both empirical criteria we used before—accuracy of uncertainty estimation and predictive capability—although it is clear that they are not on par with those generated in the Swedish application.

Figure 5 evaluates the accuracy of the uncertainty estimation, in a manner corresponding exactly to Figure 3. The binned scatter plots indicate an acceptable level of accuracy, but also that the dispersion around the diagonal line is larger than in the Swedish application and that there is a slight tendency of our measure to underestimate the level of uncertainty.¹⁵ Like in the Swedish case, accuracy is not systematically affected by incorporating poll data (panel b).

Figure 5 Comparison between average estimated election probabilities and average outcomes, cross-nationally, with and without the use of vote intention poll data.

Next, with respect to the capability to predict re-elections, Figure 6 confirms that our cross-national measure does not perform as well as our Swedish measure; the adjusted $R^2$ parameters of 0.09–0.10 (without polls) and 0.12–0.13 (with polls) are considerably lower than those reported in Figure 4. Nevertheless, when compared to 16 other existing measures, it is clear that the predictive capability of most measures is reduced in this more challenging setting, and that our re-election probabilities still represent an improvement over existing measures.¹⁶

Figure 6 Comparison of measures’ capability to predict re-election into office cross-nationally (adjusted $R^2$).

6 Concluding Remarks

This paper has demonstrated a general approach to measuring electoral competitiveness for parties and governments, which incorporates both pre-electoral and post-electoral uncertainty into a joint measurement that lies close to the theoretical concept of interest and satisfies Kayser and Lindstädt’s (2015) six useful measurement criteria. We have applied the four-step procedure in a similar manner both on the favorable case of Swedish local government elections, to illustrate its full potential, and on a cross-country sample of national elections, to assess how well it works in a more challenging setting. Unsurprisingly, our evaluations have shown that the approach works better in the more favorable environment, yet also in the cross-national case we find that the predictive capability of our measure exceeds that of all existing measures.

Like any previous measure of electoral competitiveness, ours has certain disadvantages. Specifically, it is both complex and computationally demanding, and works best in fairly information-rich settings. Furthermore, it is mostly advantageous in analyses of multiparty democracies with parliamentary or semi-presidential systems of government, and for research questions where parties (rather than, for instance, individual candidates) are the actors of primary interest.

It should also be noted that even if our approach allows researchers to calculate election probabilities that are comparable across a multitude of institutional settings, it should be kept in mind that the implications of a given election probability for a party or a constellation of parties may differ across contexts. For instance, in a setting where power is concentrated to the executive branch of government, whether or not a political party is elected into office ultimately determines how much influence it will have during the coming election period. In contrast, in settings with many checks and balances, or where important policy decisions are dependent on deliberations in the parliament, what parties end up in the government may be of lesser importance.

As emphasized throughout the paper, the specific choices that need to be made when applying our approach depend on the context and on the available data. However, the key takeaway is a general one, namely that when devising measures of electoral competitiveness in multiparty systems, scholars will do well to carefully consider not only pre- but also post-electoral uncertainty.