Metaphorical framing in political discourse through words vs. concepts: a meta-analysis

Abstract

Conceptual metaphor theory and other important theories in metaphor research are often experimentally tested by studying the effects of metaphorical frames on individuals’ reasoning. Metaphorical frames can be identified by at least two levels of analysis: words vs. concepts. Previous overviews of metaphorical-framing effects have mostly focused on metaphorical framing through words (metaphorical-words frames) rather than through concepts (metaphorical-concepts frames). This means that these overviews included only experimental studies that looked at variations in individual words instead of at the broader logic of messages. For this reason, we conducted a meta-analysis (k = 91, N = 34,783) to compare the persuasive impact of both types of metaphorical frames. Given that patterns of metaphor usage differ across discourse domains, and that effects may differ across modalities and discourse domains, we focused on one mode of presentation and one discourse domain only: verbal metaphorical framing in political discourse. Results showed that, compared to non-metaphorical frames, both metaphorical-words and metaphorical-concepts frames positively influenced beliefs and attitudes. Yet, these effects were larger for metaphorical-concepts frames. We therefore argue that future research should more explicitly describe and justify which level of analysis is chosen to examine the nature and effects of metaphorical framing.

1. Introduction

An important debate in metaphor research deals with the relationship between linguistic and conceptual metaphors. Linguistic metaphors are generally defined as cross-domain mappings in the meaning of words. Conceptual metaphors are defined as clusters of cross-domain mappings in thought (Lakoff & Johnson, 1980). Lakoff and Johnson’s conceptual metaphor theory was one of the first theories to propose a connection between metaphors in language and cognition. They argued that individuals think about abstract concepts (e.g., ideas) in terms of other, often more concrete concepts (e.g., food) to fully understand these abstract concepts. Conceptual metaphors (e.g., ideas are food) thus facilitate the understanding of abstract concepts by connecting them to concepts that are, for instance, more familiar and meaningful to the individual. Consequently, conceptual metaphor theory posits that the use of metaphors is one of the most fundamental forms of reasoning (Lakoff & Johnson, 1980). Given this hypothesis that people reason metaphorically, metaphors in language (e.g., “digest information”, “meaty book”, “warmed-over theories”) are considered subsequent manifestations of these metaphors in thought.

Many experimental studies have examined the hypothesis that people reason in conceptual metaphors. Such studies typically focus on the effects of linguistic metaphors on individuals’ attitudes and behavior in the form of metaphorical frames. Metaphorical framing constitutes the idea that metaphors fulfill one or more functions of framing as defined by Entman (1993): to emphasize specific problems, causal relationships, moral evaluations, and/or solutions (Burgers, Konijn, & Steen, 2016; Semino, Demjén, & Demmen, 2018). Metaphors can fulfill these functions of framing by hiding and highlighting how the target domain of a metaphor (e.g., ideas) is similar to its source domain (e.g., food). When individuals adopt the line of reasoning as suggested by the metaphorical frame, this is considered evidence for theories like conceptual metaphor theory because these results demonstrate that individuals indeed use metaphors as an important mode of reasoning (Lakoff & Johnson, 1980). Previous overviews of such experimental studies have shown that, when linguistic metaphors are used to conceptualize a wide variety of issues or events, they are more persuasive than their literal counterparts (Sopory & Dillard, 2002; Van Stee, 2018).

However, conceptual metaphors do not necessarily have to be expressed through linguistic metaphors. In framing research for instance, a popular frame used to describe political elections is the ‘game frame’. The ‘game frame’ emphasizes who is winning versus who is losing in the polls (Aalberg, Strömbäck, & de Vreese, 2012). By focusing on the relative positions of political candidates in the polls, the ‘game frame’ draws from the conceptual metaphor of election is a race, without necessarily being manifested by linguistic metaphors like “run for office”, “neck and neck”, and “jockey for position”. Metaphor research is thus characterized by at least two levels of analysis: words vs. concepts (Renardel de Lavalette, Steen, & Burgers, in press). While previous overviews have presented a good amount of empirical evidence for the persuasiveness of metaphorical frames compared to non-metaphorical frames at the level of language (Sopory & Dillard, 2002; Van Stee, 2018), we still know little about how these effects compare to those at the level of concepts. We therefore argue that a comprehensive overview of previous experimental studies on metaphorical framing is imperative for understanding whether and how both levels of analysis are important approaches for advancing metaphor research.

Given that patterns of metaphor usage differ across discourse domains (Steen, Dorst, Herrmann, Kaal, & Krennmayr, 2010), and that the effects of metaphorical frames on individuals’ reasoning may differ across modalities (Sopory & Dillard, 2002; Van Stee, 2018), this study focuses on one mode of presentation and one discourse domain only to maximize the ability of the study to meaningfully compare effects. With regard to the discourse domain, we focus on political discourse because this domain has been widely recognized as inherently metaphorical (Lakoff, 2002). Moreover, some scholars assign considerable framing effects to the use of political metaphors such as “inherently persuasive power” (Charteris-Black, 2011, p. 44) and “home runs of persuasion” (Thibodeau, 2016, p. 53). With regard to modality, we focus on studies using verbal experimental stimuli because these studies most often constitute the starting point when bringing together approaches on metaphor and framing (e.g., Burgers et al., 2016; Semino et al., 2018).

This study thus compares metaphorical-framing effects on political persuasion through words vs. concepts. To this end, we conducted a meta-analysis because meta-analyses synthesize previous research on the same topic based on strict selection criteria and statistical rules to produce precise estimations of effects (Cumming, 2014). We focused on: (1) calculating a weighted effect size per study by giving effect sizes of studies with more precision more weight; and (2) determining whether this effect size depends on type of metaphorical frame and multiple study characteristics. Accordingly, by comparing metaphorical-framing effects across two levels of analysis (words vs. concepts), this study improves our understanding of the nature and effects of metaphorical framing.

2. Metaphorical reasoning

One of the reasons why conceptual metaphor theory suggests that the use of metaphors is one of the most fundamental forms of reasoning is that different conceptual metaphors are thought to promote different considerations about any topic under discussion (Lakoff & Johnson, 1980). This is often described in the metaphor literature as the “framing power” of metaphors (Semino et al., 2018, p. 627). For instance, the conceptual metaphor economic problems are weather is a common way to think and talk about the economy (e.g., “economic storm”, “monetary rain”, and “credit drought”). Following conceptual metaphor theory, scholars would hypothesize that this conceptual metaphor should lead individuals to believe that economic problems are always outside of our control. By contrast, the conceptual metaphor economic problems are diseases is also often represented in how people talk about this issue (e.g., “debt virus”, “financial pain”, “money injection”), and may instead lead individuals to believe that political actors are able to “cure” economic problems. Conceptual metaphor theory (Lakoff & Johnson, 1980) thus proposes that conceptual metaphors help people to make sense of the world around them, but often in different ways.

With regard to political discourse, the theory of moral reasoning (Lakoff, 2002) proposes that different political ideologies are constructed based on different metaphorical models. According to this theory, both conservatives and liberals reason metaphorically about the nation as a family, government as the parent, and citizens as children. However, conservatives typically follow a strict-father model, whereas liberals follow a nurturing-parent model (Lakoff, 2002). Strict fathers are characterized by a focus on self-discipline and self-reliance. Consequently, conservative governments generally focus on protecting the nation and maintaining order. Nurturing parents emphasize empathy and responsibility. Liberal governments therefore generally focus on promoting fairness and caring for individuals in need. According to the theory of moral reasoning, these two metaphorical models (i.e., the government as strict father vs. nurturing parent) primarily explain why conservatives and liberals favor different political policies (Lakoff, 2002).

3. Two levels of analysis

Previous research into the theory of moral reasoning (Lakoff, 2002) has identified the strict-father versus nurturing-parent metaphorical models using two different levels of analysis. While some scholars focus on the meaning of words at the level of words, other scholars focus on the level of concepts (Renardel de Lavalette et al., in press). Scholars focusing on words (e.g., Ahrens, 2011; Cienki, 2005) identify the two models by looking for metaphorical expressions that belong to one of the two metaphorical models. They commonly first compile a list of lexical items such as nouns, verbs, and adjectives associated with the source domains of the two models, like “strength” or “purity” (strict father) and “support” or “flourish” (nurturing parent). They then determine whether these lexical items are indeed used metaphorically in moral statements in political discourse. This means that conclusions about the presence of strict-father vs. nurturing-parent reasoning in political discourse are drawn exclusively based on the presence of metaphorical expressions that correspond to these metaphorical models.

By contrast, scholars focusing on concepts (e.g., Holman, 2016; Moses & Gonzales, 2015; Ohl, Pfister, Nader, & Griffin, 2013) identify the two metaphors by looking for references to general ideas associated with the two metaphors such as self-discipline (strict father; “working hard”, “being independent”) or kindness (nurturing parent; “working together”, “helping each other”). They first develop a taxonomy or list of examples of moral expressions for each metaphor based on Lakoff’s (2002) description of the two metaphorical models. They then categorize all moral statements in the political discourse that they examine following this taxonomy. This means that the presence of strict-father vs. nurturing-parent reasoning in political discourse is determined based on the presence of expressions that more generally reflect the semantic relations underlying the metaphorical models. These expressions can be metaphorical, but they do not need to be.

While both approaches measure the same phenomenon (i.e., metaphorical reasoning), the choice for one of the two levels of analysis (i.e., words vs. concepts) can lead to different outcomes when applied to the same data. Renardel de Lavalette et al. (in press) examined whether identifying the strict-father and nurturing-parent metaphorical models through words and through concepts could be considered equally appropriate ways of identifying the presence of moral reasoning in political discourse. The authors applied both methods to the same set of political speeches by US presidents Barack Obama and George W. Bush. Using the words approach, they found that Obama and Bush used a similar amount of nurturing-parent expressions in speeches delivered between 2001 and 2010, while, using the concepts approach, Obama used more nurturing-parent expressions than Bush. Because the choice of level of analysis can impact the presence of metaphorical framing in political discourse, this choice may also influence the presence of effects of metaphorical frames compared to non-metaphorical frames.

For this reason, the current study identifies studies on metaphorical framing at the level of words as well as concepts. Applying both approaches will provide a comprehensive overview of metaphorical-framing effects in political discourse by showing whether and how metaphorical frames influence political persuasion compared to non-metaphorical frames through words versus concepts. Furthermore, applying both approaches will help to determine whether or not the choice of level of analysis is a key factor in the presence of effects. In this paper, we from now on refer to the two levels of analysis (i.e., words vs. concepts) as: (1) metaphorical-words framing and (2) metaphorical-concepts framing.

Studies on metaphorical-words framing use frames that consist of metaphorical expressions in which one domain is described by means of another domain. An example is a study by Kalmoe (2014), who analyzed the effects of using fighting words (e.g., “battle”, “fight”, “attack”) to present politics as war (vs. a non-metaphorical description) on support for political violence. Similarly, Joris (2016) examined whether using war-related words (e.g., “under fire”, “losing ground”, “casualties”) to describe the euro crisis as war (vs. a literal description) decreased support for the European Union. The metaphorical frames in both studies were thus based on the conceptual metaphor of politics is war (Lakoff & Johnson, 1980). Studies on metaphorical-words framing are characterized by a focus in the method sections on the use of linguistic metaphors in the experimental conditions.

Studies on metaphorical-concepts framing are studies that use frames that represent a metaphorical understanding of the issue under investigation. Almost one in three political-framing experiments involves this form of metaphorical framing (Brugman, Burgers, & Steen, 2017). One of the clearest examples is the ‘strategy frame’. This frame is metaphorical because it compares political elections to military tactics by, for example, focusing on politicians’ strategies to win elections (Aalberg, et al., 2012). Another example of a metaphorical-concepts frame is the ‘conflict frame’ which uses the idea of a physical fight between groups of people or countries to describe the notion of having a difference in political opinion. While the ‘strategy frame’ and ‘conflict frame’ also reflect the conceptual metaphor of politics is war, they are not by definition made up of war-related words. Instead, these frames can also be recognized by expressions like “campaigning again each other” and “using the issue to gain votes” which can be associated with war-related concepts like enemies (i.e., political opponents) and weapons (i.e., arguments).

In order to examine whether both approaches to metaphorical framing yield different results and whether the findings of this meta-analysis therefore depend on scholars’ choice of level of analysis, the first two research questions of this paper read:

• RQ1: To which degree are metaphorical frames more likely to promote political persuasion than non-metaphorical frames?

• RQ2: How do the effects of metaphorical-concepts frames on political persuasion compare to the effects of metaphorical-words frames?

Many methodological challenges are involved in designing experimental studies on (metaphorical) framing. Debates about these challenges generally center around the questions of whether or not: (1) findings are identical when participant data are collected online rather than in a controlled research setting (e.g., Clifford, Jewell, & Waggoner, 2015); (2) findings are generalizable to the general population when student samples are used rather than population-based samples (e.g., Druckman & Kam, 2010); and (3) findings differ across countries (e.g., Machery, 2010). Moreover, discussions are ongoing about whether framing effects are similar across types of topics (e.g., Bechtel, Hainmueller, Hangartner, & Helbling, 2015). All of these study characteristics could thus potentially influence the presence and/or size of effects of metaphorical frames on individuals’ attitudes and behavior.

One of the main advantages of using a meta-analysis is that it is possible to test the influence of each of these factors separately. Study characteristics such as type of research setting and type of sample can be included in meta-analyses as moderators. In this way, we gain important insights into whether (or not) one of the two types of metaphorical frames is always more persuasive than the other, or only under certain conditions. For this reason, the last research question of this paper reads:

• RQ3: Which study characteristics moderate the effects of either type of metaphorical frame on political persuasion?

4. Method

4.1. identification of studies

We conducted a systematic database search and synthesis of the literature to identify relevant studies for the meta-analysis. Publication bias may threaten the precision of effect estimations of meta-analyses because negative and non-significant results could be under-represented in the sample (Cumming, 2014). Various studies have shown that published research is often characterized by publication bias, which is why excluding the “grey literature” (unpublished studies and studies published outside widely available journals) generally leads to overestimation of effect sizes (Levine, Asada, & Carpenter, 2009; Polanin, Tanner-Smith, & Hennessy, 2016). To avoid overestimation of effect sizes, meta-analysis experts strongly recommend including studies from a wide range of academic outlets: journal articles, books, but also doctoral dissertations and conference papers (Borenstein, Hedges, Higgins, & Rothstein, 2011; Rothstein, Sutton, & Borenstein, 2005). We therefore included studies on metaphorical framing in political discourse that were published in all four types of publication outlets. The time frame was set on 2000–2017 to update the meta-analysis by Sopory and Dillard (2002), who included studies that were published between 1983 and 2000. Sopory and Dillard showed that metaphorical frames are more persuasive than non-metaphorical frames, but they focused only on metaphorical-words frames. This study not only updates their meta-analysis to the twenty-first century, but also adds metaphorical-concepts frames as a second type of metaphorical framing.

Studies were identified in five steps.¹ First, a systematic database search was conducted for publication title, abstract, and keywords in multiple electronic databases.² The database selection allowed for the collection of relevant studies across different disciplines (e.g., linguistics, communication science, political science, psychology, economics). The search string included search terms for framing, metaphor, politics, and experiments, and contained wildcards to also find plurals and alternative spellings (see online ‘Appendix A’ for details³). For studies on metaphorical-words framing, metaphorical frames were identified by examining the ‘Method’ sections of the publications for any reference to the use of metaphors as the stimulus in the experimental conditions. For studies on metaphorical-concepts framing, metaphorical frames were identified by applying the Metaphor Identification Procedure Vrije Universiteit (MIPVU; see Steen, Dorst, Herrmann, Kaal, Krennmayr, & Pasma, 2010, for the coding instructions), one of the most reliable methods for identifying metaphor in language (Krennmayr, 2013), to the names of the frames employed in the studies that were coded as potentially relevant for the meta-analysis.

A frame was coded as metaphorical when the name of the frame implied a comparison between a political domain (target domain) and a non-political domain (source domain). The source domain was identified by searching for the name of the frame in the online version of the Macmillan Dictionary – British English (online <www.macmillandictionary.com>) and by establishing its basic meaning. The target domain was inferred from the authors’ account of the framing topics. Frames were coded as metaphorical when they had a more concrete and precise meaning than the topic under investigation. For instance, the ‘game frame’, used to describe political elections, was coded as metaphorical because sports competition is the source domain through which the target domain of political elections is interpreted. Similarly, the ‘view frame’ was coded as metaphorical because this frame compares political opinions to the notion of seeing landscapes such as oceans or mountains from a particular place. In contrast, the ‘ethical frame’, used to describe political values and ideas, was coded as non-metaphorical given that both the contextual meaning and basic meaning of the word ‘ethical’ refer to the principles used for deciding whether behavior is right or wrong. Another example of a non-metaphorical frame is the ‘issue frame’, because the frame uses political concepts rather than non-political concepts like sports or landscapes to explain the nature of issues and/or which role they play in politics.

In the second step of the identification of relevant studies, duplicates of both types of metaphorical-framing publications were removed. Third, abstracts were screened against the inclusion criteria (see below). Fourth, the ‘Method’ sections of the potentially relevant publications were read to determine definite relevance. Fifth, all relevant studies were coded for the variables of interest in this study.⁴ In this stage of the study, studies were included instead of publications because publications could contain more than one relevant experimental study (e.g., Steen, Reijnierse, & Burgers, 2014, contained four relevant experiments; for an overview of the number of publications included per stage of the study, see online ‘Appendix B’).

A study was included when the following inclusion criteria were met. First, the study consisted of two or more verbal experimental conditions. Second, at least one of the independent variables in the study was metaphorical framing. Third, at least one of the dependent variables represented political attitudinal or behavioral change in response to the experimental conditions (e.g., issue support, policy preference, political cynicism) to make sure that the topics of the studies were within the domain of political discourse. Fourth, the participants were native-speaking, healthy adults, and thus represented the majority of voters. Finally, the study included a non-metaphorical message condition as a baseline for comparison in the meta-analysis. Experimental studies without a non-metaphorical message condition (e.g., Robins & Mayer, 2000; Thibodeau & Boroditsky, 2011) were excluded.

Also, when studies were published in multiple publication outlets, we included the study that was likeliest to be peer reviewed. For instance, a study published in a journal article was chosen over the same study published in a dissertation. When a study included a cross-country comparison, the samples from the different countries were coded as different studies when possible (e.g., United Kingdom and Bulgaria; Abbarno & Zapryanova, 2013). Finally, in the case of multiple frame exposures or repeated observations (e.g., Chong & Druckman, 2007), the analysis was limited to the first frame exposure or first observation only. This selection procedure yielded 91 relevant studies from 63 publications: 48 journal articles (68 studies), 1 book chapter (1 study), 9 doctoral dissertations (16 studies), and 5 conference papers (6 studies) published between 2000 and 2017 (for an overview, see online ‘Appendix C’).

4.2. variable coding

We classified the dependent variables reported in each study according to the theory of planned behavior (Ajzen, 1991), one of the most broadly applied theories to explain attitudinal and behavioral change. The theory of planned behavior predicts that behavior is determined by multiple factors: beliefs, attitudes, subjective norm, perceived behavioral control, and behavioral intention. See Table 1 for coding details.

table 1. Coding categories and reliability scores per variable

notes: κ = Cohen’s kappa; % = percent agreement; dependent variables were binary coded (1 = yes, 0 = no).

We also coded two types of study characteristics to include as moderator variables in the meta-analysis: (1) context factors and (2) framing topics. Context factors were variables associated with assessing publication bias (publication year, publication outlet) and with the challenges involved in designing experimental studies (type of sample, research setting, and cultural setting). The possible categories for the framing topics were chosen based on common newspaper categories: economic policy, foreign (vs. domestic) politics, health and environment, science and education, and security and justice. Multiple categories could apply to each study. Landau, Keefer, and Rothschild (2014), for example, focused on the issue of financial crisis as their study topic, which concerned both foreign politics and economic policy. See Table 1 for coding details.

4.3. inter-coder reliability

Inter-coder agreement was established by the first author with the assistance of a second coder (see Table 1 for the full results). After double coding 75% of the dataset (69 studies), the two coders reached “almost perfect” inter-coder agreement (Landis & Koch, 1977) for 14 out of 19 variables (e.g., participant age, student sample). The Cohen’s kappa scores for these variables ranged from 0.81 to 1.00. Inter-coder agreement was “substantial” (Landis & Koch, 1977) for the remaining five variables. The Cohen’s kappa scores for these variables ranged from 0.62 to 0.79. Disagreements were resolved by means of discussion between the coders.

4.4. analysis strategy

We analyzed the data using the Comprehensive Meta-Analysis (CMA) software (version 3.3.070; Borenstein et al., 2011). For each dependent variable, we calculated an effect size (i.e., Cohen’s d, a common measure of standardized mean difference) to compare the effects of metaphorical frames on political persuasion to those for non-metaphorical frames. The statistical measures that were most often reported were means and standard deviations per experimental condition. When necessary, we sent the authors of the publications data requests.

Our sample included metaphors on a wide variety of political topics. These metaphors implied either a positive or a negative position towards certain attitudes or behaviors. For instance, Corner and Pidgeon (2015) studied the effects of describing geo-engineering as a natural phenomenon (‘natural analogy frame’). This metaphorical frame was meant to make geo-engineering seem like a common, natural thing, and was thus meant to positively impact recipients’ attitude towards geo-engineering. By contrast, when D’Angelo and Lombard (2008) studied the effects of describing elections in terms of military strategies (‘strategy frame’), this metaphorical frame was meant to make elections seem like a combative, hostile thing, and was therefore meant to negatively impact attitudes towards news.

To establish the expected direction of each metaphorical frame, we relied on the descriptions of materials and, when available, hypotheses included in the original publications. To be able to deal with the diversity in metaphors and political positions, we coded the effect direction as positive when the dependent variable shifted in line with the position implied by the metaphorical frame (e.g., when the natural analogy frame led to a more positive attitude towards geo-engineering, or when the strategy frame led to a more negative attitude towards news), and negative when the dependent variable shifted in the other direction (e.g., when, contrary to expectations, the natural analogy frame led to a more negative attitude towards geo-engineering, or when the strategy frame led to a more positive attitude towards news).

Next, we calculated a weighted effect size per study because 48 of the 91 studies reported more than one relevant dependent variable. By calculating a weighted effect size per study, we ensured that participants would be counted only once. Finally, we wanted to apply a model to our data that would allow for generalizing findings beyond the specific studies included in the meta-analysis. A meta-analysis can be conducted using either a fixed-effects model or random-effects model. Compared to a fixed-effects model, random-effects models are more conservative. Random-effects models assume that there is not one true effect size for the studies combined, but a distribution of true effects sizes that reflects both within-study error and between-study error (Borenstein et al., 2011). The studies included in this meta-analysis differed in their research aims and designs, which makes the existence of one true effect size unlikely. For this reason, we applied a random-effects model to our data.

5. Results

The experimental studies on metaphorical framing in political discourse included in this meta-analysis were characterized by the following distributions in study characteristics. First, 34.1% of the studies used student samples and 15.4% were conducted in a controlled research setting. No less than 65.9% of studies were conducted in the United States. The other studies were conducted in a diverse set of mostly European countries. Furthermore, 34.1% of studies involved the topic of economic policy, 30.8% foreign (vs. domestic) politics, 19.8% science and education, 36.3% health and environment, and 33.0% security and justice.

With regard to the types of dependent variables measured, we found that 19.7% of the 193 dependent variables included in the meta-analysis were beliefs (e.g., factual beliefs, blame attribution, threat perception), 73.6% were attitudes (e.g., political cynicism, policy preference), and 6.2% were behavioral intentions (e.g., turnout intention). Put differently, 23 studies measured one or more belief variables, 79 studies measured one or more attitude variables, and 10 studies measured one or more intention variables. Only one study measured perceived behavioral control as epistemic political efficacy, which refers to individuals’ confidence in their own ability to determine the truth in politics (Pingree, Hill, & McLeod, 2013). The other two variables – subjective norm and actual political behavior – were not measured.

RQ1 addresses the overall degree to which metaphorical frames promote more political persuasion than non-metaphorical frames. The analysis demonstrated that the Cohen’s d for all 91 studies included in the meta-analysis was 0.11 (N = 34,783, SE = 0.02, 95% CI [0.07, 0.14], Z = 6.19, p < .001; see online ‘Appendix D’ for an overview of the overall effect sizes and study characteristics per study). As expected, the effect sizes were not homogeneous (I²(90) = 69.36, p < .001), because relevant studies were identified through both a focus on words and concepts, and the studies therefore differed in their research aims and designs. Nevertheless, this outcome revealed that both types of metaphorical frames combined were more persuasive than non-metaphorical frames. Following Cohen (1988), effect sizes (e.g., Cohen’s d, Hedges’ g) around 0.20 and below are small in magnitude, those around 0.50 are medium, and those around or above 0.80 are large. The effect size of metaphorical (vs. non-metaphorical) frames on political persuasion should thus be classified as statistically small.

We assessed potential differences in overall metaphorical-framing effects due to differences in participant characteristics. On average, 54% of the participants in the studies included in this meta-analysis were female. The average age was 35 years. The results showed no moderating effect of the percentage of female versus male participants per study (β = –0.00, SE = 0.00, k = 74, N = 26,215, 95% CI[–0.00, 0.00], p = .72, z = –0.35), or of the average participant age per study (β = –0.00, SE = 0.00, k = 59, N = 20,915, 95% CI[–0.01, 0.00], p = .22, z = –1.23). Other relevant participant characteristics such as political preference or political knowledge were not measured in sufficient studies to include in the meta-analysis. Hence, based on the data that were available, no participant characteristics were found to influence metaphorical-framing effects on political persuasion.

When we zoom in on the different types of dependent variables that were measured often enough in the original publications to study in the meta-analysis, we find differences in effect sizes. Combined, the effects of both types of metaphorical frames compared to non-metaphorical frames were significantly larger on beliefs, with a Cohen’s d of 0.29 (k = 23, SE = 0.09, 95% CI [0.12, 0.46], Z = 3.31, p < .01) than on attitudes with a Cohen’s d of 0.10 (k = 79, SE = 0.02, 95% CI [0.06, 0.14], Z = 5.23, p < 0.001) (Q = 4.38, p < .05). The results demonstrated no significant effect size for metaphorical-framing effects on intentions because the confidence interval included zero (k = 10, d = 0.10, SE = 0.13, 95% CI [–0.02, 0.27], Z = 1.69, p = .09). We would like to emphasize, however, that studies on both types of frames were not equally represented in this analysis. We will therefore also conduct this analysis for each type of metaphorical frame separately.

RQ2 asked whether metaphorical-words frames are more persuasive in political discourse than metaphorical-concepts frames, or vice versa. We found that effects were larger in studies on metaphorical-concepts framing than in studies on metaphorical-words framing. The Cohen’s d for the 36 studies on metaphorical-concepts framing was 0.22 (N = 14,667, SE = 0.04, 95% CI [0.15, 0.29], Z = 6.01, p < .001), whereas the Cohen’s d for the 55 metaphorical-words framing studies was only 0.04 (N = 20,116, SE = 0.01, 95% CI [0.01, 0.07], Z = 2.67, p < .01). These effect sizes, as well as the difference between the effect sizes, were significant (Q = 21.15, p < .001). Accordingly, when we compared the effect sizes of metaphorical frames to those of non-metaphorical frames, metaphorical-concepts frames were found to be the most persuasive type of metaphorical frame.

We found one difference in effect size between dependent-variable types. Metaphorical-words frames had a significantly larger effect on beliefs, with a Cohen’s d of 0.27 (k = 16, SE = 0.11, 95% CI [0.06, 0.48], Z = 2.55, p < .05), than on attitudes, with a Cohen’s d of 0.03 (k = 44, SE = 0.02, 95% CI [0.00, 0.06], Z = 2.12, p < 0.05) (Q = 4.90, p < .05). No significant effect size was found for intentions (k = 5, 95% CI [–0.10, 0.16], Z = 0.40, p = .69). The effect size of metaphorical-concepts frames on beliefs was a Cohen’s d of 0.33 (k = 7, SE = 0.15, 95% CI [0.04, 0.62], Z = 2.23, p < .05), compared to a Cohen’s d of 0.20 for attitudes (k = 35, SE = 0.04, 95% CI [0.13, 0.27], Z = 5.30, p < .001), but this difference in effect size was not significant (Q = 0.73, p = .39). Again, no significant effect size was found for intentions (k = 5, 95% CI [–0.02, 0.51], Z = 1.84, p = .07). These results were in line with our earlier results showing that metaphorical frames were likely to influence beliefs and attitudes in political discourse, but not intentions.

RQ3 focused on which study characteristics moderate the effects of either type of metaphorical frame on political persuasion.⁵ The results of the analyses for the studies on metaphorical-words frames showed that the effects of metaphorical-words frames on political persuasion were not moderated by any context factors (i.e., type of sample, research setting, cultural setting) or framing topics (i.e., economic policy, foreign (vs. domestic) politics, health and environment, science and education, security and justice; see Table 2). While the effects of metaphorical-words frames on political persuasion can only be considered small according to Cohen’s (1988) guidelines, this effect size proved to be relatively consistent.

table 2. Moderating effects for the persuasiveness of metaphorical-words framing (vs. non-metaphorical framing)

notes: d = Cohen’s d; k = number of studies; N = number of observations; Q = chi-square statistic for moderators; * p < .05; ** p < .01.

Finally, the effects of metaphorical-concepts frames on political persuasion were moderated by one study characteristic only: the framing topic of science and education (see Table 3). When a study topic was unrelated to science and/or education, metaphorical-concepts frames had a larger persuasive effect than non-metaphorical frames (d = 0.23, 95% CI [0.15, 0.31]) compared to when the topic concerned science or education-related issues such as the climate change debate (d = 0.11, 95% CI [0.04, 0.18]; Q = 4.87, p < .05). The effect size of metaphorical-concept frames did not change depending on context factors (i.e., type of sample, research setting, cultural setting) or other framing topics (i.e., economic policy, foreign (vs. domestic) politics, health and environment, security and justice). While according to Cohen’s (1988) guidelines the effect of metaphorical-concepts frames on political persuasion was again only small, it seemed to be quite consistent.

table 3. Moderating effects for the persuasiveness of metaphorical-concepts framing (vs. non-metaphorical framing)

notes: d = Cohen’s d; k = number of studies; N = number of observations; Q = chi-square statistic for moderators; * p < .05; ** p < .01.

5.1. publication bias analyses

Publication bias was formally tested by means of three measures: (1) Begg and Mazumdar’s (1994) rank correlation test, (2) Egger’s regression (Egger, Smith, Schneider, & Minder, 1997), and (3) trim and fill analysis (Duval & Tweedie, 2000). Both the Begg and Mazumdar rank correlation test and Egger’s regression report whether there is a linear association between sample size and effect size. Publication bias could be present when smaller studies are more likely to be published when they have larger than average effect sizes. This is the case because studies with larger than average effects are more likely to meet the criterion for statistical significance. The trim and fill analysis reports funnel plot asymmetries and estimates how many studies may be missing from the meta-analysis to adjust for publication bias.

The Begg and Mazumdar rank correlation test indicated that sample size and effect size were indeed related (Kendall’s tau = 0.23, z = 3.17, p < .01). Similarly, Egger’s regression showed that the relationship between sample size and effect size was significant (b = 1.26, SE = 0.36, 95% CI [0.54, 1.98], t(89) = 3.48, p < .001). However, the trim and fill analysis found only one potentially missing study on the left side of the funnel plot. This is a study that would have found negative and/or non-significant results. When this unpublished file drawer study would have been present in the dataset, the trim and fill analysis estimated that the Cohen’s d would not change and would remain at the value of 0.11 (95% CI [0.07, 0.14]).

We also included whether or not a study was published in a journal article as a moderator in the meta-analysis. The results showed that this variable potentially moderated metaphorical-framing effects in political discourse. Effect sizes were larger when studies were published in a journal (d = 0.13, SE = 0.02, k = 68, N = 28,584, 95% CI [0.09, 0.17]) than when they were published in another publication outlet (d = 0.05, SE = 0.04, k = 23, N = 6,199, 95% CI [–0.03, 0.12]). However, this difference was only marginally significant (Q = 3.37, p = .07). We also found a marginally significant moderating effect of publication year (β = –0.01, SE = 0.00, p = .09), indicating that effect sizes may have decreased slightly over time.

An additional way to assess publication bias is to examine how scholars deal with pressures to publish. One type of questionable research practice that is often considered a consequence of publication bias is p-hacking; the phenomenon in which some scholars “play” with their data until they find the desired effects (e.g., by selectively excluding outliers; Simonsohn, Nelson, & Simmons, 2014). We conducted a p-curve analysis to determine the evidential value of the studies included in our meta-analysis. When a p-curve is skewed to the left, this typically indicates p-hacking because most p-values are high rather than low. Our p-curve was clearly skewed to the right (Z = –12.29, p < .0001) (see online ‘Appendix F’ for the p-curve figure). This means that no evidence was found for the presence of p-hacking in experiments on metaphorical framing in political discourse. Overall, the publication bias analyses produced mixed results, but in the end suggest that publication bias could have somewhat influenced the outcomes of this meta-analysis.

6. Conclusion and discussion

The findings of this meta-analysis show that metaphorical frames are more persuasive than non-metaphorical frames in political discourse, irrespective of the specific frame type (RQ1). We found an overall Cohen’s d of 0.11 for both types of metaphorical frames combined, with the largest effects found on beliefs (d = 0.29), followed by attitudes (d = 0.10), and no effects on intentions (d = n.s.). The results also revealed a difference in effect size depending on the level of analysis (RQ2). Compared to non-metaphorical frames, metaphorical-concepts frames were found to promote over five times more political persuasion (d = 0.22) than metaphorical-words frames (d = 0.04). Only one study characteristic was found to serve as a moderator, and only for metaphorical-concepts frames (RQ3). The effects of metaphorical-concepts frames versus non-metaphorical frames were about twice as large when the framing topic was related to economic policy, foreign (vs. domestic) politics, health and environment, and security and justice (d = 0.23) than science and education (d = 0.11). Metaphorical frames thus produce relatively consistent effects on beliefs and attitudes.

6.1. theoretical implications

These findings have three major implications. First, our meta-analysis demonstrates that studying metaphorical framing at both levels of analysis is necessary to advance metaphor research because its outcomes suggests that metaphors can influence individuals’ reasoning through words as well as concepts. Accordingly, studies on both types of metaphorical frames seem to provide support for important metaphor theories like conceptual metaphor theory (Lakoff & Johnson, 1980) and the theory of moral reasoning (Lakoff, 2002), which propose that the use of metaphors is one of the most fundamental forms of reasoning. Although in different degrees, both types of metaphorical frames have been shown to be more persuasive than non-metaphorical frames in political discourse.

Second, the effect sizes found in this study challenge popular claims made about the effects of metaphors in political discourse such as “home runs of persuasion” (Thibodeau, 2016, p. 53). We found only a statistically small effect size of the effects of metaphorical frames compared to non-metaphorical frames on beliefs and attitudes. Moreover, while these claims tend to refer to the effects of linguistic metaphors on individuals’ attitudes and behavior, the findings of our study demonstrate an even smaller effect size for metaphorical-words frames than for metaphorical-concepts frames. This means that scholars should be careful not to overestimate the persuasiveness of metaphors. However, we would also like to emphasize that small effect sizes found in artificial research settings could still have substantial effects in real life. For instance, in certain political systems, receiving only a few more election votes can decide who becomes the country’s leader. In the right time and place, the right metaphor may make a difference nonetheless.

The difference in effect size between metaphorical-words frames and metaphorical-concepts frames can be explained by looking more closely at the stimulus texts used in both types of studies. In studies on metaphorical-words framing, the framing manipulations consist only of variations in a small number of words. By contrast, the framing manipulations in studies on metaphorical-concepts framing are more generally driven by the comparison between source and target domains. While metaphorical-words frames are expressed only by a certain number of metaphorical expressions that may promote metaphorical reasoning, metaphorical-concepts frames actually represent a metaphorical understanding of a particular topic as a whole. As a result, metaphors may constitute bigger building blocks of metaphorical-concepts frames. This could also explain why metaphorical-concepts frames seem to promote more metaphorical reasoning than metaphorical-words frames.

We would like to point out three reservations with regard to this second implication. First, the fact that we only looked at metaphorical framing in political discourse may have influenced our findings. Since patterns of metaphor usage differ across discourse domains (Steen, Dorst, Herrmann, Kaal, & Krennmayr, 2010), and political discourse is an inherently metaphorical domain (Lakoff, 2002), many metaphors presented in the metaphorical-framing studies may already play an important role in how individuals reason about politics. Yet, attention to metaphors as metaphors is by some scholars considered to be a necessary condition for metaphors to impact attitudes and behavior (Steen, 2017). Thus, metaphors may have larger effects only when they provide new ways of reasoning about a topic.

Second, we included only experimental studies using textual manipulations of metaphorical frames, while different modes of presentation could guide frame processing in different ways (Geise & Baden, 2014). For instance, a large body of experimental studies on the embodiment of conceptual metaphors such as affection is heat (Williams & Bargh, 2008), importance is weight, (Chandler, Reinhard, & Schwarz, 2012), and morality is cleanliness (Lee & Schwarz, 2010) has demonstrated that conceptual metaphors may also be grounded in experience. We therefore recommend that future research extends this meta-analysis on previous research of metaphorical-framing effects to other discourse domains and other modes of presentation such as the sensory-motor modality to evaluate the generalizability of the findings of this specific study to metaphor research.

Finally, the third implication of the findings of this meta-analysis is that future research on the effects of metaphorical frames should more explicitly distinguish between different types of dependent variables. In line with theories such as the theory of planned behavior (Ajzen, 1991) that predict that some variables are easier to influence than others, we found a difference in persuasiveness of metaphorical-words frames depending on dependent-variable type. The effects on beliefs were larger than on attitudes, and no effects on behavioral intentions were found. Given this difference, future research should further explore how the persuasiveness of metaphorical frames can differ between subtypes of dependent variables. For instance, the theory of planned behavior distinguishes between three types of beliefs: (1) behavioral beliefs: perceptions about what constitutes the behavior of interest; (2) normative beliefs: perceptions about the opinions of significant others regarding the behavior of interest; and (3) control beliefs: perceptions about factors that may help or hinder individuals’ ability to engage in or refrain from specific behavior (Ajzen, 2002). Metaphor research would benefit from a more comprehensive analysis of which particular aspects of individuals’ reasoning are most likely to be influenced by metaphorical frames.

6.2. study limitations

This study’s limitations are closely connected to the known limitations of the method of meta-analysis. The first limitation is the comparability of studies. Because the studies that were synthesized in this meta-analysis differed in their research aims and designs, effect sizes were not homogenous (which is not uncommon for meta-analyses; Borenstein et al., 2011). This means that the patterns of effects that were found were not completely consistent, which may reduce the generalizability of findings. However, we have mitigated this issue as much as possible by applying a random-effects models to the data, by conducting moderator analyses of relevant study characteristics, and by reporting the outcomes of the meta-analysis for the two types of metaphorical framing studies separately. In this way, we demonstrate that the findings of this meta-analysis are still sufficiently consistent across study types.

The second limitation is that the results of this study may have been influenced by publication bias. We used three measures to formally assess publication bias in our sample. Even though we included studies from a wide range of academic outlets, including doctoral dissertations and conference papers, two of the three measures indicated that publication bias may have threatened the conclusions drawn from this meta-analysis. Furthermore, effects were potentially larger when studies were published in a journal compared to other publication outlets. For this reason, the effects of metaphorical frames on political persuasion could in reality be smaller than we have demonstrated in this paper.

The third limitation of this meta-analysis is that we were able to distinguish between only three categories of dependent variables: beliefs, attitudes, and behavioral intentions. This means that relatively diverse sets of dependent variables were coded as belonging to the same conceptual categories, which could account for some of the observed heterogeneity across studies. However, effect sizes could not be computed per specific type of dependent variables (e.g., political cynicism, policy preference) because they were not measured often enough in the original publications for each type of metaphorical frame. We therefore instead classified dependent variables bases on conceptual similarity.

Even though these limitations warrant some caution in the interpretation of the study’s findings, they do not undermine the theoretical and empirical value of this meta-analysis. It is important to conduct meta-analyses because they help scholars build a cumulative base of knowledge (Cumming, 2014). This specific meta-analysis incorporated studies from a wide range of publication outlets from a diverse set of disciplines that would otherwise be rarely brought together (e.g., linguistics, communication science, political science, psychology). This study therefore offered a unique opportunity to critically evaluate and rigorously reinterpret previous research on the topic of metaphorical framing in political discourse.

In sum, the purpose of this study was to evaluate whether the presence and size of metaphorical-framing effects on political persuasion depend on the choice of level of analysis. This paper presents evidence supporting the importance of taking both the level of words and the level of concepts into account when studying metaphorical framing. With a focus on concepts rather than words, studies on metaphorical-concepts frames yielded larger effects than studies on metaphorical-words frames, suggesting that each level of analysis provides its own insights when experimentally testing the premises of important metaphor theories such as conceptual metaphor theory (Lakoff & Johnson, 1980) and the theory of moral reasoning (Lakoff, 2002). In line with previous research that brought together approaches on metaphor and framing (e.g., Burgers et al., 2016; Semino et al., 2018), we therefore argue that the two levels of analysis need to be considered complementary. Accordingly, we urge scholars in this field to more explicitly describe and justify the approach on which their research is based.