You Won’t Believe Our Results! But They Might: Heterogeneity in Beliefs About the Accuracy of Online Media

Abstract

“Clickbait” media has long been espoused as an unfortunate consequence of the rise of digital journalism. But little is known about why readers choose to read clickbait stories. Is it merely curiosity, or might voters think such stories are more likely to provide useful information? We conduct a survey experiment in Italy, where a major political party enthusiastically embraced the esthetics of new media and encouraged their supporters to distrust legacy outlets in favor of online news. We offer respondents a monetary incentive for correct answers to manipulate the relative salience of the motivation for accurate information. This incentive increases differences in the preference for clickbait; older and less educated subjects become even more likely to opt to read a story with a clickbait headline when the incentive to produce a factually correct answer is higher. Our model suggests that a politically relevant subset of the population prefers Clickbait Media because they trust it more.

Utility and media choice

A central parameter in the study of political media is the process by which the public selects media to consume. The degree to which media exposure happens incidentally or as the result of an explicit choice varies with both the individual and the larger sociotechnical context, and while incidental and explicit exposure are both large components of the media diet, the latter is particularly relevant in the “hyperchoice” context of contemporary online media (Arceneaux and Johnson 2013; Messing and Westwood 2012).

Faced with either a blank search bar or a “feed” of articles on a social media platform (e.g., Facebook) or general interest web portal (e.g., MSN News), the citizen must take some action. She can “scroll” through the feed – a prominent form of online behavior akin to browsing newspaper headlines but far more extensive in terms of content (Settle 2018) – but the central act of media consumption only takes place after she makes a selection or “clicks.” Indeed, fewer and fewer people navigate directly to media companies’ homepages, and even then they have to select an article to read or video to watch (Guess 2018). This shift in the location of media choice has changed the economics of the media industry to what Munger (2019) describes as “Clickbait Media.”

The increasing centrality of consumer choice has not been met with sufficient attention by scholars of media effects. While the wider political communication and journalism literature has addressed this topic (Hamilton 2004; Webster 2014), and there are a number of excellent books on the topic in the context of cable news (Arceneaux and Johnson 2013; Prior 2007; Stroud 2011), the political behavior literature has not fully adapted to the centrality of “clicking” in online media. ¹ . Still, Sood and Lelkes (2018) identify a key theoretical limitation. The robust evidence of the preference for information from congenial sources conflates two possible causes: the preference for congenial information and the belief in the greater accuracy of congenial sources.

Here, we present results from an experiment that transcends this limitation. We collect a sample of Italian Facebook users through a paid advertisement. The sample is thus not representative of the general population but should be quite representative of the actual population of interest: people who click on links on Facebook. Italy is a useful case study because it has in many ways been ahead of the rest of the West in terms of disillusionment with the media establishment, and the ensuing rise to power of anti-establishment politicians who embrace the credibility-through-virality that clickbait entails. ²

We first present subjects with a series of non-experimental choice tasks to estimate their individual “preference for clickbait,” and the distribution of this propensity across the sample population. This portion of the design corroborates – in a novel national context – the findings in Munger et al. (2020) that the elderly, the less educated, and the less digitally literate have a higher “preference for clickbait”; we also find this to be higher among women, frequent internet users, and supporters of the ruling coalition.

For the experiment, we present subjects with a similar choice task, but also inform them that they will read the article they rank highest. Subjects are then randomly assigned into one of two treatment conditions: being told there will be a quiz on the topic discussed in the article or being told there will be a quiz and that providing answers that are “closer to the truth” will increase their chance of winning a lottery for a financial prize; the control group is simply told they will read the article. ³

We find no evidence of direct effects of the addition of the financial incentive, but this masks significant and offsetting heterogeneous treatment effects. The treatment increased the preference for clickbait among the elderly, the less educated, and less digitally literate, while at the same time it decreased the preference for clickbait among those on the other ends of these respective distributions. The heterogeneity in “preference for clickbait” thus increases in the financial incentive for accuracy. This finding contrasts with the results in Prior and Lupia (2008) and Prior et al. (2015), which find that increasing financial incentives decreases heterogeneity in civic knowledge and perceptions of the economy. Even more importantly, the result provides strong evidence for the hypothesis that people who prefer clickbait headlines do so because they consider them more likely to be accurate. More generally, this finding supports a theory of media choice that more heavily weights perceived source accuracy than preference for congenial information.

The troubling implication of this finding in the current context is that people may not be opting to consume low-quality news sources because they are being cognitively lazy (Pennycook and Rand 2019) or because they lack the savvy to differentiate low and high quality news. Instead, our results suggest that – for a certain subset of elderly, pro-government, less educated individuals – they are opting to consume low-quality news sources because they actually trust them more.

Theory and hypotheses

Clicking on a link is central to the process of online media consumption and this is reflected in the behavior of online media firms. This creates the market dynamics of “Clickbait Media” in which each online news story must compete with thousands of other stories for the audience’s attention. Hundreds of new online media firms employ a novel business model enabled by two interrelated features of Clickbait Media: the sharing or retweeting of posts along homophilous social networks and the quantified accretion of these decisions in the form of the number of likes (or shares, or retweets) associated with a given post.

The structural reasons for the emergence of Clickbait Media are complex and outside the scope of this paper. But there exists an intrinsic connection between the business model of Clickbait Media and the style of headline that gives it that name. We argue that this style serves as a signal to readers that the media firm is not aligned with the legacy media establishment. However, it is also a signal of their business model. While the traditional media relies on “credibility” derived from journalistic standards, the “credibility cascades” of Clickbait Media obviate these standards.

The role of social recommendation in online media choice is thus larger today than in previous media contexts, but other theories of media choice remain very much relevant. Sood and Lelkes (2018) summarize the state of this literature, explaining the empirical regularity of consumers’ preference for congenial sources. The key argument is that researchers need to conceptually differentiate preference for congenial sources from preference for congenial information. The former is well-established but often naively assumed to be caused by the latter; Sood and Lelkes (2018) make the case that the preference for congenial sources is also a function of perceived accuracy.

The other major change produced by contemporary online media is the massive expansion in the number of media firms and articles from which to choose. The expansion of the media choice set reveals previously unobservable heterogeneities in consumer preference; considering Prior’s argument that the roll-out of cable television revealed that many Americans had a high “preference for entertainment” and thus opted out of viewing any television news if given the option of more and better entertainment programs (Prior 2007).

In the early days of the internet, online headlines were no different from newspaper headlines. But the development of “clickbait” headlines by web native media firms like Upworthy in the early 2010s introduced a new dimension of preference heterogeneity. Munger et al (2020) demonstrate that this “preference for clickbait” varies widely among the US population. Republicans, the elderly, and the less educated tend to prefer clickbait headlines, holding the topic constant and in the absence of source or social recommendation cues. Possible explanations for this variation are simply that these people are less “digitally literate” (Hargittai 2001; Hargittai, Piper, and Morris 2018) or lower in analytical reasoning (Pennycook and Rand 2019), and also posited as explanations for the related propensity to consume “fake news” (Guess, Nagler, and Tucker 2019).

The key limitation of these studies is the same described in Sood and Lelkes’ summary of broader theories of selective exposure: an inability to differentiate between theoretically distinct mechanisms underlying media preferences (Sood and Lelkes 2018).

Our approach to address this problem is to adapt the framework used in Prior and Lupia (2008) and Prior et al. (2015) and experimentally add incentives for correct answers, thereby raising the relative salience of the motivation for accuracy. In Prior et al. (2015), the behavior of interest is how people respond to survey questions that probe their perception of economic conditions. As in our context, these respondents face incentives that can be in conflict: the desire to be accurate and the desire to report congenial information. In these studies, the addition of an incentive for accuracy decreases the heterogeneity of the responses, suggesting that in the absence of the desire to report congenial information, people’s estimates would be centered on the truth.

The situation with contemporary media choice is different. Our theory is that some people have a preference for clickbait that is linked to a sense that legacy media is aligned with the distrusted political establishment, and that it is only from new online sources that employ the distinctive style of partisan emotional clickbait that can be trusted. Some other people have a preference for non-clickbait based on the analogous idea that only information from vetted and reputable media outlets can be trusted. ⁴ Our hypothesis is that the addition of incentives for accuracy will increase the heterogeneity in media choice, along the dimensions that predict preference for clickbait in the pre-experimental phase.

First, we aim to corroborate findings about this preference for clickbait from the US context: it is higher among the elderly, the less educated, and the less digitally literate. Here, we follow the conception of “digital literacy” proposed by Guess and Munger (2020): information discernment combined with the basic digital skills necessary to attain it. This is an active and unsettled area of research, but at the time we conducted the experiment, we combined the 6-question “abbreviated battery” proposed in Hargittai, Piper, and Morris (2019) with the 8-question “social media” battery described by Hargittai, Piper, and Morris (2019) was the most current and relevant measurement of the concept. See Appendix Section 10 for a full discussion; note that there was an error in our survey instrument and that we only in fact included 5 of the 6-question battery.

Hypothesis 1 Preference for clickbait will be higher among (H1a) the elderly, (H1b) the less educated, and (H1c) the less digitally literate.

Next, we test our main hypothesis:

Hypothesis 2 The addition of incentives for accuracy will increase the preference for clickbait among (H2a) the elderly, (H2b) the less educated, and (H2c) the less digitally literate and decrease the preference for clickbait among the young, the better educated, and the more digitally literate.

Sampling and demographic characteristics

We recruited survey subjects using a Facebook ad campaign. The ad simply promoted a survey about “news consumption” and offered the possibility to enter a raffle to win 250 to all participants who finished the 15-minute survey. The campaign ran between September 28 and October 15, 2018. Overall, around 540,000 people were exposed to the ad, 16,000 people clicked on the ad, 4,104 started the survey, and 1,754 finished it. Most attrition was in the very early stages of the survey and was not related to treatment conditions. The campaign was run using Facebook’s tool for promoting pages, which automatically “learns” the optimal target population to maximize the number of clicks to the link. As a result, we did not specify who was exposed to the ad, though we do have ex-post information on demographic characteristics.

Appendix Table 1 presents the full demographic breakdown. Women are overrepresented (72%), but the sample is diverse in terms of age and other relevant covariates. Although not representative of citizens, voters, or even Facebook users, the population contains sufficient variation in the demographics of interest. The other variables listed in Appendix Table 1, and included in our analyses, are defined in the Appendix. Interspersed with measurement questions, we also include an attention check. We ask people to answer “65” to a question to show they are not replying randomly. Of the 1,754 initial respondent, 1,537 passed the attention check. In our analysis, we restrict our focus to people who passed the (pre-treatment) attention check.

Experimental design

Prior to the experiment, we asked respondents to rank a set of fictional headlines in order from “most prefer to read” to “least prefer to read,” to estimate each individual’s taste for Clickbait Media. The treatment headlines vary on two dimensions: they are either “Clickbait” or “Non Clickbait”, and are either “Pro Government” or “Anti Government.” We also include a “Neutral” control headline that is dryly factual and non-partisan.

We follow previous research on clickbait and create partisan emotional clickbait headlines by the addition of phrases like “This will make you furious…” or “You won’t believe this! …” to headlines with a strong partisan valence. The full list of headlines (and their English translations) is found in the Appendix.

We took several steps to mimic the style of Italian media. We first studied clickbait-news outlets, following examples provided by the main Italian website for debunking fake-news butac.it. We selected only topics that were politically controversial, where each side (Pro and Anti government) had a clear stance. We then wrote headlines changing the slant and the style to provide something that would look like it was published from either a slanted newspaper (e.g., La Repubblica or Il Giornale) or from a non-traditional news outlet (e.g., InformareXResistere or Matteo Renzi News). While a reader who is not accustomed to Italian politics might not catch the tics that define the slant and style of the headline, we believe that any Italian reader could easily guess these characteristics by the headline alone.

The “Pro Government” headlines suggest a positive development for the government, for instance, that “The government finally provided a cut to expenses.” While “Anti Government” give a negative spin to the event, for instance, saying “The Government managed to cut just a few expenses.”

In addition to the four slanted headlines, Neutral headlines report events without a clear slant or clickbait style. They are designed to imitate sources with a sober style, with no implicit judgment on the Government. All five articles are on the same topic, hence people were asked to choose between different ways of spinning the same story.

The pre-treatment section consisted of two sets of 5-headline choices that needed to be ordered in terms of preference to read. We then use this ordering to construct the individual-level measure of preference for clickbait (CB-Score): we assigned 4 points if a clickbait headline was ranked first, 3 if ranked second, and so on. The maximum CB-Score is 14, which would result if a user ranked the two clickbait headlines in the top two positions for both batches (i.e., 4 + 3 + 4 + 3 points, adding up to 14). The mean value of this variable is 6.7, the median is 7. This variable is a strong predictor of the likelihood of choosing clickbait in the second part of the survey and is a crucial control for ex-ante taste for clickbait.

Appendix Table 2 shows that older, less educated, more pro-government, less digitally literate people are more likely to choose clickbait. This finding nicely replicates previous research from the USA (Munger et al. 2020).

Next, we implement our experimental manipulation in order to understand the mechanism that explains variation in taste for clickbait. In this section, subjects had to rank five headlines with the same structure as above, but they were told that they would then read the article they put in the first position.

Our experimental manipulation involves changing the message that subjects receive before selecting a headline. All of the articles cover a cut in expenditure for pensions in the House of Representatives (Camera dei Deputati), which was one of the main campaign issues for the largest party in government (Movimento Cinque Stelle) during the electoral campaign. We have three treatment conditions:

No Incentive: “Attention! In the next section we will give you the titles of five new articles. This time, we will also display the text of the article to read.”

Intrinsic Incentive: No Incentive + “After that, there will be a small quiz on the theme of the article.”

Monetary Incentive: Intrinsic Incentive + “There will be a second drawing for an Amazon voucher to reward the correct answers. The closer your answers are to the real quantities, the more likely you are to win an extra prize of €250.”

The wording of the stimuli is designed to map tightly with our theoretical interest. We say explicitly that the quiz will be on the topic of the article, not on the article itself. We also say that the answers have to be “vicine al vero”: close to the truth, not close to the article. The text of the articles varied in style but contained all of the same information necessary to answer the quiz questions correctly; crucially, subjects were unaware of the content of the article before selecting it based on the headline.

The monetary incentive is exactly as large as the raffle prize with which we recruited subjects. The five headlines can be found in the Appendix.

Results

The dependent variable in all of the following results is a binary indicator for whether the article selected in the top slot (which subjects were told that they would then be asked to read) had a clickbait headline or not. Appendix Table 1 summarizes the characteristics of those who selected the different articles. It also confirms that the choice of article was not random. Pro Government (Pro) articles were far more popular than Anti Government ones (Anti), which reflects the fact that our sample includes many more Pro Government people. ⁵

Appendix Table 2 provides evidences in favor of all three sections of Hypothesis 1. Respondents who were older, less digitally literate, and less educated were more likely to select clickbait headlines, as were women, frequent internet users, and government supporters.

Appendix Table 3 presents a manipulation check, displaying the log of the time spent choosing and reading the article as the dependent variable. We find that people spend significantly more time both choosing and reading the article when given a monetary incentive. However, the intrinsic incentive has a negative effect: people spent less time reading when they knew that there was an unpaid quiz than when they did not know there would be a quiz. It is unclear why this is the case, but we note that the effect size is only 60% as large as the increase caused by the monetary incentive.

Figure 1 plots the key interactions from the regressions reported in Appendix Table 6. ⁶ The interaction in the top panel reports the likelihood that the respondent selected a clickbait headline under the monetary incentive treatment condition as a third-order polynomial. The slope is steepest in the age ranges of 20–30 years old and 60–70 years old, while there is little difference in the 30–60 year old range.

Figure 1 Interaction between treatment and age (top panel), education (middle panel), and digital literacy (bottom panel).

The middle panel of Figure 1 plots the estimated probabilities at each level of education, binned into three levels (no high school diploma, high school diploma, some college, or more). Respondents with low or medium education were more likely to select clickbait under the monetary incentive condition. Appendix Table 5 demonstrates that this significant interaction effect is robust to a regression in which education is treated as a continuous percentile. To reiterate, the only education category for which the monetary incentive caused increased clickbait selection was the lowest category: respondents without a high school diploma.

The bottom panel of Figure 1 demonstrates that there are people at the lowest end of the digital literacy spectrum who have a markedly higher likelihood to select clickbait under the monetary incentive condition. Respondents with medium or high levels of digital literacy were less likely to select clickbait under either incentive. ⁷

Conclusion

This paper takes seriously the role of perceived accuracy in media choice, investigating the source and the depth of the bond between mistrust for traditional media and support for anti-establishment, populist politics. First, we find that the “clickbait style” associated with the anti-establishment parties currently in government in Italy is preferred by people who are older, less educated, lower in digital literacy, and more likely to support the Italian populist government. This provides face validity for our theory (and support for H1(a-c)) and corroborates results from the USA.

Our main result is achieved by manipulating the salience of accuracy in respondents’ choice of media consumption. Some models of online media choice suggest that people opt to consume low-credibility online media because of a desire for congenial information, others because they are not taking the time to think seriously about their choice, and others because they are being deceived because they lack the digital literacy to evaluate source credibility online.

However, we find that offering a monetary incentive for accuracy increases the likelihood that certain respondents (older, less educated) will select a clickbait headline (and thus support for H2(a-b) but not H2(c)). ⁸ This result is not consistent with other models of media choice. The partisan content of the choice task is held constant. Rather than rushing through unreflectively, the monetary incentive causes responds to spend considerably longer selecting an article. Indeed, the only model that is consistent with our results is that the people who prefer clickbait headlines do so because they think they are more accurate.

These results suggest that the challenge facing the media and political establishments is more serious than some observers appreciate. The standard journalistic style that is designed to signal high-credibility reporting has the intended effect for the young and educated, but a sizable portion of the Italian population – and the key constituency of the current governing coalition – makes the opposite inference.

This is a serious problem, one unlikely to be addressed by recent developments around attempts to signal article quality to readers. Indeed, publishing fact-checking articles and emphasizing institutional credibility as the main distinction between good quality and bad quality information may paradoxically serve to further alienate an already skeptical portion of the audience.