Unreported Realities: The Political Economy of Media-Sourced Data

Abstract

What is the gap between scholars’ expectations of media-sourced data and the realities those data actually represent? This letter elucidates the data generation process (DGP) that undergirds media-sourced data: journalistic reporting. It uses semi-structured interviews with 15 journalists to analyze how media actors decide what and how to report—in other words, the “why” of reporting specific events to the exclusion of others—as well as how the larger professional, economic, and political contexts in which journalists operate shape the material scholars treat as data. The letter thus centers “unreported realities”: the fact that media-derived data reflect reporters’ locations, identities, capacities, and outlet priorities, rather than providing a representative sample of ongoing events. In doing so, it reveals variations in the consistency and constancy of reporting that produce unacknowledged, difficult-to-identify biases in media-sourced data that are not directionally predictable.

INTRODUCTION

News media is a staple data source for social scientists. While scholars commonly use journalistic reporting to collect data on event occurrence (e.g., protests), casualty numbers, actors’ narratives, and government actions (see, e.g., Davenport and Ball 2002; Tilly 2005), its use has not been without significant critique (Ben Hammou, Powell, and Sellers 2023; Davenport 2009; Dawkins 2021; Dietrich and Eck 2020; Gohdes and Price 2013; Miller et al. 2022; Snyder and Kelly 1977; Wang et al. 2016). Scholars identify numerous sources of systemic bias: regime type influences media coverage (Baum and Zhukov 2015); large-scale protest events are more likely to be reported than small-scale events (Oliver and Meyer 1999); more violent events are more likely to receive coverage (Hendrix and Salehyan 2012); and geographic location influences reports of human rights abuses (Caliendo, Gibney, and Payne 1999). Journalists note that their identities influence stories they choose and are able to report (Hassan 2019, 101; Sharif 2019). Arjomand (2022) demonstrates how foreign journalists’ relationships with local fixers shape possibilities for access and the level of detail in reporting.¹ Yet researchers continue to rely on media-sourced data without appropriate caveats, overstating the robustness of results and underestimating the political and economic factors that shape the data generation process (DGP).

This letter elucidates the political economy of journalistic reporting as an overlooked, central element of the DGP of media-sourced data. It analyzes how journalists decide what and how to report—the “why” of reporting specific events to the exclusion of others—and how the larger professional, economic, and political contexts in which they operate shape coverage. It reveals broad, contingent variation in the consistency and constancy of reporting and heavily context-dependent dynamics. It thus underscores the “unreported realities” media-derived data represent: reporters’ locations, identities, capacities, relationships, outlet priorities, and need to place stories in competitive media markets, rather than a representative or systematic sample of ongoing events. These realities suggest caveats to current research and opportunities for future research.

GOING TO THE SOURCE: METHODOLOGY

Dietrich and Eck (2020) note that most prior examinations of media-based DGPs have focused on within-country comparisons of scholar-coded data (Loyle, Sullivan, and Davenport 2014; Weidmann 2016). Most of these studies take a macro approach while acknowledging that they do not address the micro- and the meso-level of journalist and media outlet behavior.² This letter complements such work using original data from 15 semi-structured, in-depth interviews with journalists working in the Middle East and Africa to illustrate how the media production process and the industry at large shape the material that scholars code as data. Most interviews occurred in 2018 and 2019; all respondents asked to remain anonymous and are quoted with permission.³ Journalists worked in both print and multimedia for international and regional English-language media outlets. Half of the interviewees simultaneously or had previously worked for local media in a regional language (e.g., Arabic and Sorani); others had worked for non-English language international outlets. Interviewees were or had worked as editors, producers, correspondents, stringers, freelancers, and fixers. Some focused exclusively on the Middle East or Africa, while others moved between contexts. Additionally, the researcher analyzed over a year of activity on the Syrian Democratic Forces’ WhatsApp group for English-language journalists, recent first-person accounts of reporting, secondary sources such as the Columbia Journalism Review (CJR), research in communications and media studies, and reports from organizations such as the Reuters Institute at Oxford and the Committee to Protect Journalists.

CONSISTENCY AND CONSTANCY

Two erroneous assumptions about media reporting, particularly in the context of events data compilations, have theoretical and empirical implications for how researchers understand both the size and direction of bias in the DGP and its political nature. These persist despite some acknowledgement that the DGP varies across contexts in initial reporting—for example, depending on press freedom—and which sources scholars use—for example, if there are reliable local media outlets.⁴

First, end users tend to assume consistency in reporting across domestic and international contexts. For example, if there is a civil war in Syria and a civil war in Iraq, beliefs that urban bias affects both cases in the same way often lead scholars to proceed as though coverage of violent events will be comparatively reliable in Damascus and Baghdad and coverage in al-Hasaka or Ninewa will be comparatively lacking.

Second, end users tend to assume constancy in reporting, or that journalists cover the same types of events in similar ways with stable rates of attention. Yet journalists operate in dynamic environments and change reporting approaches over relatively short periods of time.⁵ The assumption that journalists cover the same types of events, gather and report the same level of detail, and remain in those spaces for similarly significant amounts of time is thus nontrivial. Dorff, Henry, and Ley (2023) demonstrate, for example, that Mexican journalists have decreased the specificity of their reporting on organized crime given an uptick in violence against media. The obvious implication is that an urban protest in Mexico in 2021 may be reported very differently from the same class of event if it occurred in Mexico in 2011 or in Venezuela in 2021.

If a Tree Falls in the Forest…

The salience of variability in ground-level journalistic presence grows further as outlets commit fewer resources to overseas reporting and rely on reporters who “parachute” in only to cover specific events (Arjomand 2022, 1–3; Khalaf 2019, 259). Between 1998 and 2010, 18 US newspapers and two chains closed their overseas bureaus (Enda 2011). Martin (2012) notes that while publications such as the Washington Post and the Los Angeles Times maintained foreign bureaus, they were frequently staffed by a single person who was responsible for huge swaths of territory. He observes, “One [Washington] Post reporter, Sudarsan Raghavan in Nairobi, is listed as the paper’s ‘bureau chief in Africa.’ Raghavan is the chief of a bureau of one in Kenya. For the continent of Africa.”⁶ Schwartz (2018) corroborates this trend in noting the closure of Foreign Policy’s overseas bureaus.

Risks to journalists also shape coverage holes, particularly where safety concerns or legal means constrain press freedom. For example, Agence France Press ceased deploying journalists to rebel-held areas in Syria and stopped accepting freelance work from such regions to disincentivize extremely risky reporting. The Committee to Protect Journalists’ data demonstrates a clear upward trend in the imprisonment of journalists since 1992 (Committee to Protect Journalists 2022) (see also: Carey and Gohdes 2021; Gohdes and Carey 2017). Both factors depress the number of journalists on the ground and the likelihood of events being reported (Armoudian 2016, 2–3).

Inconsistent rates of reporting and variability in coverage depth may be inevitable implications here; overstretched journalists in precarious positions cannot equally cover each country or region on their beats. They constantly make choices about which stories to pursue and in what level of detail. They are increasingly likely to miss even major events. Clarke (2023, 302, 308) identifies bias in both the consistency and constancy of cross-national and subnational data that result. Specifically, he demonstrates that protest datasets built from local-language sources in the Middle East “identify considerably more events than most off-the-shelf datasets” and that there is wide variation in the differences between Arabic-language sources and popular event datasets (e.g., ACLED, SCAD, and NAVCO) with respect to the events reported in Egypt, Tunisia, Morocco, Syria, and Iraq. Yet freelancers and those working for local media often do not have the same resources as well-funded and protected correspondents.⁷

Reporting as a Tight-Rope Act

Fluid states of access and proximity shape reporters’ ability to identify stories, reach locations, interview participants, and publish, creating biases in data. Journalists’ relationships with key actors influence what they do and do not report. For example, during interviews the researcher conducted in the Kurdistan Region of Iraq (KRI), multiple journalists relayed that other journalists threw parties for members of the country’s military elite, which granted them special access and increased the chances that the journalists involved might avoid reporting on embarrassing issues such as government human rights violations.⁸ Sasha, a freelancer with extensive local experience, emphasized the self-preservation bias that drove many publication decisions: “a lot of journalists here…close their eyes so that they can maintain access.”⁹ Tariq, a producer for a major TV network, showed the researcher WhatsApp groups where members of the country’s security sector elite sent tips to reporters; he specified that given time, security, and resource constraints, some reporters took their stories and quotes directly from the feed.¹⁰ Government manipulation of the press is nothing new; during the Cold War, for example, many foreign news outlets relied on a Moscow-based fixer with known KGB connections, who fed journalists both true stories and propaganda on behalf of the USSR (Arjomand 2022, 48).

Journalists may also be restricted by political conditions or safety concerns that can vary unpredictably (Sinjab 2019, 199). Over half of the journalists interviewed had received credible threats to their physical safety from governments, non-state actors, or both. Several journalists, particularly those reporting from their home country, had been arrested by state agents, especially when reporting for local outlets. The majority mentioned being surveilled; one local journalist had been repeatedly tortured. Khalaf (2019, 263), a seasoned correspondent, emphasizes that “reporters under pressure can be made to feel as though they must adhere to red lines and withhold some of the most sensitive information they uncover.” Speaking about reporting from states such as China, journalist David Schlesinger underscores: “if they don’t like what you’ve written, then they’ll either throw you out or make your life difficult or call you in and yell at you. Or in the cases of what you see now with Bloomberg and The New York Times, they’ll control visa access for replacements for the bureau” (quoted in Armoudian 2016, 120). Countries may also use censorship or launch libel cases against journalists who report on certain topics (Armoudian 2016, 120). Zoran, for example, explained how differences between defamation laws in federal Iraq versus the Kurdistan region shaped his reporting; Amy discussed her blacklisting by two governments for reporting on human rights issues. These realities mean that genres of events—from elite corruption to seemingly tamer topics such as pregnancies that occur during US military deployments (Armoudian 2016, 126–8)—may be more difficult to report, receive incomplete coverage, or be completely avoided. Such topics might be locally specific; reporters who had worked in Kurdish media explicitly mentioned the sensitivities and risks involved with reporting on suicides.¹¹ These experiences underscore the intense risks local journalists often face, especially when working for local media, and reveal how such vulnerabilities can shape reporting.

MARKETS, REPORTING, AND ENDOGENOUS DYNAMICS

Just as access and resource issues challenge academic assumptions of journalists’ uniform interest in and regular interactions with events on the ground, the actors who control media outlets have their own agendas that shape reporting (Grossman, Margalit, and Mitts 2022). Incorporating external politics and economics into a political science understanding of media sources challenges assumptions of consistency and constancy that underpin scholars’ DGPs. Political communications research emphasizes that many media outlets are fundamentally corporations whose survival is predicated on responsiveness to consumers (Boydstun 2013; Cook 1998) They operate on budgets and have limited space, reporters are expensive, and readers have (changing) preferences (Norris 1995). Within the industry, there is anxiety over these pressures; a 2022 Reuters Institute report notes widespread industry concern that increasing trends toward subscription-based news services will result in media catering to wealthier, highly educated audiences by publishing material they want to read (Newman 2022, 6).

The journalists interviewed for this research, as well as the texts analyzed, reported concerns that these dynamics force media workers to oversimplify or misrepresent stories that require nuance. The trends they describe imply that many of the preset actors found in datasets (e.g., ethnic groups) may have been labeled so by editors responding to market forces and accessible audience frames, rather than on-the-ground dynamics.¹² “No one is willing to commission five weeks of work,” George commented, going on to emphasize the industry’s heavy focus on straightforward, up-to-the-minute, headline-grabbing news that fits preexisting narratives.¹³ BBC correspondent Allan Little notes that he constantly fought about framing with his editors while covering the former Yugoslavia:

I would always say. ‘the Bosnian government forces, the Bosnian government side,’ whereas they wanted to say ‘Muslim forces.’…To me it was a three-sided war in which two sides represented some sort of ethnic supremacy…and the third side represented…multi-ethnic tolerance. [My editor] called them ‘the Muslims,’ and so on [as if] they’re all the same [as the Serbs and Croats]. (Little, quoted in Armoudian 2016, 131–2)

Julie, a print journalist who worked in the Middle East for over a decade before moving into an editorial role, also underscored how perceptions of inherent newsworthiness subtly shaped reporting:

Have you heard of those calculations that cynical hacks make, that one dead [Westerner] equals five dead [Middle Easterners] equals two hundred dead [Africans]…that’s not exactly what it is, but that’s roughly [pause] you instinctively know it, when you’re on that beat. It’s not like the actual reporters think these lives are worth less, it’s about newsworthiness.¹⁴

Media outlets must navigate these incentives and constraints to maintain readership and financial viability. Whether formally or informally, outlets also increasingly link articles’ popularity to journalists’ job security and pay, incentivizing the publication of certain types of stories to the exclusion of others (Bland 2021). Mina, who worked for a wire service, relayed reporting for years from sub-Saharan Africa, but only got high online page views when she incorporated then-US President Donald Trump into her narratives. Her editor subsequently instructed her to work the Trump administration into her submissions as much as possible, shaping the kinds of stories she reported and overstating Trump’s position in the pieces printed, a process Arjomand (2022, 65–7) refers to as “frame control.”¹⁵ A 2018 article in CJR emphasizes that during the Trump administration, “[n]ewspapers, magazines, and TV news programs simply [had] less space for freelance international stories than before.” However, this effect is conditional; international stories directly tied to the controversies of the administration (e.g., Russia and North Korea) showed an uptick in reporting (Schwartz 2018). In addition to contributing to bias in events datasets, such insertions would bias scholarship based on Factiva searches or text scraping for mentions of Donald Trump, as his name was artificially inserted to drive page views, rather than representing genuine coverage.

This phenomenon extends beyond the Trump example. Danny Gold, who covered the Rohingya crisis in Burma, notes that getting articles published about “lesser known subjects” requires strategic framing. In his case, he leveraged the 2016 US presidential election and Hillary Clinton’s previous work on Burma as Secretary of State as a hook for Rohingya coverage, underscoring “…if I need to frame it that way to get out in the field and cover it, you know” (quoted in Armoudian 2016, 132–3). Ingrid, an experienced freelancer who reported on gender-related issues in the Middle East, had a Europe-based editor inform her that “no one” was interested in an article about child marriage markets unless it involved al-Qa`ida; she added mention of a tenuous link between the group and child marriage to get the story published and be paid for her work.¹⁶ Karam (2019, 233) notes of reporting on the Syrian Civil War that “The Islamic State group’s barbaric activities became the subject of global fascination and dominated the news…Freelance journalists complained that if their story did not have an Islamic State element to it, editors weren’t interested.” An implication for researchers is that endogenous market dynamics—for example, that “clickable” Islamic State stories beget more Islamic State stories and crowd out other coverage—affect research designs that rely on methods such as content analysis and machine learning because media markets, rather than events on the ground, drive coverage (including the quantity and type of events reported).

POLITICAL SCHOLARSHIP WITH MEDIA SOURCES: TOWARD NEW RESEARCH AND ENGAGEMENT

The media industry is meant to produce news, not data. Journalists do not operate to generate scientifically representative reporting, which brings into question the data uncritically harvested from such sources and assumed to be “objective,” or at least predictable. Scholars must directly engage the reality that existing biases are both stochastic and nonrandom.

This letter identifies gaps between end users’ expectations of what event data represent (consistent and constant reporting that carries systematic, identifiable biases) versus the reality of what those data reflect. That reality encompasses both access biases (e.g., associated with journalist presence, safety, and censorship) and editorial biases (e.g., linked to frame control, audience, and profitability).¹⁷ Both biases shape what outlets publish—and is thus included in datasets—and what media coverage excludes—producing silences and missing data. On one level, these findings indicate a need for more attention to DGPs in both manually coded events data and text scraping of media sources, as well as an acknowledgement of endogenous market dynamics. That is, given that bias in some datasets appears more extensive than previously reported (Clarke 2023, 307) and that there are documented regional, country-level (Dawkins 2021), and temporal variations in reporting bias, the bias and its origins themselves ought to be addressed as objects of inquiry that reveal crucial political dynamics. For scholars interested in identifying different bias patterns that affect the data, collaboration with regional specialists and interpretivist scholars could facilitate the type of data forensics necessary. The goal here would not be for such scholars to offer a “fact check” or “verification” of the data, but rather to intellectually collaborate to explore the meso- and micro-level politics—whether lack of market interest or overwhelming censorship—that shape the biases that affect specific locations and periods.

This article also reveals how the media-based material scholars treat data has its own politics; unreported realities thus create opportunities to use existing data in new ways. Research using existing datasets might examine how market opportunities shift over time; machine learning techniques might identify the rise and fall of specific media content and frames, especially in connection to major events such as elections and wars. Scholars might directly interrogate the “crowding out” mechanism associated with the Trump administration’s tenure and its long-term political consequences for democratic engagement and foreign policy. Acknowledging the challenges associated with the DGPs of media-sourced data clarifies new opportunities for critical conceptual and causal research that addresses the core of how ideas operate in political life.