Use of artificial intelligence to enable dark nudges by transnational food and beverage companies: analysis of company documents

Abstract

Objective:

To describe the use of artificial intelligence (AI)-enabled dark nudges by leading global food and beverage companies to influence consumer behaviour.

Design:

The five most recent annual reports (ranging from 2014 to 2018 or 2015 to 2019, depending on the company) and websites from twelve of the leading companies in the global food and beverage industry were reviewed to identify uses of AI and emerging technologies to influence consumer behaviour. Uses of AI and emerging technologies were categorised according to the Typology of Interventions in Proximal Physical Micro-Environments (TIPPME) framework, a tool for categorising and describing nudge-type behaviour change interventions (which has also previously been used to describe dark nudge-type approaches used by the alcohol industry).

Setting:

Not applicable.

Participants:

Twelve leading companies in the global food and beverage industry.

Results:

Text was extracted from fifty-seven documents from eleven companies. AI-enabled dark nudges used by food and beverage companies included those that altered products and objects’ availability (e.g. social listening to inform product development), position (e.g. decision technology and facial recognition to manipulate the position of products on menu boards), functionality (e.g. decision technology to prompt further purchases based on current selections) and presentation (e.g. augmented or virtual reality to deliver engaging and immersive marketing).

Conclusions:

Public health practitioners and policymakers must understand and engage with these technologies and tactics if they are to counter industry promotion of products harmful to health, particularly as investment by the industry in AI and other emerging technologies suggests their use will continue to grow.

The strategies employed by transnational food and beverage companies to promote food and beverages that are harmful to health (also known collectively as the ‘commercial determinants of health’⁽¹⁾) are increasingly scrutinised for their role in suboptimal population diets. Food and beverage marketing is a key strategy employed by the food and beverage industry to increase the desirability and acceptability of marketed products⁽¹⁾. Nudges are another key strategy; these seek to influence choices by making changes to the choice architecture (or the environment in which people make decisions) that ‘alter people’s behaviour in a predictable way without forbidding any options or significantly changing their economic incentives’⁽²⁾. Nudges are often considered in terms of how they can be used to improve diets and, in this context, include interventions such as nutrition labelling and changes to portion sizes. They do not include traditional educational efforts (for example, workshops or pamphlets), which do not change the choice architecture, nor do they include changes to prices, which change economic incentives⁽³⁾. While the food and beverage industry has employed some nudges to create healthy food environments⁽⁴⁾, nudges can also be used to exploit cognitive biases to promote the consumption of harmful products – these nudges have been termed ‘dark nudges’⁽⁵⁾.

Rapid technological change, such as the advent of the Internet-of-Things, associated generation of Big Data, and advances in artificial intelligence (AI), are providing opportunities for the food and beverage industry to influence customer behaviour in ways that are more intelligent, immersive and engaging than ever before. For example, McDonald’s fast food restaurant reportedly now programmes its drive-through menu boards to market products based on the time of day and weather and to display recommendations at the end of each transaction to prompt customers to order more⁽⁶⁾. Other food and beverage companies are also reportedly investing in their technological capabilities, with the Global Director of Digital Innovation at Coca-Cola describing AI as ‘the foundation for everything we do’⁽⁷⁾.

While the food and beverage industry is embracing the application of such new technologies and tactics, their use remains largely unexplored in the public health literature. Understanding and engaging with these will be essential for public health practitioners and policymakers to ensure appropriate actions are adopted to reduce the harmful influence of the food and beverage industry.

This study aims to describe the use of AI and other emerging technologies to enable dark nudges by leading global food and beverage companies to influence consumer behaviour.

For the purposes of this paper, the term ‘AI and other emerging technologies’ refer to the use of computers to autonomously or semi-autonomously complete tasks generally requiring human intelligence and to technologies such as the Internet-of-Things, robots and augmented and virtual reality. Table 1 provides definitions of the key related terms used in this paper.

Table 1 Key terms related to artificial intelligence (AI) and other emerging technologies

Sources:^(43–50).

Methods

We included twelve food and beverage companies in our study (Table 2). Companies were selected on the basis of being in the top five companies globally in the packaged food, soft drinks or consumer foodservice categories, using the latest available global company market shares (2019)⁽⁸⁾ at the time of selection (three companies featured in the top five companies of both the packaged food and soft drinks categories).

Table 2 Food and beverage companies included in review of company documents

For each of the twelve included companies, we used a two-step search strategy to identify evidence of the use of AI and other emerging technologies to influence customer behaviour. First, we retrieved the five most recent annual reports available on the global websites of each of the companies in March/April 2020 (ranging from 2014 to 2018 or 2015 to 2019, depending on the company), which were read in full by the lead author (RB) to inductively identify mentions of the use of AI or other emerging technologies to directly influence customer behaviour. This included the use of broad terms such as ‘artificial intelligence’ or ‘digital technology’ or more specific terms related to specific technologies such as ‘machine learning’ or ‘image recognition’. Whether the purpose of the use of AI or other emerging technologies was to influence consumer behaviour was not always explicit; in some instances, it was inferred by the lead author (RB) based on its potential influence on selection and consumption and confirmed by the second author (DN). We did not include companies’ uses of AI and other emerging technologies to optimise food production processes as this is unlikely to have a direct impact on customer behaviour. Mentions of e-commerce, where it was not apparent that AI or other emerging technologies were employed, were also excluded.

To complement the search of annual reports, we searched the global website of each company to identify any additional company uses of AI and other emerging technologies. The websites were searched in February/March 2020 using Google and the ‘site:’ search operator. The search query was developed based on scoping searches and included both broad terms related to AI and digital technology, as well as more specific terms related to certain technologies identified during scoping, as follows:

site:URL (‘artificial intelligence’ OR AI OR ‘digital technology’ OR automate OR targeted OR ‘machine learning’ OR ‘natural language processing’ OR ‘image recognition’ OR ‘facial recognition’ OR geotarget OR bot OR ‘social listening’)

The lead author (RB) screened the first twenty pages of search results for each website for relevance (based on the title and snippet text for each item) and reviewed potentially relevant items in full.

All relevant text on the reported use of AI or other emerging technologies to influence customer behaviour from annual reports and websites was extracted into an Excel spreadsheet and categorised according to the Typology of Interventions in Proximal Physical Micro-Environments (TIPPME) framework⁽⁹⁾. The TIPPME framework is a tool for categorising and describing nudge-type behaviour change interventions that has previously been used to describe the use of dark nudge-type approaches by alcohol industry-funded corporate social responsibility organisations⁽⁵⁾. The framework categorises interventions into those related to the placement of products, related objects and the wider environment and those related to the properties of products, related objects and the wider environment⁽⁹⁾. Interventions related to placement are categorised according to whether they alter availability or position, while those related to properties are categorised by whether they alter functionality, presentation, size or information ⁽⁹⁾. While we used the TIPPME framework as a guide to categorise uses of AI or other emerging technologies, we were open to the addition of additional categories if needed.

All data extraction and categorisation were conducted by one author with expertise in public health nutrition (RB) and verified by a second author with expertise in computer engineering and AI (DN). The categorised data were reviewed, discussed and agreed upon by all authors.

Ethics approval was not required for this study as it did not involve human participants, human material or human data. All data were publicly available.

Results

At least one mention of the use of AI or other emerging technology to influence customer behaviour was identified for eleven of the twelve leading global food and beverage companies included. Reports of AI or other emerging technology use were derived from fifty-seven documents, which included annual reports, news stories, press releases, transcripts from investor events, slides from presentations at investor events and privacy notices or policies. The level of detail reported on AI or other emerging technology use by companies varied.

When categorised to the TIPPME framework, examples of use of AI and other emerging technologies to influence customer behaviour were identified for both the placement and properties of products and related objects. Table 3 provides a summary of the TIPPME framework intervention types, their expected influence on selection and consumption and examples of AI-enabled dark nudges by the food and beverage industry, which are described in more detail in the text that follows. Supplementary Table 1 provides all extracted text and associated document details, for each TIPPME framework intervention type.

Table 3 Use of artificial intelligence (AI) and other emerging technologies by the food and beverage industry to enable dark nudges, identified from company documents and categorised against the TIPPME framework⁽¹⁸⁾

Placement

Availability

Company documents show how food and beverage companies are using AI and other emerging technologies to inform seemingly more rapid, targeted and data-driven changes to their product ranges. Danone, for example, described using social listening to monitor comments on social media, through which it saw a ‘developing trend of people seeking more lactose-free choices’ and, in response, introduced a new product⁽¹⁰⁾. The company described how social listening allowed consumer feedback to reach companies ‘instantaneously’ and provided ‘a game changer for honing in on consumers’ needs on a local and regional level’⁽¹⁰⁾. In another example, The Coca-Cola Company described using real-time sales data to monitor the ‘most poured’ flavour combinations from customisable soda fountains, which it then used to ‘power the company’s innovation pipeline’, including the launch of new products⁽¹¹⁾.

Company documents from a number of companies show how companies are using AI and other emerging technologies to inform product ordering and to closely monitor product levels, which can be assumed to increase the likelihood that products remain well stocked at all times. Seven & i Holdings, for example, described using AI-based ordering, which considered factors such as product-specific sales and weather-related factors, to place orders for food⁽¹²⁾, while Nestlé described using digital imaging and machine learning to monitor ice cream freezers, which allowed stores to know how much stock they had and when they would need to order more⁽¹³⁾.

PepsiCo similarly described the use of ordering algorithms to generate ‘perfect’ orders for stores, as well as investment in its ‘digital capabilities… to enable better execution of custom assortment and displays at a very granular store level’, which included the development of an app that provides ‘detailed trends at the outlet level and the store level for pack, flavor or brand’⁽¹⁴⁾. The combination of ‘local intuition with a much more data-driven understanding of the demographics and the consumer types in a particular store’ was described as allowing ‘much, much more precise’ store assortments, which were suggested to result in ‘double-digit types of growth’⁽¹⁵⁾.

Position

Company documents show how food and beverage companies are using AI and other emerging technologies to manipulate the position, and therefore likely selection and consumption of products. This is particularly evident in the manipulation of the visibility and salience of products displayed on menu boards. McDonald’s, for example, described using decision technology to alter its drive through menu boards based on the time of day, weather, current restaurant traffic and trending menu items⁽¹⁶⁾. The Coca-Cola Company similarly described a cloud-based digital signage system that allowed restaurants to update their menu boards based on real-time sales data⁽¹⁷⁾. Yum! Brands, meanwhile, manipulated the position of products on menu boards based on the characteristics of individual customers, with its company documents describing the use of facial recognition technology to narrow menu items shown to customers based on their estimated age, sex and apparent mood⁽¹⁸⁾.

Beyond the positioning of products, McDonald’s described assessing the suitability of sites for restaurants through the use of ‘advanced technology’ to analyse traffic and walking patterns, census data and other relevant data⁽¹⁹⁾.

Properties

Functionality

Company documents show how food and beverage companies are using AI and other emerging technologies to guide consumers’ use of websites, apps and ordering kiosks or drive throughs and, in particular, cross sell products to increase the size of sales. In one example, PepsiCo described its use of data and analytics to ‘highlight relevant and contextual affinities online’ in prompting consumers to purchase additional products from its online store based on their current selections (for example, suggesting a customer purchase pita chips if they have already selected hummus), a strategy it described as having been ‘effective’⁽¹⁴⁾. McDonald’s similarly described using decision technology on its drive through menu boards to ‘instantly suggest and display additional items to a customer’s order based on their current selections’ and planned to integrate the same technology in other places customers make orders, such as in its self-order kiosks and app⁽¹⁶⁾.

Company documents also show how food and beverage companies are using AI and other emerging technologies to interact with consumers in new ways, including ways which increase the ease and convenience with which consumers can order and obtain their products and which increase the channels through which companies market to consumers. Yum! Brands, for example, described developing a bot accessible via Amazon’s voice assistant Alexa, which allowed customers to ‘use their voice to order the Colonel’s chicken’ and ‘learn about promotional offers’⁽¹⁸⁾. Nestlé similarly described using ‘algorithm-rich technology’ to develop ‘conversational marketing’ experiences, such as a chatbot on Facebook Messenger that delivers special offers⁽²⁰⁾. PepsiCo, meanwhile, described its ‘snackbot’, a self-driving robot that delivers snacks ordered by university students via an app across a university campus, making snacking ‘ultra-convenient’⁽²¹⁾, Yum! Brands similarly described testing the use of autonomous robots to deliver pizza to customers⁽¹⁸⁾.

Presentation

Several company documents show how companies are using AI and other emerging technologies to alter presentation, particularly in the use of augmented reality and virtual reality to create highly engaging and immersive marketing campaigns. PepsiCo, for example, described the use of virtual reality in marketing that put Gatorade and Mountain Dew consumers ‘in the shoes of their favourite athletes’⁽²²⁾, as well as the launch of ‘an artificial intelligence-powered camera’ that ‘allows users to transform everything they see into unbelievable Cheetos-inspired creations’, which can be shared on social media⁽²³⁾. Danone similarly described launching a Snapchat lens that put evian consumers’ faces ‘in an animation of a famous evian dancing baby’⁽²⁴⁾, while The Coca-Cola Company described an app with which consumers could scan products to ‘unlock’ augmented reality ‘experiences’ featuring the ‘Coca-Cola Polar Bears’⁽²⁵⁾.

Company documents also show how companies are using AI and other emerging technologies (including automated data collection technologies like cookies and web beacons) to tailor the presentation of marketing and target specific marketing to specific consumers. For instance, PepsiCo described how ‘digitalizing’ its marketing and consumer insights allowed it to ‘tailor and target ads with greater precision’⁽¹⁵⁾, while Starbucks described using information about the location of devices consumers were using to ‘deliver more relevant advertising’, including tailored marketing offers or messages based on location, time of day and weather⁽²⁶⁾. Seven & i Holdings described using facial recognition to guess customers’ age and gender and accordingly adjust the in-store advertising displayed⁽²⁷⁾.

Discussion

Our review is the first to systematically review documents from leading global food and beverage companies (which market and sell predominantly unhealthy foods and beverages (i.e. foods and beverages high in saturated fat, sodium, sugars and energy)^(28,29) ) to describe the use of AI and other emerging technologies to enable dark nudges using an established intervention typology. We found that global food and beverage companies are using a wide variety of AI and other emerging technologies to enable dark nudges that alter the availability, position, functionality and presentation of products and objects. Nudges that alter these have the potential to increase selection and consumption by exploiting a variety of cognitive biases, such as salience bias and the principle of least effort^(30,31) . The use of AI in the identified dark nudges means that these are far more personalised and engaging than more traditional non AI-driven dark nudges, potentially driving even greater selection and consumption of unhealthy foods and non-alcoholic beverages.

Our results align with, and provide direct evidence from company documents of, some of the key features of the ‘Big Data digital food marketing system’ described in a commentary by Montgomery and colleagues, which included tracking, profiling and targeting of individual customers; geotargeting and place-based marketing; data-driven targeting by ethnicity and race; cross-device tracking and personalised advertising; in-store surveillance and point-of-purchase prompting and micro-moment messaging⁽³²⁾. Our results also align with those of Carolan, who interviewed individuals who had overseen the use of Big Data applications within food retail settings⁽³³⁾. The interviewees reported the use of data mining to understand customer trends (such as those based on weather or emerging food trends) and of beacon technology and media access code addresses to send alerts or coupons to customers when they were near to stores⁽³³⁾ – we similarly found examples of product development based on social listening and of geotargeting of marketing offers or messages. The interviewees in Carolan’s study, however, also reported increasingly sophisticated purported future uses that we did not find in our study, such as the use of high-resolution cameras and software to analyse eye movement and bodily cues and send alerts to customers while they are in store⁽³³⁾. Similarly sophisticated tactics have, however, been reported by media organisations, including the reported fitting of cameras, motion sensors and eye-tracking capabilities to frozen food aisle doors in retail settings, with information from these used to tailor the displays and promotions shown on the doors⁽³⁴⁾. In a recent report, the Center for Digital Democracy documented numerous examples of the use of AI and other emerging technologies by food and beverage companies, which, similar to our results, included the use of chatbots, augmented and virtual reality, and voice assistants⁽³⁵⁾.

The AI-enabled dark nudges we identified are ultimately likely to drive increased selection and consumption of targeted products, by altering the availability, position, functionality and presentation of products and objects and exploiting cognitive biases. For example, as noted in Table 3, nudges that alter the position of products may increase their selection and consumption by increasing their visibility and salience⁽³⁰⁾. As the companies included in our study market and sell predominantly unhealthy foods and beverages, any increase in selection and consumption of their products is likely to reduce the quality of population diets and contribute to diet-related morbidity and mortality. While our study is not able to determine whether these AI-driven dark nudges are more effective at increasing selection and consumption than more traditional non-AI-driven dark nudges there is some evidence to suggest that this may be the case. For one, the nudges traditionally used in retail settings, such as background music, lighting, smells and product placement, are relatively blunt – for example, only one background song can be played at a time and eye-level product placement is not at eye level for all customers⁽³³⁾. In contrast, dark nudges that employ AI and other emerging technologies can be far more personalised and engaging – for example, marketing offers sent to customers can be tailored based on customers’ location or the time of day or weather, as identified in our study, as well as large amounts of other data (such as information about individuals’ activities on websites, mobile phones and wearable devices)⁽³²⁾. Nudges driven by such Big Data have been termed ‘hypernudges’ due to their ability to create ‘a highly personalised choice environment’ and are suggested to be ‘highly potent’⁽³⁶⁾. AI and other emerging technologies that allow advertisers to generate and test multiple versions of advertisements (varying, for example, the advertisement copy, images and video) and deliver different versions of advertisements to increasingly targeted market segments are also being used and are reported to be ‘effective at driving increased sales and brand loyalty’⁽³⁵⁾. Company documents included in our review also indicate that other AI-enabled dark nudges have been successful in driving sales growth. For example, PepsiCo, reported that its data-driven local store assortments had resulted in ‘double-digit types of growth’⁽¹⁵⁾. Further research should explore the impact of AI-enabled dark nudges on selection and consumption of intended products and total energy intake to determine their impact on population health.

The use of AI and other emerging technologies to enable dark nudges also raises several ethical concerns. The collection of personal data by technology companies through automated data collection technologies (such as cookies and web beacons) and the on-selling of these data to the food and beverage industry allows for carefully tailored and targeted marketing campaigns. This collection of personal data to market products that are harmful to health presents issues related to consent, data privacy and misuse of data⁽³⁷⁾. As few consumers read or fully understand online terms and conditions, which are often lengthy and contain substantial ‘legalese’⁽³⁷⁾, the extent to which consumers actually consent to the collection and use of their data to inform dark nudges is highly questionable.

There are also concerns that data can be used to target unhealthy food and beverage marketing to certain populations⁽³⁸⁾, with the potential to exacerbate existing disparities in diets and health. Data related to engagement with unhealthy food marketing could also potentially be used to make inferences about long-term health risks, which could then be applied in insurance, financial services, differential pricing or other discriminatory applications⁽³⁸⁾. Strengthening of data protection and privacy laws, so that they are fit-for-purpose in a highly technological and data-driven world, should be a priority of governments around the world.

In addition, regulations that specifically reduce exposure to, and the power of, marketing of unhealthy foods and beverages should be pursued by governments, as recommended by authoritative health organisations, such as the WHO⁽³⁹⁾. The terms and conditions that underpin such efforts should be carefully considered and as comprehensive as possible to account for these AI-enabled dark nudges and ensure that transnational food and beverage companies do not undermine efforts to improve population diets. For example, a ban on unhealthy food marketing at end-of-aisle displays and checkouts in retail stores may be undermined by technology that determines a customer’s location within an aisle and delivers micro-targeted advertisements via mobile phone that offer discounts or rewards in real time⁽³²⁾.

The need for this is reinforced by evidence we found in company documents (while completing this study) of large-scale investment in AI and other emerging technologies by food and beverage companies. This included McDonald’s acquisition of entire technology companies (namely Dynamic Yield, described as ‘a leader in personalization and decision logic technology’⁽¹⁶⁾, and Apprente, ‘an early stage leader in voice-based, conversational technology’⁽⁴⁰⁾) and Nestlé’s description of its ‘Silicon Valley Innovation Outpost’, made up of a team of ‘commercial and technology experts’⁽¹³⁾. The outcome of this investment is likely to be increased use and increasingly sophisticated use of AI and other emerging technologies in dark nudges in future years.

Relatedly, countering the promotion of products harmful to health will require monitoring and scrutiny of the actions of not only transnational food and beverage companies but also technology companies. In addition to the role that technology companies take in the collection and sale of personal data to the food and beverage industry, technology companies are also partnering with the food and beverage industry to directly support AI-enabled dark nudges. For example, the conversational AI company Nuance Communications partnered with Domino’s Pizza to deploy a virtual assistant allowing ordering and conversations about menus, ingredients, store locations and operating hours⁽⁴¹⁾, while the search engine company Baidu partnered with KFC to deploy facial recognition technology to recommend menu items to customers⁽¹⁸⁾ (as described in our results).

Finally, although we have focused on how AI and other emerging technologies are being used to enable dark nudges, these same technologies also offer opportunities for public health. Tailoring and targeting marketing can be used to deliver health promotion messages to targeted population groups, while image recognition and machine learning technologies are beginning to be used to monitor the activities of alcohol companies⁽⁴²⁾, and advanced AI technologies are increasingly being used to optimise the delivery of nutrition interventions⁽⁴³⁾.

A strength of our review is the use of company documents, which provides information about the use of AI and other emerging technologies in companies’ own words. While these were from only a sample of the top global companies (by market share), these companies are likely to have the greatest resources to enable the widespread and sophisticated use of AI and other emerging technologies. While mentions of the use of AI or other emerging technologies were occasionally vague or lacking detail, authors with expertise in computer engineering and AI technologies verified that the illustrative text supported each of the included categories.

Our review was limited to documents publicly available on the sampled websites and as such is unlikely to have captured the full extent or details of the use of AI and other emerging technologies by the food and beverage industry to influence customer behaviour. Nevertheless, it provides an indication of the breadth of tactics and technologies used. Marketing literature, including grey literature, could be reviewed in future to triangulate these findings. Finally, there are a wide range of terms that may be used to describe AI and other emerging technologies. While all annual reports were reviewed in entirety, it is possible that the additional search of company websites, where we used key terms to identify relevant information, may have missed examples where alternative terms were used. However, given the likely non-technical audience for documents placed on company websites, we anticipate that most examples would have included broad terms such as ‘artificial intelligence’ or ‘digital technology’.

The use of AI and other emerging technologies to exploit cognitive biases and nudge consumers towards unhealthy foods and beverages appears widespread and sophisticated. Based on company documents, global food and beverage companies are using such technologies to enable dark nudges that alter the availability, position, functionality and presentation of products and objects and are ultimately likely to drive increased selection and consumption. Given investment by companies in AI and other emerging technologies suggests use of these technologies will only continue to grow, efforts to counter promotion of harmful products and improve population diets must be cognisant of these emerging tactics and technologies.