A computational simulation-based framework for estimating potential product impact during product design

Abstract

The impact of engineered products is a topic of concern in society. Product impact may fall under the categories of economic, environmental or social impact, with the last category defined as the effect of a product on the day-to-day life of people. Design teams lack sufficient tools to estimate the social impact of products, and the combined impacts of economic, environmental and social impacts for the products they are designing. This paper aims to provide a framework for the estimation of product impact during product design. To estimate product impact, models of both the product and society are required. This framework integrates models of the product, scenario, society and impact into an agent-based model to estimate product impact. Although this paper demonstrates the framework using only social impact, the framework can also be applied to economic or environmental impacts individually or all three concurrently. Agent-based modelling has been used previously for product adoption models, but it has not been extended to estimate product impact. Having tools for impact estimation allows for optimising the product design parameters to increase the potential positive impact and reduce potential negative impact.

1. Introduction

The world is seeing an ever-increasing call for businesses to do more than maximise shareholder value (Henderson 2018; Gelles & Yaffe-Bellany 2019). The rapid rise of the benefit corporation – a company that includes a positive impact on society as a goal – demonstrates an increased interest in the impact of products and companies (Chen & Kelly 2015). Product impact generally falls into three categories: economic, environmental and social impacts. Emerging sectors such as engineering for global development centre their mission around leveraging engineering and design to provide positive impact (Mattson & Winter 2016; Burleson & Austin-Breneman 2020). While there is increased discussion around product impact, few tools exist to estimate it or plan well for it (Schöggl, Baumgartner & Hofer 2017).

Interviews with practicing engineers show that most consider the social impact of products they design, but that effort is largely centred on ensuring that products are safe to use and not on other social impacts such as impacts on education, income or family (Pack et al. 2019). It has been acknowledged that more tools need to be developed to help designers estimate the social impact of products to make better design choices (Pack et al. 2019). To estimate the social impact of a product, a modelling framework needs to have methods for (i) social impact measurement and (ii) the adoption and diffusion of technology in society.

The contribution of this paper is a framework for estimating the effect of changing product parameters on the social impact of a product during the design phase. This is accomplished by coupling existing methods for measuring social impact with those of technology diffusion in an agent-based model (ABM). Although the framework may be used to model economic, environmental and social impact, this paper focuses mainly on social impact. This framework allows for building ABMs to estimate social impact as a design tool so that the product design team can make choices to increase the positive impact and reduce the negative impact of the product before it enters the market. The different components of this framework have previously been used separately in the literature, the framework described in this paper builds upon these methods by combining them, and then importantly using that combination to extract new knowledge about the societal response to changes in product parameters. Though no ABM can be declared fully accurate, emergent behaviour and the dynamics of complex systems can be difficult to discover without models and simulations. Thus, we can use the new framework to estimate product social impacts. Following the description of the framework, an illustration of how the framework can be used is provided using face masks and vaccines during the COVID-19 pandemic. Tools such as this framework will help design teams make decisions about what aspects of a product are most important to increase potential positive impact while limiting potential negative impact. This gives the ability to iterate on product parameters to explore the design space and find areas to improve product impact. These methods will also help in understanding relationships between product parameters and the societal response to changes in those product parameters.

1.1. Social impact measurement

The social impact of a product can be defined as its influence on a person’s day-to-day life (Burdge 2015). All products have a social impact, and the effect can be positive or negative (Norman & MacDonald 2004; Mattson et al. 2019). This social impact can happen during the development, production, use and end-of-life phases of products. The framework presented in this paper will focus on social impacts during the use of a product. Literature on measuring social impact has been growing and frameworks for assessing the social impact of products have emerged such as social impact assessment (SIA) (Fontes et al. 2018), the social life cycle assessment of products (Benoît et al. 2010), and the product impact metric (Stevenson et al. 2018). It is important to note that these frameworks were largely built on assessing current products and not on estimating the social impacts of a future product. Only recently has work emerged in the area of social impact predictive modelling (Stevenson, Mattson & Dahlin 2020). To estimate social impact one must gain knowledge of the societies that will be potentially affected (Esteves, Franks & Vanclay 2012), decide upon areas of potential social impact (Rainock et al. 2018), what stakeholders to include (Fontes et al. 2016), and relevant measures for the given category and stakeholder (Stevenson et al. 2020). These methods are important to design teams to gain an understanding of the extent to which a product affects a person’s day-to-day life rather than focusing on surrogate impact indicators such as the number of units sold.

1.2. Adoption and diffusion of technology

Theories on adoption and diffusion of technology can be split into how technology spreads across society on a macro scale and how a single individual decides whether to adopt a particular technology. The Diffusion of Innovations (Rogers 1995), with its familiar s-curve of adoption, focuses on the diffusion of technology through society on an aggregate level. Theories such as the theory of planned behaviour (TPB) (Ajzen 1991) and the unified theory of acceptance and use of technology (UTAUT) (Venkatesh et al. 2003) focus on the factors that cause an individual to adopt a technology or behaviour. To implement TPB or UTAUT, information must be gathered about individual user preferences to build a model. Although all of these have been used to predict adoption, they have not been extended to estimate the social impact after product adoption.

UTAUT is a framework that can provide good results for predicting adoption, but its complexity is a barrier to use in many cases (Williams, Rana & Dwivedi 2015). Thus, this paper focuses on extending the use of TPB to estimate social impact, due to it being more pragmatic to implement a model from secondary data, due to fewer model components than UTAUT. Adoption models have successfully utilised TPB to capture the intentions of users to adopt a technology (Pavlou & Fygenson 2006; Pakravan & MacCarty 2020a). TPB has three main components that affect an individual’s choice of adopting a behaviour: (i) attitude towards the behaviour, (ii) social norms and (iii) perceived behavioural control. Data from the TPB can be integrated into a discrete choice model to determine the probabilities of a user adopting a behaviour (Pakravan & MacCarty 2020b).

1.3. Agent-based modelling

ABM is a method for describing the behaviour of a system by modelling it as a collection of individual entities called agents (Bonabeau 2002). Each agent acts independently and makes decisions based on the agent’s definition and model parameters. Much of the value of ABM comes from observing the patterns that emerge as agents interact (Hsu 2007). This emergent behaviour can be unexpected even from a simple model with just a few rules (Axelrod 1997). ABM differs from more traditional differential modelling in that it does not seek for a top-down governing equation for modelling the system behaviour, rather it builds the system from the bottom-up with individual agents (Vicsek 2002). ABM has been used to model segregation (Schelling 1971), wind farm landowner relations (Syal, Ding & MacDonald 2020), social behaviour (Smaldino et al. 2012), decision making (Meluso & Austin-Breneman 2018) and product adoption (Kiesling et al. 2012; Rai & Robinson 2015), among many other applications (Squazzoni 2010).

Importantly, the bottom-up approach of ABM allows for combining social factors into the modelling of adoption using Discrete Choice Analysis (DCA) (He et al. 2014). By building on existing adoption models, this paper adds measures of product impact in addition to indicators for agent adoption. The addition of these impact indicators enables ABM to estimate product impacts while a product is still in the early design stage by coupling models of the society with product parameters.

2. Product impact modelling framework

Products will affect people in different ways. This will depend on the stakeholder group and the use context of the product (Reich & Subrahmanian 2020). To model the impact of a product there must be both a model for the product and a model for society (Dorrestijn, van der Voort & Verbeek 2014). By having models for the product and society, connections can be made between product parameters and how they influence the impact a product has on society. This is an example of a bottom-up modelling approach using ABM, and the authors recognise that other approaches such as a top-down differential model using system dynamics could be used if the governing relations can be understood. The framework presented here combines model types in a novel way, using ABM, to extend modelling beyond product adoption – where most models stop – to that of product impact.

An overview of the framework is presented in Figure 1. Inputs for these models were selected based on those observed in the literature for product models (Fontes et al. 2018), societal models (Moss & Edmonds 2005), social impact models (Benoît et al. 2010), choice models (Brock & Durlauf 2001) and ABMs (Bonabeau 2002). Additional inputs may be added if necessary. As shown, models for the product, society, scenario and impact are created and integrated in an ABM. The ABM output data are used to carry out the impact assessment. Product impact data are fed back to the product model to iterate on design parameters to maximise the potential positive impact of the product. Each of these components is explained in detail below.

Figure 1. Product impact framework structure. Definitions for the input models will be explained in the following sections.

2.1. Framework: product model

This framework is intended to be used during product development when the design is still somewhat flexible, but enough must be known about the product’s design to adequately estimate its performance. Items such as the performance, geometry, material and potential distribution model may be incorporated into the product model. Essential product attributes that may influence the product impact should be chosen as inputs for the ABM. For example, a designer might choose efficiency, comfort and ease of use as three potential design parameters to examine, explore and iterate. These attributes will also be used in the DCA as a part of an individual agent’s choice of adoption. Adoption modelling using DCA will be covered in depth in Section 2.5. The essential part of the product model is to find a relationship between product parameters and how they will influence both the adoption of the product and the metrics for measuring impact.

2.2. Framework: society model

The first step in building a society model is selecting a target population for the simulation. There are two types of information needed about a target population: survey data on user attitudes, behaviours and preferences, and demographic data to build a synthetic population in the model (Wassenaar et al. 2004).

Data on user preferences can effectively be collected in a survey. This survey includes how important the selected product parameters are to a potential user (Alberini 1995), as well as information related to the TPB categories of attitude, social norms and perceived behavioural control surrounding the adoption of the product (Ajzen 1991). This framework focuses on a binary choice model, where the choice is to adopt or not adopt the product. A multinomial choice model may be considered where a user will choose between competing products, but this leads to a more complex model that is outside the scope of this paper due to needing preference data about competing products. Data about user preferences are needed for creating the adoption probability equations that will be discussed in Section 2.5.

The other set of data are about the demographics and behaviour of the target population. These data are used to inform agent actions and interactions. When complete knowledge of a population is not available, a synthetic population may be used. Synthetic populations create demographic data sets for agents that match the distribution of the population of interest (Müller & Axhausen 2010). Synthetic populations allow for more accurate simulation of both the individual behaviour of agents, and interactions between agents that are affected by individual-level demographics (Gargiulo et al. 2010). This helps ABM simulate emergent behaviour due to societal behaviour (Xu et al. 2017).

Another key structure of the society model is the social network structure of the particular target population (Chen et al. 2018). The framework in this paper uses a small-world network topology (Watts & Strogatz 1998) that has individuals within the network connect to k closest neighbours, and each individual has a probability of rewiring a connection to that of another random individual from the entire population. Small-world networks have been used widely in ABM adoption and diffusion models (Cowan & Jonard 2004; Schwarz & Ernst 2012; Rai & Robinson 2015) because they have been shown to follow patterns found in real-world social networks (Latora & Marchiori 2001).

2.3. Framework: scenario model

The scenario model used in this paper defines the scope of the ABM. This includes model parameters such as the duration of the simulation, physical attributes of the world created for agents and initial conditions of inputs for the ABM (Macal & North 2005). These initial conditions may include how many people initially have a product, the time period a simulation starts within, or an external force such as a law influencing an agents’ choice to adopt. Data gathered for the society model help generate building the physical attributes of the scenario with items such as the number of households or workplaces that may need to be created in the modelled world. It should be decided if one or multiple scenarios will be examined as design teams may be interested in the product impact over different contexts.

2.4. Framework: impact model

Much of the literature on assessing the social impact of products that are on the market will also apply to estimating impact. As outlined in SIA and Social Life Cycle Assessment, categories of impact need to be chosen and then indicators selected to measure the impact categories (Benoît et al. 2010; Fontes et al. 2018). Work on predictive social impact modelling adds the importance of determining how product parameters link to impact indicators (Stevenson et al. 2020). The framework in this paper utilises the 11 categories from Rainock et al. (2018) for social impact category selection. Clearly, the number of resources required to collect data will increase with the number of selected impact categories and indicators. Therefore, it may be important for the design team to narrow down the focus of the model to the number of impact categories that will enable data collection with available resources. Data from users should inform the selection of impact categories (Fontes et al. 2018).

2.5. Framework: ABM

Inputs from the product, society, scenario and impact models are integrated into an ABM. These inputs define the world that agents exist within, and the agents themselves. One of the key characteristics of any ABM is defining how agents interact with each other; the product and the world (Bonabeau 2002), these interactions are shown by the dashed arrows labelled 1, 2 and 3 in Figure 1. Agents may influence each other, such as when an agent has a poor experience with a product and their attitude towards the product becomes more negative, and dissuades, indirectly or directly, other agents from adopting the same product. The world may influence agents through stimuli defined in the scenario model, such as by changing transportation conditions available to agents to move around the world. Finally, agents may affect the world such as when an agent emits pollutants into the modelled environment. Data about the population’s beliefs and behaviours is integrated into variables held by individual agents. Distributions of the behaviours and beliefs of the agent population should be validated against the original survey data collected early in the process. The purpose of the ABM and framework is to ascertain the impact of the product, and in order to do that the rules regarding an agent’s choice to adopt the product are paramount.

Choice models have been used previously in product development (Li & Azarm 2000), social interactions (Brock & Durlauf 2001) and ABM (He et al. 2014). To isolate the effect of different product parameters on the social impact a binomial choice model is used, where only one type of product is available at a time of simulation. A multinomial choice model, where multiple variations of a product are available at one time, may more closely mimic real-world markets, but it becomes more difficult to interpret the impact results from a single product. An agent’s choice of adoption can be modelled as a utility function as shown in Eq. (1), where $ {U}_i $ is the product utility as understood by the agent $ i $ and the utility is calculated for each of the $ n $ agents. Agent $ i $ will adopt the product if $ {U}_i>0 $ . $ {U}_i $ contains an observed portion represented by $ {V}_i $ and an unobserved portion $ {\varepsilon}_i $ . The observed portion includes the influence of all product and TPB parameters where $ j $ is the number of parameters as shown in Eq. (2). The unobserved portion includes anything that may have been not included in the model as well as any error. $ \beta $ weightings may vary for individual agents or may be constant for the entire population. Varying $ \beta $ across the population shows different weightings of the factors. For example, $ \beta $ may vary across agents in the population if their preferences are influenced by age. The $ S $ variables will vary based on the conditions of the model and by agent interactions. By assuming a logit model, the unobserved portion of Eq. (1) is assumed to follow a logistic distribution (Greene 2009). This allows for simplifying the probability of the agent $ i $ adopting the product can be defined by Eq. (3). Therefore as the model progresses and conditions change, the probability of an agent adopting the product can change due to changes in values of the $ S $ terms of Eq. (2):

(1) $$ \hskip-.2pc {U}_i={V}_i+{\varepsilon}_i\hskip30em i\hskip2pt \in \hskip2pt \left\{1,2,\dots, n\right\}, $$

(2) $$ \hskip-1pc {V}_i={\beta}_{1i}{S}_{1i}+{\beta}_{2i}{S}_{2i}+\cdots +{\beta}_{ji}{S}_{ji}\hskip2em \forall j\hskip2pt \in \hskip2pt \left\{1,2,\dots, m\right\},\hskip4em i\hskip2pt \in \hskip2pt \left\{1,2,\dots, n\right\}, $$

(3) $$ \hskip0pc {P}_i=\frac{1}{1+{e}^{-{V}_i}}\hskip29em i\hskip2pt \in \hskip2pt \left\{1,2,\dots, n\right\}. $$

2.6. Framework: evaluation of the impact

As agents adopt the product, they will experience an impact. For example, if a product improves the health of an agent, that agent may be more likely to continue working, going to school, or survive to the next period. Measurements of aggregate impact across the population should be recorded for each run of the simulation. Depending on the model it may also be important to break down the impact by different demographic categories rather than a single measure of impact for the entire population. Product parameters can be iterated within the framework to see how changes in product parameters influence the impact of the product on society. It is important to note that because of the stochastic nature of an ABM, results will vary so it is important to run the model sufficient number of times to understand the distribution of results. By continuing to iterate on the parameters, a design team can understand what factors are most important to improve to maximise potential positive impact and minimise potential negative impact.

3. Framework illustration

To demonstrate how the framework is used, an illustration is presented in this section. During the COVID-19 pandemic the impact of face masks has been widely discussed. Because face masks are a product that most of the world has become familiar with in 2020, combined with the amount of data available surrounding COVID-19, it was chosen to demonstrate the framework. This illustration shows an example of exploring the design space of face masks by linking product parameters to social impact metrics. By doing so, a design team can explore the face mask design space to understand what product parameters are most influential to improve social impact. Simplifications have been made to the model to enable comparisons between mask types. For example, there is only one type of face mask available to agents in any given simulation run. This simplifies the model to a binary choice and makes the results clear as to which face mask has the greatest total impact. The model is further simplified by focusing only in the United States because preferences may change in different countries and more data were available to construct the model based on communities in the United States. This illustration uses a susceptible, infectious, recovered (SIR) approach to modelling COVID-19 (Chen et al. 2020; Cooper, Mondal & Antonopoulos 2020). The goal of this example is not to provide the most accurate model of COVID-19 transmission, but to show relative product impacts of a product with which most people are recently familiar.

3.1. Illustration: product model

Mask attributes of effectiveness, comfort and aesthetics were selected for modelling. An assumption was made that the price of a mask was not a significant factor for mask adoption in the United States given the generally low cost of a face mask. Five mask designs with different attributes were evaluated in the model: (i) a control case with no masks, (ii) an N95 mask with high effectiveness but low comfort, (iii) a cloth mask with medium effectiveness and comfort, (iv) a neck gaiter with low effectiveness but high comfort and (v) an ideal mask that is very effective, comfortable and aesthetic. Effectiveness was rated on a 0–5 scale, with 0 being no mask and 5 a mask that filters $ 100\% $ droplets. Comfort was rated on a −5 to 0 scale with 0 being no mask, with an assumption that all masks will be less comfortable than no mask. Aesthetics were rated on a −5 to 5 scale with 5 being a mask with excellent aesthetic qualities that will make a user want to wear it, while a score of −5 will deter use. Attributes for the effectiveness of face masks were built from data by Fischer et al. (2020) and Clapp et al. (2021). The authors acknowledge that the real-world effectiveness of a mask may not match those of a laboratory study, but for this exercise, it was assumed that the real-world effectiveness of a mask at reducing the risk of COVID-19 matched the filtration effectiveness from the literature. Data on the percent of the population that uses face masks correctly varies widely from 25% (Cumbo & Scardina 2021) to 90% (Cohen et al. 2021). For the illustration, it was assumed that 80% of agents who adopt a mask will use it correctly. If an agent was not wearing the mask correctly the efficacy of the mask was assumed to be cut in half. The attribute scores of each mask case can be seen in Table 1.

Table 1. Mask attributes

3.2. Illustration: society model

The goal of the society model is to fully define the model population with demographics, preferences, movement patterns and behaviours. Synthetic population attributes were based on Cape Elizabeth, ME and Angleton, TX. Each location will use a population of 10,000 agents. Cape Elizabeth has an older median age and a lower population density than Angleton. Two different locations were selected to observe how the results might change with different population demographics. The synthetic population was created using aggregate data from the 2019 US Census American Consumer Survey. The synthetic populations were created using an Iterative Proportional Fitting approach (Gargiulo et al. 2010), used previously for simulation of infectious disease transmission dynamics in American Samoa (Xu et al. 2017).

Assumptions and simplifications were made about the society to enable modelling such as all adults to travel to their workplace, and all children attend a school in person. The relative frequency of travelling to work, school and stores was defined with data from the American Time Use Survey (United States Bureau of Labor Statistics, 2020).

Relative user preferences were obtained through existing studies on mask use demographics (Igielnik 2020) and perceptions (Howard 2020). Older members of the population were assumed to have a higher preference for mask efficacy due to higher mortality rates for that demographic. The user preferences were used to obtain the utility function for a person $ i $ adopting the use of a face mask, see Eq. (4). Eq. (4) is then substituted into Eq. (3) to obtain the probability of the agent $ i $ adopting a mask on a given day in the model. The $ \beta $ coefficients are the relative weighting of different factors by an individual and the $ S $ terms are the scoring of that variable. Values and ranges for the variables of the utility function can be found in Table 2. The attitude towards mask scores, $ {S}_{ATB} $ , were initialized using a normal distribution with a median of −1.1, −1 and −0.9, respectively based on republican, independent or democratic political affiliation and standard deviation of 1 to simulate the population initially being more unfamiliar with face mask use (Igielnik 2020). Political affiliation was chosen as a way to initialize attitudes towards masks instead of other mask use predictors such as a belief in science (Stosic, Helwig & Ruben 2021), due to the availability of political affiliation data for the synthetic population. Social norm scores were set based on the percent of connections in the agent’s network that have adopted masks with 5 being all connections have adopted masks and −5 being no connections adopting masks. Virus severity was defined by the percent of the population currently infected with the virus. These variables define the utility of a face mask for each user:

(4) $$ {V}_i={\beta}_{ATB}{S}_{ATB i}+{\beta}_{SN}{S}_{SN i}+{\beta}_{PBC}{S}_{PBC}+{\beta}_{Ei}{S}_E+{\beta}_{Ci}{S}_C+{\beta}_{Ai}{S}_A+{\beta}_V{S}_V+{\beta}_{AV}{S}_{AV}. $$

Table 2. Variables and coefficients for mask DCA

Note: Asterisks: For weights with multiple values: first value for individuals age 19–39, second value for individuals age 40–65 and third value for individuals age 65+.

Individuals in the network built social connections based on a small-world network structure (Watts & Strogatz 1998) with their five closest neighbours and up to five individuals that share the same workplace. It was assumed that there was a 10% chance of deleting a social connection and forming a new connection with a random individual in the population.

3.3. Illustration: scenario model

The scenario model defines the scope and initial conditions of the simulation. The simulation was run for 5376-time steps equivalent to 8 weeks in the simulation. Each time step is equal to 15 minutes of simulated world time. Testing how face mask product parameters impact health and safety during COVID-19 was the main goal of the scenario. Initially, 25 people out of 10,000 were randomly infected with the virus. The SIR approach chosen for the virus allows agents to take on three conditions: (i) susceptible to the virus, (ii) infected with the virus and (iii) after 10 days they will recover or die (Cooper et al. 2020). Parameters of the virus such as mortality rates by age, infectiousness and duration of illness were set according to data from the Centers for Disease Control and Prevention (COVID-19 Pandemic Planning Scenarios 2020). In this model, the scenario takes place in a closed system, therefore there are no new individuals to the population that may add cases of COVID-19. Parameters for mask mandates, the perceived behavioural control term, $ {S}_{PBC} $ in Eq. (4), were included on a 0 to 5 scale with 5 meaning that masks are required by law. A term for the availability of masks was included, the term $ {S}_{AV} $ in Eq. (4) on a scale of −5 to 5 with −5 being masks are extremely difficult to obtain and 5 being a supply of masks is provided to each household automatically. To isolate the mask design impact from policy impacts $ {S}_{PBC} $ , and $ {S}_{AV} $ were held constant throughout the simulations. $ {S}_{PBC} $ was set at 1, which would be equivalent to suggesting the public wear a mask and $ {S}_{AV} $ was set at 0 which would mean that masks are available but a supply is not given automatically to each household. These parameters would be useful if examining the influence of policy decisions on the social impact, but if the interest is only in the product parameters, they should be held constant.

3.4. Illustration: impact model

The focus of the Framework Illustration section is to demonstrate the framework used for social impact, but similar approaches could be taken with economic or environmental impact or all impacts combined. Health and safety were selected as the impact category to assess from among the categories presented by Rainock et al. (2018). To measure health and safety, indicators of the total number of COVID-19 cases and the number of deaths in the simulation were selected as the indicators to assess impact. The goal is to minimise both indicators. By viewing the results of these indicators for the different mask scenarios a design team can understand which product attributes are most important and may lead to the most positive impact.

3.5. Illustration: ABM

Most parameters of the ABM have been defined in the other input models (see previous illustration sections). The ABM for this illustration was developed using the Netlogo software package (Wilensky 1999). To define how the virus can spread within the model, interactions between individuals were defined as two agents occupying the same coordinate in space. If this occurred there was a probability based on the COVID-19 parameters that the noninfected individual would contract the virus from the infected individual. Inputs to the model for the probability of influencing an agent’s attitude positively or negatively about masks were included to simulate media campaigns, news reports, and emerging evidence of mask use. This influence probability was held constant in the simulation to better explore the design space of mask parameters. Once at a given location, the agents would move in a random walk for the duration they were in a location such as a workplace, school or store to simulate random interactions that can take place throughout the day. By integrating the four input models in an ABM it is possible to model the health and safety impacts of face masks in the context of COVID-19.

3.6. Illustration: evaluation of the impact

To understand the range of results, 100 full simulations were completed for each potential mask design. A summary of the results is presented in Table 3. The results show that there is a statistically significant difference between all mask types and no masks with p-values <0.0001 from an analysis of variance test. A mask with low effectiveness, like a neck gaiter, still provides a significant reduction in COVID-19 cases but does not provide the same impact as a more effective mask. There was not a statistically significant difference in the number of COVID-19 cases between cloth, N95 and ideal masks with p-values ranging from 0.5 to 0.99 from a Tukey–Kramer test (Ramsey & Schafer 2012). There are differences in the results for each population. Angleton has a higher population density so it generally has more cases, while Cape Elizabeth with its older population has more deaths.

Table 3. Mask simulation results

Note: Each population contains 10,000 agents. Asterisks: The adoption % is the maximum adoption rate of face masks during the simulation.

To understand the individual mask attributes of effectiveness, comfort and aesthetics, a sensitivity analysis was performed using the cloth mask type as the baseline. This mask type was selected because cloth masks have become ubiquitous during the pandemic. Each of the mask attributes was improved by one unit while the others were held constant. Results from 100 runs of the simulation sensitivity analysis for each case are shown in Table 4.

Table 4. Sensitivity analysis

Abbreviations: A, aesthetics; C, comfort; E, effectiveness.

This sensitivity analysis shows that effectiveness is the most important attribute to improve by one unit to improve the positive impact. Effectiveness and aesthetics provided a statistically significant difference in cases, with effectiveness providing the greatest effect on the number of cases. With the limited resources that a design team has, the effectiveness attribute should be where the priority lies for the improvement of the design, as opposed to comfort or aesthetics.

The estimation of coefficients in the utility equation gives rise to a level of uncertainty. To understand the implications of uncertainty in the coefficients, propagation of uncertainty techniques was used (McClarren 2018). Each coefficient in the utility function was increased one at a time by 10% and the simulation was run 100 times. Results from these simulations can be seen in Table 5. Overall, these results show that the wrong estimation of a parameter by 10% does not change the output results of the simulation in a statistically significant way when comparing it to the base case of cloth masks using a Tukey–Kramer test (Ramsey & Schafer 2012).

Table 5. Propagation of uncertainty

This is how the framework would function in the design process. A model is built factoring in the product, society, particular scenario and potential impacts. As the design team iterates on product inputs in the model they can see how the estimated impact of the product changes. By observing these changes over iterations of product parameters the design team can see opportunities to increase the positive impact and reduce negative impact.

3.7. Illustration: verification and validation

It should be noted that this paper is focused on the theory of the framework shown in Figure 1 and the purpose of this illustration is to show how the framework could be implemented. All engineering models, even fundamental principles such as Hooke’s Law, are built on assumptions and approximations. Despite inexact results, models remain useful for estimating results, understanding the dynamics of a system and the relationships of parameters in that system to the results of a model (Epstein 2008). For those using ABM, it is important to be realistic and transparent about what is known and what is not about the model to conclude from the results. Due to gathering data on face masks from multiple sources without access to individual survey responses, assumptions were made for the scoring of coefficients in the utility function. Future work could include gathering necessary data about face masks in a single survey. The drawback of this is that opinions change over time. With that in mind efforts were made to validate the model. Although it is difficult to perform macro-validation of all model outputs, micro-validation and limited macro-validation were carried out as recommended by North & Macal (2007) and Wilensky & Rand (2015). There are seven recommended areas of validation for an ABM that were carried out, requirements validation, data validation, face validation, process validation, model output validation, agent validation and theory validation.

Requirements validation

Requirements for the model included understanding the impacts of face mask parameters on the health and safety of individuals. The model outputs of COVID-19 cases and deaths provide answers to the question of the health and safety impacts of face mask use during the COVID-19 pandemic. This fulfils the requirements of the model.

Data validation

The input data for the model were obtained from peer-reviewed literature or government agencies. When assumptions were necessary they have been stated. The synthetic populations were validated to have the same distributions as the real population, and virus parameters were validated to match real-world estimates at the time of writing (COVID-19 Pandemic Planning Scenarios 2020). It was verified that mask parameters matched lab studies (Fischer et al. 2020) and user preferences (Howard 2020).

Face validation

Qualitative measures of the results match intuition based on different cases of face masks and their reduction in COVID-19 cases. This was also carried out during verification checks that the model functioned properly and that changes in values produced an appropriate response.

Process validation

Growth rates for COVID-19 match those from the literature (COVID-19 Pandemic Planning Scenarios 2020). Interactions for virus spread occur when agents occupy the same coordinate space, this corresponds with virus spread happening when close to another person.

Model output validation

Because of the complexities of the system, it would be difficult to perform full-scale quantitative macro-validation of all model outputs. Efforts have been made to validate the output of the mask adoption rates. The results from the mask adoption rate output generally align with adoption rates of 40–90% found in community observations (Haischer et al. 2020).

Uncertainty exists in all types of models, including ABM. With proper verification and validation efforts, ABM can be a useful tool to provide output estimates and to explain patterns, trends and relationships that may exist to help prioritise resources in product development.

3.8. Illustration: model reusability

The time and resources required to build an ABM such as the COVID-19 face mask illustration can be prohibitive. By extending the use of a model to multiple products the cost of producing such a model may be more worthwhile. To test the reusability of models created from the framework, the illustration was extended to include the introduction of vaccines. Face masks and vaccines are closely related products where the models can be tested on the same target population with the same impact model. This enables the scenario, society and impact models to largely stay the same. Using models for more than one product should be a consideration during the model creation as deploying the model for subsequent products is generally more rapid.

The additional product parameter to assess for vaccine introduction is the number of doses that will become available each day. This would help inform decisions about scaling up manufacturing, and its impact on virus cases. To make these changes for the addition of the vaccine, a new utility equation for the adoption of the vaccine is created. It follows a similar form to Eq. (4) with some modifications as seen in Eq. (5). New variables are generated for the attitude towards the vaccine, effectiveness of the vaccine and extent of side effects. The variable descriptions and scores can be found in Table 6. Eq. (5) is substituted into Eq. (3) to determine the probability that an individual chooses to be vaccinated if doses are available. The values for the coefficients and scores were based upon a hypothetical vaccine that is 90% effective at preventing the contraction of the virus and has minimal side effects. Initial attitudes towards the vaccine were based on survey data that was assumed to have a normal distribution (Funk & Tyson 2020). The vaccine is administered to the oldest ages first and then to younger groups. Adult agents that made the choice to be vaccinated were selected at random if doses of the vaccine were available. The vaccine did not start being available in the model until 1 week had passed model time. Full efficacy for the vaccine required two doses in the model, and it was assumed that 80% of agents that received the first dose would go on to get the second dose. After one dose the vaccine was assumed to achieve half of its full efficacy. This scenario was run for 16 weeks of simulated time to better understand differences in the results due to the vaccine administration happening over time:

(5) $$ {V}_{nV}={\beta}_{ATBV}{S}_{ATBV i}+{\beta}_{SNV}{S}_{SNV i}+{\beta}_{PBCV}{S}_{PBCV}+{\beta}_{EV i}{S}_{EV}+{\beta}_{Ri}{S}_R+{\beta}_V{S}_V. $$

Table 6. Variables and coefficients for vaccine DCA

Note: Asterisks: For weights with multiple values: first value for individuals age 19–39, second value for individuals age 40–65 and third value for individuals age 65+.

Results from executing the framework with the addition of the vaccine can be seen in Table 7. Scenarios were tested with and without mask use. From the results, we see that the vaccine does help to reduce cases, but not as quickly as face masks that may be immediately available to the entire population. The best case is to have mask and vaccine use in the model.

Table 7. Mean simulation results for 100 model runs with vaccine introduction

Once again the purpose of the illustration of model reuse was to demonstrate how a design team might extend the model to new uses with the same society model. An initial model may take weeks or months to build depending on the data requirements, but if the model can be extended to a new use it may only take a matter of a few days or less to create the new model. Some model extensions are more difficult to implement than others. Social distancing measures, for example, would be more difficult to include because of greater complexities in the movement patterns of the agents in the simulated world.

4. Limitations and future work

The limitations of the framework largely lie in the challenges of ABM. The potential challenges can be grouped into two main categories that will be discussed further: model results and resources required. Within these categories, areas will be highlighted where further work can be performed to aid in overcoming the challenges.

4.1. Model results

The validity of results from predictive models has long been called into question (Epstein 2008) due to the difficulty of validating models. In spite of this difficulty, predictive models remain a useful tool and they continue to be used in many applications such as the COVID-19 pandemic (Wynants et al. 2020) and climate change (O’Neill et al. 2017). The validation of predictive ABM on innovation diffusion can happen for general models where the results can be validated against previous historical data. A key issue with this approach is that the creation of a general model that can be validated on historic data will likely not be specific enough for new innovations (Kiesling et al. 2012). Future work can be undertaken to validate the use of the framework on a new product or intervention and measure the product impact across a sample population.

Current strategies for handling the validation of ABM involve performing micro-validation techniques and limited macro-validation (Midgley, Marks & Kunchamwar 2007; North & Macal 2007). Micro-validation can be used on individual components of the model that can be validated by the modellers or using previously validated components. Macro-validation can be carried out on a limited basis and may be restricted to qualitative validation (Bonabeau 2002; Moss & Edmonds 2005). There are cases where experimental macro-validation is not possible, such as the case of modelling the integrity of a nuclear waste repository for 10,000 years, and good practice is to state assumptions clearly and justify them (McClarren 2018). Although the quantitative validation of ABM may be limited, there is still value in creating representations of a system to highlight places where opportunities may exist to maximise positive product impact (Wilensky & Rand 2015).

Because of the stochastic nature of an ABM, results can vary for different runs of the simulation. It is important to have a sufficient number of runs to understand the distribution of results (Epstein 1999). In the case of the framework presented in this paper, a sufficient number of runs for each instance of product parameters needed to be executed. This enabled the use of statistical tests to understand where there is a statistically significant difference in the product impact between the different product parameters.

4.2. Resources required

ABM can be resource-intensive to implement and execute. A balance needs to be created between making a model that is simple enough to implement with the available resources and complex enough to provide meaningful results (Rahmandad & Sterman 2008). Creating decision models requires detailed information about the preferences of individuals. Demographic and behaviour information is needed about the target population. In some countries, this information is accessible through government surveys but is not always available for a target area. In addition to the data about individuals, enough must be known about potential product designs to understand the potential product impacts. All of this combines to make the initial data collection for this framework time intensive. To lessen the resources required, it is recommended to use available data sets whenever it is appropriate.

Depending on the size of the models, they may also be computationally intensive to execute. Although this is not always possible to completely overcome, computational resources are more accessible than ever. Reducing the model to the minimum necessary scope helps balance the trade-off between accuracy and cost.

The resources required for implementation create a situation where this framework may not be feasible for small projects or products with limited distribution. For large-scale projects where products may affect many people, taking the time to use this framework to model the product impact will help inform design decisions to have better product impact outcomes.

5. Conclusion

To estimate the social impact of products, models of the product and models of society need to be integrated. They are integrated in this paper in a computational simulation-based framework that can be used to quickly explore a product’s design space in terms of how it will likely affect society. While not shown in this paper, the framework allows for the simultaneous inclusion of economic, and environmental impacts for an even more comprehensive impact assessment. Importantly, it is acknowledged that such frameworks require significant amounts of data to capture complex details. It is shown however that model reuse is possible and practical thus reducing the long-term costs. This is particularly important as societal impacts, coupled with economic, and environmental impacts, continue to be important measures of product success.

Design teams need tools to estimate the potential social impacts of design choices during the design process. By combining existing methods for impact assessment, and technology adoption into an ABM, the framework presented provides an outline for how a design team can assess the potential impact of many design candidates before the product reaches the market, and identify opportunities of how the design can be changed to maximise positive impact. Although the exact form of a model may differ according to the specific parameters of the product, and its use, the framework guidelines of combining models of the product, society, scenario and impact can be used to estimate impact. The framework presented allows impact to be measured for different population segments, such as if the deaths in the illustration had been reported by age group or for the entire target population like the aggregate values presented above. Tools such as the framework presented in this paper can be a valuable asset for design teams for improving the impact of engineered products.