Almaatouq et al. propose an “integrative approach” to increase the generalisability and commensurability of experiments. They suggest systematically sampling and testing a subset of experiments – chosen randomly from a design space – to deduce inferences about the population of all potential experiments. Yet their metascientific approach, which loosely translates the “potential outcomes framework” underlying experiments (Holland, Reference Holland1986; Neyman, Reference Neyman1923; Rubin, Reference Rubin1977) to the science of experimentation itself, has one glaring omission (and misinterpretation of) – the role of sample diversity in reasoning generalisability. Their suggestion that “an explicit, systematic mapping of research designs to points in the design space (research cartography) ensures commensurability” (target article, sect. 3.1, para. 6) is, at best, an ex-post exercise to ensure validity, based on the myopic assumption that “there is nothing special about the subjects…in principle, what goes for subjects also holds for contexts” (target article, sect. 2.2, para. 4). In this commentary, we challenge these false notions of subsumed duality between contexts, population, and sample diversity, and highlight the importance of diversity in ensuring commensurability. We propose modifications to their design space to incorporate sample diversity.
Almaatouq et al. outline their design space as a Cartesian product of two factors – population which is “a set of measurable attributes that characterizes the sample of participants,” and context which is a “set of independent variables hypothesized to moderate the effect in question as well as the nuisance parameter” (target article, sect. 2.2, para. 2). Their conceptualisation, however, fails to account for myriad factors arising from the sample and sampling technique itself, which affects the scope of any experiment. We outline three challenges resulting from this.
First, as defined, “population” lacks accountability of representativeness, such as cultural outliers in social and behavioural science experiments, a point that has been argued extensively by critics to metatheories of behaviours (Arnett, Reference Arnett2009; Henrich, Heine, & Norenzayan, Reference Henrich, Heine and Norenzayan2010). This evident oversight on diversity undermines the role that sample features can play in introducing biases to experiments, invariably leading to methodological narrowness, generating spurious and misleading results (Gurven, Reference Gurven2018; Rad, Martingano, & Ginges, Reference Rad, Martingano and Ginges2018). An integrative framework, therefore, must measure diversity, both between- and within-countries (Ghai, Fassi, Awadh, & Orben, Reference Ghai, Fassi, Awadh and Orben2023). Without a careful consideration of representativeness in selected sampling approaches and the match of samples to population – be it through crowdsourcing platforms or distributed collaborative networks of different laboratories or sophisticated machine-learning algorithms – integrative experimental techniques will continue to yield noisy and biased results, inapplicable beyond specific population samples.
Second, integrative techniques must grapple with the limitations of not only imperfect sampling approaches but also the limiting assumptions in current disciplinary theories (Medin, Ojalehto, Marin, & Bang, Reference Medin, Ojalehto, Marin and Bang2017). This is particularly important in the context of Almaatouq et al., who cite that the “ultimate goal” of experiments is to arrive at a comprehensive theoretical understanding of experimental insights. Nonetheless, here, the authors assume (falsely) that metatheories emerging from the design space will naturally lead to heterogeneity and guarantee commensurability. While mapping theoretical boundaries and engaging in meta-metatheoretical reflections, in applying integrative experimental approaches, can be valuable for understanding the generalisability of existing theories, this does not necessarily address underlying structural issues contributing to the lack of theoretical diversity (Haeffel & Cobb, Reference Haeffel and Cobb2022). Our critique speaks broadly to need for behavioural sciences to see and reason complex adaptive systems with diverse samples (see Banerjee & Mitra, Reference Banerjee and Mitra2023; Hallsworth, Reference Hallsworth2023).
Third, acknowledging diversity is important since the costs of running integrative metaexperiments are largely unequal, thereby excluding researchers and relevant stakeholders in the Global South from generating integrated experimental insights. Since, Almaatouq et al. suggest that an “integrative approach would start by identifying the dimensions… as suggested … by prior research” (target article, sect. 3.1, para. 4), it is likely this space will then suffer from publication biases. Their claim that an integrative approach “will actually broaden the range of people involved in behavioral research” (target article, sect. 5.7, para. 1) is, at best, misguided, given this drawback. Integrative methods may be transferable but such initiatives are expected to be concentrated and accepted in Western contexts mostly (Singh, Reference Singh2022). Thus, while we share the optimism for large-scale collaborative science, we are less confident on the ability to draw robust, generalisable conclusions by relying on integrative approaches only. Overcoming the epistemic marginalisation of underrepresented groups in integrative experimental designs arguably is important to achieve this.
In view of these critiques, we propose a modification to their design space that we think is necessary to unlock the power of the integrative approach.
Our proposition relates to explicitly measuring (sample) diversity to quantify the heterogeneity within the sample, advancing the goals of Almaatouq et al. One approach might be using a scalar measure of representativeness for all population and contextual characteristics, for any given experimental point. This scalar measure can then be used to transform and reweigh the design space for generalisability. For example, the authors' conceptualisation of the population space merely accounts “for a set of measurable attributes” rather than a rich and diverse set of measurable attributes “that characterizes the sample of participants” (target article, sect. 2.2, para. 2). As such, their original design space is unduly influenced by certain population subsamples more than others. Here, sampled experimental points cannot be fully representative of all potential experiments. Nonetheless, our approach of first measuring diversity as a scalar index, to then transform these factors of the design space, just like a weighted sampling approach, increases reliability of integrative experiments (Deffner, Rohrer, & McElreath, Reference Deffner, Rohrer and McElreath2022). One limitation of this approach is that scalar measures of diversity may vary depending on the population, as what even counts as diverse samples will widely differ between the Global North and South contexts (Ghai, Reference Ghai2022). Here we call on the field to develop new ways to increase global diversity and analyse which of these might work best in optimising the design space (Tang, Suganthan, & Yao, Reference Tang, Suganthan and Yao2006).
Ultimately, given that many new sources of knowledge are likely to emerge from the Global South and that these are likely to deviate from Western-centric behavioural insights (Adetula, Forscher, Basnight-Brown, et al., Reference Adetula, Forscher, Basnight-Brown, Azouaghe and IJzerman2022), accounting for the sample's diversity will truly enhance the scope of integrative experimental methods. The future of experimental design must not only be integrative but also diverse and inclusive.
Almaatouq et al. propose an “integrative approach” to increase the generalisability and commensurability of experiments. They suggest systematically sampling and testing a subset of experiments – chosen randomly from a design space – to deduce inferences about the population of all potential experiments. Yet their metascientific approach, which loosely translates the “potential outcomes framework” underlying experiments (Holland, Reference Holland1986; Neyman, Reference Neyman1923; Rubin, Reference Rubin1977) to the science of experimentation itself, has one glaring omission (and misinterpretation of) – the role of sample diversity in reasoning generalisability. Their suggestion that “an explicit, systematic mapping of research designs to points in the design space (research cartography) ensures commensurability” (target article, sect. 3.1, para. 6) is, at best, an ex-post exercise to ensure validity, based on the myopic assumption that “there is nothing special about the subjects…in principle, what goes for subjects also holds for contexts” (target article, sect. 2.2, para. 4). In this commentary, we challenge these false notions of subsumed duality between contexts, population, and sample diversity, and highlight the importance of diversity in ensuring commensurability. We propose modifications to their design space to incorporate sample diversity.
Almaatouq et al. outline their design space as a Cartesian product of two factors – population which is “a set of measurable attributes that characterizes the sample of participants,” and context which is a “set of independent variables hypothesized to moderate the effect in question as well as the nuisance parameter” (target article, sect. 2.2, para. 2). Their conceptualisation, however, fails to account for myriad factors arising from the sample and sampling technique itself, which affects the scope of any experiment. We outline three challenges resulting from this.
First, as defined, “population” lacks accountability of representativeness, such as cultural outliers in social and behavioural science experiments, a point that has been argued extensively by critics to metatheories of behaviours (Arnett, Reference Arnett2009; Henrich, Heine, & Norenzayan, Reference Henrich, Heine and Norenzayan2010). This evident oversight on diversity undermines the role that sample features can play in introducing biases to experiments, invariably leading to methodological narrowness, generating spurious and misleading results (Gurven, Reference Gurven2018; Rad, Martingano, & Ginges, Reference Rad, Martingano and Ginges2018). An integrative framework, therefore, must measure diversity, both between- and within-countries (Ghai, Fassi, Awadh, & Orben, Reference Ghai, Fassi, Awadh and Orben2023). Without a careful consideration of representativeness in selected sampling approaches and the match of samples to population – be it through crowdsourcing platforms or distributed collaborative networks of different laboratories or sophisticated machine-learning algorithms – integrative experimental techniques will continue to yield noisy and biased results, inapplicable beyond specific population samples.
Second, integrative techniques must grapple with the limitations of not only imperfect sampling approaches but also the limiting assumptions in current disciplinary theories (Medin, Ojalehto, Marin, & Bang, Reference Medin, Ojalehto, Marin and Bang2017). This is particularly important in the context of Almaatouq et al., who cite that the “ultimate goal” of experiments is to arrive at a comprehensive theoretical understanding of experimental insights. Nonetheless, here, the authors assume (falsely) that metatheories emerging from the design space will naturally lead to heterogeneity and guarantee commensurability. While mapping theoretical boundaries and engaging in meta-metatheoretical reflections, in applying integrative experimental approaches, can be valuable for understanding the generalisability of existing theories, this does not necessarily address underlying structural issues contributing to the lack of theoretical diversity (Haeffel & Cobb, Reference Haeffel and Cobb2022). Our critique speaks broadly to need for behavioural sciences to see and reason complex adaptive systems with diverse samples (see Banerjee & Mitra, Reference Banerjee and Mitra2023; Hallsworth, Reference Hallsworth2023).
Third, acknowledging diversity is important since the costs of running integrative metaexperiments are largely unequal, thereby excluding researchers and relevant stakeholders in the Global South from generating integrated experimental insights. Since, Almaatouq et al. suggest that an “integrative approach would start by identifying the dimensions… as suggested … by prior research” (target article, sect. 3.1, para. 4), it is likely this space will then suffer from publication biases. Their claim that an integrative approach “will actually broaden the range of people involved in behavioral research” (target article, sect. 5.7, para. 1) is, at best, misguided, given this drawback. Integrative methods may be transferable but such initiatives are expected to be concentrated and accepted in Western contexts mostly (Singh, Reference Singh2022). Thus, while we share the optimism for large-scale collaborative science, we are less confident on the ability to draw robust, generalisable conclusions by relying on integrative approaches only. Overcoming the epistemic marginalisation of underrepresented groups in integrative experimental designs arguably is important to achieve this.
In view of these critiques, we propose a modification to their design space that we think is necessary to unlock the power of the integrative approach.
Our proposition relates to explicitly measuring (sample) diversity to quantify the heterogeneity within the sample, advancing the goals of Almaatouq et al. One approach might be using a scalar measure of representativeness for all population and contextual characteristics, for any given experimental point. This scalar measure can then be used to transform and reweigh the design space for generalisability. For example, the authors' conceptualisation of the population space merely accounts “for a set of measurable attributes” rather than a rich and diverse set of measurable attributes “that characterizes the sample of participants” (target article, sect. 2.2, para. 2). As such, their original design space is unduly influenced by certain population subsamples more than others. Here, sampled experimental points cannot be fully representative of all potential experiments. Nonetheless, our approach of first measuring diversity as a scalar index, to then transform these factors of the design space, just like a weighted sampling approach, increases reliability of integrative experiments (Deffner, Rohrer, & McElreath, Reference Deffner, Rohrer and McElreath2022). One limitation of this approach is that scalar measures of diversity may vary depending on the population, as what even counts as diverse samples will widely differ between the Global North and South contexts (Ghai, Reference Ghai2022). Here we call on the field to develop new ways to increase global diversity and analyse which of these might work best in optimising the design space (Tang, Suganthan, & Yao, Reference Tang, Suganthan and Yao2006).
Ultimately, given that many new sources of knowledge are likely to emerge from the Global South and that these are likely to deviate from Western-centric behavioural insights (Adetula, Forscher, Basnight-Brown, et al., Reference Adetula, Forscher, Basnight-Brown, Azouaghe and IJzerman2022), accounting for the sample's diversity will truly enhance the scope of integrative experimental methods. The future of experimental design must not only be integrative but also diverse and inclusive.
Financial support
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Competing interest
None.