The concept of a design space is Almaatouq et al.'s major and important contribution to solving the incommensurability problem that arises in the social and behavioral sciences. The incommensurability problem, which the authors claim is caused in part by the promotion of “one-at-a-time” experiments that are conducted in theoretical isolation from other relevant experiments, has resulted in “irreconcilable theories and empirical results” (target article, sect. 1, para. 5). The kind of incommensurability at issue here is the inability to compare the same effect of interest across separate experiments. Call this dimensional commensurability. To address this, the design space's core features include (i) identification of plausibly relevant dimensions of the phenomenon of interest and (ii) assignment to each possible experiment a coordinate based on the dimensions the experiment is designed to investigate. Commensurability between experiments is thus supposed to be “baked in,” since all experiments directed at answering a given question can be compared in the design space, allowing for more nuanced theories that take into account varying dimensions and contexts. Although we agree that widespread use of a design space could help address such dimensional incommensurability, namely, the many variables that make social and behavioral theories particularly complex, our worry is that this strategy does not yet alleviate another major source of incommensurability: Conceptual discontinuity between research projects.
Implicit in the process of assigning design space coordinates to experiments is the assumption that each dimension will track the same concept across experiments. If an experiment E 1 investigates some dimension d and another experiment E 2 also claims to investigate d, then commensurability requires not only that d is identified as a variable or effect of interest in both cases but also that d is conceptually identical in both E 1 and E 2. In other words, merely using equivalent terms to refer to the same purported dimension d does not yet achieve conceptual identity. In seeking to assign the results of E 1 and E 2 to the design space, what justifies their respective locations? Taking their variables of interest at face value will yield one set of coordinate assignments. But if the concepts that underpin these variables are not the same, then their subsequent relation in the design space may be inaccurate or misleading. As has been highlighted elsewhere in the literature on experimentation in the social and behavioral sciences (e.g., Scheel, Tiokhin, Isager, & Lakens, Reference Scheel, Tiokhin, Isager and Lakens2021), investigating human behavior requires well-defined concepts to ensure that observations and measurements accurately and adequately capture the phenomena of interest. This kind of incommensurability, call it conceptual incommensurability, affects both the validity of any given experiment as well as the ability to effectively compare two different experiments.
To use one of the target article's examples, possible dimensions of the design space for experiments on “group synergy” will include individual-level traits such as “average skill,” “social perceptiveness,” and “cognitive style,” as well as group-level variables including “communications technology” and “incentive structure.” Mapping how these variables interact through a shared design space would considerably improve understanding of group synergy. But what the design space does not address is the ambiguity within singular concepts that both guides the research project (“group synergy”) and defines its relevant parameters (“average skill,” “social perceptiveness,” etc.). If it turned out that a set of experiments, each purporting to test the effect of social perceptiveness on group synergy, was differentially conceptualizing social perceptiveness, then it is not clear how the results of these experiments could be commensurable. To put it simply, can research group A be sure that research group B conceptualizes “social perceptiveness” in the same way? Plotting a set of experiments in a design space can obscure underlying conceptual discontinuities. There is reason to believe that research conducted under different guiding strategies might inherit conceptual discordance from the outset, precluding the construction of a design space in the first place (Lacey, Reference Lacey2005). Even prior to running experiments, the process of designing a space of possible experiments assumes that researchers will subsequently operate using those same concepts. This may turn out to be true, but not by fiat. Thus the design space, while addressing one aspect of the incommensurability problem (whether the stated effects of interest are the same) overlooks another (whether the concepts are equivalent).
Our concern is that the conceptual identity of the variables determining the design space must be ensured, not merely taken for granted. Without such conceptual identity, results are not guaranteed to be commensurable, and social and behavioral evidence will not be reconciled across experiments. Disagreements in two measurements cannot be resolved if the source of the discrepancy is unclear. Is it a problem in the measurement methodology? Is it a problem in the accuracy of the measurements themselves? Or is the problem that the measurements are either not capturing the relevant concept or not measuring the same concept across different studies? Feest (Reference Feest2022), for instance, argues that experimental psychologists must address three distinct reactivity challenges – all related to the ways psychological subject matter have dispositions to react to experimental contexts – in designing their experiments; Almaatouq et al.'s call for integrative experiment design across a design space suggests another challenge to designing experiments where genuine results can be distinguished from artifacts.
Thus, our suggestion is that, for commensurability to obtain in the design space as the authors advocate, researchers must first precisely specify the operative concepts. Such specification is challenging, as it involves what philosophers, historians, and scientists have called the problem of coordination: Coordinating measurement of directly observable entities with quantities of interest that can only be inferred from the observable entities (Kellen, Davis-Stober, Dunn, & Kalish, Reference Kellen, Davis-Stober, Dunn and Kalish2021). Since there is good reason to think that the specification of concepts used across the social and behavioral sciences does not occur on a systematic basis (e.g., Bringmann, Elmer, & Eronen, Reference Bringmann, Elmer and Eronen2022; Scheel et al., Reference Scheel, Tiokhin, Isager and Lakens2021), the use of concepts to generate the proposed design space will inherit the same conceptual incommensurability.
Debra Titone
[Opens in a new window]Debra Titone
, 
You have
Access
The concept of a design space is Almaatouq et al.'s major and important contribution to solving the incommensurability problem that arises in the social and behavioral sciences. The incommensurability problem, which the authors claim is caused in part by the promotion of “one-at-a-time” experiments that are conducted in theoretical isolation from other relevant experiments, has resulted in “irreconcilable theories and empirical results” (target article, sect. 1, para. 5). The kind of incommensurability at issue here is the inability to compare the same effect of interest across separate experiments. Call this dimensional commensurability. To address this, the design space's core features include (i) identification of plausibly relevant dimensions of the phenomenon of interest and (ii) assignment to each possible experiment a coordinate based on the dimensions the experiment is designed to investigate. Commensurability between experiments is thus supposed to be “baked in,” since all experiments directed at answering a given question can be compared in the design space, allowing for more nuanced theories that take into account varying dimensions and contexts. Although we agree that widespread use of a design space could help address such dimensional incommensurability, namely, the many variables that make social and behavioral theories particularly complex, our worry is that this strategy does not yet alleviate another major source of incommensurability: Conceptual discontinuity between research projects.
Implicit in the process of assigning design space coordinates to experiments is the assumption that each dimension will track the same concept across experiments. If an experiment E 1 investigates some dimension d and another experiment E 2 also claims to investigate d, then commensurability requires not only that d is identified as a variable or effect of interest in both cases but also that d is conceptually identical in both E 1 and E 2. In other words, merely using equivalent terms to refer to the same purported dimension d does not yet achieve conceptual identity. In seeking to assign the results of E 1 and E 2 to the design space, what justifies their respective locations? Taking their variables of interest at face value will yield one set of coordinate assignments. But if the concepts that underpin these variables are not the same, then their subsequent relation in the design space may be inaccurate or misleading. As has been highlighted elsewhere in the literature on experimentation in the social and behavioral sciences (e.g., Scheel, Tiokhin, Isager, & Lakens, Reference Scheel, Tiokhin, Isager and Lakens2021), investigating human behavior requires well-defined concepts to ensure that observations and measurements accurately and adequately capture the phenomena of interest. This kind of incommensurability, call it conceptual incommensurability, affects both the validity of any given experiment as well as the ability to effectively compare two different experiments.
To use one of the target article's examples, possible dimensions of the design space for experiments on “group synergy” will include individual-level traits such as “average skill,” “social perceptiveness,” and “cognitive style,” as well as group-level variables including “communications technology” and “incentive structure.” Mapping how these variables interact through a shared design space would considerably improve understanding of group synergy. But what the design space does not address is the ambiguity within singular concepts that both guides the research project (“group synergy”) and defines its relevant parameters (“average skill,” “social perceptiveness,” etc.). If it turned out that a set of experiments, each purporting to test the effect of social perceptiveness on group synergy, was differentially conceptualizing social perceptiveness, then it is not clear how the results of these experiments could be commensurable. To put it simply, can research group A be sure that research group B conceptualizes “social perceptiveness” in the same way? Plotting a set of experiments in a design space can obscure underlying conceptual discontinuities. There is reason to believe that research conducted under different guiding strategies might inherit conceptual discordance from the outset, precluding the construction of a design space in the first place (Lacey, Reference Lacey2005). Even prior to running experiments, the process of designing a space of possible experiments assumes that researchers will subsequently operate using those same concepts. This may turn out to be true, but not by fiat. Thus the design space, while addressing one aspect of the incommensurability problem (whether the stated effects of interest are the same) overlooks another (whether the concepts are equivalent).
Our concern is that the conceptual identity of the variables determining the design space must be ensured, not merely taken for granted. Without such conceptual identity, results are not guaranteed to be commensurable, and social and behavioral evidence will not be reconciled across experiments. Disagreements in two measurements cannot be resolved if the source of the discrepancy is unclear. Is it a problem in the measurement methodology? Is it a problem in the accuracy of the measurements themselves? Or is the problem that the measurements are either not capturing the relevant concept or not measuring the same concept across different studies? Feest (Reference Feest2022), for instance, argues that experimental psychologists must address three distinct reactivity challenges – all related to the ways psychological subject matter have dispositions to react to experimental contexts – in designing their experiments; Almaatouq et al.'s call for integrative experiment design across a design space suggests another challenge to designing experiments where genuine results can be distinguished from artifacts.
Thus, our suggestion is that, for commensurability to obtain in the design space as the authors advocate, researchers must first precisely specify the operative concepts. Such specification is challenging, as it involves what philosophers, historians, and scientists have called the problem of coordination: Coordinating measurement of directly observable entities with quantities of interest that can only be inferred from the observable entities (Kellen, Davis-Stober, Dunn, & Kalish, Reference Kellen, Davis-Stober, Dunn and Kalish2021). Since there is good reason to think that the specification of concepts used across the social and behavioral sciences does not occur on a systematic basis (e.g., Bringmann, Elmer, & Eronen, Reference Bringmann, Elmer and Eronen2022; Scheel et al., Reference Scheel, Tiokhin, Isager and Lakens2021), the use of concepts to generate the proposed design space will inherit the same conceptual incommensurability.
Financial support
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Competing interest
None.