Getting lost in an infinite design space is no solution

Abstract

Almaatouq et al. argue that an “integrative experiment design” approach can help generating cumulative empirical and theoretical knowledge. Here, we discuss the novelty of their approach and scrutinize its promises and pitfalls. We argue that setting up a “design space” may turn out to be theoretically uninformative, inefficient, and even impossible. Designing truly diagnostic experiments provides a better alternative.

Almaatouq et al. argue that research findings in the behavioral and social sciences are rarely conclusive and even less integrative or cumulative. They claim that these insufficiencies largely originate from a flawed approach to research design and the “one-at-a-time” procedure of planning and conducting experiments. Almaatouq et al. propose an alternative: “Integrative experimentation.”

While we share many of the concerns raised in the target article, we have reservations about the proposed alternative. First, the “integrative experimentation” idea is – in principal – not novel. Similar demands for integrative, diagnostic, and cumulative experimentation have been voiced repeatedly in the past (e.g., Brunswik, 1955; Campbell, 1957; Meehl, 1978; Platt, 1964). That said, truly cumulative research endeavors and rigid theory testing has not really been a strength of psychological science so far. Walter Mischel once called this the “toothbrush problem” of psychological science: “Psychologists treat other peoples’ theories like toothbrushes – no self-respecting person wants to use anyone else's” (Mischel, 2008). Not much has changed since then – at least not until the replication crisis hit psychology full force.

Second, and more importantly, constructing the “design space,” the first step in Almaatouq et al.'s integrative experimentation approach, is practically very difficult, if not impossible. Where should the dimensions that constitute a design space come from? One problem is that these dimensions are either theoretical constructs themselves or at least defined and informed by theoretical assumptions, and that their number is potentially infinite. Consider the group synergy/group performance example discussed in the target article: Design space dimensions such as “social perceptiveness” or “cognitive-style diversity” are theoretical constructs. The theories that underlie these concepts provide definitions, locate them in a nomological network, and allow the construction of measurement tools. The latter is particularly important because a measurement theory is necessary to test the construct validity of operationalizations (measures and/or manipulations) for a given construct (e.g., Fiedler, McCaughey, & Prager, 2021). So, each design-space dimension turns out to have its own design space. Constructing design spaces may therefore lead to infinite regresses.

Third, and related to our previous point, setting up a design space is necessarily contingent on both the focal hypotheses as well as all accompanying auxiliary assumptions, some of which are relevant for testing (and building) a theory, while others are not. For instance, operationalizing “cognitive-style diversity” either as the sum of the within-team standard deviation across cognitive styles (Aggarwal & Woolley, 2013) or as the average team members' intrapersonal cognitive-style diversity score (Bunderson & Sutcliffe, 2002) may be relevant for testing a measurement theory, yet irrelevant for testing a substantive theory. Almaatouq et al.'s claim that “All sources of measurable experimental-design variation are potentially relevant, and decisions about which parameters are relatively more or less important are to be answered empirically” (target article, sect. 3, para. 1) disregards the (in our view, important) distinction between conceptually relevant and conceptually irrelevant design space dimensions (see Gollwitzer & Schwabe, 2022).

Fourth, all design-space decisions necessarily require a universal metatheory. Such a metatheory does not exist, and it is unlikely to appear in the future. To be clear: We do not oppose to the idea of integrative experimentation. Indeed, we second Almaatouq et al.'s (and Walter Mischel's) claim for more theory building and integration (“cumulative science”). But without a metatheory that can help us define the design space, a theoretically guided and targeted “one-at-a-time approach” may eventually yield more solid cumulative evidence than a theory-free “integrative experimentation” approach.

Fifth, and finally, since resources and the number of participants required for conducting experiments are limited, randomly combining potentially infinite design factors is not efficient. Atheoretical sampling from the design space requires a tremendous amount of resources, which are better spent testing theories in a most rigid and truly falsification-oriented fashion.

Instead of randomly sampling from an infinite design space, a more useful rationale for setting up an integrative experimental series requires acknowledging a hierarchy regarding the informativeness and discriminability of potential design factor decisions (Fiedler, 2017). Design factors that can – either logically or theoretically – be defined as more crucial than others (e.g., because they relate to a core assumption of a theory or provide a strong test of the boundary conditions of a theory) should be prioritized over those that are more marginal. For instance, if a theory predicted that interacting groups outperform nominal groups more strongly in conjunctive than in additive tasks, an experimental series might focus on varying conjunctive and additive tasks (or varying different forms of group interaction), while holding other features, which are less relevant to a rigid test of this hypothesis, constant (e.g., culture) or treating these features as random factors (e.g., sample characteristics).

Indeed, modern learning methodology can support the identification of the most informative and diagnostic design factors. However, such models do not learn from random and independent data, but from results depending on design factors that were a priori defined, scaled, and considered relevant (or irrelevant) as well as design-space boundaries set by the experimenters. To conclude, conducting conclusive research means (1) improving the precision of formulated hypotheses, (2) specifying metahypotheses from which (at least some) auxiliary assumptions and boundary conditions can be deducted, and (3) pursuing a systematic, exclusive, and diagnostic testing strategy. From our perspective, systematic research design does not mean searching through a maximally affordable design space, but carefully designing experiments that exclude explanations, alternative phenomena, or assumptions. This goal can rather be achieved by systematic exclusion than exploration of possibilities (Wason, 1960). Scaling up such diagnostic experimental series would not require changing the methodological paradigm: On the contrary, truly integrative, metatheoretical, conclusive, and cumulative research is nothing less than proper execution of a long-known and widely accepted experimental methodology.