Diversity of contributions is not efficient but is essential for science

Abstract

Dominant paradigms in science foster integration of research findings, but at what cost? Forcing convergence requires centralizing decision-making authority, and risks reducing the diversity of methods and contributors, both of which are essential for the breakthrough ideas that advance science.

The integrative experiment design approach advocated by Almaatouq et al. represents an intervention to accelerate the convergence of research findings in the social and behavioral sciences. Observations from the evolution of scientific fields over centuries lead to questions about whether the results of such an intervention would be uniformly positive. According to Kuhn (1962), all scientific fields go through initial periods, sometimes spanning centuries as in the case of physics, during which many concepts and competing models are proposed. This continues until a breakthrough insight reconciles discrepancies and establishes a dominant paradigm around which the field coheres. Dominant paradigms enable what Kuhn (1962) calls “normal science,” coordinated efforts to refine the paradigm and build evidence; however, the power structure surrounding a dominant paradigm can suppress alternative perspectives, making it difficult to prompt its reconsideration.

These observations from the history of science suggest potential unintended consequences of the intervention to accelerate convergence that Almaatouq et al. propose. Two sources of concern are the power structures that typically evolve to maintain organizing paradigms, and the potential they have to overly constrain the breadth of inputs considered, both of which can be problematic for a young science focused on diverse, multifaceted phenomena.

Who decides?

Pfeffer (1993) cautioned that fields with higher levels of consensus get that way via a core group of elite scholars who wield control. Imposing a framework to foster consensus requires some mechanism for decision making. For instance, what variables are included or receive more attention? When two groups of researchers have converged on the same topic, who gets the naming rights to the theoretical space? These issues are often sorted out via peer-review processes and citation of papers, which admittedly is not “efficient” but incorporates the judgments of many other researchers in the field, based on their assessment of the evidence. And, contrary to the authors' claim that no integrating frameworks exist, we point to a few recent examples in work on team process (i.e., Marks, Mathieu, & Zaccaro, 2001) and team structure (i.e., Hollenbeck, Beersma, & Schouten, 2012) that have been built upon by others based on the evidence supporting them. In the approach proposed by Almaatouq et al., the dimensions are “mapped” onto the design space before the experiment is run, by a “cartographer” – but how does this occur?

The solution – as proposed – is for machine learning to make such decisions for us. While an elegant solution sidestepping the potential of an individual or group of individuals making the decisions, machine-learning algorithms by their very nature have bias baked into them (e.g., Fu, Aseri, Singh, & Srinivasan, 2022). Furthermore, while machine learning undoubtedly has a substantial role to play in many areas of science, machine-learning models can only analyze the information provided, and are typically not able to identify variables that have not yet been considered but should be. While Almaatouq et al. would argue that such machines are flexible, and adaptive to change, we see this as overly optimistic. Indeed, across multiple disciplines and experiments, we can say with some certainty that a status quo – once set – is very difficult to change, as Kuhn's (1962) observations of the difficulty of challenging “dominant paradigms” demonstrates.

Another significant challenge stems from the strong incentives for researchers to introduce novel ideas in order to advance their careers. As Almaatouq et al. acknowledge, these incentives are at odds with efforts to promote convergence since there are few rewards for researchers who contribute to “normal science.” Though the authors attempt to brush this aside by pointing to examples from physics, it is important to note that fields requiring major infrastructure investment also tend to be more hierarchical, and also have significant struggles with other issues such as sexual harassment as well as gender gaps in participation and career length (Huang, Gates, Sinatra, & Barabási, 2020; National Academies of Science, Engineering, and Medicine, 2018). Thus the efficiency that can come from centralizing decision-making authority to accelerate convergence also risks introducing some of the known problems associated with consolidating power (Pfeffer, 1993).

Limiting diversity

The imposition of a framework for fostering convergence not only risks creating problematic power dynamics but also limiting the diversity of ideas in undesirable ways. Almaatouq et al. argue that their framework enables different studies to make their measures “commensurable.” They claim this can facilitate the integration of research using different methods, however, it most naturally lends itself to the use of the “high-throughput” techniques they mention, typically online experiments, to generate the volume of data needed for sampling the design space. This is likely to result in more uniformity in the methods and measures used. While some may see this as desirable, we point out that when different studies using different methods yield convergent patterns of results, the field can have greater confidence in those effects, as advocates of “full-cycle research” (e.g., Chatman & Flynn, 2005) point out. Conversely, findings using the same measures and methods might make the results of different studies “commensurable,” but could mask limits to generalizability. Indeed, just as we have made great strides to sample beyond undergraduate students, we need to continue to push scientists to replicate and extend their work beyond that of online samples, as such samples are limited in their ability to capture rich behavioral outcomes. We also need to continue to broaden connections across contexts and disciplines to enable surprising new breakthroughs to emerge (Shi & Evans, 2023).

The social and behavioral sciences are at an exciting nexus. Diversity is finally gaining traction: Historically underrepresented groups are bringing new theory and ideas to our historically homogenous field. Taken-for-granted knowledge is being falsified, or shown to only apply to the dominant groups. Exciting perspectives are just now being brought to fruition. To borrow from the author's terminology, the “unknown unknowns” are just starting to emerge due to the burgeoning diversity in the field. Will kicking off an intervention to force convergence, facilitated by machine learning, bake today's bias into algorithms that stymie the diversity that is just starting to take hold in our fields (Daft & Lewin, 1990)?