Confidence in research findings depends on theory

Abstract

Almaatouq et al. view the purpose of research is to map variable-to-variable relationships (e.g., the effect of X on Y). They also view theory as this mapping of variable-to-variable relationships rather than an explanation of why the relationships occur. However, it is theory as explanation that allows us to reconcile disparate findings and that should guide application.

We agree with Almaatouq et al. that the integration of disparate research findings is often inefficient or fails to occur entirely. However, their proposed solution is based on an inadequate but widely shared conception of the nature of theory and its importance for application.

Almaatouq et al. assume the aim of research is to map variable-to-variable relationships (e.g., the effect of X on Y), and that current research has failed to do so adequately due to experiments that use incommensurate variables. Their approach is to use the literature and experience including previous experiments to identify a large number of variables that form a “design space” of experimental outcomes. By means such as sampling and predicting outcomes, boundaries in this space can be established that specify disparate sets of outcomes. This process results in a “theory” of how variables work together in complex ways. Almaatouq et al. remark that their integrative approach may strike many as atheoretical. In fact, it is atheoretical in that the focus is entirely on variables as opposed to explanatory theoretical constructs.

For Almaatouq et al. theory is a mapping of variable-to-variable relationships. Their approach entirely ignores the need for an explanation of why the relationships occur. Progress in research requires both the observation of variable relationships and their explanation. Observation informs theory and theory informs observation. Neither is sufficient, no matter what the scale of observation.

To illustrate their approach, Almaatouq et al. discuss the phenomenon of group “synergy.” They write that research on the topic often reaches conflicting conclusions, with some studies finding that groups outperform individuals and other studies finding that individuals outperform groups. They lament that, “researchers in this space have no way to articulate how similar or different their experiment is from anyone else's. As a result, it is impossible to determine… how all of the potentially relevant factors jointly determine group synergy…” (target article, sect. 1, para. 4).

However, the idea that research can determine how “all of the potentially relevant factors” influence the effect of one variable on another is ill-founded. Effects are always contingent, and the moderating variables that influence them are unbounded and evolving. Attempting to circumscribe an effect is a vain pursuit. Only theoretical explanation can resolve this problem.

Consider Almaatouq et al.'s example of the sort of systematic examination involving “commensurate” studies they favor, an investigation of moral dilemmas, “inspired by the trolley problem,” by Awad et al. (2018). They write that one of the findings is that the “ethical preference for inaction is primarily concentrated in Western cultures” (target article, sect. X, para. X). However, how stable is this finding?

Interestingly, Awad, Dsouza, Shariff, Rahwan, and Bonnefon (2020), using data collected from the same source as Awad et al. (2018), examine the actual trolley problem rather than studies inspired by it. They find that the preference for the inaction alternative (e.g., for not switching the trolley to kill one person in order to save five) tends to be greater in Eastern cultures than in Western ones. Moreover, even within these cultures, there is variation, such that some countries in the Eastern culture have a greater preference for inaction than those in the Western culture. Separately they note that acceptability of the action alternative has increased over time. Given the variance of the findings and their instability over time, what can be their relevance if not understood by means of underlying theoretical constructs?

Our argument is that the value of a finding lies in its ability to lead to theoretical understanding. For example, a higher-level theoretical explanation arising from the effects observed in moral dilemma experiments might center on a theoretical construct such as norms of social responsibility. We might hypothesize that this construct explains why the effects occur and why they might not occur in different cultures and contexts with different norms of social responsibility. Any theory is of course subject to revision through evaluation of additional evidence, but the essence of theory lies in the development of explanatory constructs that are not tied to any specific set of variables (Calder et al., 2021). And it is theory, not previously observed variable relationships, that should guide application (Calder, Phillips, & Tybout, 1981; Gal & Rucker, 2022, 2023) and that can reconcile seemingly disparate findings.

Consider, for example, the attraction effect (Huber, Payne, & Puto, 1982), where adding a choice alternative (the decoy), which is similar to but inferior to one of the alternatives (the target) but not the other (the competitor), increases the choice of the similar but superior alternative. This effect occurs reliably when the features of the choice alternatives are presented numerically but not when one of the features is presented perceptually. These findings indicate when the effect occurs, but not why numerical and perceptual information produce different effects. As a result, interest in this paradigm waned.

Introducing a theoretical framework that casts the task employed to demonstrate the attraction effect as involving cognitive resource allocation addresses this issue. The hypothesis is that choice is disambiguated by adopting an effort conservation goal that is accessible for numerical but not the more-complex perceptual information provided a means of documenting a perceptual attraction effect. It entailed reducing the effort required to make the decoy accessible as a comparison standard for the target and thus adoptable in making a choice (He & Sternthal, 2023). This framework has also been shown to account for why repeating a single persuasive message has a different effect than repeating different statements that are either truthful or false, and why the depletion effect and its reversal occur (Calder, He, & Sternthal, 2023). The introduction of a theoretical framework thus results in cumulative knowledge across seemingly disparate effects and paradigms.

We contend that confidence in the explanatory power and scope of theory is critical to reconciling seemingly disparate findings and to application in the social and behavioral sciences. Confidence in research findings arises through theoretical understanding not from attempting to map variable to variable relationships.