Individual differences do matter

Abstract

The integrative experiment design proposal currently only relates to group results, but downplays individual differences between participants, which may nevertheless be substantial enough to constitute a relevant dimension in the design space. Excluding the individual participant in the integrative design will not solve all problems mentioned in the target article, because averaging results may obscure the underlying mechanisms.

Many of us probably have experienced that fear responses to certain stimuli, such as spiders or snakes, are not the same for everyone, even when we are in the same situation. The difference between such individual responses is evidently not negligible compared to those elicited by situation-specific conditions. However, this is what Almaatouq et al. suggest when they downplay the role of individual differences in their otherwise excellent proposal of integrative experiment design (target article, sect. 2.2, para. 4). In psychology, the so-called person–situation debate has been going on for more than a century, and Almaatouq et al. obviously belong to the “situation” faction. Although influential studies such as the famous work of Mischel (1968) have given the “person” faction a backlash since the second half of the twentieth century, there has been a recent resurgence in personality psychology that emphasizes the importance of individual differences (see Roberts & Yoon, 2022). Also in other fields, the importance of the individual has been recognized for quite some time. Recent approaches towards personalized medicine and therapy are probably the most prominent example. A PubMed search for these keywords in title or abstract yields over 33,000 results within the last 20 years. Personalized approaches can be found, for example, in treatment of certain forms of leukemia (Bazinet & Kantarjian, 2023) or in autoinflammatory diseases (Miner & Fitzgerald, 2023). In other fields, the importance of the individuality of behavior has been recognized as well: Even Drosophila shows idiosyncratic behavioral differences, which are caused by developmental variations in brain wiring rather than genetic factors (Linneweber et al., 2020).

However, it is not necessary to resort to personality to see that individual differences found experimentally in a single experiment can be as large as those possibly found by changing the situation, thus contradicting Brunswick's assumption cited in the target article. In magnitude reproduction tasks, participants often overestimate small magnitudes and underestimate large ones, a perceptual bias termed central tendency (Hollingworth, 1910). The central tendency is quantified by 1-slope of reproduced magnitude plotted over stimulus magnitude. In a duration reproduction experiment (Glasauer & Shi, 2022) with randomized stimuli, individual differences in central tendency ranged from 0.1 to about 0.7 (average 0.44), thus covering almost the whole range from veridical reproduction (central tendency 0) to stimulus independence (central tendency 1). In a different experimental situation, when the temporal order of the trials followed a random walk, the average central tendency decreased to 0.11, which means that in this condition duration reproduction was almost veridical.

This example shows that individual differences within one situation can be as large, or even larger, as differences caused by a change of situation. Thus, individual differences can be large enough to be eligible as separate design dimension. However, targeted sampling along that dimension is hardly possible – which criterion would tell us which individual to test? Moreover, the inclusion of the individual as explicit dimension in the design space may often be not viable, because individuals correspond to discrete variables, and “meaningfully distinguishing between the various settings of a discrete variable could require dozens or even hundreds of descriptors” (Eyke, Koscher, & Jensen, 2021). Notably, this is a problem that might also affect the proposed dimension population, which could also be composed of many more different descriptors than can be tested.

The usual way to deal with individual differences is to model them as random effect (Yarkoni, 2022). In this view, individual differences are treated as variation of unknown origin without interest in the question. Thus, the solution to the problem of individual differences is to simply constrain the theories to the group level so that individual variation eventually averages out when groups are sufficiently heterogeneous and large.

In the study mentioned above (Glasauer & Shi, 2022) instead of treating individual differences as random effect, we could explain them by a Bayesian model which assumes that participants entertain individual beliefs about how sequential trial-by-trial stimuli are generated. The same model also predicted the massive change in central tendency from one condition to the other (0.44 down to 0.11) based on the individual differences identified in the first condition, that is, without changing the individual model parameters. Thus, behavioral differences observed in different experimental situations do not necessarily indicate actual changes in participants’ characteristics (such as beliefs, or personality). This latter conclusion does not depend on whether our theoretical model is correct: The model demonstrates that it is possible.

Thus, while a group-level theory might explain the situation-dependent change, this approach would not allow for a theory that links observed differences to a variation in individual characteristics. The same holds for Drosophila behavior: Considering only average behavior could not reveal that individual differences are not just random but have the distinct reason of being caused by variation in neuronal wiring.

A possible solution for including individual differences as important information in the integrative experiment design proposal could be to consider the interindividual variability of the variable of interest as additional input for the sampling procedure. Large variability could on the one hand indicate that the particular context of an experiment is not sufficiently constrained, thus leaving too much space for individual differences, indicating design dimensions that have not been included. On the other hand, in the example above the point in design space that resulted in small variability was hiding possible interindividual differences, and thus, from the perspective of theory building, the point with large variability might be the more interesting one. Thus, a sampling procedure that considers interindividual variability could help in defining regions in design space that provide either situations with homogeneous behavioral results, or situations unraveling differences that are of interest for any theory interested in the individual.