The integrative approach advocated by Almaatouq et al. starts with mapping a research field onto an n-dimensional design space that defines the universe of relevant experiments – what they call “research cartography” (target article, sect. 3.1 para. 2). They suggest that the design space's dimensions can be extracted from available taxonomies, prior experimental research, and practical experience. However, as they acknowledge, this approach is vulnerable to unknown unknowns: Taxonomies, prior experiments, and practical experience may all fail to identify important dimensions which should be included in the design space.
Here, we focus on one way of identifying unknown unknowns: High-throughput natural description. This approach may help research cartographers to uncover missing dimensions of the research design space, at a cost comparable to the later stages of the integrative experiment design.
To appreciate the value of high-throughput natural description, consider cases where researchers noticed a discrepancy between the experimental stimuli and the naturalistic variation of these stimuli. For instance, Schutz and Gillard (Reference Schutz and Gillard2020) showed that many experiments studying nonspeech auditory perception used flat tones as stimuli, despite the fact that such tones are unrealistic: Their content lacks dynamic changes found in the temporal structure of naturalistic sounds. Experiments that included such naturalistic content made novel discoveries about the auditory system. For example, a study of audiovisual integration showed that tones with a temporal structure similar to impact sounds, like the sound of a xylophone, but not flat tones, which lack temporal variation, were reliably integrated with visual information when participants judged tone duration (Schutz & Kubovy, Reference Schutz and Kubovy2009).
Similarly, Dawel, Miller, Horsburgh, and Ford (Reference Dawel, Miller, Horsburgh and Ford2021) and Barrett, Adolphs, Marsella, Martinez, and Pollak (Reference Barrett, Adolphs, Marsella, Martinez and Pollak2019) showed that many experiments studying face perception used highly standardised and posed facial configurations which are not representative of the real-world variation in facial configurations. When naturalistic facial configurations are used in experiments, reported findings differ from previous results. For example, using naturalistic facial stimuli, Sutherland et al. (Reference Sutherland, Oldmeadow, Santos, Towler, Michael Burt and Young2013) found that facial first impressions have three underlying dimensions (trustworthiness, dominance, and youthfulness/attractiveness) instead of just two (trustworthiness and dominance), as previously reported when standardised facial stimuli were used (Oosterhof & Todorov, Reference Oosterhof and Todorov2008; Todorov, Said, Engell, & Oosterhof, Reference Todorov, Said, Engell and Oosterhof2008).
In these examples, researchers noticed and resolved some discrepancy between the variation of experimental and real-world stimuli. Such an approach, while useful, does not completely solve the problem of unknown unknowns. This is because there may be many more real-world variations in stimuli that could update one's understanding of a phenomenon, if they were introduced in experimental designs. However, a researcher cannot identify them unless they have a thorough description of real-world variation.
One solution to this issue is “high-throughput natural description”: The systematic collection and annotation of large, representative corpora of real-world stimuli to identify unknown unknowns.
An example in the field of emotion perception demonstrates the value of this approach. By collecting and annotating 7 million pictures of faces and 10,000 hours of filmed video from the internet, Srinivasan and Martinez (Reference Srinivasan and Martinez2018) discovered that the emotion-category labels of disgust, anger, sadness, and happiness are associated with 1, 5, 5, and 17 “distinct” facial configurations, respectively. Such variation in the range of facial configurations conveying different emotions was an unknown unknown in the research cartography of emotion perception, and studies investigating responses to facial configurations expressing certain emotion categories have yet to investigate responses to the entirety of the observed variation, to the best of our knowledge (Barrett et al., Reference Barrett, Adolphs, Marsella, Martinez and Pollak2019). Thus, high-throughput natural description can aid in defining the design space of relevant experiments via the identification of unknown unknowns.
However, this solution is not an easy fix to the problem of unknown unknowns. Large-scale naturalistic observation is logistically challenging. Obtaining 7 million images of faces from the internet is in itself difficult, but the difficulty ramps up if researchers wish to obtain a sample of faces from more diverse sources. Furthermore, large-scale annotation can be as challenging as large-scale naturalistic observation. For example, creating a corpus of 7 million faces that is useful for answering different research questions requires annotating the images for meaningful dimensions. Coding action units (specific facial muscle movements) manually via human annotators in these images can require expertise, or can take years when the dataset is extremely large (Benitez-Quiroz, Srinivasan, & Martinez, Reference Benitez-Quiroz, Srinivasan and Martinez2016; Srinivasan & Martinez, Reference Srinivasan and Martinez2018). Furthermore, the pool of annotators must itself be (very) large, not only to deal with the size of the corpus, but also to identify relevant individual and cultural variations in the way coders perceive the dimensionality of the stimuli.
In sum, while high-throughput natural description aids in the identification of unknown unknowns of a research design space, it introduces significant logistical challenges. However, these challenges can be surmounted via a combination of mass collaboration, automation (a use case is already present in the aforementioned emotion perception example where Srinivasan & Martinez, Reference Srinivasan and Martinez2018, use a computer vision algorithm to annotate action units in the internet images; Benitez-Quiroz et al., Reference Benitez-Quiroz, Srinivasan and Martinez2016; Yitzhak et al., Reference Yitzhak, Giladi, Gurevich, Messinger, Prince, Martin and Aviezer2017), citizen science (Awad et al., Reference Awad, Dsouza, Kim, Schulz, Henrich, Shariff and Rahwan2018, Reference Awad, Dsouza, Bonnefon, Shariff and Rahwan2020; Hilton & Mehr, Reference Hilton and Mehr2021), and gamification (Long, Simson, Buxó-Lugo, Watson, & Mehr, Reference Long, Simson, Buxó-Lugo, Watson and Mehr2023). In fact, Almaatouq et al. already propose that these aforementioned solutions could be deployed in the later stages of the integrative experiment design
Nonetheless, the application of these solutions for executing high-throughput natural description should not be ignored, as they amplify concerns about the up-front costs and inclusivity of the integrative approach. Few research groups may have the resources to implement an integrative experiment design, and fewer groups still may be able to solve its unknown unknowns problem during the research cartography stage. While we are enthusiastic about the ideas in the target article, we believe it is necessary to be explicit and constructive about the requirements of an integrative experiment design approach.
Debra Titone
[Opens in a new window]Debra Titone
, 
You have
Access
The integrative approach advocated by Almaatouq et al. starts with mapping a research field onto an n-dimensional design space that defines the universe of relevant experiments – what they call “research cartography” (target article, sect. 3.1 para. 2). They suggest that the design space's dimensions can be extracted from available taxonomies, prior experimental research, and practical experience. However, as they acknowledge, this approach is vulnerable to unknown unknowns: Taxonomies, prior experiments, and practical experience may all fail to identify important dimensions which should be included in the design space.
Here, we focus on one way of identifying unknown unknowns: High-throughput natural description. This approach may help research cartographers to uncover missing dimensions of the research design space, at a cost comparable to the later stages of the integrative experiment design.
To appreciate the value of high-throughput natural description, consider cases where researchers noticed a discrepancy between the experimental stimuli and the naturalistic variation of these stimuli. For instance, Schutz and Gillard (Reference Schutz and Gillard2020) showed that many experiments studying nonspeech auditory perception used flat tones as stimuli, despite the fact that such tones are unrealistic: Their content lacks dynamic changes found in the temporal structure of naturalistic sounds. Experiments that included such naturalistic content made novel discoveries about the auditory system. For example, a study of audiovisual integration showed that tones with a temporal structure similar to impact sounds, like the sound of a xylophone, but not flat tones, which lack temporal variation, were reliably integrated with visual information when participants judged tone duration (Schutz & Kubovy, Reference Schutz and Kubovy2009).
Similarly, Dawel, Miller, Horsburgh, and Ford (Reference Dawel, Miller, Horsburgh and Ford2021) and Barrett, Adolphs, Marsella, Martinez, and Pollak (Reference Barrett, Adolphs, Marsella, Martinez and Pollak2019) showed that many experiments studying face perception used highly standardised and posed facial configurations which are not representative of the real-world variation in facial configurations. When naturalistic facial configurations are used in experiments, reported findings differ from previous results. For example, using naturalistic facial stimuli, Sutherland et al. (Reference Sutherland, Oldmeadow, Santos, Towler, Michael Burt and Young2013) found that facial first impressions have three underlying dimensions (trustworthiness, dominance, and youthfulness/attractiveness) instead of just two (trustworthiness and dominance), as previously reported when standardised facial stimuli were used (Oosterhof & Todorov, Reference Oosterhof and Todorov2008; Todorov, Said, Engell, & Oosterhof, Reference Todorov, Said, Engell and Oosterhof2008).
In these examples, researchers noticed and resolved some discrepancy between the variation of experimental and real-world stimuli. Such an approach, while useful, does not completely solve the problem of unknown unknowns. This is because there may be many more real-world variations in stimuli that could update one's understanding of a phenomenon, if they were introduced in experimental designs. However, a researcher cannot identify them unless they have a thorough description of real-world variation.
One solution to this issue is “high-throughput natural description”: The systematic collection and annotation of large, representative corpora of real-world stimuli to identify unknown unknowns.
An example in the field of emotion perception demonstrates the value of this approach. By collecting and annotating 7 million pictures of faces and 10,000 hours of filmed video from the internet, Srinivasan and Martinez (Reference Srinivasan and Martinez2018) discovered that the emotion-category labels of disgust, anger, sadness, and happiness are associated with 1, 5, 5, and 17 “distinct” facial configurations, respectively. Such variation in the range of facial configurations conveying different emotions was an unknown unknown in the research cartography of emotion perception, and studies investigating responses to facial configurations expressing certain emotion categories have yet to investigate responses to the entirety of the observed variation, to the best of our knowledge (Barrett et al., Reference Barrett, Adolphs, Marsella, Martinez and Pollak2019). Thus, high-throughput natural description can aid in defining the design space of relevant experiments via the identification of unknown unknowns.
However, this solution is not an easy fix to the problem of unknown unknowns. Large-scale naturalistic observation is logistically challenging. Obtaining 7 million images of faces from the internet is in itself difficult, but the difficulty ramps up if researchers wish to obtain a sample of faces from more diverse sources. Furthermore, large-scale annotation can be as challenging as large-scale naturalistic observation. For example, creating a corpus of 7 million faces that is useful for answering different research questions requires annotating the images for meaningful dimensions. Coding action units (specific facial muscle movements) manually via human annotators in these images can require expertise, or can take years when the dataset is extremely large (Benitez-Quiroz, Srinivasan, & Martinez, Reference Benitez-Quiroz, Srinivasan and Martinez2016; Srinivasan & Martinez, Reference Srinivasan and Martinez2018). Furthermore, the pool of annotators must itself be (very) large, not only to deal with the size of the corpus, but also to identify relevant individual and cultural variations in the way coders perceive the dimensionality of the stimuli.
In sum, while high-throughput natural description aids in the identification of unknown unknowns of a research design space, it introduces significant logistical challenges. However, these challenges can be surmounted via a combination of mass collaboration, automation (a use case is already present in the aforementioned emotion perception example where Srinivasan & Martinez, Reference Srinivasan and Martinez2018, use a computer vision algorithm to annotate action units in the internet images; Benitez-Quiroz et al., Reference Benitez-Quiroz, Srinivasan and Martinez2016; Yitzhak et al., Reference Yitzhak, Giladi, Gurevich, Messinger, Prince, Martin and Aviezer2017), citizen science (Awad et al., Reference Awad, Dsouza, Kim, Schulz, Henrich, Shariff and Rahwan2018, Reference Awad, Dsouza, Bonnefon, Shariff and Rahwan2020; Hilton & Mehr, Reference Hilton and Mehr2021), and gamification (Long, Simson, Buxó-Lugo, Watson, & Mehr, Reference Long, Simson, Buxó-Lugo, Watson and Mehr2023). In fact, Almaatouq et al. already propose that these aforementioned solutions could be deployed in the later stages of the integrative experiment design
Nonetheless, the application of these solutions for executing high-throughput natural description should not be ignored, as they amplify concerns about the up-front costs and inclusivity of the integrative approach. Few research groups may have the resources to implement an integrative experiment design, and fewer groups still may be able to solve its unknown unknowns problem during the research cartography stage. While we are enthusiastic about the ideas in the target article, we believe it is necessary to be explicit and constructive about the requirements of an integrative experiment design approach.
Acknowledgments
T. K. would like to thank Dr. Julie Grèzes for briefly discussing the current state of the face perception and social cognition literature.
Financial support
S. A. M. is supported by NIH DP5OD024566. J.-F. B. acknowledges support from grants ANR-19-PI3A-0004, ANR-17-EURE-0010, and the research foundation TSE-Partnership.
Competing interest
None.