Almaatouq et al. propose that to improve the generalisability and efficiency of the research in the social and behavioural sciences, experiments should be systematically selected from a large design space. The authors argue that this approach requires the application of the paradigm to large samples and thus extensive collaboration. However, we believe that large samples are useful but not necessary to systematically probe the topography of experimental space. Recently, a conceptually comparable approach called neuroadaptive Bayesian optimisation (NBO) has been developed in neuroimaging. NBO does not require large samples and can operate on an individual level in real time. We propose this approach can also be generalised to the social and cognitive sciences to further the design proposed by Almaatouq et al.
NBO combines real-time analysis of participants’ responses and machine learning into a closed-loop design to find the experimental parameters that maximise the target (cognitive or brain) measure while sampling across a large design space (Lorenz, Hampshire, & Leech, Reference Lorenz, Hampshire and Leech2017) (Fig. 1). Bayesian optimisation (BO) is an active sampling technique that learns from input data collected during a single-experimental testing session. The algorithm uses an acquisition function that balances exploration and exploitation to select an experiment (e.g., a particular combination of characteristics of the presented stimulus) for presentation to the participant from the experiment design space (Brochu, Cora, & de Freitas, Reference Brochu, Cora and de Freitas2010). Target measures are collected and modelled across the design space using Gaussian process regression. Through an iterative process of sampling the search space, a surrogate model of the target response across the full design space is estimated and updated with every sampling. BO will select the next experiment in the design space in an explorative manner early during the session (areas that have not been sampled yet and for which the uncertainty regarding the relationship with the target measure is high) and an exploitative manner towards the end of the session (areas that show the maximum brain/cognitive response and have been sampled previously such that predictability of the response is high). The NBO converges on the area of the design space that maximises the target measure for that individual participant within a few iterations, if the target measure is reliable and the effect size is sufficient. In one early study, researchers constructed an experimental space based on a meta-analysis of existing literature (as suggested by Almaatouq et al.), and used NBO with real-time fMRI analysis to identify cognitive tasks that maximally dissociated between frontoparietal attention networks at the individual level (Lorenz et al., Reference Lorenz, Violante, Monti, Montana, Hampshire and Leech2018). Thus, NBO brings an integrative experiment design approach to the level of the individual through incorporating the tools of active learning with real-time data analysis.
Figure 1. Overview of the integrative experiment design proposed by Almaatouq et al. (inner circle) and how it is operationalised in the neuroadaptive Bayesian optimisation (NBO) approach described in the current proposal (outer circle). This figure was created with Biorender.com.
We argue that integrative experiment designs as proposed may have parallels across multiple fields, and interdisciplinary exchange may be fruitful. NBO is currently being applied in developmental science to examine social development in infants (Gui et al., Reference Gui, Throm, da Costa, Haartsen, Leech and Jones2022; Wass & Jones, Reference Wass and Jones2023). Developmental researchers typically preselect a limited range of experimental conditions or stimuli based on specific theories. Similar to issues in social science identified by Almaatouq et al., this has limited progress because each study only probes selected questions and the relationship between theoretical models or different experiment designs remains uncharacterised. In contrast, integrative experiment designs allow experimenters to map brain or behavioural responses across a larger experimental space, including providing out-of-sample predictions for stimuli that are unsampled. This enables developmental researchers to simultaneously test multiple developmental theories through considering their predictions for the variation in responses across a larger space, in accordance with Almaatouq et al.'s proposal. This approach can be extended to other multidimensional spaces with different concepts mapped across dimensions of the search space, for example to explore the influence of contingency or sensitivity on infant attention.
Beyond testing theories in basic science, we think integrative experiment designs may also have clinical utility (Lorenz et al., Reference Lorenz, Hampshire and Leech2017, Reference Lorenz, Johal, Dick, Hampshire, Leech and Geranmayeh2021). We currently lack objective biomarkers with clinical utility for psychiatric conditions (Loth, Reference Loth2023). As in the social sciences described by Almaatouq et al., the field of psychiatry is challenged by reproducibility issues that stem from heterogeneity in participant populations, selection of single tasks based on particular theories, and broad analytic flexibility of the resulting data. This leads to difficulty with integrating clinical findings from different studies and hinders biomarker discovery. Integrative experiment designs allow researchers to expand the search space across multiple different tasks or analysis pipelines (Lorenz et al., Reference Lorenz, Hampshire and Leech2017). Adaptive designs can then be used to select the task and/or pipeline that shows greater individual deviation from population norms for a particular person, similar to Almaatouq et al.'s proposal of mapping individual-level traits across a design space. For example, NBO has recently been used to identify tasks sensitive to residual network function in individual patients with stroke and higher dissimilarity in responses in patients compared to controls (Lorenz et al., Reference Lorenz, Johal, Dick, Hampshire, Leech and Geranmayeh2021). Similarly, an experimental space can be constructed through characterising the similarity space of the output of different neuroimaging analysis pipelines and identify a pipeline with the most experimental sensitive contrasts, illustrating one way in which the “cartographer” can assign numerical coordinates to different locations in space (Dafflon et al., Reference Dafflon, da Costa, Vasa, Monti, Bzdok, Hellyer and Leech2022).
In summary, we argue that the approach proposed by Almaatouq et al. has relevance beyond the boundaries of social and behavioural sciences, and can be extended to the individual level by deploying active learning using real-time feedback during the data collection session itself (Fig. 1). Current implementations in developmental and clinical samples indicate that NBO is particularly promising in samples characterised by high heterogeneity. Further, the NBO approach can be generalised to the social sciences through use of real-time behavioural data collection (i.e., decisions, reaction times, or opinions). Integrative experiment approaches applied at the individual level using real-time data collection, such as NBO, will allow us to conduct reliable research that does not depend on large sample sizes and that can be applied in screening and clinical programmes, answering to Almaatouq et al.'s call to produce unbiased scientific findings that apply to particular real-world contexts.
Almaatouq et al. propose that to improve the generalisability and efficiency of the research in the social and behavioural sciences, experiments should be systematically selected from a large design space. The authors argue that this approach requires the application of the paradigm to large samples and thus extensive collaboration. However, we believe that large samples are useful but not necessary to systematically probe the topography of experimental space. Recently, a conceptually comparable approach called neuroadaptive Bayesian optimisation (NBO) has been developed in neuroimaging. NBO does not require large samples and can operate on an individual level in real time. We propose this approach can also be generalised to the social and cognitive sciences to further the design proposed by Almaatouq et al.
NBO combines real-time analysis of participants’ responses and machine learning into a closed-loop design to find the experimental parameters that maximise the target (cognitive or brain) measure while sampling across a large design space (Lorenz, Hampshire, & Leech, Reference Lorenz, Hampshire and Leech2017) (Fig. 1). Bayesian optimisation (BO) is an active sampling technique that learns from input data collected during a single-experimental testing session. The algorithm uses an acquisition function that balances exploration and exploitation to select an experiment (e.g., a particular combination of characteristics of the presented stimulus) for presentation to the participant from the experiment design space (Brochu, Cora, & de Freitas, Reference Brochu, Cora and de Freitas2010). Target measures are collected and modelled across the design space using Gaussian process regression. Through an iterative process of sampling the search space, a surrogate model of the target response across the full design space is estimated and updated with every sampling. BO will select the next experiment in the design space in an explorative manner early during the session (areas that have not been sampled yet and for which the uncertainty regarding the relationship with the target measure is high) and an exploitative manner towards the end of the session (areas that show the maximum brain/cognitive response and have been sampled previously such that predictability of the response is high). The NBO converges on the area of the design space that maximises the target measure for that individual participant within a few iterations, if the target measure is reliable and the effect size is sufficient. In one early study, researchers constructed an experimental space based on a meta-analysis of existing literature (as suggested by Almaatouq et al.), and used NBO with real-time fMRI analysis to identify cognitive tasks that maximally dissociated between frontoparietal attention networks at the individual level (Lorenz et al., Reference Lorenz, Violante, Monti, Montana, Hampshire and Leech2018). Thus, NBO brings an integrative experiment design approach to the level of the individual through incorporating the tools of active learning with real-time data analysis.
Figure 1. Overview of the integrative experiment design proposed by Almaatouq et al. (inner circle) and how it is operationalised in the neuroadaptive Bayesian optimisation (NBO) approach described in the current proposal (outer circle). This figure was created with Biorender.com.
We argue that integrative experiment designs as proposed may have parallels across multiple fields, and interdisciplinary exchange may be fruitful. NBO is currently being applied in developmental science to examine social development in infants (Gui et al., Reference Gui, Throm, da Costa, Haartsen, Leech and Jones2022; Wass & Jones, Reference Wass and Jones2023). Developmental researchers typically preselect a limited range of experimental conditions or stimuli based on specific theories. Similar to issues in social science identified by Almaatouq et al., this has limited progress because each study only probes selected questions and the relationship between theoretical models or different experiment designs remains uncharacterised. In contrast, integrative experiment designs allow experimenters to map brain or behavioural responses across a larger experimental space, including providing out-of-sample predictions for stimuli that are unsampled. This enables developmental researchers to simultaneously test multiple developmental theories through considering their predictions for the variation in responses across a larger space, in accordance with Almaatouq et al.'s proposal. This approach can be extended to other multidimensional spaces with different concepts mapped across dimensions of the search space, for example to explore the influence of contingency or sensitivity on infant attention.
Beyond testing theories in basic science, we think integrative experiment designs may also have clinical utility (Lorenz et al., Reference Lorenz, Hampshire and Leech2017, Reference Lorenz, Johal, Dick, Hampshire, Leech and Geranmayeh2021). We currently lack objective biomarkers with clinical utility for psychiatric conditions (Loth, Reference Loth2023). As in the social sciences described by Almaatouq et al., the field of psychiatry is challenged by reproducibility issues that stem from heterogeneity in participant populations, selection of single tasks based on particular theories, and broad analytic flexibility of the resulting data. This leads to difficulty with integrating clinical findings from different studies and hinders biomarker discovery. Integrative experiment designs allow researchers to expand the search space across multiple different tasks or analysis pipelines (Lorenz et al., Reference Lorenz, Hampshire and Leech2017). Adaptive designs can then be used to select the task and/or pipeline that shows greater individual deviation from population norms for a particular person, similar to Almaatouq et al.'s proposal of mapping individual-level traits across a design space. For example, NBO has recently been used to identify tasks sensitive to residual network function in individual patients with stroke and higher dissimilarity in responses in patients compared to controls (Lorenz et al., Reference Lorenz, Johal, Dick, Hampshire, Leech and Geranmayeh2021). Similarly, an experimental space can be constructed through characterising the similarity space of the output of different neuroimaging analysis pipelines and identify a pipeline with the most experimental sensitive contrasts, illustrating one way in which the “cartographer” can assign numerical coordinates to different locations in space (Dafflon et al., Reference Dafflon, da Costa, Vasa, Monti, Bzdok, Hellyer and Leech2022).
In summary, we argue that the approach proposed by Almaatouq et al. has relevance beyond the boundaries of social and behavioural sciences, and can be extended to the individual level by deploying active learning using real-time feedback during the data collection session itself (Fig. 1). Current implementations in developmental and clinical samples indicate that NBO is particularly promising in samples characterised by high heterogeneity. Further, the NBO approach can be generalised to the social sciences through use of real-time behavioural data collection (i.e., decisions, reaction times, or opinions). Integrative experiment approaches applied at the individual level using real-time data collection, such as NBO, will allow us to conduct reliable research that does not depend on large sample sizes and that can be applied in screening and clinical programmes, answering to Almaatouq et al.'s call to produce unbiased scientific findings that apply to particular real-world contexts.
Financial support
This work is supported by EU SAPIENS (grant number 814302), the Economic and Social Research Council (grant number ES/R009368/1), and SFARI GAIINS (grant number 10039678). The results leading to this publication have received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement number 777394 for the project AIMS-2-TRIALS. This Joint Undertaking receives support from the European Union's Horizon 2020 research and innovation programme and EFPIA and AUTISM SPEAKS, Autistica, SFARI. IMI disclaimer: The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results. Any views expressed are those of the authors and not necessarily those of the funders.
Competing interest
None.