Uncovering hidden patterns of design ideation using hidden Markov modeling and neuroimaging

Abstract

The study presented in this paper applies hidden Markov modeling (HMM) to uncover the recurring patterns within a neural activation dataset collected while designers engaged in a design concept generation task. HMM uses a probabilistic approach that describes data (here, fMRI neuroimaging data) as a dynamic sequence of discrete states. Without prior assumptions on the fMRI data's temporal and spatial properties, HMM enables an automatic inference on states in neurocognitive activation data that are highly likely to occur in concept generation. The states with a higher likelihood of occupancy show more activation in the brain regions from the executive control network, the default mode network, and the middle temporal cortex. Different activation patterns and transfers are associated with these states, linking to varying cognitive functions, for example, semantic processing, memory retrieval, executive control, and visual processing, that characterize possible transitions in cognition related to concept generation. HMM offers new insights into cognitive dynamics in design by uncovering the temporal and spatial patterns in neurocognition related to concept generation. Future research can explore new avenues of data analysis methods to investigate design neurocognition and provide a more detailed description of cognitive dynamics in design.

Introduction

Design cognition has been a significant area of interest in design research. Traditional approaches to studying design cognition typically relies upon subjective and qualitative techniques. Researchers need to infer, or participants need to report, the internal processes in the designer's mind that align with design behavior through observations, questionnaires, or interviews (Chiu and Shu, 2011; Dinar et al., 2015). Such approaches allow the research to be performed in-situ or in controlled experiments. However, these approaches are limited by their intrinsic subjective nature and extensive qualitative data processing requirements (Chiu and Shu, 2011; Hay et al., 2017). To overcome some of these limitations and combine more quantitative methodologies in design cognition research, an emerging research area in the design research community, often referred to as “design neurocognition”, is seeking to apply techniques from cognitive neuroscience to measure brain activity related to design and advance knowledge of design cognition (Liu et al., 2018; Goucher-Lambert et al., 2019; Hu and Shealy, 2019; Gero and Milovanovic, 2020; Vieira et al., 2020; Zhao et al., 2020; Balters et al., 2023; Hay et al., 2022).

Functional magnetic resonance imaging (fMRI) is one of the neuroimaging techniques used to measure design neurocognition. fMRI offers a more direct understanding on the whole-brain neurocognitive processes that correlate with design behavior and support design activity. Classical analysis of fMRI data usually focuses on a pre-specified “event” (e.g., event-based design matrix) or time points (e.g., specific time window or sliding window). Significant assumptions are required in the pre-specification relating temporal and spatial information to uncover meaningful links between brain activity and participant behavior in response to experimental tasks. Additionally, this type of analysis leads to a loss of information from the entire dataset, especially the dynamics in the process. In this work, an unsupervised machine learning technique, hidden Markov modeling (HMM), is used to automatically infer states and their spatial and temporal patterns in underlying fMRI data related to design cognition without prior specifications on event-based design matrix or time window for fMRI data analysis.

HMM is a generative model that describes data in a temporal sequence of a finite number of discrete states. Prior research in both design and neuroscience domains has demonstrated that using HMM provides valuable insights into temporal patterns in varying types of data, for example, a short-timescale sequence in design behavior data (McComb et al., 2016, 2017a, 2017b), and dynamic patterns (states) of neural activation in large-scale resting-state fMRI data (Vidaurre et al., 2017, 2018). A prior study by the authors also used HMMs to extract distinct states in the fMRI data and find differences in neurocognitive patterns between participants with different performance levels (Goucher-Lambert and McComb, 2019). In that prior work, participants were assigned to high- and low-performing groups based on idea fluency (i.e., the number of concepts generated in a fixed time). Half of the designers with higher design fluency were assigned to the high-performing group, while the other half were assigned to the low-performing group. Significant differences were found between these two groups in the number of solutions generated in every 15-second block. Differences were also observed in the state occupancy between the high- and low-performing designers (Goucher-Lambert and McComb, 2019).

However, the neural activation patterns associated with the distinct states identified in the prior work (Goucher-Lambert and McComb, 2019) are still unknown. There is a lack of understanding of the specific brain regions involved in each neurocognitive pattern plus corresponding cognitive functions. The current work builds on (Goucher-Lambert and McComb, 2019) by investigating the patterns of neural activity, linking them to physical locations in the brain, and inferring the cognitive functions associated with each of the 12 states discovered in prior work. The findings suggest that the states extracted from fMRI data using HMM are linked to varying brain regions and associated with different cognitive functions that provide meaningful explanations for different performances in concept generation.

Background

This work employs neuroscience experiments (i.e., fMRI) and a machine learning technique (i.e., HMM) to explore dynamic neurocognitive patterns related to design concept generation. This section first introduces design research that applied fMRI to understand brain activities during design and concept generation. Then, critical brain regions and large-scale networks associated with the concept generation process are summarized. This section also discusses HMM and its application to neuroimaging data and design research.

fMRI and design neurocognition

A growing body of research is using neuroimaging techniques to investigate brain activities relevant to design in multiple phases, for example, design concept generation (Fu et al., 2019; Goucher-Lambert et al., 2019; Hay et al., 2019; Hu et al., 2019, 2021; Shealy et al., 2020), design decision-making (Goucher-Lambert et al., 2017b; Hu and Shealy, 2020, 2022), and open design or problem-solving (Zhao et al., 2020; Vieira et al., 2022b). A variety of neuroimaging techniques have been employed to measure design neurocognition, such as electroencephalography (EEG), functional near-infrared spectroscopy (fNIRS), and functional magnetic resonance (fMRI). EEG and fNIRS are portable in data collection but limited in spatial resolution. EEG cannot pinpoint the specific brain regions where the electrical signal comes from (Burle et al., 2015). fNIRS usually has a limited number of light sensors and a shallow penetration depth, so it is restricted to cover only the outer cortex (Quaresima and Ferrari, 2019). In contrast, fMRI provides excellent spatial resolution and rich information on brain activity through whole-brain scanning. However, a limited number of fMRI studies have investigated design or concept generation considering the lack of mobility and high cost of operation in an fMRI experiment (Hay et al., 2022).

One of the first fMRI study related to design was performed by Goel and Grafman (2000) which explored the difference between architects with and without lesion to the prefrontal cortex, and found that the right dorsolateral prefrontal cortex (PFC) was necessary for ill-structured representation and computation in room space design. Another early study that adopted fMRI to investigate design was by Alexiou et al. (2009). This study found distinguishing cognitive functions and brain networks when performing architectural room layout tasks in two forms: (1) ill-defined and open design and (2) well-defined and constrained problem-solving. The study also identified that higher activation in the right dorsolateral prefrontal cortex was associated more with open design than problem-solving (Alexiou et al., 2009), which was confirmed by a recent EEG study that extended Alexiou et al. (2009)'s work by investigating the open design tasks at three distinct stages and found increased activation in ideation stages in alpha 2 and beta 3 band in the PFC (Vieira et al., 2022b). Another two fMRI studies related to design decision-making include Sylcott et al. (2013) and Goucher-Lambert et al. (2017b) that used fMRI to understand product preference judgment when users made trade-offs between different design variables (e.g., form, function, and environmental impact) and found varied brain regions associated with each of the decision attributes.

Design concept generation, or design ideation, is arguably the most critical phase for injecting creative inspiration and shaping the creativity of subsequent design phases (Cross, 2001; Yang, 2009; Hay et al., 2019). The design research community is increasingly interested in using neuroimaging methods to understand performance (e.g., quantity, quality, creativity, etc.) and cognitive processes related to design concept generation. Ellamil et al. (2012) used fMRI to investigate the cognitive difference between creative generation and evaluation. The results found that the medial temporal lobe was central to the generation of novel ideas while evaluation mainly involved the executive regions for affective and visceroceptive evaluative process. Hay et al. (2019) compared the neurocognitive activation during concept generation between open-ended and constrained design ideation tasks but found no significant difference between the two conditions. However, they did identify increased activation in the left cingulate gyrus and right superior temporal gyrus during ideation. Fu et al. (2019) studied the neurocognitive patterns associated with design fixation in concept generation. They found increased activation in areas associated with visuospatial processing (e.g., left middle occipital gyrus and right superior parietal lobule regions). Goucher-Lambert et al. (2019) investigated design concept generation with and without the support of inspirational stimuli (e.g., text-based analogies) and identified two separate patterns of brain activation: one is associated with the successful application of inspirational stimuli to generate design solutions via insight in the temporal and parietal lobes, and the other is the currently unsuccessful and external search for insights in the primary visual processing-related brain regions.

Important brain regions and networks for ideation and insights

Even though only a limited number of fMRI studies have been performed to understand design concept generation (Alexiou et al., 2009; Ellamil et al., 2012; Sylcott et al., 2013; Goucher-Lambert et al., 2017b; Fu et al., 2019; Hay et al., 2019), ideation (i.e., concept generation) and insights are widely studied in the neuroscience literature that used fMRI (Blumenfeld et al., 2011; Benedek et al., 2014; Green et al., 2015; Beaty et al., 2016; Heinonen et al., 2016; Shen et al., 2018; Benedek and Fink, 2019) or design neurocognition studies that used other neuroimaging techniques (Shealy and Gero, 2019; Hu et al., 2021; Vieira et al., 2022a, 2022b). The process of generating insights and new ideas requires complex cognitive processes of attention, cognitive control, and memory (Fink et al., 2007; Benedek et al., 2018; Benedek and Fink, 2019). Some brain regions and large-scale brain networks have been shown to play critical roles in supporting ideation and insight. Prior research highlights activity within the brain regions from the default mode network (DMN) and executive control network (ECN) as being particularly influential (Ellamil et al., 2012; Beaty et al., 2016; Heinonen et al., 2016). DMN–ECN interactions also occur during cognitive tasks that involve generating and evaluating creative ideas (Ellamil et al., 2012; Beaty et al., 2016), and the dynamic transitions between default and control network are facilitated by the salience network (Uddin, 2015; Beaty et al., 2018).

DMN predominantly includes the medial prefrontal cortex (mPFC), the posterior cingulate cortex (PCC), and the medial and inferior parietal cortex. DMN activity may engage in spontaneous and associative processes, such as self-generated and internally-directed thought during mind wandering, mental stimulation, and episodic memory retrieval (Beaty et al., 2020). Such self-generated and internally-directed cognition contributes to concept generation by deriving useful information from long-term memory (Beaty et al., 2016, 2020). Prior neuroimaging studies found strong activation within the DMN related to creative processing by analogy (Beaty et al., 2016, 2020; Benedek and Fink, 2019). For instance, the mPFC shows higher activation during the novel generation of words with analogies (Green et al., 2015). Likewise, activation in the PCC is associated with creative idea generation through metaphor production (Benedek et al., 2014).

The ECN mainly comprises the dorsolateral prefrontal cortex (DLPFC) and the anterior cingulate cortex (ACC). The ECN has been linked to the support of internal representation, working memory, and relational integrations in creative cognition literature (Gilhooly et al., 2007; Beaty et al., 2016; Heinonen et al., 2016). The PFC, especially the dorsolateral PFC, is heavily involved in encoding of relational information and executive control when retrieving information from working memory (Green et al., 2010; Blumenfeld et al., 2011). Working memory is necessary to focus attention on and maintain executive control over elements related to concept generation (De Dreu et al., 2012). A prior study found activation in the dorsolateral PFC, especially in the left hemisphere, is dominant in concept generation (Shealy and Gero, 2019). The ACC activity is also a consistent finding in creative analogical thinking tasks for executive processes of response conflict and response selection between different ideas (Green et al., 2015).

Insights also rely on memory. The temporal cortex, a brain region in charge of semantic and episodic memory, is often involved in creative insight (Shen et al., 2017). Temporal regions, especially the medial temporal lobe, have been closely linked to the function of breaking mental sets and establishing remote and novel associations, which then can trigger insight experience (Zhao et al., 2013; Shen et al., 2018). Prior design neurocognition research also found higher activation in the temporal regions during creative ideation (Ellamil et al., 2012; Hay et al., 2019) and concept generation with inspirational stimuli (Goucher-Lambert et al., 2019). Other brain regions, such as the primary visual processing-related brain region in the occipital lobe, show activation in creative processing as well. While it is usually connected to participants being unable to solve problems with insights (Kounios et al., 2006), design fixation without new ideas (Fu et al., 2019), or a continued external search without insights (Goucher-Lambert et al., 2019) in design cognition.

Application of HMM in neuroscience research

Previous research in design neurocognition (mentioned in Sections “fMRI and design neurocognition” and “Important brain regions and networks for ideation and insights”) provides valuable information related to concept generation. However, most studies followed classical fMRI data analysis methods that depend on significant assumptions. The temporal and spatial information regarding the fMRI data needs to be assumed beforehand to extract meaningful statistics linking brain activity to participant behavior in response to tasks (e.g., a design matrix that specifies time of event in general linear model methods). These analysis techniques are locked to specific time points (e.g., when the neural process of interest occurs) and do not uncover connections between brain regions that may be correlated in space and time. These methods might be limited when the neural process of interest (e.g., ideation) is complex and not easy to pre-specify. In addition, the dynamics in the fMRI data are hard to capture when using classical methods. To catch the dynamic information in design cognition without making assumptions on the structure of the data, HMM is adopted in this work to automatically infer states in fMRI data related to design cognition without prior assumptions.

HMM uses a probabilistic approach to describe the data as a dynamic sequence of discrete states with a flexible definition of distribution (e.g., Gaussian, Wishart, or any other family of the probability of distribution). HMM can model time-series fMRI data in a temporal structure of the inferred brain states, each with specific spatial activation patterns. Applying HMM to fMRI data assumes that (1) fMRI data can be reasonably modeled in a discrete number of states with Markovian dynamics. (2) At each point in time, these states are reflective in the form of probabilities, and only one active state is assigned based on probability. (3) The current state being occupied is only dependent on the last state, not the previous history of state activation (Vidaurre et al., 2017; Vidaurre, 2021). The model allows for the analysis of how likely a state being occupied at a particular time point, how much time is being spent in each state, and how certain a state is transitioning to another state. Such recurrent patterns and dynamics in brain activation data throughout entire datasets can be uncovered using HMM. It provides a more reliable estimation of brain activation patterns and overcomes the insufficiency when a short time window is pre-specified for classical statistical analysis (Vidaurre et al., 2018). Another benefit is that HMM enables the detection of the transient occurrence of a state and switches between the states when the visits of the states are relatively short in time, which is usually missed in classic analysis methods (Vidaurre et al., 2018). Based on the flexibility and analysis power, HMM has been applied to fMRI data (Anderson et al., 2010, 2016; Anderson, 2012; Suk et al., 2016; Baldassano et al., 2017; Vidaurre et al., 2017, 2018; van der Meer et al., 2020; Vidaurre, 2021).

The earliest fMRI studies that adopted HMM were by Anderson et al. (2010, 2016) and Anderson (2012). This study used HMM to distinguish the period of time and mental states (e.g., encoding, planning, solving, and responding) when students engaged in mathematical problem-solving (Anderson et al., 2016). Baldassano et al. (2017) applied HMM to fMRI data and detected event boundaries during narrative perception through shift between brain activation states without stimulus annotations. HMM was also applied to decode brain states in resting-state fMRI data for clinical application (Suk et al., 2016). Vidaurre et al. (2017) used HMM with the large datasets (resting-state fMRI data from 820 subjects) in the Human Connectome Project (HCP) to achieve richer and more robust conclusions about the dynamic nature of brain functional connectivity. Here, the results demonstrated that activation data can be well represented in discrete states which are hierarchically organized in time, and the dynamic transitions between these states are far from random. More recently, van der Meer et al. (2020) applied HMM to fMRI data collected during movie viewing. The HMM captured a sequence of well-defined functional states plus dynamic transitions that were temporally aligned to specific features of the movie in the study. In summary, previous research has demonstrated HMM as a viable approach to represent brain activation data in a variety of contexts for which information regarding recurrent patterns of activity is of interest. The goal of the current work in this paper is to uncover brain activation patterns and cognitive functions that emerge and transit between different states during design concept generation.

Application of HMM in design research

Another critical motivation for applying HMM to neuroimaging data on design ideation comes from prior work that has demonstrated HMM as a valuable tool for capturing patterns and sequence in design behavior data. HMM was adopted by the authors in prior work to represent and stimulate sequential patterns of design behaviors when designing for additive manufacturing (Mehta et al., 2020) and solving configuration problems, including the design of truss structures or internet-connected home cooling systems (McComb et al., 2016, 2017a, 2017b; Brownell et al., 2021). Design is a dynamic process in a sequence of stages or activities (Howard et al., 2008; Gericke and Blessing, 2011; Cramer-Petersen et al., 2019). In engineering design, the capacity of designers to learn and employ sequences (temporal patterns of activity) has long been of interest to design researchers (Gericke and Blessing, 2011; McComb et al., 2016, 2017b; Cramer-Petersen et al., 2019). Prior research explored sequence in design at different levels of abstraction (McComb et al., 2016). The level of abstraction refers to the sequencing levels in design based on the ordering of design stages (more abstract and generalized), specific tasks, or design operations (less abstract and more detailed-specific). For example, the higher level of abstraction as design stages that tend to occur at the longer timescales (e.g., customer needs assessment, conceptual design, detailed design) (Atman et al., 2007; Goldschmidt and Rodgers, 2013), and a lower degree of abstract at a shorter timescale as specific design tasks and operations (e.g., adding a member, adding a joint, resizing a member, etc., in the design of truss structures) (Rogers, 1996; Sen et al., 2010; Brownell et al., 2021). Sequencing at short timescales and low abstraction directly impact design proficiency (Brownell et al., 2021) or performance (McComb et al., 2016, 2017b). However, this level of abstraction and timescales has not well studied in the engineering design literature (McComb et al., 2017a). The current work presented in this paper aims to fill this gap by exploring the states in neurocognition as imaged through fMRI. The spatial and temporal patterns are investigated from a neurocognitive aspect. The results identify and assess a short-timescale sequence of different states in neurocognition that has not previously been examined in engineering design research. Here, sequence refers to the temporal patterns and transitions in neurocognitive activation and functions. This intersection of neuroimaging, design concept generation, and analysis using HMM provides a novel contribution to design cognition literature.

Methods

This study investigates the patterns of neural activation and possible cognitive functions associated with each of the 12 states related to design concept generation identified in prior work (Goucher-Lambert and McComb, 2019). The fMRI datasets, data processing procedures, and HMM are introduced in Sections “Design concept generation task and fMRI experiment”, “fMRI data collection, pre-processing, and brain parcellations” and “Hidden Markov modeling”, respectively. Section “Localizing the brain activation in each HMM state” describes the method for localizing the brain activations and inferring possible cognitive functions associated with each state.

Design concept generation task and fMRI experiment

This study used the fMRI dataset collected in a prior design by Goucher-Lambert et al. (2019) in which participants engaged in concept generation tasks with or without the assistance of inspirational stimuli. Inspirational stimuli are examples provided to designers to enhance creativity and innovation during conceptual ideation (Goucher-Lambert and Cagan, 2019). These stimuli were sourced in prior work by extracting common and uncommon words from crowdsourced solutions using a text-mining technique. Their distance to the problem (near or far) was determined based on word frequency and bidirectional path length textual similarity (Goucher-Lambert and Cagan, 2019).

In the fMRI experiment, designers (i.e., engineering and design students) completed the 12 design problems and developed as many solutions as possible in an MRI scanner. For each design problem, designers were given a total of 2 min, separated into two 60-s blocks, and asked to develop as many solutions as possible in each block. From the beginning of each block, all designers were presented with word sets drawn from inspirational stimuli (inspirational stimuli condition, near, or far stimuli) or containing words from the design problem without inspirational stimuli (control condition). A total of five inspirational stimuli were displayed: three words displayed at the same time (Word Set 1) from the beginning of the first block and the remaining two words displayed simultaneously (Word Set 2) from the beginning of the second block. The purpose is to make the presentation of inspirational stimuli alternate throughout the task and provide new stimuli if participants had exhausted their use of the inspirational stimuli presented in the first block. An example problem and inspirational stimuli can be found in Figure 1. Each of the 12 design problems had a unique set of inspirational stimuli for all three conditions (near, far, and control). The experiment conditions were counter-balanced to provide an even distribution of problem-condition pairs for each designer. Figure 1 shows the experiment process. Only fMRI images collected during the whole session of the design concept generation periods (highlighted in Figure 1, without any specification on the time points of Word Set 1 or Word Set 2) were included in the HMM. The full details of the design problems, inspirational stimuli, and fMRI experiment can be found in Sections “Important brain regions and networks for ideation and insights” in Goucher-Lambert et al. (2019).

Fig. 1. Design concept generation experiment process with an example problem and corresponding inspirational stimuli.

fMRI data collection, pre-processing, and brain parcellations

A total of 21 engineering students were recruited and completed the fMRI experiment. Figure 2 illustrates the steps for the fMRI data collection, pre-processing, and preparation for HMM training. fMRI data collection and pre-processing were performed in the prior work. Detailed information on participants, fMRI equipment, data acquisition, and data pre-processing (Steps A and B in Fig. 2) can be found in Sections “Application of HMM in neuroscience research” and “Application of HMM in design research” in Goucher-Lambert et al. (2019). Data processing in the current work includes Steps C, D, and E in Figure 2.

Fig. 2. fMRI data pre-processing and preparing. Steps A and B were performed in the prior work. The current study processed and analyzed the fMRI data in Steps C, D, and E.

A multi-stage process was applied to prepare the pre-processed fMRI time-series data into lower-order spatial representations for the purpose of more rapid HMM training, illustrated in Figure 2c,d. The first step was down-sampling each fMRI image from the resolution of 54 × 64 × 50 (in a total of 172,800) voxels to 27 × 32 × 25 (in a total of 21,600) voxels to avoid overfitting (Anderson, 2012). Then, the processing pipeline and techniques used by Smith et al. (2014) and Vidaurre et al. (2017, 2018) were applied in this study to prepare HMM inputs. Principal component analysis (PCA) was used to reduce fMRI data to its dominant constituents with a dimension of 50 parameters for each subject. The last step was to perform independent component analysis (ICA) with pre-specified constraints (i.e., parcellation in Fig. 2d). The max-kurtosis ICA algorithm was applied to project the data into a 50-dimension time-series using the 50-parcellation template from the Human Connectome Project (HCP). The whole-brain fMRI data was parcellated into the activation data within 50 functional distinct areas using the pre-validated spatial maps (Medolic_IC) from HCP, which include spatial information of the 50 spatially independent components in the brain (Beckmann, 2012). Previous researchers used the large-scale resting-state fMRI data in the HCP and provided this data-driven functional parcellation of human brains with high stability (Beckmann and Smith, 2004; Smith et al., 2014, 2015). A final standardization was performed to the 50-dimension time-series fMRI data aggregated among all participants so that the training data for the following HMM have a mean of 0 and a standard deviation of 1.

Hidden Markov modeling

The normalized fMRI time-series datasets from all participants were concatenated in the temporal dimension and used to train HMM to generate a group-level sequence of a finite number of states with varying patterns in neural activation. Specifically, the HMM was trained with emissions in Gaussian distribution, which was used in prior fMRI studies (Vidaurre et al., 2017, 2018) and is appropriate for the fMRI data used in this study. Here, each state was represented by the average modes of brain activation that are emitted or enacted with some degree of variance in Gaussian distribution. The HMM-MAR (Hidden Markov model–multivariate autoregressive) toolbox (Vidaurre et al., 2016) was used to accomplish the analysis. Estimations on parameters of state distributions, progression through states, and transition probability matrix were conducted by using the HMM-MAR toolbox. A state matrix (S_12×50) showing the state distribution across the 50 brain parcellations for the 12 states was calculated for further activation localization (detailed in Section “Localizing the brain activation in each HMM state”).

The appropriate number of states for a HMM is usually determined within an iterative procedure (McComb et al., 2017b; Pohle et al., 2017). A range of varying numbers of hidden states from 2 to 32 was tested for the HMM training, and log-likelihood values were compared among all the models. Here, log-likelihood is a measure of model accuracy, describing the probability that the observed data was produced by the trained model. The resulting differences in log-likelihood values between models were negligible, providing no basis on which to choose the number of states. As a result, 12 was determined as the number of states and used for model training in prior work (Goucher-Lambert and McComb, 2019) and the current study to align with previous literature in neuroscience applying 12-state HMM to neuroimaging data (Vidaurre et al., 2017, 2018).

Localizing the brain activation in each HMM state

The 12 HMM states from Goucher-Lambert and McComb (2019) were used in the current work for the investigation of the brain activation patterns related to concept generation. As mentioned in Section “Hidden Markov modeling”, each state was represented by the average mode of brain activation, so a state matrix (S_12×50) with mean values of activation was calculated and used. The state matrix has 12 row vectors that stand for 12 states. Each row vector contains 50 contributing indices, which are mean values from a Gaussian distribution and represent the average contribution from the corresponding parcellation. The state matrix was used to project the activation back into a higher-dimension activation matrix with more voxel elements. The mathematics is represented in Eq. (1).

(1)$$X = S \times A.$$

A mixing matrix (A_50×32,767) including the voxel compositions of the 50 parcellations was provided by the HCP (Ugurbil and Van Essen, 2017) and applied to the states matrix (S) here for the generation of high-dimension and whole-brain activation matrix (X_12×32,767) associated with the 12 states. Here, 32,767 represents the dimension length of the standard 32k surface meshes provided by the HCP mixing matrix template (16-bite integers and limited to 32,767 in each dimension) (Elam et al., 2013). Then, the activation for each state (a row vector in X) was coded and converted into appropriate CIFTI-2 format files. Doing so enabled the visualization of each HMM state in an activation heatmap using the HCP visualization and discovery tool wb_view (Marcus et al., 2013).

An investigation of the physical locations in the brain and possible cognitive functions associated with the HCP 50 parcellations was performed to better understand the activation patterns of the HMM states. Specific Montreal Neurological Institute and Hospital (MNI) coordinates for the center point of each parcellation were extracted in the wb_view tool. The extracted MNI coordinates for each parcellation were localized into brain regions and Brodmann areas using the Biolmage Suite tool (Papademetris et al., 2006). Then a meta-analytical database of fMRI studies, NeuroSynth, was used to map between the parcellation MNIs and associated cognitive functions (Yarkoni et al., 2011). NeuroSynth operates by using combined text-mining, meta-analysis, and machine-learning techniques to generate probabilistic mappings between cognitive functions and neural activation in the brain region with corresponding MNI coordinates (Yarkoni et al., 2011). The cognitive functions in NeuroSynth are coded into specific psychological terms, such as working memory, retrieval, visual, or large-scale brain networks. A total of 14,371 fMRI studies have been used in NeuroSynth for a robust and reliable inference mapping between brain regions and cognitive functions (Yarkoni et al., 2011; Yakoni, 2022). NeuroSynth has been used in previous research to localize brain regions of interest and identify common cognitive functions in fMRI datasets related to design (Goucher-Lambert et al., 2017a). This coordinate-to-term mapping approach was used in the present work to infer cognitive functions associated with each parcellation and then each HMM state. The psychological terms with a high likelihood of associating with the activation in the MNI coordinate (represented by a posterior probability P(term | activation) from Naïve Bayes Classification higher than 0.75) were selected as cognitive functions associated with the parcellation. Eventually, for each state, the key parcellations (i.e., parcellations with top 3 contributing indices to the state in the state matrix) and their associated cognitive functions (i.e., psychological terms extracted from NeuroSynth) were identified for further interpretation of the state.

Results

Using the methodologies outlined in Section “Methods”, this study investigates the patterns of neural activation that are associated with each of the states discovered by Goucher-Lamber and McComb (2019). Cognitive functions associated with each of the HMM states were inferred based on meta-analysis from NeuroSynth. State transfers between the HMM states were also uncovered and interpreted.

Patterns of neural activation associated with the 12 states

The 50 parcellations acquired from the HCP were localized to specific brain regions and Brodmann areas for further interpretation. Six parcellations were removed from the summary since the activation (i.e., z-scores) were negligible. A summary of associated brain regions for the other 44 active parcellations can be found in Table A1 in the Appendix. In addition, possible cognitive functions described by the psychological terms extracted in NeuroSynth, associated with each parcellation, are also listed in Table A1.

To directly illustrate the neural activation patterns associated with each HMM state, brain activation heatmaps of the 12 states were created using the wb_view tool and presented in Figure 3. The activation map for each state was generated by projecting the state matrix for the 50 parcellations back to high-dimension activation within each voxel element, which is described in Section “Localizing the brain activation in each HMM state”. As shown in the activation heatmap, distinct locations in the brain and patterns of activation are associated with the 12 HMM states. State 4 has significantly higher activation than other states, mainly in the prefrontal cortex and motor cortex. States 1, 7, and 11 show negative activation in a wide range of brain regions. Other states show strong activation in either the PFC, temporal cortex, or occipital cortex. For example, States 2, 8, and 10 show strong activation in the occipital and temporal cortex, while State 6 mainly involves activation in the PFC.

Fig. 3. Activation heatmap for the inferred 12 HMM states from the aggregated fMRI data. The states are characterized by their mean activation that projected from the 50-dimension parcellations to whole brain space.

When using the HMM approach, the activation pattern for each state has a linear relationship with the activation in the brain parcellations, represented in the state matrix. Figure 4 uses a color-coded state matrix to represent the contribution indices of the 44 active parcellations to each state. The 44 parcellations were reordered and clustered based on the cortex they are in to more clearly show the activated cortex for each state. A few parcellations include more than one cortex in the human brain, and therefore appear along the y-axis of the figure multiple times.

Fig. 4. Contribution indices of the parcellations to each state. The color represents the value of contribution from the parcellation to the state. The parcellations are reordered and clustered based on the cortex.

As shown in Figure 4, State 4 shows higher activation levels than other states, including in the prefrontal cortex, temporal cortex, parietal cortex, and motor cortex. Another finding is that some states show stronger activations in one or two cortexes than other brain regions. For example, States 2 and 5 are more involved in the occipital and temporal cortex; State 6 has stronger activations in the prefrontal cortex than other regions. States 3 and 10 show their major activation in the occipital cortex. States 1 and 11 are less activated but have major activation in the occipital cortex; State 7 also shows less activation in most brain regions except for activation in the occipital cortex, cingulate cortex, and prefrontal cortex.

Key parcellations for each state and possible cognitive functions

To identify physical brain locations of major activation for each state and infer cognitive functions, the top 3 parcellations of the state (ranked by the contributing indices in the state matrix) were identified. Cognitive functions of the parcellations, coded as concise physiological terms, were extracted using a coordinate-to-term approach based on the meta-analysis from NeuroSynth (Section “Localizing the brain activation in each HMM state”). Table 1 here lists the top 3 parcellations for each inferred state, plus their physical location in the brain, and associated cognitive functions from meta-analysis.

Table 1. Key parcellation to each state and possible cognitive functions

DMN, default mode network; CEN, central executive network.

Table 1 shows distinct patterns and physical locations of activation in the 12 HMM states. The physical locations of the top 3 parcellation for each state provide a consistent mapping with the state activation heatmap in Figure 3 and the color-coded state matrix in Figure 4. For example, State 4 shows higher activation in a wide range of brain regions. To be more specific, the major activation is in the dorsolateral PFC and posterior parietal cortex from the ECN, which is generally associated with executive control of working memory (Chatham et al., 2011), middle temporal cortex, and bilateral supplementary areas for motor tasks (Chu and Black, 2012). Another example is State 6 that mainly involves activation in the PFC. The major activated brain regions of State 6, shown in Table 1, are predominately in the PFC, including the dorsolateral PFC, ventromedial PFC, and inferior frontal gyrus, which are usually involved in rule-based reasoning (Rudorf and Hare, 2014; O'Bryan et al., 2018), comprehension (Gernsbacher and Kaschak, 2003), and the executive control function from the ECN (Chatham et al., 2011).

In addition to the consistent mapping, Table 1 also filters the major activated brain regions in the states that are less active and hard to notice. For instance, State 1 shows significant activation in the occipital cortex that is critical for visual processing (Clarke and Miklossy, 1990). State 7 involves activation in the occipital, orbitofrontal, and posterior cingulate cortex from the DMN. DMN usually engages in rest state or spontaneous and associative processes (Beaty et al., 2020). For State 2, except for the activation in the temporal and occipital cortex, the rostrolateral PFC is also a major brain region of activation. The restrolateral PFC is generally associated with rule-based reasoning (Hobeika et al., 2016; Paniukov and Davis, 2018).

Regardless of the specific activation patterns, most states combine collection of widespread brain regions that are functionally connected within large-scale networks. The associated networks here mainly include ECN, DMN, visual network, and motor network. The 12 inferred states share some consistent cognitive functions related to these brain networks. For instance, semantic processing and memory retrieval are two frequent functions listed in Table 1. Semantic processing refers to a human's ability to use, manipulate, and generalize knowledge to support verbal and non-verbal behaviors (Ralph et al., 2017). Memory retrieval is the process that involves the interactions of triggers/cues and stored memory traces (Frankland et al., 2019). Most states, except for States 1, 3, and 10, involve activations that are closely associated with either executive control of working memory or spontaneous associative processing for semantic and retrieving processes.

Another shared cognitive function in multiple states here is visual processing. All states, except for States 4, 6, and 12, show major activation in the primary visual processing-related brain regions. Finger tapping is also a common cognitive function in a few inferred states, including States 3, 4, 5, 9, and 10. This function from the motor network is involved because the experiment asked participants to click on a button when they generated a concept. A baseline correction with the fMRI data during the n-back task was used to remove the noise associated with movement in the experiment. However, there can still be activation associated with motivational or imaginary finger movement before or when designers clicked the button.

Likelihood of state occupancy and state transitions

Among the 12 states identified in Goucher-Lambert and McComb (2019) for the aggregated fMRI data related to concept generation, seven states, the state probability matrix suggests States 1, 2, 3, 4, 6, 7, and 11, show a higher probability of occupancy than the rest states (i.e., States 5, 8, 9, 10, and 12). These less-occupied states might represent random activation patterns less relevant to the design task. Figure 5 shows the time-varying occupancy probability of the seven states that are highly likely to occur in the process of concept generation. Among these states, States 2, 4, 6, 7, and 11, are more likely to be occupied, especially State 4, with the highest likelihood of being occupied than other states.

Fig. 5. The probability of occupancy in the seven states that are more likely to be occupied in the process of concept generation.

The dynamic pattern between the 12 states was represented using possible switches between the 12 states. Only strong transitions with a probability higher than 10% were included in Figure 6a. Strong diagonal elements suggest that participants are likely to stay in a single state across several brain image acquisitions. Other strong off-diagonal elements show a dynamic pattern and transition between different states. These transition paths with a transition probability greater than 10% are highlighted and included in Figure 6b.

Fig. 6. Strong transitions (probability > 10%) between states (a) and transition paths with high probability between states (b).

As shown in Figure 6b, the states that are least likely to be occupied (i.e., States 5, 8, 9, 10, and 12) have a high probability of transitioning to States 4, 6, 7, and 2, but not to States 1, 3, and 11. As mentioned, these less-occupied states might represent random activation patterns less relevant to the design task. This transition might represent a shift from a random state back to the active states for concept generation, especially to States 2, 4, and 6. These states involve activations in the lateral PFC from the ECN. The executive control functions associated with these states can inhibit cognitive processing on irrelevant information and amplify attention for internal representation of insights. Among other active states, there are some state switches with higher probability, for example, State 6 to State 4 (31%), State 1 to State 6 (22%), State 2 to State 11 (21%), State 11 to State 6 (17%), and State 7 to State 2 (16%). These transition paths between the key states suggest possible dynamic and recurring patterns in neurocognition related to concept generation.

Discussion

This study used a HMM approach to uncover the spatial and temporal patterns in fMRI data related to design concept generation. Using this approach, 12 distinct states, with dynamic switches between each other, were automatically inferred from the data. Specific activation patterns in each state were linked to different physical locations in the brain and varying cognitive functions based on meta-analysis. Furthermore, the state transition routes and difference in state occupancy between the high- and low-performing designers can provide meaningful explanations to their different design performances.

Associations and distinctions between the key states

Among the 12 distinct states, several key states showed a higher likelihood of being occupied and transiting than the other states, including States 2, 4, 6, and 7. Consistent cognitive functions associated with these states are semantic processing and memory retrieval (Burianova and Grady, 2007; Goldberg et al., 2007). These two cognitive functions echo the associative theory of creativity (Mednick, 1962) and a common view on analogical reasoning (Forbus et al., 1995) that support the creative process. Here, analogical reasoning is the inference inspired by the source, and applied to a target (Forbus et al., 1995; Chan and Schunn, 2015; Goucher-Lambert et al., 2019). Semantic processing supports the generation of new ideas by offering a semantic knowledge base of facts and concepts for screening and selection (Mednick, 1962; Beaty et al., 2020; Gerver et al., 2022). According to the associative theory of creativity, people who have a loosely structured semantic knowledge base are better at creative tasks because they are more capable of forming associations with remote semantic distance (Mednick, 1962). Considering the semantic nature of inspirational stimuli provided in the design task, semantic processing can play a critical role for participants to cognitively process the semantic similarity and making associations between the inspirational stimuli and the design solutions. Memory retrieval is an essential step that enables searching and recognizing a useful and relevant concept stored in designers’ memory (Gomes et al., 2006). Successful retrieval of memory can then be used in the subsequent generation of solutions to the design problem. The findings emphasize the importance of semantic processing and memory retrieval to design concept generation with inspirational stimuli. More specific characteristics of semantic processing and memory retrieval, for instance, semantic similarity, divergent or convergent semantic processing, and memory retrieval cues, plus their correlates with ideation performance can be studied with more details in future research.

Even though these states have shared cognitive functions, they involve varying physical locations of activation in the brain. Figure 7 illustrates the key brain regions (Brodmann areas) of activation for the four major States. The differentiated activation patterns of these states suggest potentially different roles for semantic and retrieving processing. Considering the temporal patterns in occupancy likelihood, these states might represent difference sequences in cognition related to concept generation.

Fig. 7. Key brain regions of activation for States 6, 4, 7, and 2. The brain regions (Brodmann areas, BA) with the top 3 contribution indices (shown in Table 1) for the states are highlighted in corresponding locations with the BA number.

State 6 might be responsible for stimuli encoding and goal defining

The activation pattern of State 6 is mainly within the inferior frontal gyrus (Brodmann area—BA 44) and supramarginal gyrus (BA 40), which are mainly involved in semantic and (specifically) verb comprehension (see Table 1), and dorsolateral PFC (BA 46) for rule and demand processing. Activation in the BA 44 and BA 40 is often linked to verb processing, especially for comprehension (Bak et al., 2001; Giraud et al., 2004; Sahin et al., 2006; Newman et al., 2009). Dorsolateral PFC is critical for representing and maintaining information related to goals and rules to guide behavior (Bunge et al., 2003; Wallis and Miller, 2003). Considering the distinct increase in the likelihood of occupancy of State 6 directly after the introduction of the inspirational stimuli (Word Set 1 at 0 s and Word Set 2 at 60 s), a possible interpretation of State 6 is to comprehend and encode the stimuli for goal defining.

State 4 appears to be generating new concepts inspired by the stimuli

In contrast, State 4 mainly shows activation from the ECN (including the dorsolateral PFC and posterior parietal cortex). Activation within the ECN is heavily involved with executive controls of internal retrieving information from working memory and relational integration (Curtis and D'Esposito, 2003; Gonen-Yaacovi et al., 2013). Several neuroimaging studies found significantly higher activations in the dorsolateral PFC and posterior parietal cortex in support of relational integration (Green et al., 2010; Blumenfeld et al., 2011) and creative generation task (Kowatari et al., 2009; Gonen-Yaacovi et al., 2013). The middle temporal gyrus (BA 37), in charge of semantic and episodic memory in creative insight (Shen et al., 2017) and formation of novel associations from analogy (Hao et al., 2013) is also activated in State 4. Prior work that applied the general linear modeling (GLM) approach to the same fMRI data as the current study found that temporal brain activation were closely associated with insights inspired by the stimuli as well (Goucher-Lambert et al., 2019). A possible interpretation of State 4 is generating new concepts with the inspirational stimuli. The activation in the motor network of State 4 might be associated with motivational or imaginary finger movement before designers confirmed the insights in their minds and planned to report the generation of a new concept.

State 7 might switch between internal and external attention

The main brain regions involved in State 7 include the inferior occipital gyrus for external visual processing (Clarke and Miklossy, 1990), orbitofrontal cortex for internal memory retrieving (Young and Shapiro, 2011; Farovik et al., 2015), and PCC, a core backbone for DMN. The PCC is typically linked to a central role in supporting internal-directed attention for episodic memory retrieving and future planning (Buckner et al., 2008). However, there are still debates regarding the exact functions of PCC in the neuroscience literature. A comprehensive review on the role of the PCC in neuroimaging studies found its possible role associated with switching between internal and external attention (Leech and Sharp, 2014). State 7 might serve to sustain insightful thoughts by flexibly switching from the external visual process to internal retrieval of memory to generate concepts or a reverse switch from the internal controlled process to external attention to the design space.

State 2 seems to contribute to solution evaluation and goal monitoring

Like State 6, a critical function for State 2 is rule-based reasoning. The specific brain region is the rostrolateral PFC. Rostrolateral PFC has been identified as a brain region in support of high-order cognitive functions in rule-based analogical reasoning (Christoff et al., 2001; Hobeika et al., 2016), and memory retrieval (Westphal et al., 2016). In particular, rostrolateral PFC plays an evaluative role in rule-based reasoning (Hobeika et al., 2016; Paniukov and Davis, 2018). This evaluative role seems to hold true when designers assess whether their associations are appropriately made, or their solutions meet the demand when generating concepts with the support of inspirational stimuli. State 2 might represent concepts assessments and evaluations. Additionally, higher activation in the occipital cortex is also involved in State 2 which suggests external attention to the design problem or stimuli.

It should be noted that these interpretations of states were made based on reverse inference. The claims about particular cognitive processes were inferred from reasoning backward from the observed brain activity rather than directly testing. However, the meta-analytic framework applied in this work using NeuroSynth can potentially address possible problems of reverse inference by enabling researchers to conduct quantitative reverse inference on a large scale of studies. These interpretations of states only represent possible explanations based on the state occupancy, associated brain regions and cognitive functions. Future research should investigate this link between design cognitive processing and neurocognitive patterns more directly to examine the interpretations. Another possible limitation is that only group-level inference was performed using temporal concatenation for group-level analysis on states occupancy and transitions. Subject-level analysis can be reconstructed in future research to explore individual characteristics in neurocognition related to concept generation. More detailed and richer descriptions on the dynamic patterns and transitions among the key states can be also explored based on individual data analysis.

Performance-differentiated characteristics in state occupancy and cognitive functions

States 6, 4, 7, and 2 represent recurring patterns in neurocognition related to the use of the stimuli and generating new concepts. The prior research also found high-performing designers (i.e., designers with higher idea fluency) showed higher occupancy probability in these states. Figure 8 shows the differences in state occupancy likelihood averaged in every 15 s between the high- and low-performing designers. High-performing designers show a higher likelihood of occupancy in States 2, 4, 6, and 7, which are mainly associated with activation in the brain regions from the large-scale networks of ECN and DMN. ECN and DMN are two brain networks widely studied in creative cognition literature (Beaty et al., 2016). ECN and DMN, plus their coupling activation, are believed to play inevitable roles in tasks that demand creative processing, such as divergent thinking (Heinonen et al., 2016), analogical reasoning (Hobeika et al., 2016), creative idea generation (Beaty et al., 2015), and art creating (Kowatari et al., 2009).

Fig. 8. Likelihood of state occupancy difference between the high-performance and low-performance designers.

On the contrary, low-performing designers showed a higher likelihood in States 1, 3, and 11 in the duration of concept generation after introducing the stimuli. State 1 mainly shows activation in the occipital cortex, so its possible role is visual processing for external information when there is no clue or insight from internal processing or participants are unable to generate new concepts under time or other constraints. State 3 also involves activation in the occipital cortex. Prior research has linked an increase in visual processing with participants being unable to solve problems with insight (Kounios et al., 2006), design fixation without new ideas (Fu et al., 2019), or an unsuccessful external search without insights (Goucher-Lambert et al., 2019). The state might represent a continued external search for inspiration when participants cannot retrieve helpful information from memory. State 11 seems to have similar activation patterns as State 2. However, the level of activation has significantly decreased. This diminished activation pattern in State 11 might render the corresponding cognitive functions not as effective as State 2. Other less-occupied states, including States 5, 8, 9, 10, and 12, might represent random activation patterns less relevant to the design task and are not discussed here.

The performance differentiated characteristics in neurocognition suggest potential leverage points in design fluency and creativity training. For instance, training or interventions in education can target improving neurocognitive ability in the ECN and DMN for semantic processing and memory retrieval while controlling unnecessary visual processing or eye movements. More research in design and education can take advantage of neuroimaging methods to shed light on strategies or practices that improve design performance by offering a new layer of data and insightful knowledge of hidden brain activities related to design cognition.

Noticeably, the classification of high- and low-performing designers was based on idea fluency, which means high-performing designers generate new concepts more quickly and fluently. High-performing designers might be quicker to encode the stimuli and define the goal, and then retrieve information from memory and generate the targeted concepts through reasoning. Idea fluency is a critical measure for creativity in ideation (True, 1956; Mirabito and Goucher-Lambert, 2021). However, a limitation is that only idea fluency was compared, while other metrics, such as novelty, quality, and feasibility, are not included in this analysis. This can be seen as a challenge posed by utilizing fMRI as a method for studying design, as capturing full design concepts (e.g., through think aloud protocols, or drawing/typing) is quite challenging in the MRI environment. Future research should explore mechanisms to capture the generated concepts and explore how other creativity metrics correlate with dynamics of design neurocognition, while accounting for possible data quality concerns that may emerge (e.g., via motion artifacts). Additionally, this work mainly investigates design neurocognition related to concept generation, which is believed to be a key activity in the design process shaping the creativity of subsequent design phases (Cross, 2001; Yang, 2009; Hay et al., 2019). However, design is a complex process involving multiple stages and activities, and spanning in varying time durations. There is a substantial need for more design research to explore behaviors and neurocognition related to different stages of design and the dynamic patterns in this process as well.

Possible transition routes related to concept generation

Several possible transition routes can be observed from the transition matrix in Figure 6b plus the temporal sequence of occupancy for each state in Figure 5. Three possible routines are highlighted in Figure 9. There is a distinct increase of likelihood in States 1, 6, and 11 right after introducing the stimuli (shown in Fig. 5), and the transition probability is high from State 11 to State 1 (10%), State 1 to State 6 (22%), and State 11 to 6 (17%) (shown in Figs. 6, 9). There seems to be a transition route (path 1 in Fig. 9), including States 11 – 1 – 6 or States 11 – 6. Considering the activation patterns and cognitive roles of these states, this route might be associated with a process that participants catch sight of the stimuli/verbs, then pass the visual information to the prefrontal cortex for encoding the stimuli and defining the goal of the problem.

Fig. 9. Three possible transition routines with high transiting probabilities between the different states.

After stimuli encoding and goal defining, the information will transit from State 6 to State 4 (31%) for analogical reasoning and generation of concepts. Then another transition route, a loop including State 4 – 7 – 2 – 4, might represent a recurring process of insights. Once an insight occurs, a switch from State 4 to 7 (13%) might help designers achieve a quick shift from the internal retrieving process to external attention to the stimuli. Then, the transition from State 7 to 2 (16%) suggests the cognitive processing of solution evaluation and goal monitoring to initiate a new round of concept generation in State 4. This transition route (path 2 in Fig. 9) may represent the successful use of the stimuli, leading to insights and generating new concepts.

In addition to the transition from State 2 to 4, the transition from State 2 to 11 also has a high probability (21%, see Fig. 7). Thus, there is a high probability that the transition loop State 6 – 4 – 2 intersects with the other transition path of State 11 – 1 – 6. There can be another transition cycle including State 4 – 7 – 2 – 11 – 1 – 6 – 4 in the process of concept generation (see path 3 in Fig. 9). States 11 and 1 here represent an extended processing in the external attention system and visual-related regions. State 6 is involved for re-encoding the stimuli and redefining the goal for the problem. This transition route might happen when participants are at an impasse during problem solving. When they are not able to retrieve more useful information and new insights from internal search, they switch their attention systems and attempt to pay more attention to the external environment for insights with visual processing. They might even need to re-encode the stimuli and re-define the goals to generate other concepts. This transition route appears to be indicative of a continued and less successful external search process for inspiration.

Implications for future work combining HMM and design neurocognition

Overall, the findings presented in this work demonstrate that HMM is a well-suited approach to recognizing the recurring patterns of both spatial and temporal dynamics in design neurocognition. HMM can capture rich information contained in the entire fMRI dataset. It also bypasses some problems and statistical limitations in classical methods for fMRI analysis. Classical methods usually rely on significant assumptions regarding the timing of activation and brain regions of interest. For example, the sliding window approach assumes a pre-specification of the timescale at which the neural activation occurs. This pre-defined temporal window limits its statistical power to detect the dynamics in neurocognition (Hindriks et al., 2016; Vidaurre et al., 2018). In contrast, there are no assumptions related to the underlying model structure when using the HMM approach. Therefore, latent patterns (states) can be automatically inferred in a completely unsupervised way, which makes HMMs suitable for exploratory analyses of neurocognition data relative to design.

Using HMM leads to the findings that echo prior design neurocognition literature and show consistency regarding the highly activated brain regions associated with concept generation and insights (Rudorf and Hare, 2014; Shen et al., 2017; Goucher-Lambert et al., 2019; Gerver et al., 2022). Here, the data-driven functional parcellation of human brains from a large dataset provides more stability in the HMM inputs. Additionally, the HMM methodology enriches knowledge in design neurocognition by unveiling the dynamic switches between the states with varying spatial and temporal patterns related to design concept generation. Prior neuroscience studies have used a similar HMM approach to investigate resting-state fMRI data and found that the transitions between states or networks are far from random (Baker et al., 2014; Vidaurre et al., 2017, 2018). The current work used HMM and captured the transient and dynamic switches between the discovered states that meaningfully characterized possible sequences in cognition for generating concepts. The state switches also offer insightful explanations of the dynamic neural patterns that influence performance in concept generation.

A limitation of the HMM inference used in this work is the prior specification on the number of states K. The log-likelihood values with different selections of K (e.g., from 2 to 32) did not significantly change when performing the model selection. So the choice of 12 states was chosen to better align with prior neuroimaging studies that applied HMM to fMRI data (Vidaurre et al., 2017). However, the findings (e.g., low occupancy likelihood in some states) suggest that a lower number of states may present a better trade-off between richness and redundancy and should be explored in future work. In addition, other model selection methods, such as model evidence via the free energy used in Bayesian inference techniques, can be adapted to select an appropriate number of states (Baker et al., 2014).

In summary, the results show the power of using HMM to uncover the neural patterns of design. This study unveils different states in neurocognition with dynamic spatial and temporal patterns and helps to construct a more insightful understanding of design neurocognition. The current work focused on the activation patterns of the discovered states related to concept generation. Network patterns or functional connectivity is another focus in the creative cognition research community. HMM also provides benefits to network analysis in fMRI data (Vidaurre et al., 2017, 2018). Future research can move from isolated activation toward exploring broad patterns in neural activation networks. The results from future research are expected to show how large-scale networks in the brain and functional connectivity contribute to design ideation.

Conclusion

This study used a HMM approach to uncover the spatial and temporal patterns in fMRI data related to design concept generation. The underlying fMRI data were collected when participants generated solutions to open-ended design problems in two concurrent blocks, each lasting 60 s. Twelve distinct states, with dynamic transitions between each other, were automatically inferred from the HMM method. Specific activation patterns associated with each state were identified and linked to varying brain regions and cognitive functions. The HMM states with higher likelihood of occupancy show more activation in the brain regions from the executive control network, the default mode network, and the middle temporal cortex. Multiple cognitive functions (e.g., semantic processing, memory retrieval, executive control, and visual processing) are involved in the key states in neurocognition related to concept generation. Highly possible transitions between the states in neurocognition are identified and suggest possible transitions between different cognitive processes (e.g., from visual processing to rule-based reasoning, from internal retrieving process to external attention). The functions of the states in neurocognition offer meaningful explanations on the different patterns between designers with high and low idea fluency. To summarize, this study shows the potential of HMM in identifying spatial and temporal patterns in the fMRI data related to design cognition. HMM offers a deeper understanding of the dynamics in neurocognitive processing and brings new knowledge to the design cognition community. Researchers in design neurocognition, not limited to those using fMRI but also EEG or fNIRS, can take advantage of HMM or other relevant machine learning techniques to provide a more detailed description of brain dynamics in design cognition.