Improving human understanding and design of complex multi-level systems with animation and parametric relationship supports

Abstract

Complex systems are challenging to design, particularly when they contain multi-level organizations with non-obvious relationships among design components. Here, we investigate engineering students’ capacity to search for optimal nanoscale biosystem designs with stochastic component and system behaviors. The study aims to characterize information types that facilitate human learning and improve their complex system understanding and design proficiency. It is hypothesized that learning parametric system relationships and/or inter-level causal mechanisms improves design proficiency; these relationships and mechanisms are teachable through software interfaces. Two contrasting learning/design interfaces were developed that presented differing information types: an interface with performance charts that emphasized parametric relationship learning and an interface with agent-based animations that emphasized inter-level causality learning. Users improved on pre-/post-learning design tasks with both interfaces; users who demonstrated inter-level causal relationship understanding, which occurred primarily with the animation interface, had greater improvement. All users were then presented contrasting animations of systems with opposing emergent behaviors, resulting in many more participants demonstrating an understanding of inter-level causal behaviors. These findings reveal the difficulties in understanding and designing multi-level systems and that interactive software tools may convey crucial information that supports engineering design, particularly with respect to the development of reasoning skills for how system components relate across levels.

1. Introduction and motivation

Complex systems typically consist of many levels of components and interactions, which makes them challenging to design (Ottino 2004). Non-obvious inter-level relationships commonly emerge among components and behaviors across scales in complex systems, and complex bio-based systems with these properties are becoming a growing area of design interest (Spatz 2005; Young & Alper 2010; Cheng & Lu 2012; Korin et al. 2012; Tan et al. 2013; Egan et al. 2015c ). Challenges for designing complex systems may include difficulties in human understanding of the system (Chi 2005; Hmelo-Silver, Marathe & Liu 2007), and application of knowledge toward optimal design configurations (Egan et al. 2015a ). In this study, a cognitive investigation is conducted by collecting measurements detailing a user’s reasoning processes and studying the influence of external tools on design proficiency via effects on reasoning processes. Although multi-level system design is a broad issue in design, we specifically focus on measuring a user’s ability to optimize multi-level biosystems as an indication of their design proficiency, and how it is influenced by their multi-level understanding of the system. This investigation is conducted to determine what types of information may aid a designer’s learning and understanding of a complex system, and how that understanding may improve their ability to proficiently design the system.

The investigation is informed by recent cognitive and engineering studies that demonstrate the usefulness of software tools including multi-level graphical user interfaces (GUIs) and agent-based renderings in improving human understanding and design of systems (Chi et al. 2012; Li & Gunal 2012; Zhang et al. 2012; German, Feigh & Daskilewicz 2013; Jordan et al. 2013; Woodruff, Reed & Simpson 2013). Specifically, we seek to demonstrate how differently configured GUIs and agent-based simulations can facilitate the learning of inter-level causal mechanisms (Chi 2005) (i.e., how perturbations at one level of organization influence the system behavior at another level) and parametric relationships (Egan et al. 2015a ), and how this information about inter-level causal mechanisms translates to improved human performance during multi-level design optimization tasks. Through asking users questions that probe their qualitative understanding of inter-level casual mechanisms throughout their design tasks, it is possible to connect measures of a user’s understanding of a system to their performance in using design tools.

Myosin motor protein systems are chosen as a representative complex biological system because they possess characteristics general to many complex multi-level systems such as emergent behavior (Egan et al. 2013). Although there are many definitions for emergence, in the context of myosins and other biological systems, emergence may be considered the global system behavior that is a result of collective interactions of lower-level components (Egan et al. 2013). Using this definition, myosin systems are quite different than traditionally engineered complex systems, such as airplanes, that do not rely on collections of highly redundant parts to function (i.e., myosin-based systems require many myosin motor proteins to function, while airplanes may operate with one motor; there are also debates concerning whether airplanes should be considered complex systems) (Ottino 2004).

Emergence itself is not rare, especially in nature when considering cells organizing into tissues and the social/group behaviors of organisms; artificial systems such as Conway’s Game of Life demonstrate that novel behaviors may emerge from relatively simple systems and components (Standish 2001). Generally, engineered systems are designed in ways to avoid extreme forms of emergence that are difficult to control and potentially unpredictable, although most engineered systems will have some form of emergent states depending on their inputs. In the traditional fields of systems engineering, a metropolitan light rail system and an aircraft carrier may both be considered complex systems with emergent behavior, and analysis becomes even more complicated when considering a system of systems, such as an entire metropolitan transport system (Keating et al. 2003). Systems of systems generally consist of nested subsystems, which motivates the need for multi-level reasoning and engineering approaches for bridging local to global levels (Sage & Cuppan 2001).

Further, design with emergent behaviors, especially across levels, often requires non-obvious changes to a system to improve functioning. For instance, in traditionally designed systems with direct behaviors, the change in one design parameter results in a direct change to a performance output coupled to the design parameter. However, in emergent systems, the direct change to design components not only affects the individual design component, but also results in changes to how they interact with collective components that can alter the global emergent behavior of the system. Myosin-based protein systems that possess such complex system attributes are particularly relevant to engineering because of their ubiquity in natural systems (Rome 2005; Huber et al. 2013) and relevance in novel technologies (Koenderink et al. 2009; Neiman & Varghese 2011).

Myosin-based systems are also well-suited for cognitive studies, because they are unfamiliar to traditional engineering disciplines, thereby providing an ideal system for investigating how engineers learn and demonstrate understanding of inter-level causality that requires specialized domain information not known by participants prior to the study. It is also possible to model emergent and highly complex myosin system behavior with three myosin design variables and one systems design variable, such that the system is presented in a manner consistent with engineering design paradigms. Such an approach has been demonstrated in recent agent-based myosin simulations that model complex inter-level behaviors that extend beyond what is currently achievable through experimental measurements or analytical predictions (Egan et al. 2015b ). In these studies, parametric relationships refer to relationships such that a change in a design input results in a measurable change in an output behavior of the system, and inter-level relationships refer to relationships such that a change in an individual myosin design parameter alters the behavior of a system as a whole. Additionally, inter-level relationships exist to describe how differences in a systems level variable, such as the number of myosins in a system, affect the behaviors of individual myosins. Further elaborations on these relationships, and their analogy to designs in the traditional mechanical engineering domain, have been discussed in past work using structure–behavior–function representations (Egan et al. 2013).

Previously, we have modeled and designed myosin systems using agent-based molecular simulations (Egan et al. 2013, 2015b ) and developed GUIs to measure human design proficiency in myosin-based optimization tasks (Egan et al. 2015a ). The GUIs and tasks tested users’ design proficiency with regard to parametric design; parametric design is representative of an important design activity relevant to design search and optimization (Sim & Duffy 2003). Parametric design is also particularly relevant in computer-aided design applications for fine-tuning designs that have pre-defined forms (Roller 1991; Simpson et al. 2007; Goel et al. 2012; Yassine 2012). Our previous GUI studies focused on parametric design relationships for how individual myosins and their organization in a system may influence emergent system performance, but only modeled myosin systems that operated with a consistent emergent behavior. In this paper, we now consider two qualitatively distinct emergent system behaviors that are influenced by the design of individual molecules and the system as a whole. These features lead to inter-level causal relationships that influence the sensitivities of design variables on performance, which is an essential aspect of many complex multi-level systems. This paper also builds from cognitive-based studies that have demonstrated GUIs can aid in learning complex system relationships (Chi 2005; Hmelo-Silver et al. 2007), and extends beyond these findings to demonstrate what types of learning also result in better design of a system.

The primary steps of investigation in this paper are to develop GUIs that present different types of information to a user, determine what understanding a user gains through learning with this information, and examine whether such learning promotes improved design proficiency. The interfaces are extensions of our past myosin-based design GUI and concentrate on presenting information to a user that is parametric in nature, through use of the existing interface, or revealing of inter-level causal relationships, through inclusion of agent-based simulation renderings. These GUIs facilitate the collection of measurements from pre-/post-learning assessments of engineering students’ design proficiency and understanding of inter-level causalities, which enable inferences about user learning and performance from a cognitive perspective. The cognitive considerations in this investigation extend beyond human–computer interaction studies by collecting textual responses from participants to determine their thought processes and understanding beyond purely quantitative measurements collected with the software interface.

Although we present in-depth details for modeling and designing myosin systems (particularly in Background Sections 2.4–2.6 and Appendices A.1 and A.2), the general methods and findings of this study can be understood without a detailed understanding of how myosins in general or the particular myosin system modeled here works. The core understanding required of myosins’ behavior is that the design of myosin-based systems across scales causes the system as a whole to behave in one of two possible emergent states. These consist of whether a filament propelled by myosins has a consistent velocity or an intermittent velocity—the filament velocity is intermittent during periods when all myosins in a system are momentarily not exerting force on the filament. The couplings among variables to describe these behaviors across scales are complicated and counter-intuitive when even a few design inputs are considered (Egan et al. 2015b ), which are simplifications of even more complicated biophysical behaviors (Harada et al. 1990). The outcome of the paper is to demonstrate strategies for aiding user learning, understanding, and design of complex systems from a cognitive perspective that could inform the construction of software tools and design methodologies for supporting complex systems design.

2. Background

The background is divided into seven sections that focus on human understanding of complex systems, use of agent-based simulations for learning, use of animations as teaching supports, the complexities of myosin systems with varied emergent behaviors, modeling of designed myosin systems, and past myosin-based GUI studies.

2.1. Human understanding of complex systems

Studies in measuring human understanding of emergent systems with micro-/macrolevels have demonstrated that misunderstandings of emergent behaviors (e.g., diffusive fluid flow) are more strongly held than misunderstandings of direct behaviors (e.g., blood flow in the circulatory system) (Chi 2005). People who lack understanding of emergent systems often have fragmented system understandings, such as being able to understand aspects of component behaviors but not how they relate across scales. Most commonly, this fragmented understanding occurs through attempts to explain emergent systems as direct processes, rather than distributed behaviors. Studies comparing expert and novice understandings of ecosystems and lungs have demonstrated that experts have greater understandings of behaviors within and across levels, which enables them to more effectively reason about how perturbations affect system functioning (Hmelo-Silver et al. 2007). Such findings are relevant to design problems because engineers must be capable of effectively manipulating a system and its components toward a desirable goal state. These considerations can significantly aid in human design endeavors concerning parametric design, which is known to be difficult for humans even when only a few coupled parameters are involved (Hirschi & Frey 2002). Typically, humans find success in these systems when approaching them utilizing simple decision-making rules, such as manipulating one variable at a time while leaving other design variables constant (Kuhn et al. 2008). Using these strategies, designers may be able to effectively learn and make decisions that improve design outcomes (Egan et al. 2015a ).

2.2. Software interfaces for engineering design

Past engineering design interfaces have been constructed to address the challenges of complex systems design, especially the need to consider multiple objectives in searches with visualizations (Zhang et al. 2012; Woodruff et al. 2013) and trade-offs in multi-dimensional and multivariate data (German et al. 2013). Progress in effective GUI approaches is relevant to design because engineers must be capable of effectively manipulating a system and its components toward a desirable goal state—an effective GUI should enable the engineer to learn essential aspects of complex systems that aid in design. One successful GUI approach utilized the parameterization of designs such that engineers could manipulate design variables to design a system toward an optimal state (Simpson et al. 2007), which is a strategy adopted in this present study. This study expands on the findings from these previous studies through considering systems that are highly stochastic in nature with multiple outputs. These systems are then communicated to users via an agent-based simulation rendering that can provide parametric insights to a user in addition to behavior concerning the structures, behaviors, and functions of a system (Vattam et al. 2011). Such an approach of including cognitive considerations in the GUI design is similar to previous engineering interfaces for investigating design trade-offs that pinpoint designs based on human cognitive principles (Li & Gunal 2012). There is also a need for modeling complex biological systems, as evidenced by recent GUIs developed for cell and gene modeling in synthetic biology contexts, which could contribute to engineering design of bio-based technologies (Villalobos et al. 2006; Chandra, Bergmann & Sauro 2009). These recent synthetic biology interfaces motivate the need for an effective GUI approach for myosin-based complex systems as they are mechanical in nature, which contrasts with these past bio-based GUI approaches that have concentrated on biochemical system modeling.

2.3. Agent-based simulations as teaching supports

Inter-level causal mechanisms can be learned through software tools, such as agent-based simulations, which is demonstrated in studies focusing on users’ understanding of multi-level phenomena relating to fluid diffusion (Chi et al. 2012) and aquarium ecosystems (Vattam et al. 2011; Jordan et al. 2013). Agent-based simulations, in this context, operate through graphically rendering the actions of independent autonomous computational objects in a virtual environment representative of a physical system (Dessalles, Ferber & Phan 2008). These simulations are particularly well-suited for representing multi-level phenomena because individual agents can represent the bottom-most level of a system, such as molecules in the case of fluid diffusion. Agents then perform actions during each simulation step according to rule-based logic, and the results of their interactions may produce recognizable patterns of global emergent behavior, such as two liquids mixing. One of the greatest benefits of agent-based simulations is that users can manipulate a system by altering parameters among agents and receive visual feedback of emergent behaviors, which greatly supports understanding of complex multi-level systems and may be extendible to aiding in the design of complex bio-based systems.

2.4. Animations as teaching supports

Medical education has had great success in utilizing animations for learning, which present an alternate route for utilizing software tools to maximize learning in comparison to agent-based methods (Ruiz, Cook & Levinson 2009; Van Merriënboer & Sweller 2010). Animations can present users with the critical information they need to learn, rather than requiring them to actively search and manipulate a system while learning. Animations are more effective when optimized through cognitive load theory (Van Merriënboer & Sweller 2010), which suggests that a user has a restricted amount of information that they can accommodate at a time, due to possessing a limited amount of working memory. To optimize learning, intrinsic cognitive load (i.e., complexity of information in the given problem that must be processed) and extrinsic cognitive load (i.e., any information that does not contribute to learning) should be minimized while germane cognitive load (i.e., the resources a user has for learning and building of cognitive schemas) is maximized. In the case of myosin systems where two contrasting global behaviors may emerge, side-by-side animations are possibly an effective tool (Alfieri, Nokes-Malach & Schunn 2013) because they reduce cognitive load as a user no longer has to compare the current state of a system to a representation in their working memory, as required in agent-based learning environments.

2.5. Emergent behaviors of myosin systems

Myosins are nanoscale motor proteins that often work in groups to mechanically drive emergent behavior in biological systems (Howard 2001). Individual myosins operate by converting chemical to mechanical energy and stochastically attaching and detaching to an actin protein filament they propel—the propulsion of the filament is essential for contracting muscles or moving molecules across locations. The emergent propulsion behavior depends on how often at least one myosin in a system is attached to the protein filament it is propelling; during periods of time when at least one myosin is attached, the filament is assumed to translate due to the myosin exerting force on the filament, but during times when no myosins are attached, the filament remains static (Harada et al. 1990; Egan et al. 2015b ).

A general finding from biological studies (that may also be considered an engineering heuristic) suggests that a filament has continuous emergent behavior when the probability that at least one myosin is attached, $P_{\mathit{att}}$ , is greater than 90%, which generally occurs when at least two myosins are attached to a filament on average (Harada et al. 1990). This heuristic implies that emergent translation behavior may be considered continuous when $P_{\mathit{att}}>90\%$ and intermittent when $P_{\mathit{att}}<90\%$ , thus presenting two contrasting systems level behaviors. In intermittent systems, the time-average velocity of the system is much lower than the instantaneous velocity of the filament because of the many periods of time when no translation is occurring—filament translation occurs less frequently for lower values of $P_{\mathit{att}}$ . These assumptions are only valid for myosin systems that operate in a highly viscous environment; in low viscosity environments, the system terminates due to diffusion when myosins do not maintain contact with actin. The contrast in time-average velocity for continuous and intermittent systems is illustrated graphically with our agent-based simulation (Egan et al. 2013, 2015b ) in Figure 1. In the agent-based simulation, myosins obey stochastic rules that enable their chemical and mechanical interactions with an actin filament in a virtual environment.

Figure 1. Simulated (A) continuous and (B) intermittent myosin filament translation. Each frame consists of an actin filament and myosins anchored on a microscope slide. Animations are available at http://youtu.be/BEqoOBddteI.

These emergent behaviors present an interesting design challenge, namely that design parameters have different influences on performance based on the emergent behavior of the system (Egan et al. 2013, 2015b ). For instance, designing myosins with higher chances of attaching to a filament has little effect on time-average system velocity when emergent behavior is continuous, but greatly increases time-average velocity for intermittent systems because at least one myosin is attached more often. Therefore, the ability to recognize a system’s emergent behavior and understand effects of design variables on performance within and across levels may greatly improve a human designer’s proficiency in optimizing these systems.

2.6. Myosin modeling and design

Biological studies have demonstrated techniques for synthetically altering myosins (Manstein 2004), which affect their mechanical (Anson et al. 1996) and chemical (Murphy & Spudich 1998) behaviors, and we recently developed engineering models for exploring the optimization of myosin systems (Egan et al. 2013, 2015a ). In our model, myosins were represented through a structure–behavior–function paradigm, which required design of individual myosin structures to be followed by agent-based simulations that recreated their behaviors in order to predict global performance (i.e., function). Three myosin design parameters were considered: (1) a myosin’s lever arm length $\boldsymbol{l}$ , (2) a myosin’s attachment rate $\boldsymbol{k}_{\mathit{on}}$ , and (3) a myosin’s detachment rate $\boldsymbol{k}_{\mathit{off}}$ , in addition to the number of myosins $N_{\mathit{myo}}$ interacting with a filament. It was found that each parameter has unique effects on performance, for metrics including system velocity, energy use, and the average number of attached myosins. The agent-based simulations agreed with predictions from mathematical models and were suggested to offer a more complete perspective for how changes in myosin structures and behavior affect emergent system performance. Therefore, it is possible that users exposed to agent-based simulations could learn inter-level relationships that extend beyond basic input/output parametric models where underlying behavior is obscured.

2.7. Myosin-based design with GUIs

Parametric modeling of myosin design inputs and influences was analyzed in our past GUI studies to assess engineering students’ capabilities in searching for optimal solutions in myosin design tasks (Egan et al. 2015a ) (Figure 2). Details are presented here because we adapt those tools in the current study. In the prior study, the maximum values for each input were a myosin lever arm length $\boldsymbol{l}$ of 20 nm, myosin attachment rate $\boldsymbol{k}_{\mathit{on}}$ of $4000~\text{s}^{-1}$ , myosin detachment rate $\boldsymbol{k}_{\mathit{off}}$ of $2000~\text{s}^{-1}$ , and a maximum number of myosins $\boldsymbol{n}_{\mathit{myo}}$ of 100. The lever arm, attachment rate, and detachment rate were represented in a non-dimensionalized quantity to avoid overly confusing units and values for users unfamiliar with the system. These variables were weighted from 0.2 to 1.0 of the parameter’s maximum value according to the five possible input possibilities tied to each slider. The sliders were tuned such that a user increasing a slider input linearly resulted in its corresponding myosin design input increasing linearly (e.g., if a slider was at its lowest value of 0.2 and then moved to its value of 0.8, and the corresponding myosin input was lever arm length, it would increase from 4 to 16 nm). Because there were four sliders and five input positions for each slider, 625 designs were possible, which is a fairly complex search space for a user who is limited by time and number of total design evaluations. As a user searched for an optimal design, their search was recorded on plots and tabulated for easy reference.

Figure 2. Myosin GUI with demonstration available at http://youtu.be/fJTZX1JfRXw.

The resulting design space was quite difficult for users to navigate when considering the counter-intuitive parameter interactions present and limited time a user had to search the space during our timed user studies (users were allowed to input only ten designs). The feedback provided from the GUI improved users’ design proficiency; however, users still improved their design less often than expected by random chance. The study also demonstrated that users did not learn parametric relationships across problems. Users had improved performance when the GUI enforced the utilization of greedy search logic according to relationships learned while solving a given optimization problem, thus suggesting that parametric knowledge is beneficial to solving design problems.

3. Aims and significance

The primary aim of the study is to determine from a cognitive perspective what types of information can improve a user’s learning and understanding of a multi-level complex system and subsequently improve their proficiency in design tasks. In studying this aim, a cognitive-based model for how users may learn information concerning parametric design relationships and inter-level causal mechanisms from contrasting software interfaces is presented. These software interfaces are tuned to provide different information, such that measurements conducted enable inferences of users’ learning, understanding, and design proficiency according to the proposed cognitive-based model. The cognitive-based model describes how exposure of the user to a GUI may enable learning of different types of information that are beneficial for design. Hypotheses are developed for testing the cognitive-based model, and findings are expected to lead to significant contributions in understanding and improving user design proficiency for complex multi-level systems.

3.1. Aims for investigating proposed cognitive model

Our goals are to investigate from a cognitive perspective how engineers may learn information to improve their parametric and multi-level understanding of a system, and whether such understanding translates to improved design proficiency. Our past myosin GUI study empirically demonstrates that users may be capable of obtaining parametric knowledge and implementing it within a design problem for improved performance (Egan et al. 2015a ). Additionally, past studies with our agent-based simulations have suggested that they could improve a user’s proficiency in reasoning across levels (Egan et al. 2013), which is supported by findings from other cognitive studies of agent-based learning supports for aquarium ecosystems (Vattam et al. 2011; Jordan et al. 2013). These prior studies did not examine whether inter-level understanding improves a user’s ability to design a system. We therefore propose a cognitive-based model for how human design proficiency could be improved through exposure to differing GUIs that emphasize the learning of parametric design relationships or inter-level causal mechanisms. Through having two GUIs with different types of information presented and measurements of the understanding gained by each user, it is possible to study how each type of information contributes to design proficiency.

In particular, to test parametric learning effects, a GUI could enable users to explore and learn the parametric relationships among design variables by demonstrating how inputs affect performance outputs through graphical changes across many plots in a chart-oriented interface similar to the GUI in Figure 2. To test inter-level learning effects, a GUI could enable user discovery of how changes in design inputs affect intermittent and continuous global system behaviors through an animation-oriented interface, where users could change design inputs and receive visual feedback of how myosin behavior changes in an agent-based rendering akin to Figure 1. Both of these types of reasoning are then expected to lead to improved optimization design task proficiency as suggested by the cognitive-based model shown in Figure 3. It describes how strategies in presenting different information via GUIs enables different types of learning that may result in improved design proficiency.

Figure 3. Cognitive-based model for how differing GUIs enable learning that aids design.

In Figure 3, large arrows indicate the primary learning tracks targeted by each GUI interface. However, it is difficult to completely decouple these types of learning from each other as the exposure to charts could improve a user’s understanding of inter-level causal mechanisms or the exposure to animations could improve a user’s learning of parametric design relationships (as indicated by the small arrows). Because each GUI presents different forms of information to a user, it is essential to conduct multiple types of measurements at multiple points in time to determine the types of learning that take place throughout the study and how they affect design outcomes. A challenge in our development of hypotheses and testing is to determine the effect of these phenomena through separate assessments of optimization design task proficiency and a user’s learning of inter-level causal mechanisms. In Sections 4.1 and 4.2, we further elaborate on GUI construction and cognitive study protocols for evaluating the following three hypotheses that test the Figure 3 model: (1) learning via charts or interactive simulations will improve user design task performance, (2) users exposed to animated renderings of agent-based simulation behavior will demonstrate understanding of inter-level causal relationships, while users exposed to chart GUIs will not demonstrate such understanding, and (3) users who demonstrate an understanding of inter-level causalities will perform better on design tasks.

Through these three hypotheses, all arrows in the diagram representing effects are evaluated either directly or indirectly, thus validating or refuting the proposed Figure 3 cognitive-based model. More specifically, the first hypothesis tests the overall tracks with large arrows stating that exposure to the GUI results in learning that improves design task proficiency, without regard to whether that improved performance was a result of learning parametric or inter-level relationships. The second hypothesis specifically tests whether each GUI leads to learned inter-level causal mechanisms; here it is only expected that the animation GUI supports inter-level causal learning because it reveals underlying mechanics that influence emergent system behaviors hidden in parametric representations. The final hypothesis tests whether improved design performance is achieved through learning of inter-level relationships, as opposed to all gains in performance being due to increased parametric knowledge, which has already been studied and suggested to improve user design proficiency (Egan et al. 2015a ).

The assessment of these hypotheses is essential as it enables the decoupling of the types of relationships users learned throughout the study (i.e., parametric or inter-level), which highlights how the different types of information presented by the contrasting GUIs affect user design proficiency. It is possible to assess these hypotheses using a combination of between-subject and within-subject comparisons. Between-subject comparisons among users exposed to different interfaces enable inferences concerning how information types influence design proficiency, while within-subject comparisons of users throughout the study can demonstrate the effects of learning on improvements in individuals’ reasoning. Through these measurements, it is possible to infer which types of information users learned, and how it affects their design outcomes.

3.2. Significance

This study seeks to understand how engineers can learn and understand types of information that result in improved proficiency in designing complex multi-level systems. Through measuring the effects on GUI exposure on user design proficiency, it is possible to infer what types of information a user learned and how specific understandings contribute to searching for optimal solutions in myosin-based design tasks. Each of the three proposed hypotheses tests the magnitude of different learning and application pathways in Figure 3, which serves to validate the Figure 3 cognitive-based model and reveal effective routes for presenting information through software tools that aid engineers in complex multi-level system design. If the aims of this study are achieved, then new insights will be reached concerning learning and design of complex systems from a cognitive perspective, which could translate toward new strategies for developing software aids for instructing and improving engineering design performance.

4. Experimental methodology

To test the hypotheses, a parametric-oriented chart GUI and a multi-level-oriented animation GUI were constructed. An experimental protocol was developed that assesses user understanding and design proficiency as they solve design tasks and answer questions using their respective GUI.

Figure 4. Myosin GUIs for learning with (A) charts with video demonstration available at: http://youtu.be/6Ke5I-Rm1Co and (B) animations with video demonstration available at http://youtu.be/f0G7uc3Empk.

4.1. Graphical user interfaces

Two GUIs that emphasized learning through charts (Figure 4A) and animations (Figure 4B) were developed to conduct a cognitive study and validate the Figure 3 model. To maintain consistency with past myosin GUI studies (Figure 2) and to compare effects of learning from each interface, all interfaces were constructed similar to past myosin design GUIs. Modifications were made to the GUIs to produce two contrasting interfaces that facilitate the testing of different Figure 3 cognitive-based pathways through altering how graphical output of the GUIs is presented to a user. These design GUIs consist of four design inputs and up to two performance outputs in the chart interface or an agent-based rendering that updates in real time for the animation interface. All design inputs are manipulated by users via sliders that have five possible values for each myosin and systems design input, with maximum values for each parameter being a myosin lever arm length $\boldsymbol{l}$ of 20 nm, myosin attachment rate $\boldsymbol{k}_{\mathit{on}}$ of $4000~\text{s}^{-1}$ , myosin detachment rate $\boldsymbol{k}_{\mathit{off}}$ of $4000~\text{s}^{-1}$ , and a maximum number of myosins $N_{\mathit{myo}}$ of 100 or 30 depending on the task (tasks prior to learning support have a 100 myosin maximum so that intermittent behavior is uncommon). The performance of evaluated systems was calculated using mathematical equations (§ A.1) that predicted the global time-average filament velocity, the energy utilized via adenosine triphosphate (ATP) use of the myosins, and the average number of myosins attached to the filament.

The chart GUI was configured through supplying users quantitative feedback of system performance via plots of independent design inputs and dependent performance outputs, thus allowing the user to change myosin design inputs and see measured changes in emergent system behavior, as demonstrated in Figure 4(A) screen capture (video available at http://youtu.be/6Ke5I-Rm1Co). In Figure 4(A), directions for learning are in the top left, the time a user is allowed to utilize the scene before it is advanced automatically is in the bottom left, and the design inputs with a static graphical representation of the current system are in the center left. The middle portion of the screen consists of eight plots that represent each of the four independent design variables and two performance outputs that are calculated via mathematical equations. On the right of the screen is a summary that provides users a basis for what parametric relationships they should learn for their next task, which aids in maximizing a user’s germane cognitive load.

The second GUI configuration presented users animated feedback of the agent-based molecular simulation when design inputs are varied (video available at http://youtu.be/f0G7uc3Empk), as demonstrated in Figure 4(B) screen capture. The animation GUI is similar to the chart GUI of Figure 4(A), except that the plots and static myosin representation are replaced with an agent-based simulation that updates in real time as users manipulate design inputs. During periods of time when no myosins are attached during the simulation, the system becomes transparent, which should draw the attention of the user to the intermittent system behavior that occurs when less than two myosins are attached on average, thus highlighting crucial information indicative of inter-level relationships.

Through each of these two interfaces, a user is expected to learn different types of information about the system by manipulating inputs and recognizing their effects on output performance. For instance, with the animation interface, a user could add more myosins to the system and notice that if the filament was originally intermittently moving, it would begin moving more often as there are more periods of at least one myosin being attached. In the chart interface, a user would see the filament velocity parameter increase on the $y$ -axis as the number of myosins is increased in the system if it was changing from intermittent to continuous emergent behavior. However, it may not be obvious to the user in the chart interface why the velocity increased in this intermittent case, but not when the system is already continuous since they do not directly observe simulated myosin and global filament behaviors—such inter-level revelations are more apparent from the animation interface. Further details about how users may expect parameter changes to influence the performance of a system are available in § A.2. A drawback to the animation interface is that the user must view the simulation for a period of time before recognizing how a perturbed design input has affected the behavior of the system. Such a delay leads to a greater effort and less opportunity for the user to gain insights concerning parametric relationships within the same time period as a user with the chart interface.

Thus, in each interface, users have some chance to learn both parametric and inter-level causal mechanisms but the effort required to infer (i.e., cognitive load) and availability of information differ. These differences in information presented by the interfaces are an intentional aspect of the experimental design. Specifically, it enables comparison of how different types of information learned from the contrasting interfaces influence design proficiency and shows how the decoupling of parametric learning differs from that of inter-level causal learning with regard to design outcomes. Without these differences in the GUIs and experimental approaches, it would be difficult to determine whether users gained better design proficiency as a result of gained parametric knowledge or inter-level causal understanding.

4.2. Cognitive study protocol

In order to conduct cognitive studies with the contrasting GUIs, each user follows a sequence of scenes that present useful information, design tasks, learning interfaces, or multiple-choice/free response questions. To test the hypotheses proposed in Section 3.1, there are one of two scene sequences presented to each user that are identical except for users being exposed to the animation or chart GUI during particular learning scenes and question/answer scenes. In our study, 31 mechanical engineering undergraduates in a senior design class participated for course credit and were randomly assigned to either track, resulting in 15 participants in the chart condition and 16 participants in the animation condition. Details concerning each track are illustrated in Figure 5 and elaborated on throughout this section.

Table 1. Six optimization tasks were developed with varied goals and constraints. Solver score is indicative of results from a random search of the design space (higher scores reflect better designs found)

Figure 5. Flow chart of cognitive study protocol.

Users in both conditions begin with a software tutorial that explains how to use the interface to design myosins and answer questions. Next, three baseline myosin design tasks are presented to the user via generic variable names of Input A/B/C/D and Output A/B instead of the different myosin design inputs and outputs they represent. The purpose of the generic variable names is to minimize learning of parametric relationships prior to exposing users to learning scenes. The mapping of the generic names to myosin parameters is changed for each task. The maximum number of myosins for these tasks is set to 100, such that intermittent filament behavior is not significant and therefore not learned by a user. Users are allowed to input ten designs within 4 min and receive feedback immediately upon evaluation of that design. All users throughout this study successfully input ten designs for all tasks. These design tasks are provided in Table 1 and consist of the functional requirements considered during a user’s optimization task that represent a myosin technology.

The table shows the output goal (users are instructed to minimize or maximize their design performance with respect to this output) and the additional constraints on output performance (users are instructed that their designs cannot violate any output constraints) for each design task. If a design violates a constraint, it is designated infeasible and given a score of zero. Three design task difficulties were created by increasing the number of output variables and constraints (although only one output is graded as the goal output, additional constraints on secondary output variables inherently reduce the set of acceptable possibilities in the larger design space). The three types of difficulties are 1-output–1-constraint, 2-outputs–1-constraint, and 2-outputs–2-constraints. To create a benchmark for later comparisons against user scores, a random solver was used to propose ten unique designs for each task and provided an average score on a scale of 0 to 1, which is normalized to the global optimum for each task; the average solution of the random solver was found by having it solve the task repeatedly until the standard error in the measurement of its mean score for a given task was negligible. A score of 1 is the highest score possible, and user scores were also normalized according to this metric (e.g., if the global optimum solution is a maximum velocity of $4~{\rm\mu}\text{m}~\text{s}^{-1}$ , and a user’s best design has a velocity of $1~{\rm\mu}\text{m}~\text{s}^{-1}$ , they receive a score of 0.25; the reciprocal is taken for minimization tasks).

A baseline assessment of whether users could demonstrate inter-level causal reasoning was taken by asking them to answer what would happen to system velocity if an unloaded system of 100 myosins was redesigned with only 10 myosins or with myosins of a higher attachment rate. This question is an assessment of their understanding of inter-level causal relations because a system with many myosins should have a large number of myosins attached on average (and therefore be continuous), so removing a large number of myosins would likely slow the system and drive it toward the intermittent regime, while increasing the myosin attachment rate would have no effect. In all of the qualitative assessment sections, the inter-level relationship of global velocity to the average number of myosins attached to a system is used as a metric because it is easily defined using the heuristic regarding how many myosins are attached on average (two or more lead to a continuous system, less than two leads to intermittent behavior). Whether a system is continuous or intermittent has profound effects on parameter sensitivities essential for effectively searching the design space (§ A.2).

Users are then provided one of the learning interfaces based on their assigned condition before solving their next set of learning tasks with functional requirements provided in Table 1. The maximum number of myosins in each design is reduced to 30, which means that the system is much more likely to become intermittent. Afterward, users’ ability to describe inter-level causalities is assessed a second time by asking several multiple-choice questions about a system presented according to their study condition.

Table 2. Four qualitative reasoning assessment phases

These questions are shown in Table 2, and present the user a system either in the intermittent or continuous regimes and ask how the time-average filament velocity should change based on perturbing a system from an initial state of being continuous or intermittent; design perturbations in the questions were designed to have an obvious effect on a user who understands the inter-level causal mechanisms of the system. In Table 2, the design input values in the table reflect the slider value in the GUI for the myosin design variables and how many myosins were present in the system. At this point in the experimental design, users who demonstrate an understanding of the inter-level causality are assumed to have also had that knowledge for their final design task (completed just before the second assessment of inter-level understanding). This assumption enables an examination of how inter-level causal knowledge affects design task proficiency (which is necessary for answering the third hypothesis from Section 3.1). A third qualitative assessment was then conducted in which users were given a hint through provision of the correct parametric answer to the multiple-choice questions prior to demonstrating their reasoning in support of that answer. This hint enables the determination of whether improved parametric knowledge may support proper description of inter-level causalities (second hypothesis from Section 3.1).

Questions were designed as in Table 2 for a balance in difficulty across each assessment, but questions varied in specifics to avoid users becoming familiar with specific questions being asked. All questions were tuned to test a user’s inter-level causal reasoning through asking how a change in system design may result in a change in filament velocity, although the precise variables for each question changed. Balancing of questions was controlled through having one question in each assessment require knowledge that the time-average filament velocity should change notably, while the other question in each phase required knowledge that the design perturbation would not significantly alter filament velocity. Additionally, users were not provided feedback concerning whether their answer was correct or not at the end of each assessment, which further promotes the independence of measurements gathered from each phase. The purpose of having multiple assessment phases is to determine the growth in user understanding of inter-level causalities throughout the course of the experiment, and to specifically address the second hypothesis of the study.

Additionally, as a control, parametric questions were asked during the second and fourth assessment phases to determine whether users properly understood whether changes in attachment rate or numbers of myosins affected system outputs. Including such questions enabled an assessment of whether students misunderstood inter-level causal mechanisms due to the non-obvious nature of the emergent behavior, or because they did not understand parametrically how changes in design inputs affect design outputs. Examples of questions for these controls are ‘Does adding more myosins to a system increase the average number of attached myosins in a system?’ and ‘Does increasing the attachment rate of each myosin increase the average number of attached myosins in a system?’ Users answered these questions through checkboxes to indicate ‘yes’ or ‘no’ answers.

Last, users in both groups are presented contrasting animations before a final qualitative reasoning assessment. The animations show users the systems side by side in the intermittent and continuous emergent behavioral regime, thus providing a clear distinction to the user for how systems designed in two different ways produce two different global patterns of behavior. Users are also provided the engineering heuristic that if about two myosins are attached on average, the system will remain continuous while systems with much fewer myosins attached on average have intermittent behavior (and therefore lower time-average velocity). The purpose of the contrasting animations is to determine what percentage of users are able to correctly learn and explain these crucial inter-level causal mechanisms from a learning interface with reduced cognitive load and salient characteristics highlighted. The question details for this final assessment are also provided in Table 2. The complete sequence of scenes for the two study conditions provides an opportunity for testing each proposed hypothesis, through comparison of effects of each learning interface, as well as how users learn and apply parametric and inter-level causal knowledge of the system.

5. Cognitive study results

Cognitive study results are reported in three sections that reflect the three hypotheses proposed in Section 3.1, with the first focusing on the extent that learning supports from the GUI improve user design proficiency. A measurement of how many users demonstrated proper understanding of inter-level causal relationships is then provided. Last, the design proficiency of users who demonstrated understanding of inter-level causalities is compared to those who did not. These results are aimed to specifically address and decouple different aspects of the Figure 3 cognitive-based model, particularly with respect to the extent that each interface enables learning of parametric and inter-level causal relationships, and how such understanding translates to improved design.

5.1. Both types of learning improve design proficiency

The first hypothesis was learning via charts or interactive simulations will improve user design task performance. The hypothesis was analyzed by aggregating data from each optimization task separately, and then averaging the solution quality of a user’s best solution for that task with all other users in their condition. Solution quality was determined by first comparing a user’s goal output value to the global optimum for a task and providing it a relative score between 0 and 1 (all designs that did not meet constraints had a score of 0; the global optimum has a score of 1). The solution quality was then calculated by finding the difference between the user average relative objective function and the average score found by a random solver (Table 1) to facilitate absolute evaluation and performance comparison across problem types. Each problem type is defined by the number of outputs/constraints that describe its design space. Tasks were generated such that those of the same type are representative of similar search difficulties (Egan et al. 2015a ), thus providing a means of comparison on pre-/post-learning tasks. The average solution quality of users is presented in Figure 6 for each task and learning condition.

Figure 6. Average user solution quality in each learning condition for all tasks.

In comparing baseline and post-learning tasks, the average solution quality improved post-learning for all tasks, which supports the hypothesis, as demonstrated by Figure 6 pair-wise comparisons with $p$ -values reported. When $p\leqslant 0.05$ , it is suggested that the error rate in reporting a false positive is less than 5%, which is generally considered as statistically significant. The results suggest that the information conveyed by each GUI resulted in nearly equal improvement in design proficiency. Improvement of users increased as task complexity grew, thus motivating an ever increasing need for software tools as systems grow increasingly more complex, which remains consistent with previous findings for myosin GUI studies (Egan et al. 2015a ). Interestingly, the charts and animations did not appear to support design performance by changing design strategies, at least not on the strategies previously shown to improve performance on these tasks (e.g., only changing one input at a time, searching near their current best design, minimizing/maximizing inputs), which were assessed through post-processing of user input data in relation to their received output (Egan et al. 2015a ). Thus, the benefit may occur through the knowledge used in existing strategies rather than via changing strategies. The difference in performance among those in different chart/animation conditions is not significant for any pair-wise measurement.

These results demonstrate the overall influence of each GUI for improving design task proficiency. They demonstrate that agent-based models, which past studies have shown to improve user understanding of systems (Vattam et al. 2011), can also improve user design proficiency of the systems. The relatively equal impact of each method on design quality has limited theoretical importance, as it does not imply that the two methods have equal influences on users since it is possible that different underlying cognitive mechanisms of users are affected by each interface (such as performing better due to gains in parametric relationships or qualitative understanding). Further analysis of the user performance with regard to the Figure 3 cognitive-based model is required to determine how gains in design performance are attained by users in each of the study groups, despite differing types of information being emphasized by each GUI.

5.2. Animations support inter-level understanding

Throughout the cognitive study, user understanding of inter-level causalities was assessed to address the second hypothesis: users exposed to animated renderings of agent-based simulation behavior will demonstrate understanding of inter-level causal relationships, while users exposed to chart GUIs will not demonstrate such understanding. In each assessment phase, users were asked two questions of how filament velocity of a system would change if redesigned, as presented in Table 2. The questions were designed such that the perturbations would obviously have a large effect or negligible effect on system velocity to a user with knowledge of how inter-level relationships of the system affect its emergent behavior.

Users indicated their answer via a multiple-choice box (either filament velocity increases notably or remains about the same) and typed their reasoning in a free response box that was only analyzed for users who correctly answered the multiple-choice question (with the exception of assessment phase one, where no multiple-choice question was given). Free response answers were tagged as a correct demonstration of understanding inter-level causality if users referred to the stopping/starting behavior of the filament being related to having at least one myosin attached. Examples of user responses that were tagged as correct were The increased number of myosins results in more time during which at least 1 myosin is attached and therefore the filament is being pushed forward and Average filament velocity increases because when there are more myosins there is less of a chance the filament will not be moving as a result of no present myosins. These were deemed as correct because both answers relate the individual myosin level (that one must be attached) to the emergent phenomenon of continuous or intermittent filament motion. Some examples of answers tagged as incorrect, despite users indicating the correct parametric relationship, were Since the amount of force is increased and On average the filament is going to move at the rate at which the myosin pulls, which is independent of how often it attaches. The first of these answers is incorrect because the force of the system has no influence on its velocity in our model where time-average force among myosins equals zero. The second answer is incorrect because it fails to take into account the average velocity of the filament, which requires consideration of times when it is translating and static, despite it having the same instantaneous velocity every time at least one myosin is attached.

Understanding of inter-level causalities was measured four times through averaging the percentage of correct free response answers of users in each condition during each phase. No users indicated a proper understanding during the baseline assessment phase prior to learning sessions. The next two assessments occurred after the learning sessions (the first being directly after, and the second occurring once users were provided the correct parametric relationship via the correct multiple-choice answer but still had to provide an explanation). The final assessment occurred immediately after the contrasting animations were presented. The percentage of correct answers for users in each condition is presented in Figure 7.

Figure 7. Percentage of correct responses from users in both learning conditions for demonstration of inter-level causal understanding.

Users in the animation group correctly demonstrated understanding of inter-level causality about 33% of the time directly after the learning tasks, thus indicating that a significant number of users learned $(p\leqslant .02)$ as a result of interacting with the animation interface, when compared to none of the users demonstrating an understanding during the baseline assessment. No users in the chart condition demonstrated understanding at this point, thus supporting the hypothesis that the agent-based simulation supports learning inter-level relationships while the chart-based GUI does not support learning inter-level relationships $(p\leqslant .04)$ . However, despite supporting the hypothesis, only a small portion of users demonstrated understanding. After the parametric hint was provided, there was no significant improvement; one user in the chart condition explained the stochastic system behavior correctly, which suggests that it was not entirely implausible for users in that condition to formulate theories in line with correctly explaining the inter-level causal mechanisms. However, such a result suggests that parametric reasoning processes are not sufficient to teach users inter-level causal mechanisms.

After the contrasting animations were presented to users, approximately half of the users in each condition correctly explained the inter-level causality, which is a significant increase for both the chart $(p\leqslant .001)$ and the animation condition $(p\leqslant .03)$ . This pattern further supports the hypothesis that agent-based simulation renderings are effective in teaching inter-level causality, and are more effective when cognitive load is reduced via contrasting animations (Alfieri et al. 2013). Because average user score was only about 50%, our findings reinforce prior reports that emergent systems are difficult to understand (even for engineering students) and misconceptions about these systems are robust to learning interventions. The growth of user understanding in both groups, however, shows the success in the software interface’s capabilities for producing a cognitive shift in how some users in each group understand and reason about inter-level causal mechanisms essential to system performance.

Past research has also found that inter-level causal understanding is particularly difficult for learners (Chi et al. 2012). In that prior work, most students were able to learn about molecular diffusion using agent-based simulations along with other instructional tools. The lower number of learners gaining proper understanding from simulations in this study may be attributed to shorter overall learning periods, in addition to the greater complexity of the myosin system in design contexts when compared to diffusion systems.

Additionally, control questions (Section 4.2) were asked to determine whether users understood parametric influences of the system, such that they knew an increase in the number of myosins or their attachment rate resulted in an increase in the average number of attached myosins in the system. User groups scored between 73% and 94% correct on these types of questions, which further demonstrates the difficulties in learning inter-level causal behaviors even when correct parametric understanding of the system is achieved.

5.3. Inter-level understanding improves design performance

The final hypothesis tested was users who demonstrate an understanding of inter-level causalities will perform better on design tasks. This hypothesis was investigated by separating the users in the animation condition into those who did and did not demonstrate an understanding during the second assessment. Users in the chart group were ignored for testing this hypothesis because they demonstrated no inter-level causal understanding immediately after the design tasks and had access to differing information compared to users in the animation groups, which would not enable a controlled comparison of how inter-level understanding influences design task performance independent of a user’s gained parametric knowledge. Group performance was then compared on the final design task (Figure 8). Only the final design task was selected because the first qualitative assessment immediately followed it and thus was the closest measure of inter-level understanding during design. Performance on the baseline task was also investigated to rule out variable differences among participants related to design ability.

Figure 8. Design performance of users from animation condition who did/did not demonstrate inter-level causal understanding.

The results show that during the baseline, there was not a significant difference $(p\leqslant .47)$ in design performance among the two groups; thus when no users understood inter-level causality their design task performance was similar. Afterward, users who had demonstrated understanding of inter-level causal relationships performed better and found designs very close to the global optimum when compared to users who had not demonstrated understanding $(p\leqslant .01)$ . This finding is statistically significant and supports the hypothesis that users who demonstrate an understanding of inter-level causality perform better in design applications related to that understanding. The result is particularly important because it isolates the understanding of inter-level causal mechanisms of the system as beneficial to the design process, in comparison to learning parametric relationships among variables. This finding also highlights inter-level causal relationships as an important form of information that can improve engineers’ ability to design, as opposed to learning other types of information about the system that improve a user’s understanding (e.g., learning the detailed mechanochemistry of myosin motors would improve user understanding, but not necessarily translate into improvements in designing a system effectively).

6. Discussion

The discussion section focuses on the importance of inter-level causal reasoning, the two routes for improving design proficiency conducted in this study, and the support of contrasting animations in promoting inter-level causal reasoning.

6.1. Importance of inter-level causal reasoning

Complex systems are often characterized by multi-level relationships, and our results demonstrate that when inter-level understanding of a system is isolated from other forms of knowledge, such as parametric relationships, it has a significant effect on design search success. It is possible that knowledge of inter-level relationships was helpful in the case of myosin design because parameter sensitivities are altered depending on the design of the system as a whole; for instance, adding more myosins only increases the system velocity when the global behavior is intermittent. This finding is important because it demonstrates that through knowledge of how a complex system is organized, designers may become more proficient. In other words, there are not just levels that make up the complex system, but inter-level relationships that when identified and understood may support designers in forming effective decisions in an otherwise intangible system.

These findings reinforce conclusions from other cognitive studies (Hmelo-Silver et al. 2007; Chi et al. 2012), such that learning underlying behaviors and connections in a system is beneficial in comparison to more structure–function or black-box presentations that obscure underlying information that designers could use effectively. Such findings are important as they isolate the types of information that directly aid users in designing complex systems effectively, as opposed to information that aids in understanding other aspects of the complex system, such as nuances in myosin biochemistry that may not translate to improving user proficiency in parametric design contexts.

6.2. Validity of cognitive-based model for improving design proficiency

A cognitive-based model for improving user design proficiency was proposed via Figure 3, which suggested that presenting information emphasizing parametric and inter-level relationships to users could promote learning and understanding that translates to improved design proficiency. Two contrasting GUIs were proposed as methods for testing and supporting the model. Both GUIs were demonstrated to improve user design proficiency equally on pre-/post-learning tasks; however, they emphasized different aspects of learning parametric and inter-level relationships. Results demonstrated that only users with the animation GUI learned inter-level relationships; therefore, results imply that all of the pre-/post-learning benefits from the chart condition occurred through increased parametric knowledge. Users in the animation condition were then assessed as two separate groups: those who demonstrated knowledge of inter-level relationships and those who did not. The results suggest that users in the animation condition who only learned parametric relationships did not improve as much as those in the chart condition, and users who may have learned both parametric and inter-level relationships improved the greatest.

These results support the Figure 3 cognitive-based model and suggest that the chart GUI leads to a large effect on parametric relationship learning, which improves design proficiency but has little influence on user learning of inter-level relationships. The animation GUI on the other hand has a smaller effect than the chart GUI for parametric relationship learning and about one third of users gain the benefit of inter-level relationship learning. Therefore, these results suggest that the equal overall design performance improvements provided by each interface (Figure 6) are achieved through differing magnitudes of cognitive-based pathways illustrated in Figure 3. These findings are also supported by previous study reports that parametric knowledge supports design proficiency (Egan et al. 2015a ), but these new results now demonstrate that parametric knowledge by itself does not provide inter-level reasoning (as demonstrated by the third assessment phase), which is beneficial in complex systems design tasks. Most importantly, both types of knowledge are important for improved design proficiency and engineers should learn both parametric and inter-level relationships when confronting complex multi-level design challenges. Further research could consider whether the fidelity of interfaces influences user learning and affects design outcomes, such as the agent-based model demonstrating more realistic representations of the system when compared to the chart interface. Such investigations would require new hypotheses and questions that isolate such considerations, and future studies could also investigate GUIs that are hybrids of the two approaches or convey information beyond the GUIs considered in this study.

6.3. Support of contrasting animations

Because learning of inter-level relationships is beneficial to the design tasks beyond what is easily learned through parametric relationships, it is essential to have effective ways of conveying such information to designers. Such learning can be facilitated by communicating salient inter-level causal relationships in a system via contrasting animations. When users were exposed to contrasting animations, nearly half of all users in the study regardless of their condition were capable of demonstrating inter-level system knowledge. Such findings demonstrate the effectiveness of the contrasting animation approach and also the difficulty for human designers to learn salient and necessary complex system characteristics. Future work could consider more effective tools for teaching engineers inter-level causal mechanisms of complex systems, possibly through reducing the cognitive load in methods presented in this paper or through teaching important system characteristics in multiple phases that are easier to understand.

7. Conclusions

This study aimed to determine how different forms of information presented via GUIs may support engineers’ learning and understanding of parametric relationships and inter-level causalities that influence emergent system behavior, and whether such understanding leads to improved design proficiency for complex multi-level systems. Users consisted of mechanical engineering undergraduates and their design proficiency was assessed through completion of myosin optimization tasks with a design GUI. The supports of learning parametric relationships and inter-level causalities were decoupled by having users exposed to either a GUI that emphasized learning of parametric relationships through interactive charts or learning of inter-level causalities through agent-based simulations rendered in real time as animations. Qualitative assessment tests were utilized to determine whether users had identified and understood inter-level causal mechanisms crucial to the design space. The experimental design enabled inferences regarding cognitive-based learning of individuals through a combination of selective exposure to different GUI interfaces and the resulting effects on measured performance.

The results for pre-/post-learning tasks demonstrated that both GUIs improve user design proficiency nearly equally, although about a third of the users in the animation condition also demonstrated a correct understanding of inter-level causalities while no users in the chart group demonstrate such understanding. The users in the animation group that correctly identified inter-level causalities also had greater improvement on pre-/post-learning tasks, thus suggesting that learning of inter-level causal system mechanisms supports multi-level design tasks. Contrasting animations of the system designed in two different behavioral regimes were provided to users in both conditions to conclude the study, which resulted in about 50% of users in each study condition demonstrating an understanding of inter-level causal mechanisms.

These findings suggest that successful design of complex systems may benefit from reasoning processes that extend beyond knowing the parametric relationships in a system, namely that engineers may benefit from considering the complex organization of the system they are attempting to design. Here, our study suggests that methods for teaching inter-level causal mechanisms to engineers are crucial to consider when promoting human understanding for effective complex systems design. Our results demonstrated that the most proficient engineers may have learned both parametric and inter-level relationships and outperformed users who only benefited from parametric design knowledge. Therefore, the most effective route to engineering complex systems may be a combination of traditional engineering approaches for teaching parametric relationships bolstered by considerations of how a complex system’s unique organization influences the search of its design space.

As a whole, these findings demonstrate the challenges in user understanding and reasoning about inter-level causal and parametric relationships in complex multi-level systems, and that software tools can promote learning of these relationships. Particularly, agent-based simulations may support users in discovering behaviors of a system that are often hidden in parametric representations of design spaces. Gains in understanding can then promote better human performance in complex systems design applications, where non-obvious emergent behaviors that were demonstrated in myosin-based systems are common to many complex systems that are challenging for designers.