Development of a game-based learning tool for applied team science communication in a virtual clinical trial

Abstract

Educational tools for application of team science competencies in clinical research are needed. Our interdisciplinary group developed and evaluated acceptability of a virtual world game-based learning tool simulating a multisite clinical trial; performance hinges on effective intrateam communication. Initial implementation with clinical research trainees (n=40) indicates high satisfaction and perceived relevance to team science and research career goals. Game-based learning may play an important role in team science training.

Introduction

Team science in clinical and translational research is increasingly required for complex multidisciplinary projects [1]. Because this discipline is a relatively new field, the competencies for team science learners are not well defined [2]. Moreover, a recent analysis of Clinical and Translational Science Award institution curricula indicated that available didactic resources for team science training were adapted from other disciplines and not directly pertinent to clinical research [3]. In addition, best practices for training in the application of team science competencies are lacking, such as facilitated practice in interprofessional communication. Therefore, there is a need for educational tools that foster skills for effective clinical research teams.

Game-based learning, particularly through digital media, can increase motivation to learn and to build skills in several fields [4, 5]. A number of recent reviews indicate positive impact of games in knowledge acquisition and skill development as compared with traditional teaching methods [4, 5]. Skills in interprofessional communication are fundamental to team science competencies, and games can potentially connect group communication to immediate performance feedback. Team games also allow participants to assess the relative value of strategies for collaboration, including decisions regarding assignment of leadership, attunement to individual- Versus team-level performance metrics, and strategies for collaborative planning.

Virtual worlds afford opportunities to simulate applied scenarios, and, in distance learning, can enhance the social element of didactic content. Game-based learning in virtual worlds to support teamwork has been used in management [6, 7], and medical teaching [8]. To explore the potential of this methodology in clinical research, we designed, developed and evaluated acceptability of a team science education game centered on communication and collaboration in a multisite clinical trial. We also describe key decision points during development, which might help educators develop other clinical and translational research education games.

Methods and Results

Design Phase

We established an interdisciplinary design group that included clinical research educators, computer scientists, organizational psychologists, instructional designers, graphic artists, and end users. The group engaged in an 8-month iterative development process, building from a partnership between UC San Diego Altman Clinical and Translational Research Institute and VirBELA, Inc., a creator of virtual worlds for collaboration and education. The design group’s initial plan was to develop a virtual world that addressed broad team science and leadership competencies, with a series of “mini-games” that simulated multiple steps in the clinical trial process (e.g., staff selection, Gantt charts). These ideas were presented to our targeted end users (i.e., predoctoral and postdoctoral scholars and junior faculty engaged in clinical research training and their instructors) with storyboards, and feedback revealed tensions between devising a simulation experience to teach the elements of clinical trials Versus gameplay that focused more on the interactions and strategies for communication. Feedback indicated that linking communication and collaboration strategies with performance in a stylized clinical trial would best capitalize on the unique strengths of the virtual world. Moreover, the initial design was intended to connect progressive experiences over multiple sessions. Pragmatic considerations regarding feasibility led to the conclusion that the most scalable approach would be a shorter game that could be embedded in multiple courses containing team science content. We then developed a facilitator guide and didactic content connecting game experience with the learning objectives.

The choice of a multisite clinical trial as a setting was selected over other options (e.g., collaborative planning for a novel study) because of the variety of roles in a clinical trial, applicability to broad range of learners, and the inherent necessity to simultaneously consider individual, “site” and trial-wide performance in a multisite trial. Detailed game elements, including the duration, perspective, scoring and performance metrics were specified in a game concept document [9]. Assessment and refinement of the beta-version involved about 15 meetings with 40 end users and administrators, and over 500 hours of software development time.

Game Learning Objectives and Design

The learning objectives of the completed game are to (1) identify interdependencies between professionals in a clinical trial needed to achieve a shared goal, (2) reflect on communication as a critical component of individual role performance to support team-level performance, and (3) learn strategies for using trial accrual data to facilitate team coordination discussions aimed for performance improvement.

The game is played in a virtual world 3D campus programmed with the commercially available Unity 3D engine. Learners first download the application (available for Macintosh or Windows operating systems). They next register an account and create an avatar to represent themselves. After team members view game-play instructions, they enter the game, which can be calibrated to last either 5 or 10 minutes. The object of the game is to recruit, enroll, and complete the protocol for study participants during a timed period. The game is played by teams of 4 learners, with each individual team member taking 1 of the 4 “roles” which represent professionals in a clinical trial: Recruiters, who are responsible for identifying and prescreening potential study participants among a general population; Screeners, who identify eligibility of recruited individuals based on inclusion and exclusion criteria; Coordinators, who manage the day-to-day completion of study-related activities; and Principal Investigators (PIs), who use budget-limited resources to improve processes and manage overall team performance (see Fig. 1). Each team may also include an optional nonparticipant Observer who does not play a role but is present in the virtual world. Each player works semi-independently in a distinct virtual space (with the exception of the PI, who is able to navigate freely throughout the virtual setting). The minimum number of players is 4, and up to 4 teams or “sites” may play the game for a maximum of 16 simultaneous players.

Fig. 1 Screenshots of team roles. From top left: Recruiter, Screener, Coordinator, Principal Investigator. At the top of each screen, time remaining, budget, and enrollment metrics are displayed.

Although players are tied to their individual tasks, they are able and encouraged to communicate with their teammates using Voice over IP. The game is designed with several potential opportunities for “bottlenecks” in managing study flow, as well as several relevant points of information that are not revealed fully to all players, such as the impact of various recruitment strategies on the complexity of the screener’s tasks and the negative impact of participant “wait times” on attrition. Thus, intrateam communication is critical in successfully completing the game. Once the allotted time has run out, team members reconvene in a debrief “room,” in which they view indicators of their performance in the clinical trial, including the number of participants recruited, screened, retained and completed (see Fig. 2). Learners can compare their outcome with other teams, introducing competition to increase engagement. The teams can play multiple times and experiment with different communication and teamwork strategies in order to improve their performance.

Fig. 2 Screenshot of the debrief room. After completion of the game, team performance metrics are displayed and discussed in comparison with other teams.

In-Class Facilitation

We developed the game to be played within a 1.5–2-hour facilitated discussion, with discussion connecting game experience with relevant principles in team science. The following in-class agenda was employed:

1. (1) Group discussion (10 min): learners first reflect on their experiences in effective and ineffective teams, in research teams or, without such experience, in clinical teams.

2. (2) Team science and clinical trials (10 min): trends in increased prevalence of multiauthored manuscripts and multi-PI funding proposals [1], as well as the increasing complexity of research projects, are discussed. Typologies of scientific teams, from unidisciplinary to interdisciplinary and transdisciplinary [1] provide context for trainees to reflect on their current clinical and research teams.

3. (3) Communication and coordination in team science (10 min): using a brief case study in workplace communication [10], facilitators and trainees discuss the effect of communication in team performance. Common conflicts and challenges in team science [11], as well as potential mitigation strategies, are discussed.

4. (4) Gameplay (30–40 min): two 5 or 10 minute rounds of the game are played, with time for debrief and strategy-building between rounds. Mid-point debrief discussion elucidates pitfalls in gameplay and communication, strategies for improved performance, and team dynamics.

5. (5) Final debrief (10 min): the facilitator guides trainees through a discussion of the generalization of their game experience with “real-world” teams.

6. (6) Team science resources (10 min): the discussion ends with additional reading and topical resources available for trainees, which include an NIH-sponsored toolkits and field guides [12, 13], resources to assess team readiness and performance [14, 15] and conflict mitigation information [16].

The freely downloadable software program, facilitator guide, and didactic materials are available through http://www.ctri.ucsd.edu/education/Pages/GameBasedLearning.aspx.

Evaluation, Analysis, and Results

Our initial focus of evaluation is acceptability, and we have implemented the game in different groups of learners as a single session embedded in research training curricula at UC San Diego, including: (1) a career development seminar for MD or PhD junior faculty in clinical research, such as KL2 scholars (n=15), (2) a credit-bearing applied lab-based course in a Master’s in Clinical Research with medical students (n=20), and (3) an interdisciplinary T32 training program for junior faculty members with MDs or PhDs (n=5). Evaluations were gathered voluntarily and anonymously via Qualtrics, eliciting Likert-type satisfaction ratings and open-ended questions soliciting perceived benefits and ideas for improvement and future application; our primary outcome was user satisfaction with the class session. Given the focus on acceptability, analyses were descriptive and no data on current or prior team participation or long-term outcomes were collected. These data were collected anonymously as part of routine evaluations and therefore the study was not subject to human research protections. Data analysis was approved by the UC San Diego HRPP, under Exempt Status.

Responses indicate a high level of satisfaction with the session (mean of 8.8 out 10, SD=1.6), perceived relevance to the understanding the topic of team science, as well as usefulness to research careers (Table 1). Qualitative feedback aligned with the learning objectives in providing an applied experience in coordination and communications. Suggestions for improvement included modifications to game instructions and aspects of the user controls. Learners identified potential extensions of the game in the context of stimulating discussion on roles and communication in actual research teams.

Table 1 Scholar evaluation data (n=19)

Discussion

Clinical and translational research is increasingly a “team sport,” and our online game-based learning tool delivers an applied learning experience tailored to clinical research learners. Extensive discussions with learners, simulation and game experts, and education experts led us to modify our original plan for a simulation in favor of a “gamified” solution. As a result of these modifications, the primary goal of the tool was refocused to collaboration and teamwork. Our game-based tool contains relevant content in clinical trial implementation, but the learning objectives are focused on communication skills and strategies fundamental to team science. Simple game elements such as scores and competition enhance the experience and increase motivation to improve performance.

Our results support feasibility, acceptability, and perceived relevance to team science in small sample of learners. While this is encouraging, quantitative and qualitative data on the impact of the game on skill acquisition and team behavior are needed, as an approach to problem-based learning in clinical and translational research education [3]. The field is still in its infancy, and there is a need to define which learning objectives are best suited for games, establish best practices for evaluating impact on competencies, and optimize methods for enhancing transfer of training to real-world experience. Our implementation of game-based learning integrated live instructor facilitation, and therefore future research could evaluate whether automated approaches to facilitation (such as written reflection) could be employed to increase scalability. The application we described was targeted to early-stage learners in clinical research, and we are identifying how games might foster skills in existing teams, as well as among clinical research coordinators whose learning needs are poorly understood [3]. We are also identifying data that can be extracted from game-based learning sessions, since important aspects of communication can be objectively measured (e.g., team communication patterns). These objective data could form the basis for performance-based assessment and real-time feedback [17]. Finally, our development process highlighted the importance of an interdisciplinary approach to program development, use of game concept documentation, and an iterative development process with frequent end-user engagement.