Introduction
Contemporary science relies on diverse forms of expertise and specialized methodologies to solve complex health challenges.Reference Gray 1 – Reference Falk-Krzesinski 8 Collaboration has become essential, necessitating partnerships among scientists trained in different fields.Reference Wuchty, Jones and Uzzi 6 , Reference Bennett, Gadlin and Levine-Finley 9 – Reference Buscemi, Steglitz and Spring 14 Reports issued over the past 14 years by the National Academies of Sciences (NAS) 13 , 15 – 17 emphasize the need to remove barriers to successful team science (TS), defined as the study and practical application of principles about how to assemble, optimize, and lead groups of researchers who collaborate on a shared endeavor to achieve more than additive results.
NAS called for “interdisciplinary team-based approaches in training, education, and research” and “innovative education and training models and methodologies, which include a focus on team science.” 16 Yet, although a need to improve the effectiveness of science teams is widely recognized, available training opportunities remain scarce. The paucity of resources to train science teams contrasts with the abundance of both in-person and online resources available to train clinical practice teams. 18 – Reference Weaver, Dy and Rosen 20 In contrast, only a few academic institutions offer in-person training workshops or a course in team science to train research teams.Reference Grewal 21 – Reference Read 23 Several federal funding agencies offer occasional creatively facilitated week-long residential IDEAS labs (also called Sandpits or Innovation Labs) that assemble geographically distributed researchers to explore new interdisciplinary science. These closely mentored, in-person training formats carry a high cost and participant burden, however. Digital dissemination of online, self-administered training materials warrants consideration as an alternative or complementary training platform. By eliminating needed costs for travel, coordination, and real-time faculty salaries, digital delivery holds potential to make TS training more broadly accessible.
In 2011 with NIH support, we developed COALESCE (CTSA Online Assistance for Leveraging the Science of Collaborative Effort) as a learning resource for scientists planning to undertake cross-disciplinary, collaborative health science research. Although several printable guidesReference Bennett, Gadlin and Levine-Finley 9 and online repositories of TS resourcesReference Vogel 24 are available, COALESCE remains the only e-learning training resource. A free, open-access online educational tool located at teamscience.net, COALESCE hosts a series of online learning modules shown in Table 1: the four original modules created in 2011, four additional modules added in 2018, and one module currently under development. All use an interactive format and aim to improve users’ knowledge, attitudes, and self-efficacy for team science skills.Reference Cairncross and Mannion 25 These particular learning objectives are based on Bloom’s taxonomy,Reference Bloom 26 which has guided the development of traditional curricula and web-based training in many disciplines,Reference Felder 27 including medicine.Reference Bloom 28 The current evaluation focuses on the first four modules developed for the site: the Science of Team Science (SciTS) module and three scenario-based modules (basic, biomedical, and clinical team science) because these have been accessible on COALESCE long enough to accumulate users for analyses. The SciTS module takes the form of an interactive encyclopedia: its short videos and animations introduce considerations involved in assembling, managing, and evaluating a team. Users move freely throughout the SciTS module, choose questions of interest, and watch experts answer based on the team science evidence base. A more guided structure characterizes the three multimedia scenario-based modules that address the praxis (i.e., practical application) of team science knowledge. Here users assume the role of a senior scientist, an early-stage scientist, or a research development officer, while navigating research challenges in behavioral, basic biomedical, and clinical medical sciences, respectively. They make decisions about choosing and managing a team, while receiving feedback that lets them learn from mistakes in a safe environment.Reference Schank 29
Table 1. COALESCE Module descriptions
The goal of this study was to characterize COALESCE (teamscience.net) usage and users, and to evaluate whether completing the modules improves team science knowledge, attitudes, and self-efficacy about team science skills.
Materials and Methods
Because the study involved research on the effectiveness of an instructional strategy, the Northwestern University Institutional Review Board (IRB) judged the study to be exempt research. User data from October 20, 2011, through December 4, 2017, were analyzed (preceding changes made to the site after this date). Website utilization was tracked via Google Analytics data on page views, acquisition channels, and new versus returning visitors. To access the interactive learning modules, visitors needed to register with a valid e-mail address and answer several questions about gender, discipline, profession, and primary interest in the practice versus the science of TS (SciTS). Registrant characteristics and survey responses of those who took module pre- or post-tests were tracked through a password-protected administrative portal hosted on Northwestern University servers.
Upon entering a COALESCE module, the user was invited to take a voluntary online pretest. The pretest was not re-offered if the user declined the invitation and entered the module (so that pre-assessment responses would not be contaminated by exposure to educational material in the module). After completing the module, the user was invited to take the same assessment as a posttest. Each of the four modules had unique test questions developed by the research team to match its training content, but all modules assessed knowledge about team science, attitudes toward team science, and self-efficacy about team science skills. The knowledge assessment for all modules contained 10 multiple-choice questions, asking about information conveyed in that particular module. Attitudes assessments used three to five questions on a five-point Likert scaleReference Likert 30 on which users expressed the favorability of their opinions about team science. Five-point Likert scales were also used for the skills assessments that included between three and seven questions: users rated their confidence (self-efficacy) to act effectively in team roles associated with that module. Once having begun the pre- or posttest for a module, users could not skip questions; options were to complete or exit the test. After completing the post-assessment for any module, users were invited to take a satisfaction survey consisting of four questions about the anticipated impact of the modules on their future research, favorite and least favorite sections of the tool, and suggested new modules.
Statistical Analysis
An anonymized dataset containing descriptive characteristics and pre- and posttest scores of registered users was downloaded from the COALESCE (teamscience.net) server. Data were analyzed using SPSS software. Demographic characteristics of those included in the analyses because they took paired pre- and posttests, versus those excluded because they took only pre- or posttests, were compared via Chi-Square tests for categorical variables and t-tests for continuous variables. Main study outcomes were team science knowledge, attitudes, and self-efficacy about TS skills for each of the four modules. Data were analyzed using repeated measures ANOVA testing the effect of time (pre- vs. posttest) on the outcomes for the modules. Because the modules’ effects on learning were of primary interest, cases were included in analyses only if they had both pre- and posttest scores for a given module. Interactions between time and disciplinary group (biomedical vs. other field) also were examined to test whether users with a biomedical background learned more than users from non-biomedical fields.
Results
Site Traffic and Navigation
Between October 20, 2011, and December 4, 2017, the COALESCE website was visited by 16,280 new unique users and had 31,294 page views (not including pages viewed within the portal that housed the learning modules) accomplished in 21,716 sessions. A majority of visits were brief: 65.4% of sessions involved visiting a single page and 67.3% involved spending 10 s or less on the site. On the other hand, 6461 sessions (29.8%) involved spending more than 1 min on the site. As is shown in Fig. 1a, the number of new users of the site grew steadily from 2011 to 2015, when recruitment of new first-time users peaked. As Fig. 1a also shows, the number of new users visibly declined between 2015 and 2017, but the number of page views continued to increase, prompting us to upgrade the site in the ways discussed below. As Fig. 1b indicates, a majority of users (50%) accessed COALESCE (teamscience.net) directly by typing the URL into their browser, clicking on a bookmark, or following a link in an e-mail or other electronic documents. Another group of users (29%) was referred via external links from another website, such as sites hosted by the NIH and universities featuring teamscience.net as a resource. The remaining users (20%) accessed the website by way of search engines. Despite some variation over time, direct access remained the most common way for users to access COALESCE (teamscience.net).
Fig. 1. Traffic and navigation through COALESCE. The amount of traffic COALESCE (teamscience.net) received between its launch on October 20, 2011, and December 4, 2017, as evaluated using Google Analytics. A. Visits and Page Views, 2011-2017. New Users are those navigating to the site for the first time in a two-year period via a specific device and specific browser. Page View is a view of a page on the site that is being tracked by the Analytics tracking code. B. Ways Users Accessed COALESCE, by Year (2011-2017). Direct: visitors who reached the site by entering “teamscience.net” into the web address bar of their browser, clicking on a bookmark, or following a link in an email or other electronic document. Referral: visitors who clicked on a link from a referring site to teamscience.net. Search: visitors who reached the website via search engines like Google, Bing, and Yahoo.
Study Sample
Figure 2 shows the flow of participants through the study. Of 16,280 unique visitors to teamscience.net during the study period, 16% (n=2686) chose to register and provide demographic information. A majority of registered users identified as female (66%), reported working in academia or education (72%), and expressed primary interest in the practice (rather than the science) of TS (67%). Approximately half (47%) of registered users identified as being in a biomedical field.
Fig. 2. Participant flow through COALESCE (teamscience.net) evaluation study. Diagram shows the progression of COALESCE (teamscience.net) users from discovery of the site, through registration, through completion of a pre- or posttest, through completion of both pre- and posttest assessments of knowledge, skills, and attitudes for each of four modules.
Among COALESCE registrants, 37% (n=989) voluntarily completed at least one pre- or posttest (Fig. 2). Of those completing a pre- or posttest, 22% (n=219) completed both a pre- and posttest for at least one module, allowing their change in performance to be examined. There were no differences in gender, educational attainment, employment sector, or primary team science interest between participants included in the analyses and those excluded for lacking a paired pre- and posttest. Those included in the analyses were more likely than those excluded to have been trained in a biomedical field versus a different discipline (X 2 (1) = 5.11, p < 0.05). They were also somewhat younger (m = 38.10, SD = 11.47 vs. m = 41.39, SD = 12.70 years, t (899) = 3.30, p < 0.005) and had several fewer years of professional experience (m = 9.04, SD = 7.69, vs. m = 12.14, SD = 10.41 years, t (921) = 3.95, p < 0.001). The number of users completing each COALESCE module is shown in Table 2, together with the average change in team science skills, attitudes, and knowledge that users displayed after completing each module. All four modules increased users’ self-confidence about being able to perform team science skills, and all effect sizes were large. Attitudes toward team science grew more positive after users completed the behavioral, basic biomedical, and clinical modules, showing small to medium effect sizes, but attitudes did not improve significantly after taking the SciTS module. Likewise, knowledge scores improved after completing the behavioral, basic biomedical, and clinical modules, and effect sizes were large, but knowledge did not change significantly after taking the SciTS module (Table 2). As Table 3 indicates, no significant interactions were found between discipline (biomedical vs. other) and change in skills and attitudes for any module. After taking the clinical module, users from a biomedical background appeared to show significantly greater improvement in knowledge than those from other disciplines (F(1, 75) = 7.72, p=0.0069), but this effect failed to meet the criterion for statistical significance (p=0.0042) after Bonferroni correction for multiple comparisons.
Table 2. Change in team science skills, attitudes, and knowledge after completing COALESCE (teamscience.net) modules *
* Data shown for COALESCE users who completed both pre- and posttest for any module. Skills and attitudes questions, scored on a 1–5 Likert scale, are shown as mean (SD). Knowledge scores are shown as mean percentage of correctly answered questions. p-values derived from repeated measures ANOVA show significance of change from pre- to posttest. TScience: team science
Table 3. Comparison of change from pre- to post-assessment score for all modules by users from biomedical versus other disciplinary backgrounds *
* Skills and attitudes rated on 1–5 Likert-type scale and shown as mean (SD). Knowledge scores reflect mean percentage of correctly answered multiple-choice questions. p-values based on repeated measures ANOVA reflect interaction between discipline and pre- to posttest performance change. Team Science: Science of Team Science.
Of those who completed a paired pre- and posttest, 77 also completed a satisfaction survey. Across all modules, a majority of respondents (84%) reported that completing COALESCE training was very likely to impact their future research (Fig. 3). Only a small number of participants (5%) reported that completing the modules was unlikely to impact their future research. Although all of the modules received positive reviews, the SciTS module accumulated the greatest number of suggestions for improvement. Recommendations focused primarily on how to improve the module’s flow and usability.
Fig. 3. Satisfaction Survey Response. After completing each COALESCE module, users were asked, “How likely is this module to impact your future research?” Response options were: very likely, to some extent, and unlikely. The percentage of respondents endorsing each response option for each module is shown earlier. TS, team science.
Discussion
COALESCE (teamscience.net) remains the first and only open-access, online training for team science in the health and medical professions, and has been considered a gold standard team science training resource.Reference Hesse 31 , Reference Aronoff and Bartkowiak 32 However, the tool’s effect on team science learning, attitudes, and skills had not previously been evaluated. From its launch in 2011–2017, COALESCE (teamscience.net) attracted 16,280 unique new visitors who engaged in 6461 sessions lasting more than 1 min, indicating broad dissemination. Users showed significant improvement in knowledge, attitudes, and skills after taking any of the three scenario-based modules in basic biomedical, behavioral, or clinical science, indicating that these modules successfully meet the learning objectives for which they were designed.
Learners’ self-confidence about their ability to perform team science skills also increased after using the SciTS interactive encyclopedia module, but knowledge and attitudes were unchanged. For the most part, biomedical and other health professionals improved comparably after taking the modules. The one exception involving differential knowledge change after taking the clinical module was no longer significant after correction for multiple comparisons, suggesting that the finding may be spurious. Finally, most learners believed that the experience of taking the modules would positively impact their future research. In aggregate, these findings support the premise that an online e-learning tool such as COALESCE (teamscience.net) offers an acceptable, effective, low-cost, readily disseminable resource to improve team science outcomes.
Across the website’s lifespan, most users have accessed COALESCE directly, by typing “teamscience.net” into a browser or following a link in an e-mail. There also has been a gradual increase in the proportion of visits directed to the website through search engines and external referrals, such as the National Cancer Institute’s Team Science Toolkit, NIH Intramural Training and Education, and university CTSA webpages. The site’s sustained usage is likely attributable to continuing public interest in team science, since no marketing campaign was undertaken for COALESCE.
Consistent with the developers’ intended audience, the modal COALESCE user comes from a biomedical discipline and is employed in an academic/educational setting. Despite identifying largely as academic researchers, very few users expressed interest in learning about the scientific evidence base that grounds team science. Instead, most say they came to the site in order to learn the practical aspects of how to do research in scientific teams.
Importantly, taking any of the four modules increased learners’ confidence that they had the skills needed to engage effectively in team science. Moreover, learners with backgrounds in biomedicine or a different discipline showed generally comparable improvements in team science knowledge and attitudes after taking the scenario-based modules. In aggregate, the current findings support the appraisal of COALESCE (teamscience.net) as an effective resource for both biomedical and non-biomedical scientists who seek learning opportunities in cross-disciplinary, collaborative research.
SciTS was the only COALESCE (teamscience.net) module not to improve team science knowledge or attitudes; it also received the most suggestions for improvement. Designed differently than the other more directed, scenario-based modules, SciTS is an interactive encyclopedia (i.e., a buffet-style repository of information wherein users select questions that interest them from banks of videos, animations, and activities). User feedback suggests that some felt overwhelmed by the number of choices available to them and the lack of a clear structure guiding progression through the materials. Therefore, in 2018, this module was modified by adding a table of contents and a tracking system to help users navigate through the information. No structural changes were made to any of the three scenario-based modules because knowledge, attitudes, and skills improved after taking any of them.
Nevertheless, several factors prompted an upgrade to the teamscience.net site in 2018, in addition to the improvements to the SciTS module described earlier. One factor involved changes in site traffic and navigation patterns that we observed between 2015 and 2017. We attributed the decline in new users at least partially to the increasing use of mobile technologies to access the Internet. Because COALESCE (teamscience.net) had been programmed in Flash, rather than HTML, mobile device users could not reliably access its video content. Meanwhile, the increased number of page views during the same period suggested a need for greater server capacity. Other technical upgrades performed in 2018 met user requests for disability access, enhanced privacy, improved user experience, and Shareable Content Object Reference Model (SCORM) compliance – a set of standards for e-learning technology.
Also, given growing concern with ensuring the real-world translation and uptake of research findings 33 and specific recommendations from the CTSA Team Science Domain Task Force (DTF) (predecessor to the current CTSA Collaboration and Engagement DTF), the scientific scope of the COALESCE (teamscience.net) tool was extended to highlight the application of team science principles to community-engaged research. As advised by the DTF, four modules created with public health experts and community advocates for the NIH-sponsored evidence-based behavioral practice project (www.ebbp.org) were added to COALESCE (teamscience.net).Reference Spring 34 – Reference Spring 36 The added modules address shared clinical decision making, collaborative community decision making, stakeholder perspectives, and implementation science. Neither these four modules nor the technical changes made in 2018 are included in the present evaluation because they are too new to have accumulated sufficient data for analysis. Finally, a new module addressing how academic investigators can identify and engage community stakeholders in research collaborations is also now under development for addition to the COALESCE (teamscience.net) suite of modules. This module, funded by NCATS, has been requested repeatedly since the First Annual International SciTS Conference in 2010, when COALESCE was introduced. Development of the new module and addition of modules on implementation science and community collaboration also is consistent with the expanded scope of CTSA DTF team science efforts to engage community as well as academic partners.
Limitations
There are limitations to the present study that stem primarily from sample selection and testing. The all-volunteer sample was comprised of those who found the site of their own accord, not in response to active recruitment or incentivization. Hence, they are likely to have been more favorably disposed, or at least more curious about team science, than the average member of the population. Also in terms of sampling, only 37% of those who registered on the teamscience.net website completed a module pre- or posttest, and only 22% of these completed both the pre- and posttest for any module, albeit demographic differences between these two subgroups were modest. Hence, the present results are drawn from a relatively small, highly motivated, selected sample and may not generalize to the average person. On the other hand, whereas the threat to validity from selective sampling would be expected to bias test performance in a positive direction, the threat from repeated testing would be expected to work in the opposite direction. That is, a highly selected and motivated sample would be expected to achieve high pretest scores, introducing ceiling effects that made posttest improvement difficult to detect. Nevertheless, improvements were observed on all measured team science outcomes after taking any scenario-based module. Another limitation is that although most learners reported that the experience of completing the modules would influence their future research, it was not possible to assess long-term objective effects on respondents’ performance in interdisciplinary teams, a limitation that partly reflects lack of consensus about optimal performance metrics.Reference Falk-Krzesinski 37
Despite these limitations, there remains a clear and sustained need for team science training resources in the health and medical fields. Given the shortage of in-person training options, combined with the expense and logistical challenges involved in creating them, COALESCE (teamscience.net) provides a much-needed, accessible e-learning opportunity to introduce team science training to interested researchers. Given evidence that taking any of the scenario-based modules enhances team science knowledge, attitudes, and skills, our recommendation is for learners to begin using COALESCE (teamscience.net) by taking any of the clinical, behavioral, or basic biomedical research modules. Because a change in knowledge was not observed after taking the SciTS module and because many users found the experience overwhelming, rather than attempting to navigate the entire module, we advise using it as an interactive encyclopedia to be consulted when questions arise about the evidence base for team science.
Conclusions
COALESCE (teamscience.net) remains the first and only open-access, online training in TS for health researchers. Initial evaluation of the training modules indicates widespread dissemination and positive outcomes. The COALESCE modules were updated in late 2017-early 2018 based on these results, user feedback, and CTSA DTF recommendations. COALESCE is well positioned to play a valuable role in the future of team science training. Updates to its accessibility, content, and evaluation systems can be expected to further bolster its utility in supporting the practice and science of team science across the translational research spectrum.
Acknowledgements
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Research reported in this publication was supported by the National Institutes of Health’s National Center for Advancing Translational Sciences, Grant Numbers 3UL1RR025741 and UL1TR001422 and Office of Behavioral and Social Sciences Research, Contract N01-LM-6-3512, Online Tool to Support CTSA Research Workforce Development in Behavioral Team Science. No financial disclosures were reported by the authors of this paper.
Financial Disclosure Information
Bonnie Spring has no financial disclosures. Ekaterina A. Klyachko has no financial disclosures. Phillip W. Rak has no financial disclosures. H. Gene McFadden has no financial disclosures. Donald Hedeker has no financial disclosures. Juned Siddique has no financial disclosures. Leland R. Bardsley has no financial disclosures. Angela Fidler Pfammatter has no financial disclosures.
Open access
