Introduction/Background
Translational science (TS) describes the spectrum of research activity from preclinical investigations that elucidate the biological basis of health and disease to clinical and policy interventions that improve the health of individuals and populations. Training programs in TS aim to equip individuals with the knowledge, skills, and abilities (KSAs) that are needed to move discoveries along this translational spectrum. Concurrently, TS training also prepares individuals to glean insights from the bedside and community to inform discovery or basic science within their major research focus. In 2011, the Clinical and Translational Science Awards (CTSA) Consortium provided direction for TS training programs by releasing the CTSA Core Competencies, a set of 97 KSAs grouped into 14 thematic areas (e.g., Study Design, Research Implementation). The CTSA Core Competencies reflected expert opinion on the foundational KSAs for master’s degree candidates in clinical and translational research [1]. In practice, however, it is not efficient or necessary for each TS investigator to have in-depth knowledge and applied expertise in all 97 KSAs, nor is it feasible to measure competence in all the KSAs. Rather, individual investigators likely need different levels of mastery depending on where their work resides within the translational spectrum and the composition of their research team.
The TS landscape has evolved since the release of the CTSA Core Competencies. There is increasing recognition that complex TS is most often carried out by teams of scientists, clinicians, research staff, participants, and other stakeholders, each possessing complementary insights and expertise. Training for these multiple roles in “translational team science” poses new challenges for educators and mentors to identify the specific KSAs and level of mastery required for different team members. There has also been a call to streamline the original 97 KSAs to reduce overlap, improve their applicability, and to provide a framework of core knowledge needed to support the career development of the next generation of TS investigators. Educators now recognize that the background, experiences, and research interests of individuals seeking training in TS are best served by “customized curricula” [Reference Rubio2]. This concept is in accord with strong encouragement by the NIH to guide graduate and postdoctoral training. Through the use of tools such as individual development plans (IDPs) [3], learners are encouraged to identify their knowledge, skill, or experience gaps, and their personal and professional goals, and then to align their planned training accordingly, rather than adhering to rigidly structured or mandated curricula.
To support more individualized education and training while ensuring some consistency of attainment of KSAs, we have adopted a conceptual model of training described as Personalized Pathways. Applying this model, we have reassessed the individual KSAs and recommended levels of mastery required depending on the desired career role or TS “phenotype.” Here, we describe our first steps toward developing a framework to elucidate how the original 97 KSAs might be tailored to the individual career development goals of TS trainees to optimize efficiency and efficacy of TS training.
Materials and Methods
Study Design and Sample
We conducted a cross-sectional, web-based survey of the CTSA educational community who are involved in education and training activities. The survey was distributed to members of the National Center for Advancing Translational Sciences CTSA Workforce Development and Methods/Processes Domain Task Forces (DTFs). The Workforce Development DTF focuses on programs that prepare the TS workforce to advance TS discoveries, while the Methods/Processes DTF addresses the process of conducting TS, including optimal ways to prepare team members. We used a snowball sampling technique by asking members of these DTFs to forward the survey link to others at their institution who counsel individuals on research career development across the different career phenotypes. Respondents could complete multiple surveys if they wished to respond for more than one career phenotype.
Study data were collected and managed using REDCap electronic data capture tools hosted at the University of North Carolina at Chapel Hill [Reference Harris4, Reference Harris5]. The survey was launched in January, 2017 and was open until September, 2017. The survey was reviewed by the Biomedical Institutional Review Board of the University of North Carolina at Chapel Hill and deemed Not Human Subjects Research, exempt from further review (UNC IRB#17-0013).
Survey Development
Definition of TS phenotypes
The goal of this survey was to provide insight on how educators prioritize the 97 KSAs when advising individual trainees on the level of mastery needed to conduct research in different parts of the TS spectrum. Thus, it was necessary to define a priori TS career pathways, or phenotypes, that represent the broad range of TS careers. After deliberation, and based upon roles referenced in prior CTSA competency publications and on expert opinions of educators who counsel early-stage researchers, six phenotypes were identified: Preclinical Bench, Clinical Research, Community-Engaged, and Public Health/Big Data Researcher, Dissemination/Implementation, and Data Sciences/Analytics (Table 1). Each phenotype with corresponding descriptions of career goals was felt to be distinct with characteristics that supported adequate discrimination from other phenotypes. In addition, the six phenotypes covered the breadth of TS career pathways.
Table 1. Clinical/translational research phenotypes, or researcher types, defined by career goal
This paper reports on the three career phenotypes that represent the largest number of survey responses: Preclinical Bench, Clinical, and Community-Engaged Researcher.
Identification of mastery levels
The initial list of mastery levels considered by the Harmonizing Competencies workgroup was adapted from the “Learning Vector” model by Stritter [Reference Stritter6]. The Stritter model identifies three stages of learning (Exposure, Acquisition, and Integration) that describe the progression of a trainee from dependence on educators (or mentors) to set training goals and to model professional standards, to independence in defining career and training goals. The workgroup made two changes to the original definitions in the Stritter model to fit the context of TS. The definitions were modified to further delineate the role of mentors and also to clarify that an individual’s desired role within a research team (e.g., principal investigator vs. team member) determines their TS learning goals.
The three revised stages of learning and their definitions were subsequently discussed and iteratively refined. First, “stages of learning” was changed to “mastery levels,” which was perceived to better describe the goal of TS training. Second, Stritter’s stage of “Acquisition” was changed to “Application” to more directly convey that trainees in this stage are applying learned concepts to their specific research projects. Third, a category of Not Applicable (N/A) was included to capture the view that a KSA is not required for a specific phenotype (Table 2).
Table 2. Final mastery levels
Survey structure
After achieving consensus on the phenotypes and levels of mastery, a 100-item survey instrument was developed. The first three questions asked training director/expert educator respondents to identify their research training role, the specific TS career phenotype they selected to address in the survey, and where they believed this phenotype fell within the translational spectrum (T1–T4). The remainder of the survey was presented as a matrix listing the 97 original KSAs, grouped by their pre-established thematic areas (e.g., Study Design, Research Implementation) in columns with the response options for the level of mastery (Exposure, Application, Integration, Not Applicable) across in rows. Respondents were requested to answer how they would advise a trainee regarding the level of mastery in each of the original competencies based on the trainee’s research career phenotype. Respondents were also given the opportunity to provide free text comments (Supplementary material A).
Data analysis
Frequency tabulations and weighting for all responses were done using Excel 2016. Krippendorff’s alpha scores were then calculated in SPSS using the KALPHA SPSS Hayes macro to evaluate which TS career phenotypes were most similar in rankings [Reference Hayes and Krippendorff7, Reference Krippendorff8].
Ranking KSAs
As described above, for each of the 97 KSAs, respondents indicated whether trainees should (1) be introduced to the competency and meaningful/relevant vocabulary (Exposure), (2) be able to use the competency to initiate and implement a specific research endeavor under the supervision of a mentor or collaborator (Application), (3) have in-depth training in the competency with a goal for the learner to become independent (Integration), or (4) not receive training in the competency because it is not needed for the phenotype (Not Applicable). Through group consensus, we considered KSAs in the Integration category to be the most important, followed by those in the Application category. As such, for each KSA, we assigned a weight of 1 full point to Integration, 0.5 points to Application, 0.25 points to Exposure, and 0 points to Not Applicable. We then multiplied the proportion of responses in each category by the category weight and summed the results across all categories. For example, as it applies to the Clinical Researcher phenotype, if 80% rated the KSA “prepare an application for the IRB” at the Integration level, 10% at the Application level, 10% at the Exposure level, and 0% as Not Applicable, the weighted score for that KSA was calculated as (80 × 1.0) + (10 × 0.5) + (10 × 0.25) + (0 × 0.0) = 87.5. Then, for each phenotype, we ranked the KSAs by the weighted scores and divided the KSAs into tertile groups. The KSAs in the top tertile were considered to be the highest priority for that phenotype (Supplementary material B).
Results
Survey Responses
A total of 90 surveys were initiated; 85 (94%) surveys were completed and 5 (6%) surveys were incomplete because they did not include a phenotype or have complete responses for at least one thematic area (Fig. 1). For the completed surveys, respondents included KL2 Program Directors (42%), other individuals involved in training (29%), training program directors (20%), and CTSA principal investigators (5%). Table 3 provides the distribution of training roles across the phenotypes.
Fig. 1. Survey distribution and completion.
Table 3. Training role of survey respondents by phenotype
* Column totals may not sum to 100 due to rounding.
† Roles defined by respondents include Workforce Specialist, Director of Education, Mentor, Assoc. Director of KL2, Assoc. Director of Curriculum, Prof. of Biomedical Informatics, Masters Program Instructor.
For the analysis, we focused on the three career phenotypes with the largest number of responses – Preclinical Bench, Clinical, and Community-Engaged.
Comparison of Mastery Level Rankings Across Phenotypes
There were 10 KSAs that ranked in the top tertile across the 3 phenotypes. These KSAs addressed research methodology, the responsible conduct of research, building a team, and communication of research findings. Fig. 2 provides the specific KSAs within each category. For the middle tertile, nine KSAs were common across the three phenotypes reflecting KSAs in statistics, informatics techniques, regulatory precepts, and ensuring quality assurance and control procedures (Fig. 3). A complete list of the rankings for all the 97 KSAs by phenotype is found in the Supplementary material.
Fig. 2. Top tertile knowledge, skills, and abilities (KSAs) for Preclinical, Clinical, and Community-Engaged Researcher phenotypes.
Fig. 3. Middle tertile knowledge, skills, and abilities (KSAs) for Preclinical, Clinical, and Community-Engaged Researcher phenotypes.
The Krippendorff alpha score comparing the rankings for the Clinical and Preclinical Bench phenotypes was 0.64 (95% confidence interval 0.26–0.58). For the Clinical versus Community-Engaged phenotypes, it was 0.51 (95% confidence interval 0.24–0.75), and for the Preclinical Bench versus Community-Engaged phenotypes, it was 0.14 (95% confidence interval 0.00–0.47). These scores suggest that there is moderate agreement between the Preclinical Bench and Clinical phenotype KSAs and lesser agreement between Preclinical and Community-Engaged phenotype KSAs.
There were four KSAs whose weighted scores placed them in different tertiles for each phenotype: describe relevance of cultural and population diversity in clinical research design; recognize demographic, geographic, and ethnographic features within communities and populations when designing a clinical study; advocate for multiple points of view; and critique studies for evidence of health disparities (Table 4).
Table 4. Knowledge, skills, and abilities (KSAs) with different tertile ranks between phenotypes
Ranking of Competencies Within Phenotypes
Preclinical Bench Researcher phenotype
In addition to the KSAs that ranked in the top tertile for all phenotypes, other KSAs in the top tertile for preclinical researchers were related to the design and conduct of research and KSAs in team management. Specific examples are formulating research questions, evaluating data sources and data quality to answer study questions, fostering innovation and creativity, building a team, and managing conflict. Other first tertile KSAs reflected skills associated with responsible conduct of research (e.g., mentoring, reporting, and investigating research misconduct, role of peer review in funding, animal safety in research, and conflict of interest).
The second tertile KSAs represented a range of thematic areas, from understanding and applying statistical concepts, instructing beginning scientists, and additional KSAs in team management. KSAs that ranked in the lowest tertile of importance for preclinical investigators were those on community-engaged research, technology commercialization, policy-making (e.g., meta-analyses, guideline development), and use of the electronic health record.
Clinical Researcher phenotype
For clinical researchers, additional KSAs in the top tertile addressed KSAs that were similar in theme to the other phenotypes but the specific KSAs were different. The themes represented were the design of research and assessing validity, the responsible conduct of research, and study management. The middle tertile KSAs unique to clinical researchers represented a variety of thematic areas, including working with human subjects, mentoring, statistical concepts, team management, and collaboration. Lowest tertile KSAs were those related to adult learning strategies/principles, commercialization, and trends in technology and informatics.
Community-Engaged Researcher phenotype
KSAs in the top tertile for community-engaged researchers were more varied in their thematic areas than those for the other two phenotypes. While top tertile KSAs also included a focus on research methods/design and responsible conduct of research, this tertile also included KSAs in identifying health disparities, team management, recognizing the features that define communities and populations, proposal development, and principles of community-engaged research. Middle-level KSAs included statistical concepts, quality assurance and control, and additional KSAs in team management. Lowest tertile KSAs for community-engaged researchers included identifying observations to form the basis of clinical trials, ethical care and use of animals, commercialization, and KSAs related to adult learning strategies.
Discussion
The purpose of this study was to identify how educators prioritize the CTSA Core Competencies when counseling trainees in specific TS career pathways. The results indicated that the KSAs in the top tertile in all three phenotypes were those related to research methodology, the responsible conduct of research, team building, and communication of research findings. There was also agreement on a set of KSAs in the next tertile for all three phenotypes. These were primarily focused on biostatistics, however, also included regulatory precepts, quality control and assurance, and translating the implications of research findings. Individual KSAs related to team science were ranked highly across the top and middle tertiles for all phenotypes. When there was variation among the rankings across phenotypes, the KSAs either reflected topics directly related to the methods or populations specific to that phenotype of translational researcher (e.g., human subjects topics for Clinical Researchers) or were instances where the weighted score of the KSA may have been similar across phenotypes, but the actual score placed it in different tertiles based on the tertile cutoff.
The high rankings of KSAs related to team science suggest that TS is envisioned as a true team endeavor in which team members bring different levels of mastery and expertise to the team. As a result, a high priority for training is teaching individuals to collaborate effectively with team members from different disciplines. This may also be the reason why certain KSAs were ranked in the middle tertile of importance, because individual team members or collaborators other than the principal investigator may contribute more heavily in these content areas but investigators needed to have more in-depth understanding of the KSAs in order to effectively engage and communicate with team members and outside experts.
Individuals were offered an opportunity to provide additional narrative comments. Twenty-seven comments were provided. Of these, 12 referenced difficulty answering the question about the translational category. Seven comments were about the CTSA competencies themselves, questioning the meaning or wording of specific competencies, stating that competencies were overlapping, too numerous, inappropriate for master’s-level training, or that they were incomplete for a specific phenotype. Five comments mentioned team science; specifically (1) three comments noted that team members were needed to “fill the holes” in expertise so that the team as a whole should have Integration-level knowledge, (2) the survey was difficult to answer from the perspective of a trainee who is in a support role within a team, and (3) that an option should have been presented to respond to the survey from the perspective of a community partner. Three comments were made about the survey itself, stating that the survey was “somewhat fatiguing,” “restrictive,” and the phenotypes were too broad.
This study has several limitations. The survey was distributed to only two groups with more generalized expertise focused on education and training rather than querying expert practitioners in each of the phenotypes (e.g., informatics or biostatistics). We explicitly chose to do this to limit bias from a disciplinary perspective. Second, we elected to focus on educators and mentors rather than bringing in the “learner perspective” through querying the graduates of CTSA-funded career development programs who are currently in TS careers. These graduates may provide insight into their own training and what levels of mastery are required in their subsequent research. Third, we do not know the characteristics of individuals who chose not to respond, and this may have introduced information bias if these individuals have different opinions on the mastery levels, or if they use an alternate set of KSAs to guide training. The length and format of the survey posed difficulty for some respondents as noted by the text comments. Finally, the survey was not linked to specific individuals. Thus, we cannot report if any individuals completed the survey twice (e.g., for different phenotypes), in which case the responses would not be independent.
Despite these limitations, we believe this study provides an initial framework for further work on how these KSAs can be used to inform personalized training pathways for three TS phenotypes, independent of any specific graduate degree program and at any stage of learning. This process has essentially reduced the number of priority KSAs for each phenotype, with options to add certain KSAs from the other tertiles as needed. For trainees and mentors, the framework could guide prioritization of specific content areas for training. In turn, this could inform the IDP by clarifying the goals of specific activities. For individuals who are early in their training and yet to choose a career phenotype, focusing group training activities on the subset of competencies important to all phenotypes may be particularly useful. The high priority of research methodology skills and an understanding of responsible conduct of research across all phenotypes emphasizes that these areas are an essential foundation for future research. For individuals who are more advanced in their careers, the framework might guide additional training to support work in new domains across the translational spectrum, to identify team members with complementary expertise, and/or to enhance TS collaborations.
Future work should build on this framework by obtaining sufficient numbers of responses for all six phenotypes to better define their training priorities, the commonalities to other phenotypes, and the distinctive training needs. An expanded sampling frame, to include the perspectives of expert investigators and successful mentors within each phenotype as well as of past trainees and other stakeholders and translational team members, could better inform the relationship between training and actual mastery in practice. Likewise, educators might also apply this framework to the broader workforce (e.g., team members in academia and the community, administrators, or program managers) and explore specific ways to best use these priorities in guiding the process of IDP creation. Ultimately, additional work is needed to determine if this approach to training is more efficient and achieves better outcomes of sustainable research careers.
Acknowledgments
The authors wish to thank Karen Grabowski at the University of Rochester Center for Leading Innovation and Collaboration for administrative support during preparation of the manuscript. We also want to thank Clarence Potter at the NC TraCS Institute for his help with REDCap programming. This work was supported in part by the National Center for Advancing Translational Sciences (NCATS), National Institutes of Health, through Grant Award Numbers: UL1TR001111 (Pusek), UL1TR002537 (Knudson), KL2TR002646 (Tsevat), KL2TR001854 (Patino), UL1TR001417 (Chaplin), KL2TR002492 (Ingbar), KL2TR001432 (Umans), UL1TR002733 (Jackson), and the University of Rochester Center for Leading Innovation and Collaboration (CLIC), under Grant U24TR002260. CLIC is the coordinating center for the Clinical and Translational Science Awards (CTSA) Program. This work is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Disclosures
The authors have no conflicts of interest to disclose.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/cts.2019.445
Open access
