Training in patient-centered outcomes research for specific researcher communities

Abstract

A number of publications have discussed approaches to training the scientific workforce in comparative effectiveness research (CER) and patient-centered outcomes research (PCOR). To meet this need, funders have offered resources for developing educational materials and establishing training programs. To extend these efforts into specific researcher communities, the Agency for Healthcare Research and Quality developed an R25 Funding Opportunity Announcement that called for basic, advanced, and experiential training for a specific researcher community in collaboration with associated program partners. This paper describes the strategies developed by the 5 subsequently funded programs, their specific researcher communities and program partners, and the challenges associated with developing in-person and online programs. We focus on lessons learned that can be translated into developing training programs nationwide and on training for the special populations of interest. We also discuss the creation of a sustainable network for training and the conduct of comparative effectiveness research/patient-centered outcomes research in targeted communities.

Introduction

Motivated by the need to improve healthcare efficiencies and individual treatment recommendations, a number of organizations and researchers have stressed the need for a national research agenda in comparative effectiveness research (CER) [1–3]. More specifically, the Institutes of Medicine characterized CER as research that informs clinical decisions between at least 2 treatment strategies at the population and subgroup levels in terms of benefits and harms important to patients that use appropriate methods and data sources in real-world settings [4]. The push for CER was supported by a number of funding efforts, including over $1B from the American Recovery and Reinvestment Act.

Subsequent establishment within the Patient Protection and Affordable Care Act of 2010 of the Patient-Centered Outcomes Research Institute (PCORI) further emphasized the significance of CER with a more specific emphasis on the individual patient, and the outcomes that matter to that patient [5]. More specifically, PCOR was defined as research that informs an individual’s prognosis and treatment options [6]. PCORI funds research studies on treatment options, disparities, healthcare systems, gaps in communicating and disseminating knowledge, large pragmatic studies and methodology for PCOR [7].

The successful conduct of CER and PCOR requires a critical mass of investigators trained in the necessary methodologies. Providing that training in sufficient volume and quality has been recognized as a major barrier to achieving such goals [8,9]. The challenges associated with workforce training are exacerbated by the complexity of relevant methods and the need for expertise across a wide range of methodologies, including stakeholder engagement, research synthesis, clinical decision-making, observational methods, causal inference, pragmatic trials and training in epidemiology, biostatistics and health services research. The literature on these methods is complex and continuously evolving and the number of sufficiently qualified investigators has been lacking [10]. To address these needs, a number of related funding opportunities have been released through mechanisms including Center of Excellence [11] and K12 training grants [12].

In July of 2013, the Agency for Healthcare Research and Quality (AHRQ) released a funding opportunity announcement for an R25 Program on “Researcher Training and Workforce Development in Methods and Standards for Conducting Patient-Centered Outcomes Research Studies” [13]. The funding opportunity announcement required development of an innovative multi-component education program to build capacity specific to “the educational needs of a respective professional field, employment setting, and/or researcher community,” where the definition of what constitutes a researcher community was left to the applicant’s discretion. The applicant was required to work with program partners and develop a program with basic, advanced, and experiential training.

This manuscript describes approaches and associated lessons learned in the planning and initial development stages after 3 years of funding for the 5 funded R25 programs. These include strategies for collaborating with the targeted researcher communities and program partners, developing (mostly online) approaches for basic and advanced training, and overall approaches to experiential training. We also focus on special populations of interest associated with the selected researcher communities, including under-represented minority groups and lay audiences, as they pose particular challenges in developing such expertise. Experiences from the development phase of these programs can help guide future online education and workforce development in PCOR and other areas of clinical and translational science.

Materials and Methods

Researcher Communities and Main Program Components

The 5 funded R25 programs exhibit diverse goals, approaches, and researcher communities. Table 1 describes the audiences and approaches for basic, experiential, and advanced training.

Table 1 Audiences and approaches for comparative effectiveness research (CER) and patient-centered outcomes research (PCOR) training in R25s grants

Brown, Brown University; Einstein, Albert Einstein College of Medicine; U Pitt, University of Pittsburgh; U Texas MDACC, University of Texas MD Anderson Cancer Center; U Wash., University of Washington.

The “Patient-Centered Outcomes Research Innovations at Montefiore and Einstein (PCOR PRIME)” at Albert Einstein College of Medicine partnered with Montefiore Medical Center (a community-oriented academic medical center with a focus on care management, including a Pioneer Accountable Care Organization) to train clinical researchers in the medically underserved community of the Bronx. The program focuses on fundamental methods via a certificate program, an advanced “learning collaborative,” where faculty and students interactively discuss advanced methods, and a fellowship at Montefiore’s care management company, Montefiore Care Management Organization (CMO), where PCOR Certificate graduates gain hands-on experience.

The Evidence Synthesis Academy at Brown University trains researchers in systematic reviews (SR) and related biostatistical, computational, and information technology. It partners with researchers, governmental and non-governmental healthcare organizations, consumers and other non-traditional groups, including payers, professional societies, patients and advocacy groups, industry, librarians, and journalists. The program offers a combination of didactic, asynchronous, peer, and experiential methods, and seeks to develop freely available content, a peer learning network, and a customized fellowship program.

The “Expanding National Capacity in PCOR through Training” (ENACT) Program at the University of Pittsburgh trains investigators from 6 institutions within the NIH-funded network of Research Centers in Minority Institutions, including Charles R. Drew University, Howard University, Meharry Medical College, Morehouse School of Medicine, University of Hawai’i, and University of Puerto Rico Medical Sciences Campus. Two online courses offer training in writing (1) a PCOR concept proposal and (2) a full project proposal. A cohort of 5 Fellows each year worked on a project (presentation, manuscript, or grant proposal) and spent 1 month in Pittsburgh for PCOR training, networking, and critical feedback on their projects.

The “Methods Training in Patient-Centered Cancer Outcomes Research” Program at the University of Texas MD Anderson Cancer Center offers comprehensive training for cancer researchers through the CERTaIN (Comparative Effectiveness Research Training and Instruction) program in partnership with the American Society of Clinical Oncology (ASCO) and the University of Texas Health Science Center School of Public Health. Its overall goal is to increase the relevant workforce through the development of methodology cores in knowledge synthesis and translation, observational research, pragmatic trials, and evaluation of healthcare delivery. The program components include on-demand online lectures with interactive components, and advanced training through ASCO webinars and in-person workshops. In the experiential component, fellows and junior investigators specialize in one core area with a personalized mentoring plan.

The “Patient-Centered Outcomes Research Partnership” (PCORP) Program at the University of Washington represents a stakeholder-driven educational and experiential training program to train scientists and clinicians. The program focuses, in part, on community-based healthcare for the underserved populations of American Indians and Alaska Natives. Other key populations include clinicians and community practitioners in the WWAMI (Washington, Wyoming, Alaska, Montana, and Idaho region), quality improvement experts, and research managers at a large Puget-Sound-based healthcare system. Trainees receive mentored experiential training by experts at the University of Washington and partners at the Southcentral Foundation of Alaska, Partnerships for Native Health (at Washington State University), Swedish Medical Center, the University of Hawai’i, Sanford Health (a large rural focused healthcare network based in South Dakota), the Washington, Wyoming, Alaska, Montana, and Idaho Practice and Research Network, and the Northwest Participant and Clinical Interactions Network, which connects diverse populations to local, high-quality clinical research. The program components include a tailored CER/PCOR curriculum and in-person training, and online training modules. The experiential component seeks to train 24 participants from our partner institutions and organizations. More specifically, it will help develop pilot studies, mentoring, and an advanced peer-training network. Innovative approaches are also being implemented for program evaluation.

Results

Goals and Structure of the Basic and Advanced Courses

This section describes the current state of the basic and advanced training programs (3 y into a 5-y grant), and the associated strategies for developing the courses. Although each of the courses is evolving over time, the subsequent discussion provides useful strategies for developing ways to approach PCOR training in a specific researcher community.

Albert Einstein College of Medicine

To build upon existing training in traditional clinical research methods, Einstein initially recruited trainees for their “basic” certificate from graduates of its Masters program in clinical research, or similar programs within the Clinical and Translational Science Award Programs (CTSAs) or schools of public health. The survey of potential learners showed a high level of interest for an asynchronous online format. Online learning was new for faculty and students, and despite creation (with expert external consultations) of a high-quality online program, learners were unable to meet the program’s demands while attending to their other professional responsibilities, and thus the majority discontinued the program before completion. The program is currently revising the format to be a PCOR track within the Masters program. Einstein’s “advanced” component is a series of faculty seminars on emerging PCOR design and analytic methods. Cosponsored by the Center for Comparative Effectiveness Research and the Center for Quantitative Sciences, it is open to faculty from across these institutions as well as other New York City institutions (largely through the CTSA program).

Brown University

The Evidence Synthesis Academy is developing flexible educational resources to accommodate professionals in the workforce, balancing rigorous methodological standards with accessibility. The rapid expansion of information technology (across trials, and use of registries, electronic health records, and other observational studies) requires specialized expertise in extracting the useful information from this mass of data. Thus, core curriculum provides basic and advanced training in primary data studies and SR of different types of primary evidence. The Academy aims to train producers and consumers of SR for efficacy, safety, and diagnostic testing, using a variety of delivery modalities (ie, short courses, webinars, and online formats) in both traditional and flipped-classroom [14] environments. Users may access online materials independently or in partnership with instruction from Academy faculty. To date, the Academy has offered 6 synchronous courses with more than 400 participants, resulting in over 100 Learning Modules, Tutorials, Case Studies, and Learning Activities of various lengths and topics. The Academy also offers journal clubs and discussion forums on advanced methods and applications to provide fellows, users and experts with opportunities for interaction to move the field forward. The existing curriculum will be supplemented with additional courses and modules, whose content will be determined by the needs of its learning community and partner organizations.

University of Pittsburgh

The ENACT Program has developed and offered 2 separate 4-month courses. The platform for these courses originally used the online Acatar Learning Environment, but now uses Google Docs and Google Drive to facilitate easily sharing materials. The fundamental course was conceived as instruction on the “vocabulary of PCOR” and its associated methods, with 10–12 week-long modules on key concepts, designs, and methods. The course has subsequently evolved, based on the input of the students and program partners, to focus more directly on writing a PCOR concept proposal (in the first course) and a full funding proposal, with a more in-depth description of the study designs and methods (in the second course). Both courses use a “flipped classroom” approach, where the “lecture” is conducted by viewing assigned videos from the PCORI Standards Academic Curriculum (produced by Johns Hopkins University), the University of California at Davis (UC-Davis), and the Ohio State University. Weekly live sessions provide further didactic training and the opportunity for questions and discussion.

The University of Texas MD Anderson Cancer Center

The basic learning component aims to develop and implement online didactic courses and peer-to-peer learning forums, including modules in Knowledge Synthesis, Patient-Centered Outcomes Research, Observational Studies and Registries, and Pragmatic Clinical Trials and Healthcare Delivery Evaluations. Each module has several on-demand lectures that can be completed independently. CERTaIN also offers 1-hour Webinars in conjunction with ASCO on methodological topics (eg, qualitative research). The advanced learning components offer in-person hands-on workshops, some of which are open to researchers in all fields (not just cancer) to cover primarily methodological training. Examples include a hands-on workshop (in a computer lab) on conducting a SR, which was attended by researchers from across Texas. The advanced learning program targets participants with a commitment to complete a study in the immediate future. Participants may then choose to engage in the experiential training program.

University of Washington

The University of Washington (UW) program has many similarities to the University of Pittsburgh program. It enrolls a cohort of ~8 scholars (the first 3 y included 9, 10, and 10 scholars)—2 from each of its partner organizations (initially 4 and then 5 organizations by the second cohort). The scholars apply with the basic research idea in mind in the winter of the year before their 2-year program begins. The program begins with 4 modules of online training, which consists of readings about basic health services research, evidentiary methods, CER and PCOR basics, and patient-reported outcomes. Each module has a moderated discussion board to develop a baseline of knowledge and some connection with students. Scholars then participate in a week-long institute that provides both didactic and workshop experiences utilizing peer and faculty review of scholar proposals several times during the week and a presentation at the end of the week. This serves as the foundation for scholar proposals going forward. The summer institute provides didactic sessions on study design, evidence synthesis, stakeholder engagement, economic analysis, patient-centered and patient-reported outcomes. It also has panel sessions with colleagues from the area in relevant fields (eg, quality improvement data for research), and a panel of patient and clinician stakeholders. Scholar assignments for the next 2 years include additional online modules tailored to their needs, such as CER/PCOR among vulnerable populations. The advanced program targets faculty and postdoctoral fellows in the greater Seattle area, which has a large pool of interested methodoogists. For the first year, a quarter-long series of seminars was conducted jointly with the Program in Health Economics and Outcomes Methodology focused on observational and experimental approaches to CER studies. Sessions were recorded and have been converted into a plan for a full day summer workshop on advanced statistical methods in observational data research.

Goals and Structure of the Experiential Training

The following text describes the current state of the PCOR experiential training programs 3 years into the 5-year grant. These approaches are also evolving over time, but may again provide useful approaches for strategies to use in experiential training of a specific researcher community.

Albert Einstein College of Medicine

Einstein’s “experiential” component places faculty within the CMO, a population-oriented care delivery system that houses Montefiore’s Pioneer Accountable Care Organization. Faculty must have sufficient funding to spend 2 days per week for at least 3 months working onsite at the CMO, under the tutelage of the CMO’s Director of Research and Evaluation, along with support from the program’s faculty and the CMO data analysts. Example projects, which focus on the Fellows’ area of interest, have included models to predict and enhance medication adherence among patients with advanced congestive heart failure (from a cardiologist funded by the New York State Department of Health) and methods to reduce inappropriate emergency room utilization postpartum (from an obstetrician funded by a K-award from the Einstein CTSA).

Brown University

The Academy provides experiential learning opportunities predominantly focused on development, implementation, and interpretation of SRs and meta-analyses through (1) in-person experiential learning and (2) online cohort mentored learning. The onsite experiential learning program engages both individuals and research teams to work with faculty and staff on producing SRs. Each fellow or team meets with program faculty to develop a work plan which includes Academy coursework and mentorship by program faculty, and plans for development of a specific product (ie, a SR protocol, conduct a SR). The online cohorts also focus on a specific product, and utilize online learning modalities. Courses feature learning modules developed by the Evidence Synthesis Academy and weekly interactive sessions (eg, group discussions, workshops) facilitated by course faculty via web conferencing. Teams of learners participate in these 5–6 week cohort courses, designed to train participants in developing protocols or conducting SRs. Courses and learning activities developed with stakeholders support the community’s ongoing research and training needs. This model supports development of a community proficient in evidence synthesis methods, poised to disseminate these skills through their professional networks and provide opportunities to hone and expand skills without disrupting their current professional responsibilities.

University of Pittsburgh

Cohorts of 5 Fellows are enrolled annually. Fellows developed their PCORI-style proposal in the advanced course (January through April), and then (over the next 2 mo) identified a plan for translating their draft PCORI proposal into a feasible project for the 1-year Fellowship. They then travel to Pittsburgh to participate in intensive PCOR training (which includes seminars, networking activities, and feedback on their project concepts) for 2 weeks in June and 2 weeks in September. They also receive statistical support for data analysis. The end goals of the Fellowship may include a presentation at a national meeting, a manuscript, development of pilot data, or a grant submission. Eventually, the ENACT Program and its partners seek to achieve a wider national impact on the ability to conduct PCOR at Research Centers in Minority Institutions and other Minority-Serving Institutions.

University of Texas MD Anderson Cancer Center

The experiential components of CERTaIN target audiences with some research experience (eg, junior faculty), but who need additional training and mentoring in a specific area within CER/PCOR. Participants are selected from those who participate in the advanced workshops, and who are committed to completion of an appropriate research project within 1–2 years. Participants sign a contract that commits them to follow the outlined steps (eg, literature review, protocol development, data acquisition, etc.) within a given timeline. A CERTaIN investigator partners with the participant, according to their area of expertise, providing monthly mentoring onsite or through conference calls. Additional resources are provided by the investigators and research staff (eg, assistance with literature reviews or statistical analysis). Participants are required to have a manuscript draft by the end of the training period. Goals and mentoring plan are set according to individual needs.

University of Washington

As previously noted, the PCORP scholars develop a very rudimentary project as part of entry into the program. This project is extensively reviewed and revised during the PCORP Summer Institute, and is the basis for the subsequent experiential training. Working on a frequent (at least quarterly) basis with their mentors (one from UW with specific and relevant CER/PCOR methods expertise, and one site-partner-mentor to ensure that the organization can move the project forward), the scholar is expected to develop and conduct the pilot project by the end of the 2nd year after entering PCORP.

Discussion

Lessons Learned

The programs for PCOR training described here involve many challenges that differ across programs but, while each of the R25 grants targets a different researcher community, some key lessons have emerged that are common to two or more of the sites. We list them and explain how we deal with each of these lessons.

Online training by itself has certain well-known limitations and challenges [14]. In particular, maintaining attendance for an optional online training course represents a significant challenge. Despite the added convenience, the “tyranny of the current work in-box” tends to take precedence over training that features more flexible rules and lacks immediate consequences for not participating. Addressing these challenges requires working closely with program partners and understanding the needs of the specific researcher community. Advisory groups and/or institutional leadership of the associated program partners have helped to find the most feasible approaches that could reduce these barriers. For instance, beta-testers for Brown University’s program have encouraged inclusion of more hands-on work and incorporating development of a specific evidence synthesis product. As another example, the Minority-Serving Institutions collaborating with the University of Pittsburgh emphasized the need for training specific to grant writing and funding. Optimally training a specific researcher community thus requires true partnerships in all stages of the planning.

Finding a flexible and accessible online platform poses another challenge. Platforms typically fall into 3 categories: free with limited capability, institutional, and commercial. All 5 of these programs have access to online learning through educational platforms at the institutions, but most of these restrict content and users to students enrolled in the institutions. These platforms require customization, and in some cases do not work for outside learners taking not-for-credit courses. A second type of platform (eg, YouTube) allows users to post content for free. These may have limited capacity, however, and may lack features such as searchable content and interactive quiz capabilities. Finally, commercial portals such as Acatar provide integrated support, but require license fees and restrictions on use that make sustainability a challenge.

Some of these R25 training programs have needed to create new educational materials, whereas other programs have largely leveraged existing educational materials, such as the PCORI Methodology Report, the PCORI Academic Curriculum, and other existing videos. Developing and implementing programs such as these in the future will be greatly facilitated by the continued expansion of publically available resources. On the other hand, the methods required to successfully conduct CER and PCOR are ever evolving and require that such training programs be periodically updated and offered repeatedly. For instance, methods for optimal use of large data networks, causal inference methods, and implementation and evaluation of large pragmatic trials are all emerging and changing rapidly.

The existing institutional environment plays a key role in the speed and direction with which programs develop. Programs within institutional environments with well-established infrastructure dedicated to professional research training and education, such as a CTSA, often take advantage of existing programs, fellowships and relationships to add on additional training and support. Programs without such infrastructure must focus initially on identifying and developing tools such as web platforms, educational platforms, advertising, administration, technical support, faculty and institutional buy-in and a change in culture toward professional education. Although programs building on existing infrastructure and educational programs can certainly ramp up more quickly, the programs without these systems have more flexibility to develop innovative approaches that might serve their needs better. In many cases, these programs may focus more or less on outside professional groups. Both paradigms are useful and can inform each other, leading to cross-fertilization of curriculum, target audiences, and educational processes.

Experiential training is necessary for those relatively new to CER and PCOR. For instance, engaging relevant stakeholders to formulate the initial research question may not be a familiar process, even for experienced researchers. Trainees typically begin with more of an efficacy question and an explanatory, rather than pragmatic approach [15]. Another common first attempt looks more like program evaluation without a clear CER/PCOR objective. Consistent and frequent feedback on real projects is therefore necessary. These experiences, especially paired with a strong mentor and organizational commitment, are invaluable to translating the concepts into practice.

Several of the programs focus largely on training minority populations and associated institutions and organizations. The focus on minority populations was thought to be particularly important for PCOR for several reasons. First, much of what these organizations already do utilizes community-based approaches and naturally addresses patient-centered research questions. However, infrastructure for conducting research, associated resources, and access to large patient populations may be scarce at these institutions and organizations, thus further emphasizing the significance of partnerships with other organizations. Finally, addressing needs specifically relevant to health disparities aligns well with priorities of AHRQ, PCORI, and other PCOR-focused funding groups. Being cognizant of all such factors and involving program partners in all aspects of developing the training program represents a critical step toward successful completion of training and associated projects.

Large organizations with responsibility for the public’s health are another important audience. These R25 programs have worked with Federal Agencies such as the CDC and professional organizations such as the American Society for Clinical Oncology that develop clinical practice guidelines to train their employees and members in PCOR and CER and in specialized methods such as evidence synthesis.

Building a Sustainable Training Network in Patient-Centered CER

The motiving Request for Application (RFA) from AHRQ and the subsequently funded programs aimed to develop expertise and infrastructure within a specific researcher community. While this goal necessitates implementation of specific training, as described above, it also requires further efforts to sustain this impact over the long term and translate expertise gained into real changes and infrastructure at the associated organizations of program partners. Therefore, in addition to the programs specifically developed under this funding mechanism, other efforts are being developed in parallel to leverage collaborations to facilitate better opportunities for future funding and develop more permanent networks.

For instance, some of the programs are creating mock study sections with collaborations between smaller and larger institutions and leveraging skills gained through these R25 programs. This often fills a need for the program partners since they may not have a sufficient volume of faculty experienced in writing and reviewing PCOR projects to provide necessary guidance and feedback. The collaborations established have also facilitated sharing of information for educational purposes, such as discussing the structure of and resources for grant-writing classes. Dissemination of publically available training material, within and between the programs described here, and guidance on navigating those resources already available are also important steps.

Another concept that is being explored is creation of a virtual CER/PCOR Center(s) to provide resources, host discussions and seminars, manage processes such as the Mock Review sessions, and facilitate collaboration across multiple institutions. Although each of the programs funded under this RFA have somewhat different goals and researcher communities, making resources publically available can facilitate sharing across such centers and maximizing efficiency. Individual institutions have done this successfully to maximize available resources and promote better opportunities for funding [16]; the idea of a virtual center would be to expand the concept across institutions in a synergistic fashion. Mechanisms to fund such an institute would need to be developed, but might include leveraging CTSA-related programs, commitments from the universities and associated healthcare organizations, and/or return of indirect rates to the virtual group.

A critical aspect of all CER/PCOR programs is that collaborations between the different institutions within each of these 5 programs are bi-directional and mutually beneficial for all organizations involved. Although each program is funded out of one lead institution, PCOR necessitates collaboration and engagement between scientists, stakeholders, and patients. The program partners representing these researcher communities represent critical stakeholders; without their engagement and input at all steps, none of the individual institutions can effectively conduct PCOR in that given area/community. As highlighted by PCORI in their engagement rubric [17], collaboration with stakeholders should be based on reciprocal relationships, co-learning (where both perspectives learn from the engagement process), partnerships, and transparency, honesty, and trust. Building a long-term sustainable program in PCOR should incorporate these principles.

In addition to the collaborative efforts within each of these 5 programs, opportunities also exist for collaborations between the different CER/PCOR programs. For instance, notices for publicly-available webinars created within some of the programs have been shared with, and taken by participants from the other programs. In some cases, the specific training topics also closely overlap between programs (eg, SRs, which is a major theme for Brown University and also a specific topic for MD Anderson). The principal investigators from each of these programs meet quarterly and discuss these issues and associated strategies on an ongoing basis. Further collaboration across programs is anticipated over the last 2 years of the funding as educational products are produced by the individual programs.

Conclusions

Future training programs in PCOR, CER, and other areas of clinical and translational research are likely to continue evolving within the world of online programs and formats that are more open to and specifically designed for specific segments of the applicable scientific workforce. The PCOR training programs described here provide a useful model, or set of models, for approaches that can effectively move into that space of workforce training. We therefore feel these experiences can benefit other researchers seeking to develop such training programs and educational materials, and hope that documenting such experiences will lead to developing further collaborations and more useful resources in the future.