Community engagement to pilot electronic patient-reported outcomes (e-PROs) in early intervention: Lessons learned

Abstract

Background

Electronic data capture is essential to advancing family-centered coordinated care in early intervention (EI). The purpose of this paper is to report on EI service coordinator response to piloting an electronic parent-reported outcome (e-PRO) assessment as part of their routine workflow, including lessons learned that may inform future phases of e-PRO implementation.

Methods

This second pilot study involved families enrolled in a large EI program (n=1040 families) in concert with their implementation of a statewide quality improvement initiative for care plan development and outcomes reporting. A total of 22 EI service coordinators and supervisors were engaged in 3 phases: initial e-PRO intervention, peer-mentor enhancement, and standard recruitment protocol.

Results

Implementation of the e-PRO intervention and peer-mentoring enhancement yielded low enrollment rates over the first 6 months (n=17). A standard recruitment protocol has resulted in enrollment growth (n=83) towards the targeted enrollment rate (n=832).

Conclusions

This study reports on early insights for building and sustaining a productive academic-community partnership for e-PRO implementation to support family-centered coordinated care. Lessons learned from this academic-community partnership with respect to strategies for enhancing community significance, collaboration, return, and control are discussed as they inform further development of this intervention before scale-up.

Introduction

Nationally, 2–3% of infants and toddlers with special healthcare needs use early intervention (EI) services annually [1]. Part C of the Individuals with Disabilities Education Act authorizes states, with the incentive of financial support, to establish statewide systems of developmental and therapeutic care (e.g., physical therapy) to infants and toddlers with developmental delays and disabilities. Family-centered care, a federally mandated EI standard, involves customizing care so that families have options for how they contribute in decision-making to plan and monitor their child’s care [2]. Family-centered care activities include conducting a family assessment to determine family priorities as well as creating an EI care plan with the family.

Coordinating family-centered EI care is challenging due to provider workload and competing priorities [3–5], as well as their reliance on paper-based documentation that makes it difficult to efficiently capture and use data to coordinate family-centered care [2, 3]. National EI care coordination challenges may result in lower access to EI services, fragmentation of provider-driven care, and higher costs, thereby reducing the likelihood that children and families equitably access and use EI services [3, 6, 7]. Hence, there is broad consensus on the need to improve EI care coordination [3, 6, 7].

In response to this need, a national EI framework for electronic data capture system development was recently established [8]. EI electronic data capture systems are expected to improve state and local program capacity for efficiently collecting, analyzing, and using data on service access, use, and outcomes to support program operations. However, academic-community partnerships are needed to ensure that EI electronic data capture systems deliver on their promise of providing stakeholders with cost-effective, time efficient ways to coordinate care, report on meaningful outcomes, and conduct clinical research for quality improvement.

In states like Colorado, EI has adopted electronic data capture and is working to improve its capabilities [9]. Health information technology (HIT) strategies can enhance existing electronic data capture systems so that EI programs can better report on their services to the state (e.g., referral and service outcomes) [10] and better coordinate and evaluate family-centered care [11]. For example, electronic portals for families is a HIT strategy that may help EI teams to efficiently capture data from families to design and monitor EI care plans that reflect family priorities [12]. Families currently contribute data on their priorities for improving their child’s functioning through an assessment process that is typically interview based [13], optional [14], and often burdensome to providers and families [15].

Electronic patient-reported outcomes (e-PROs) may be a viable HIT strategy to enhance an EI electronic data capture system [16]. An e-PRO is an electronic assessment of health status that is completed by the patient (e.g., areas of participation difficulty). E-PROs may support program operations by expediting quality EI family assessment completion for family-centered care plan development and outcomes monitoring. An e-PRO also may help individual families evaluate their child to prepare for their child’s annual meeting, so that they are ready to help plan next steps in the child’s care [17]. Information outlined in a family’s e-PRO report align with the family assessment portion of an individualized family service plan (IFSP) that EI providers routinely complete to learn about a family’s priorities and strategies for their child. Lastly, e-PRO data can also provide valuable common data elements for conducting patient-centered outcomes research to inform continuous quality improvement within and across EI programs [18]. However, to our knowledge, e-PROs have not yet been used as a HIT enhancement for an EI specific electronic data capture system.

Since organizational behavior is a key force in shaping adoption of systematic changes in clinical practice [19], it is important to engage frontline personnel and advisory groups as stakeholders to explore the uptake of e-PROs for EI electronic data capture system enhancement [20]. EI service coordinators act as case managers for families and coordinate evaluations, care planning, and outcomes monitoring activities. Therefore, they are poised to serve as subject matter experts to evaluate an e-PRO HIT solution for helping them achieve an ideal workflow.

The purpose of this paper is to report on EI service coordinator response to piloting an e-PRO assessment as part of their routine workflow, including lessons learned that may inform future phases of translational research on this topic.

e-PRO Intervention Premise and Design

The overarching goal of our T2 translational research project is to evaluate the feasibility and value of implementing e-PROs into an EI process of care [21]. These e-PROs are designed to (1) improve EI care coordination by providing individual caregivers with an electronic option to actively partner in planning care and monitoring outcomes and (2) expand EI program capacity for enhancing their electronic data capture system to pursue robust patient-centered outcomes research for quality improvement.

This translational research project involves partnership between academic and community stakeholders to form the research team (i.e., an academic-community partnership) and has been carried out in 2 phases to date. The first phase included partnership with an EI program in Denver Metro (annual enrollment: 80 families). Results suggested that implementing e-PROs into EI practice was, in general, feasible, and that doing so could advance clinically relevant knowledge by estimating the relationship between EI service use and outcomes from the family perspective [22]. Specifically, e-PRO data collection was feasible for nearly half of the enrolled families when it was introduced during routine EI home visits, and e-PROs were completed by families in entirety with no missing data. At that time, EI providers suggested integrating e-PRO data collection during periodic EI progress evaluations rather than service visits to better integrate it within routine care.

The current phase was rolled out by the largest EI program in Denver Metro (annual enrollment: 1040 families) in concert with their implementation of a statewide initiative called GO4IT (see Fig. 1). GO4IT is designed to improve accountability and quality of EI services by incorporating the 3 global child outcomes (e.g., acquiring and using knowledge and skills) that are reported on annually as an organizing framework to design care and monitor outcomes. GO4IT emphasizes that all information collected about a family during EI service provision should be utilized to design and monitor a high-quality care plan [23]. Since this e-PRO gathers family input about their child’s current and desired functioning, it was incorporated into EI service coordinator workflow to reinforce the GO4IT intervention for actively enrolled families.

Fig. 1 Early intervention (EI) care process with the implementation of GO4IT and electronic patient-reported outcome (e-PRO) intervention. Standard process of care is shown with gray fill. The e-PRO intervention is shown with dotted fill. IFSP, individualized family service plan.

The e-PRO intervention included EI provider webinar training, written resources, and rewards (see Fig. 2). In total, 22 EI service coordinators and supervisors completed a 90-minute video-conference training on project rationale and protocol. This training was designed and delivered together with EI program leadership, delivered in a small group format, and included didactic and interactive components so that EI providers could provide feedback to further optimize the study protocol (e.g., streamlining steps for account creation). Each trained EI provider was issued a $50.00 gift card and written materials to support their implementation of the recruitment protocol (e.g., flyer, script). EI service coordinators were also rewarded with snack parties for meeting monthly recruitment goals as estimated based on the total target enrollment.

Fig. 2 Electronic patient-reported outcomes (e-PRO) intervention as integrated into early intervention provider workflow.

Surprisingly, implementation of the e-PRO intervention yielded low enrollment rates over 4 months (target n=276, actual n=9), though similar and high e-PRO completion rates (83.3%) as compared with the first project phase. Anecdotal feedback from service coordinators suggested unmet training needs and workload concerns, resulting in only 5 out of 22 EI service coordinators who successfully recruited families.

Peer-Mentoring Enhancement to Increase Enrollment in e-PRO Intervention

In response to low enrollment, we co-designed a peer-mentoring intervention enhancement with EI program leadership. The central premise of the peer-mentoring enhancement was to empower EI service coordinators who were successfully recruiting families to mentor those who were struggling to do the same.

Four EI service coordinators were invited to serve in the role of peer mentor. Peer mentors were identified based on their success in recruiting one or more families and in consultation with EI program leadership about their likelihood to effectively manage and derive value from peer-mentoring responsibilities.

As shown in Fig. 3, a peer-mentoring enhancement encompassed directive peer mentorship on site. Each peer mentor was randomly assigned 4–7 EI service coordinators to make contact with every 2 weeks, at minimum. Study staff created short videos to reinforce the recruitment protocol. Peer mentors then conducted knowledge checks of video content, obtained enrollment updates and mentee feedback regarding recruitment, and relayed mentee feedback to the project team to inform subsequent videos and protocol enhancements. Mentors were incentivized for their time with a $200 gift card payment, invitation to formally contribute to dissemination efforts, and credit towards their performance reviews. EI supervisors also monitored service coordinator engagement during regularly scheduled monthly meetings.

Fig. 3 Peer-mentorship enhancement to electronic patient-reported outcomes (e-PRO) intervention.

Despite implementation of the peer-mentorship enhancement, enrollment rates remained low (from n=9 to n=17, July to October 2017). Key challenges reported by mentors were lack of time to incorporate peer mentoring into their workflow, lack of comfort in adopting the peer-mentor role, and difficulty adhering to a protocol for research purposes without the guarantee of long-term e-PRO adoption in practice. These challenges may have been exacerbated due to the e-PRO intervention and peer-mentor enhancement coinciding with GO4IT implementation.

In response to continued low enrollment, the protocol was modified to a standard recruitment protocol, whereby a designated community site contact was trained and then paired with research staff to complete recruitment, resulting in significant enrollment growth (from n=17 to n=83, October 2017 to March 2018; see Fig. 4).

Fig. 4 Enrollment trends by protocol phase. e-PRO, electronic patient-reported outcomes.

Lessons Learned in Conducting Community-Engaged Research (CER) in EI

To our knowledge, this is the first attempt to explore the use of community engagement principles when implementing e-PROs as a HIT strategy to improve family-centered EI care. We have gained considerable insight about the EI setting by carrying out the e-PRO intervention and peer-mentor enhancement. These insights will inform the design of a successful community-engaged protocol during scale-up phases of implementation research, which is preferred for sustainability of the innovation in routine care [21].

Therefore, in the remainder of this section, we interpret our experiences in designing and carrying out this pilot project as guided by the community-engaged research principles of (1) community significance, (2) community collaboration and control, and (3) community return [24]. We describe our use of each community engagement research principle during protocol design and discuss how we will operationalize that research principle in subsequent scale-up phases of this work.

CER Principle 1: Community Significance

Ethical and high-quality community-engaged research answers questions that are meaningful to the community partner [24]. The EI program’s commitment to implementing GO4IT and the e-PRO intervention together reflects their strong interest in answering the question of how to improve their EI care plans so that they consistently focus on functional goals and reflect family priorities. The community partner decided on implementing the e-PRO intervention at the same time as GO4IT because the e-PRO data from families could support the EI program’s efforts to obtain family input when implementing GO4IT. They also have electronic data capture system capability for long-term implementation.

While the EI program expressed interest and were responsive to the e-PRO innovation, they may have needed to better communicate the need for change to staff for clearer expectations during project implementation [25]. Whereas GO4IT was clearly communicated to EI staff as a required and permanent change to their workflow to support known program values, the e-PRO intervention was introduced as an optional research project that was related to GO4IT. EI staff were therefore trained on 2 projects that shared a similar purpose but differed in their scope and valence. Mentors often reported that service coordinators did not integrate the optional e-PRO intervention into their changing workflow right after the training. The project was eventually deprioritized to keep up with GO4IT implementation, which required significant changes to their workflow.

Three communication strategies may improve future phases of work. First, EI staff training should more clearly communicate how the intervention aligns with current values, beliefs, and attitudes about the care they provide [25]. The academic-community partners assumed that service coordinators would understand how the e-PRO intervention links to their values by training them on the logistics of carrying out the protocol, such as by showing EI staff how to view the information gathered by the e-PRO. However, more training time is needed to better articulate how the e-PRO results reinforce the value of family-centered care when care planning.

Second, without the guarantee of longer-term adoption, many service coordinators struggled to invest effort in recruiting families due to time constraints, a common barrier encountered by organizations implementing change [19, 26, 27]. The academic partners and EI program leadership explained that the e-PRO intervention will require an increased time commitment from the service coordinators without the guarantee of saving them time in the longer term, as decisions about e-PRO adoption would hinge on project results. EI staff training is being redesigned to emphasize rationale and requirements for research engagement so there are clear expectations about the benefits and costs of this type of partnership [25].

Third, there is need to recruit a broader base of community support to successfully manage the implementation of an intervention that requires a change in EI provider workflow [19, 25]. While EI program leadership championed GO4IT and e-PRO intervention implementation, supervisors were not consistently incorporating the e-PRO intervention when communicating with service coordinators at their monthly check-in meetings. Future phases of work should ensure that more diverse EI leadership (e.g., service coordinators, supervisors) within the organization have active roles in championing the e-PRO innovation.

CER Principle 2: Community Collaboration and Control

Community collaboration is a critical component of high quality and ethical community-engaged research [24, 28]. Academic and community partner input for joint decision-making gave each group an opportunity to voice their needs and preferences, resulting in transparency about research goals and methods [28].

Academic and community partners have demonstrated a high level of collaboration for the study protocol. Both parties initiated collaboration on mutually beneficial terms. Specifically, EI service coordinators acknowledged that research engagement was a new experience for which they had limited supporting infrastructure, and the academic partner acknowledged their need to better engage service coordinators for recruitment at a large EI program. Additionally, the design of the e-PRO intervention was informed by EI provider input in the first pilot phase [22] and the EI program partner decided on protocol timing and implementation strategies for the current phase. The design of the peer-mentorship enhancement included joint decision-making around (1) mentor selection, (2) what peer mentors would do, (3) how mentee feedback would be gathered and relayed back, and (4) peer-mentor incentives.

While community collaboration has been high, there are several ways that the academic partners could improve their efforts to ensure successful academic-community collaboration in subsequent scale-up phases. First, the academic partners can play a larger role in educating their community partners about the research process in depth, but in laymen’s terms in order to foster understanding of the reasoning for the study design, short and long-term impact, and the vital need for research participants [29]. While the academic partners discussed clinical impact in the 90-minute training sessions, this training may not be adequate when working with EI programs that have little to no prior research experience. There is need to develop accessible resources for those new to research.

Second, the academic partners may benefit from conducting a formative evaluation to obtain insight into an organization’s strengths, weaknesses, and level of readiness for change before scale-up implementation [19]. These insights can then be used to guide the design of scale-up implementation efforts [19].

Third, greater service coordinator involvement may be needed to prepare for the resistance to implementation that is common and should be expected [25]. Service coordinators can generate strategies to address resistance before implementation. For example, service coordinators collected and relayed anonymous feedback from mentees with ease, but they expressed concern about holding their peers accountable by monitoring their recruitment statistics and testing them on their knowledge of recruitment processes. In this case, service coordinators anticipated that monitoring the performance of their mentees would be responded to with resistance by mentees because it would create judgment about mentee job performance. Peer mentors are better suited to serve as opinion leaders rather than supervisors. Peer opinion leaders are individuals aware of the need for change [30], comfortable adopting ideas to address the need for change [30], and who impact the adoption of innovation through their “representativeness and credibility” among their peers [25].

Community control includes joint ownership of data [24]. The EI program will be given access to the final data sets from this project. In addition, the team secured additional extramural funds to archive data from this project as a limited use public data set, allowing community access. Demographic and service use data elements to be included in the data source were decided upon by the EI program. There are plans to secure similar funding to archive the second final de-identified data source from this project, inclusive of information on child and family characteristics, service use, and e-PRO data. Community control also includes a strengths-based focus [24]. To further ensure community control, the team will continue to positively reinforce recruitment efforts in subsequent phases (e.g., providing monetary reimbursement and professional development opportunities for peer mentors, incorporating mentee feedback and questions into recruitment materials and informational videos) to emphasize community strengths and contributions [24]. This strengths-based approach will be familiar to EI service coordinators because this is a key principle to delivering family-centered care [2].

CER Principle 3: Community return

Ethical community-engaged research hinges on the community receiving returns for their research engagement effort [24]. This project is designed to provide 3 types of returns to the EI program: (1) a database that will include patient-reported outcomes data on currently enrolled families for program-specific outcomes research, (2) automated electronic reports for families to customize (e.g., child photo and name) and share with their EI providers to facilitate family-centered care planning towards functional outcomes, and (3) increased workforce capacity for research engagement.

Given the success of the standard recruitment protocol for e-PRO data collection (see Fig. 4), it is likely that this project will yield a significant return in the form of a robust EI database for testing links between service use and patient-reported outcomes. In addition to this programmatic return, we have identified areas for improving community return for individual families and providers. Although incorporating the e-PRO report into care planning was considered an immediate return for both families and EI providers to use in care planning, service coordinators often expressed a lack of understanding in how they could use information in the e-PRO report for the IFSP without increasing meeting length. Research staff developed a written resource for service coordinators to visually map the e-PRO content to the IFSP (see online Supplementary Appendix), but service coordinators had forgotten about this resource and often were reminded of it during mentor meetings. If service coordinators had understood how the e-PRO aligned with the IFSP, they may have been more motivated to engage in recruiting families to use it [25], moving forward, academic partners could introduce this resource during the 90-minute training to provide service coordinators with greater detail on how to incorporate data from the PEM report into the IFSP. Similarly, case studies may be helpful in providing hands-on training for service coordinators on the research processes involved when developing, testing, and applying e-PROs, thereby increasing workforce capacity for research [25].

Conclusion

In an era of increased accountability for EI programs, it is important to capture and use data from families when determining the effectiveness of services on outcomes that matter to children and families. Academic-community partnerships are vital to designing e-PRO interventions that can be easily integrated into existing EI systems of care. EI community engagement in research efforts allows for the design of enhancements to e-PRO intervention and implementation that can make e-PRO interventions sustainable in the longer term. This study reports on early insights for building and sustaining a productive academic-community partnership for e-PRO implementation that may support program capacity for: (1) family-centered care planning and outcomes monitoring, and (2) patient-centered outcomes research to drive continuous quality improvement towards high-quality family-centered care and positive outcomes for children and families who access EI services [9]. Lessons learned with respect to community significance, collaboration, return, and control have been identified to inform the design of scale-up efforts.