A consult service to support and promote community-based research: Tracking and evaluating a community-based research consult service

Abstract

Purpose

This study describes the design, operation, and evaluation of a community-based research (CBR) consult service within the setting of a Clinical and Translational Science Award (CTSA) institution. To our knowledge, there are no published evaluations of a CBR consult service at a CTSA hub.

Methods

A CBR consult service was created to support faculty, health-care providers/research coordinators, trainees, community-based organizations, and community members. A framework was developed to assess the stages of client engagement and to foster clear articulation of client needs and challenges. A developmental evaluation system was integrated with the framework to track progress, store documents, continuously improve the consult service, and assess research outcomes.

Results

This framework provides information on client numbers, types, services used, and successful outreach methods. Tracking progress reveals reasons that prevent clients from completing projects and facilitates learning outcomes relevant to clients and funding agencies. Clients benefit from the expert knowledge, community connections, and project guidance provided by the consult service team, increasing the likelihood of study completion and achieving research outcomes.

Conclusion

This study offers a framework by which CTSA institutions can expand their capacity to conduct and evaluate CBR while addressing challenges that inhibit community engagement.

Introduction

Community-based research (CBR) is critical in translating health-care research into the public domain to improve the health and well-being of the community and effect social change [1–3]. CBR is founded on collaboration between researchers and community members in all aspects of the research process with the intent to meet community-identified needs. CBR differs from traditional academic research in that CBR is carried out with the community rather than on the community [4]. Multiple institutions, including the National Institutes of Health Clinical and Translational Science Awards (CTSAs), recognize the critical need to conduct research collaboratively with communities [5, 6]. However, cultural disconnects and mistrust function as potential barriers to community partnerships with academic researchers [7].

Although the National Institutes of Health mandates that CTSA hubs maintain long-term bidirectional relationships with their communities, many researchers had little idea how to undertake this endeavor. Three studies were conducted under the auspices of the national CTSA. In the first, a study of community engagement strategies at the 12 original CTSA hubs identified capacity-building and research engagement strategies across a very broad range, from little or no community input to high input, such as community-researcher teams [8]. In the second, best practices were organized into 5 domains for community engagement: building/strengthening relationships with communities; collaboratively strengthening research agendas with communities; strengthening research methods; building and sharing resources; and engaging in outreach and dissemination [6]. A third study documented barriers to community-partnered participatory research success in the California consortium: communities have little influence over the research problems that are studied; key stakeholders are left out of the development of the research project; cultural differences and health disparities often limit engagement; and dissemination of research findings seldom reaches and/or is meaningful to communities [9]. Six recommendations were also proposed: enhance training in community research methods and capacity building; provide resources such as consultation services to help garner additional funding; provide training on research methodologies that are both pragmatic and adaptive; host community forums to allow the community to identify their own goals and assist them to form partnerships with academic researchers; develop a CTSA-specific funding mechanism to support the time needed for academicians and community leaders to cultivate the necessary relationships; and finally, develop and maintain an easily accessible online directory of individuals across all academic units involved in community-based participatory research, organized by area of expertise rather than discipline [9].

The purpose of this paper is to describe the implementation and evaluation of a CBR consult service designed to improve the liaison between the Case Western Reserve University Clinical and Translational Science Collaborative (CTSC) hub and our Northeast Ohio Community. To our knowledge, this is the first published report of an evaluation of a CBR consult service.

Methods

Established in 2004, the Center for Reducing Health Disparities (CRHD) helps direct the Community Research Partnership Core of the CTSC at Case Western Reserve University in Cleveland, OH. The CRHD provides a free consult service for academic and community researchers conducting CBR. The consult service is designed to promote CBR and foster research support for its clients.

Description of Clientele

The consult service is made available to faculty members, health-care providers, research study coordinators/managers, college and university students, graduate and postgraduate trainees (eg, clinical fellows), community organizations and community members conducting CBR, and/or community-directed research within Greater Cleveland. Clients learn about the service via word of mouth, and through local institutional review boards, academic and research-related Web sites, newsletters, and various local presentations by CRHD staff and faculty.

Consultation Team

The consultation team is composed of a cross-disciplinary group of experts with diverse academic, medical, research, and cultural backgrounds. The team provides expertise in the full research spectrum and supports the expansion of research with communities in Greater Cleveland. Each member provides a unique perspective and knowledge set to address the client’s specific research needs. The diversity of perspectives that results from the consultation team can be valuable for clients, especially those in the early project development stages. To adequately meet the needs of the client, the consultation team focuses on 3 essential components: (1) meeting clients where they are in the research process, (2) enhancing CBR capacity, and (3) cultivating academic-community partnerships. The team is led by Ashwini Sehgal, M.D., Co-Director of the CRHD. The team meets weekly at MetroHealth Medical Center, Cleveland, OH.

Nature of Assistance Provided

Project-related assistance can be obtained based on the consult service client’s specific needs and research parameters. A variety of services are offered, including assistance with the following: research project development and design; survey development or refining data collection tools; Spanish translation of research study documents; developing focus groups; identifying potential study participants and recruitment methods; writing grant applications and identifying funding sources; program evaluation; completing and submitting Institutional Review Board (IRB) applications; collecting data, data input/analysis, report writing, and database development; needs assessments; designing and developing a community advisory committee; providing letters of support for grant applications; identifying appropriate dissemination strategies for research findings; and identifying and adopting best practices and other assistance not specified previously (eg, providing contact persons for follow-up). These services are provided to help clients resolve research design and implementation questions in a manner that is sensitive to and inclusive of community needs and concerns.

The Consult Service Process

The consult service process is composed of initiation and review, service provision, follow-up, and evaluation. To initiate a consult meeting, an individual contacts a member of the consultation team. Alternatively, the consult team may reach out to clients directly and offer the team’s services. The prospective client then receives an electronic Community-Based Research Request for Consult Form to complete. The form requests information regarding the client’s professional role, research question, project scope, project goals and objectives, and the specific service requested. Completion of the form serves 2 purposes: first, it encourages the prospective client to define her or his questions and needs as clearly and succinctly as possible; second, it provides members of the consult team with advance review of the request in order to prepare appropriately. After the completed form is reviewed by the team, the individual is contacted to schedule a face-to-face consult meeting.

Initial consult meetings average an hour in length, and many clients require subsequent follow-up interaction. After the initial meeting, a written summary is composed and sent to the client, along with a link to a Post-Consult Meeting Satisfaction Survey. The survey captures feedback on the client’s overall meeting experience, and helps CRHD staff identify potential areas of consult service improvement. Six to twelve months after completion of the client’s project, the client is emailed a link to a long-term follow-up evaluation survey to learn about the status of the client’s project and its outcomes.

All forms are sent using REDCap (Research Electronic Data Capture), which is a secure web application designed to manage online surveys and databases [10].

Tracking and Evaluation

The consult service began tracking clients in 2011. To effectively track the progress of research projects, a 6-level “Stages of Engagement” model was developed (Table 1). A data entry form allows the consult team to assess where the client is on the Stages of Engagement, and determine which steps in the research process the client has completed from the point of initial contact (stage 1) to research project completion (stage 6). This tracking and evaluation system provides valuable feedback to the CRHD at every step in the consult service process.

Table 1 The six Stages of Engagement

To effectively track the progress of research projects, a 6-level “Stages of Engagement” model was developed to allow the consult team to assess where the client is in their project and to determine which steps in the research process the client has completed from the point of initial contact (stage 1) to research project completion (stage 6).

Tracking and evaluation of the consult service required documentation of 6 components: (1) client identity, (2) client needs, (3) steps taken, (4) referrals/connections made, (5) client feedback, and (6) project outcomes. Process evaluation involves gathering data on client identity, client needs, client satisfaction, activities, and steps taken by the consult service. Outcome evaluation gathers data on connections made (partnerships/collaborations), client feedback, and project outcomes. Study data were collected and managed using REDCap electronic data capture tools hosted at Case Western Reserve University [10].

Four instruments were designed in REDCap to document the above information: (1) a Community-Based Research Request for Consult Form; (2) a Stages of Engagement Data Entry Form; (3) a Post-Consult Meeting Satisfaction Survey; and (4) a Long-Term Follow-Up Post Consultation Form (see supplementary material for instruments and REDCap data dictionary). The following describes the timing and how each instrument functions.

Community-Based Research Request for Consult Form

This instrument gathers detailed client information, including client names and contact information, details about the type of client (eg, faculty, community organization, health provider, or trainee/student), and the client’s organizational or institutional affiliation. The form asks the client for information on their research question or the nature of the project for which they are requesting help. The request form also asks how the client learned about the service so that the consult service team learns which outreach methods work best and which methods attract specific types of clients. Finally, the request form includes a checklist of available services. This educates the client about available consult options and helps the team tailor the consult meeting to the needs of each client.

Stages of Engagement Data Entry Form

This form tracks detailed information on client progress at each stage of the project as a client progresses from start to completion. This data entry form is completed by staff members of the consult service. The CRHD has defined each stage according to Table 1. The Stages of Engagement data form has date fields for each stage so that the CRHD can calculate the elapsed time spent in each stage as well as the rate of progression through the stages. This enables the team to assess whether certain types of clients take longer than others to complete specific stages, allowing the consult service to add additional services or tailor supports to specific client’s needs. Document upload functions are included in multiple stages to capture the meeting summary (stage 2), description of project plan (stage 4), implementation plans (stage 5), and final report or outcomes (stage 6). If a client fails to complete their project, the consult service attempts to determine the underlying reason. They capture the reason for failure to complete on the data entry form, thereby allowing the consult service to identify potential barriers and to design better supports for their clients.

Post-Consult Meeting Satisfaction Survey

This is a satisfaction survey that is sent to each client 2 weeks after the initial consult meeting and 1 week after the client receives the meeting summary. It helps the consult service to continually improve the service for subsequent clients by asking whether the meeting’s location and time were convenient, whether the consult team addressed all the client’s questions, and whether the consult team was courteous and professional. The survey also probes the client’s opinion of the meeting summary as to whether it was timely, whether it clearly stated all the action items from the meeting, and whether it was useful. The survey asks whether the client’s project goals have changed as a result of the meeting or the meeting summary, and, if so, how they have changed. Finally, the survey asks whether the consultation was helpful overall and whether or not the client would recommend the consult service to someone else.

Long-Term Follow-Up Post Consultation Form

This survey gauges the clients’ overall satisfaction with the consult service process as well as captures specific outcomes that are unique to the client and the services they received. The form is sent 6–12 months after the service is provided or upon project completion. If the status of the project shows that it was not completed, the reasons can be captured here. Outcomes are also requested in order to gauge the degree to which the consult service met the client’s overarching goals. The survey asks about outcomes of the consult service that are of interest to funders of the CTSC program, such as whether or not the project led to successful grant funding, and reached under-represented minorities or groups with health disparities. Finally, the survey asks about the client’s overall satisfaction with the entire consult service process.

Results

For our evaluation of the consult service, we were interested in learning what kinds of clients were using the service, how they learned about the service, what services were used, and what outcomes were being realized as a result of using the service.

Client Information

We were interested in: (1) who is using the consult service and their affiliation; (2) whether there were more clients of one type than another; and (3) whether certain types of clients were completing their projects more than were others. Clients came from 5 different universities, 3 hospitals, and 14 community members/organizations. Between 2011 and July 2015, the total number of consultation requests received was 71. Clients consisted of 26 faculty, 17 graduate students or trainees (eg, clinical fellows), 15 community-based organizations, 2 community members, 2 health-care providers, and 9 research study coordinators/managers. Across all clients, the most common services requested included assistance with research project design, advice on survey development, Spanish translation services, and help with focus groups. Service use was not evenly distributed, however, such that different client types tended to use different services. The number of consult services based on client role is shown in Fig. 1. Faculty most often requested help with research project design and focus groups. Community organizations and members used evaluation, survey development, and data collection. Trainees tended to use more research design and survey development. Health providers and research study coordinators most often used Spanish translation and survey design.

Fig. 1 Consult service breakdown by client role. Consult services are listed down the left side in order from most used (top) to least used (bottom). Client types are listed across the top. Bar length represents the number of clients of a particular type who used each consult service. IRB, Institutional Review Board.

Consult Service Outreach

We posed 3 questions about outreach to determine how clients access the service and which methods are most successful: (1) How do clients find out about the consult service? (2) Which outreach methods are most effective?, and (3) Can the answers to the first 2 questions be used to improve outreach?

On the request for consult form, clients identified themselves and their affiliations as well as how they heard about the consult service. As shown in Fig. 2, most clients learn about the consult service via word of mouth. The second most often utilized method was direct contact from the consult service. In order to use this information to improve outreach, we examined which types of clients used which methods to learn of the consult service. Word of mouth was the primary method employed by all types of clients except community organizations, which learned of the consult service primarily via direct contact with consult service staff (data not shown). Since community organizations are not generally in regular contact with academic faculty or health providers, it makes sense that outreach to these organizations may need to be more directed and purposeful. The direct contact method of outreach appears to result in more consults from community organizations, and also has the added benefit that once a relationship with the consult service is established clients repeatedly return for further assistance. Of the 15 community organizations that have been clients of the consult service, 4 (26%) have been repeat clients one or more times on new projects. The consult service received only one other repeat client, who was a health provider.

Fig. 2 Most clients learn about the consult service via word of mouth. The graph shows the number of clients (n=71) who identified a particular method of learning about the consult service. Some clients identified more than one method. Outreach methods are ordered from the most used (top) to the least used (bottom). CRHD, Center for Reducing Health Disparities; IRB, Institutional Review Board; CTSC, Clinical and Translational Science Collaborative.

Client Project Completion

Project tracking (n=71) promotes learning the outcomes, and documenting reasons for incomplete consults promotes improved consultations to avoid the pitfalls that lead to project failure before completion. Examining clients’ completion rates (ie, clients who completed stage 6), faculty, community, and health providers/research managers showed that 25%–35% failed to reach completion, whereas 41% of trainees’/students’ projects failed to complete. This was attributed to the fact that many clinical resident trainees are assigned temporary research projects as part of their training. Thus, when the allotted time for those projects ends, residents often return to clinical training whether or not they have completed their research study.

To visualize the completion of stages along the continuum from stage 1 to stage 6 and to document reasons for premature project terminations, Fig. 3 displays which stages were completed for each client and notes the reasons for project termination without completion. This analysis allowed the consult service to assess which reasons were given for failure of completion, whether specific reasons were more commonly found with certain types of clients and, finally, whether there were interventions that would reduce the likelihood of project termination before completion. One intervention under consideration is to create a “fast track” for research fellows who have a particularly short timeline to complete their research projects.

Fig. 3 Tracking the status of consult projects by completed, ongoing, and incomplete/terminated projects promotes addressing of roadblocks to completion. Each line represents a single client (n=71) and the length of the line represents the progress through the six Stages of Engagement (along x-axis). Within each category of client [(a) faculty, (b) student/trainee, (c) community organization/member, (d) research managers/health providers] completed client projects are shown by black dots, ongoing projects by gray dots, and open dots represent projects that terminated before completion. Each incomplete/terminated project is labeled with a reason for noncompletion.

Getting to Outcomes

In order to determine whether a particular client’s project resulted in meaningful outcomes, we needed to define those specific outcomes. This required development of a logic model for the CTSC, a process that involved input from diverse CTSC stakeholders [11]. Once the logic model and outcomes were articulated, we revisited client consult service projects to assess whether the outcomes in the logic model had been achieved. This was carried out through examination of the results of the client projects that had been identified by the clients via the long-term follow-up survey and through selected phone interviews.

Among specific examples of consult services provided, and their outcomes, are the following:

• ∙ An academic researcher requested assistance recruiting young African-American men with stroke or transient ischemic attack for a study on risk factors. The consult service was able to link the researcher to a number of community organizations for participant recruitment. The researcher successfully completed recruitment and subsequently presented a poster abstract, based on the research, at an international stroke conference. The work is now published in 2 scholarly journals.

• ∙ A community organization addressing chronic disease needed to develop funder-mandated evaluation materials to demonstrate program effectiveness. The consult service assisted the group in developing validated measurement tools, and the organization was able to secure continued grant support from the funder.

• ∙ A community member needed funding support to conduct a pilot study on the effectiveness of massage therapy. The consult service helped the individual obtain training in the responsible conduct of research of human subjects, train additional therapists in research conduct, create IRB applications, and secure funding to conduct the study. Outcomes included presentation of the results at a scientific meeting, submission of a grant to fund a larger follow-up study, and development of a practice-based research network of licensed massage therapists. This has enabled additional research studies to be conducted to examine the effectiveness of massage therapy with other medical conditions.

• ∙ A graduate student needed assistance in translating survey materials into Spanish for a study examining oral health care in the LGBT community. The consult service helped the client with translating the documents and obtaining a third-party certification that the translation is accurate in order to obtain IRB approval. Spanish-translated documents reached up to 25 respondents.

• ∙ A research study coordinator/manager needed assistance in recruitment approaches to develop a community advisory board for isolated and/or difficult-to-reach populations. The client also needed multiple research documents translated into Spanish. The consult service was able to assist in all of the requested services, and the client successfully obtained critical input from the necessary populations. Translated documents were disseminated to over 1000 individuals. A phase III clinical trial is underway.

Future Studies

The consult service has been tracking projects since 2011 and continues to accumulate data that can be mined for answers to additional questions such as: (1) Is there any relationship between the number of services provided to a client, the type of client, and the likelihood of completing their project? (2) When the consult service refers clients to additional contacts, do these lead to actual collaborations and, subsequently, to more relevant CBR outcomes? Many consults involve referral of the client to one or more additional resources who may consist of community organizations, experts in specific research areas, other CTSA-related expertise (eg, biostatistics), or individuals with significant connections to populations of interest. We are approaching some of these questions using network analysis, to look at how connectedness to services, experts, or community affects project outcomes.

Discussion

Our evaluation of this consult service revealed that clients come from various contexts, are at different stages of the research process, and find that for most clients the CBR consult service was helpful in moving their project forward.

The 4 top services together constituted 50% of all consult service requests: research project design, survey development, grants assistance, and data collection. Clients frequently expressed the need for help addressing the challenges related to participant recruitment. This could reflect the high priority researchers place on recruitment, but may also point to gaps in existing guidance on how to effectively engage the community. There was also a noticeable demand for the consult service from outside of university and hospital settings, especially from nonprofit organizations conducting needs assessment and program evaluations. This finding underscores the CRHD goal of building research capacity within community-based settings.

Three practical lessons can be derived from development and implementation of the consult service. First, the consultation team and client must build, at the outset, consensus on the specific scope of service. Open communication is important in order to establish a relationship and to develop, implement, and improve research projects; and a clear definition of the problem or challenge to be presented to the team is vital to providing useful assistance. This is not surprising given the general evidence about the need to communicate and clarify as a part of any problem-solving process, but its specific application to a research consult service should be underlined.

Second, the researcher’s level of engagement must be assessed throughout the consult process, and identified as a barrier or asset to subsequent problem resolution. Although the Stages of Engagement were designed to assist the consultation team in tracking project progression, the model is also useful as a tool for understanding possible factors that may inhibit a client from moving forward with the project and/or with the service, or identifying specific stages at which the client might be insufficiently engaged. Researcher-related factors include inadequate identification of the problem or need, a shift in the client’s project direction or goals, or deficits in the client’s skill or knowledge to carry out consult service recommendations. Project-related factors include lack of necessary personnel to carry out consult service recommendations, a lack of funding, or challenges related to successful interaction with an IRB. Although unforeseeable issues may arise, it is important to assess, on an ongoing basis, the client’s level of engagement, institutional support, and resources available when providing assistance, so that clients and the team can mutually develop realistic expectations of assistance and consult service outcomes.

Third, the continually changing clinical and translational research environment means that any continuing evaluation of the service will be an ongoing developmental process. Based on changing client needs, services have been added, and expertise on the consult service team adapts to those changing needs. New ways to demonstrate community engagement outcomes to funding organizations will focus on the role of the service in facilitating collaborative networks and possibly expanding the geographic and demographic outreach to the community from the academic center.

Finally, it should be noted that there were limitations in our evaluation. First, the post-consult meeting survey was developed in January 2013. Therefore, meeting feedback data from clients utilizing the service before that date were not available. Second, thus far, fewer than 49% of clients who were sent the long-term evaluation survey completed and returned it. This hampers follow-up for completion and prevents determination of outcomes. Thus, it is important to note that results from the surveys received may not be completely representative of all those who received assistance through the service. Although time-consuming, phone calls to past clients can be beneficial in learning outcomes.

In conclusion, the community-based research consult service is a valuable resource for many types of clients and provides critical tools and information related to CBR projects. The “Stages of Engagement” model functions as a useful template for tracking consult service engagement and outcomes. Individualized consults with a broad team of members generates a range of diverse perspectives unique in completing projects. The service involves the interaction between researchers or community organizations and a team of individuals knowledgeable about CBR topics, including issues related to project development, analysis, and dissemination of results to the community. We are continuing ongoing promotion of this service to ensure that local institutions and the community are aware of its availability and utility. Our evaluation suggests that clients benefit by (1) gaining the collective knowledge of the experts comprising the team, (2) learning the process of doing CBR, including the required steps to reach completion, and (3) gaining a project management mentality promoting translational research outcomes. Future studies will determine additional interventions to assist particular types of clients in completing projects and achieving desired outcomes.