An uneven spread: A review of public involvement in the National Institute of Health Research's Health Technology Assessment program

Abstract

Objectives: This study comprises a review of public involvement strategies across the five stages of research management in the UK's HTA program at the end of a 10-year period. These five stages are: identification of topics; prioritization of these topics as researchable questions; commissioning of research; monitoring of projects throughout their implementation; and publication and dissemination of findings.

Methods: Internal HTA documentation was analyzed alongside narrative analysis of semi-structured interviews of program staff, and a rapid review of published literature.

Results: Public involvement strategies have developed with the growth of the HTA program but were spread unevenly across the five stages of research management. Public involvement was present in identification, strongest in prioritization, present in commissioning but minimal in monitoring and absent in publication and dissemination.

Conclusions: The HTA program has developed public involvement strategies but mainly in prioritization. Further research is required to ascertain where public involvement can be most appropriately used and to evaluate its impact.

In recent years there has been growing interest and concern with the relationship between the “world of research” and the “real world” it seeks to inform (14). In the United Kingdom, the Health Technology Assessment (HTA) program is part of the National Institute of Health Research (NIHR) and managed by the NIHR's Evaluation, Trials, and Studies Coordinating Centre (NETSCC) (5;16). Unlike some other HTA programs internationally, it commissions primary research (mainly randomized trials) as well as evidence synthesis reports. The program produces independent research about the effectiveness of different healthcare interventions, identifies the most important questions that the UK National Health Service (NHS) needs answering, and commissions research it thinks is most important.

Discussions around public involvement are moving beyond justifying the need for involvement and beginning to explore when, how, and to what value public involvement is deployed (1;6;15;17;18). Our case study of the NIHR HTA program seeks to contribute to the first two questions, and consider the latter. Others have evaluated the efficacy of public involvement in influencing research commissioned by the NIHR HTA program (10). This article will explore how public involvement is spread throughout the management of the program, and articulate broader lessons of interest to others. It is important to note that we offer here a summary at the end of a 10-year period.

BACKGROUND

The program funds in excess of £50 m research activity per year (up from around £10 m five years ago) and aims to be both “needs-led” and scientifically excellent. It serves the needs of patients and the NHS through careful prioritization of research to answer important questions. Rigorous scrutiny and peer-review of research proposals, monitoring of research in progress, and the editorial function of a successful journal series ensure the validity of the research. It now has three main funding streams: commissioned research where the program specifies a research question and investigators apply for funds to carry out the required research, researcher-led where researchers submit unsolicited funding applications to the program, and a program of contracted research for the National Institute for Health and Clinical Excellence (NICE). The third of these is beyond the scope of this study because the public is involved through NICE's topic selection appraisal and processes rather than those of the HTA program. The NIHR HTA program operates along five functional stages of management. These are identification, prioritization, commissioning, monitoring, and publications and dissemination, outlined in Table 1.

Table 1. The Five Stages of Research Management in the UK HTA Programme

Public involvement was first piloted in the program in 1997 and strategies readjusted in light of the pilot's feedback from participants (9). An evaluation of public involvement in the HTA program was conducted in 2003–04 and found public involvement increased when recruitment effort from the program was greater (10). Evidence demonstrating the efficacy of public involvement on research is only slowly emerging (3;11;14). A previous article (in this journal) reviewed the HTA program's public involvement strategies over a four year period from 1998 (12). Since then, the HTA program has expanded alongside a growing landscape of health research, which has been matched by increasing emphasis on public involvement to reflect these changes (14), in line with the policy of the NIHR (7). In the period reviewed, public involvement strategies were steered by the Public Involvement Strategy Group, made up of public members. The HTA program also worked closely with INVOLVE, the NIHR funded national advisory group for public involvement in health research. These groups helped strengthen public involvement in quantity and quality of engagement.

A Note on Terminology

Public involvement has been conceptualized as an ethical obligation with emphasis in the UK on democratic citizenship and stake-holding as part-owners of the NHS (2), captured in the earlier terminology of “consumer involvement,” now “patient and public involvement” (4). The organization INVOLVE promotes and supports public involvement in health research in the UK (4;14). Adopting INVOLVE's terminology, the HTA program defined “consumers” as “patients, carers, long-term users of services, organizations representing consumers' interests, members of the public who are potential recipients of health promotion programs, and groups asking for research because they believe they have been exposed to potentially harmful circumstances, products or services” (12). This definition was then transferred to the new terminology “the public.” For brevity this article will refer to “public involvement” (rather than “consumer involvement” or “patient and public involvement”) incorporating the definition above.

METHODS

Our review focuses on the spread of public involvement strategies within the HTA program by the end of 2009, it is a cross-sectional review rather than a historical longitudinal evaluation from the 1997 pilot. Narrative analysis of semi-structured interviews of key HTA managers and public contributors to the program conducted by the Patient and Public Involvement Manager constructed the story of the development of public involvement in the HTA program. The managers interviewed were from the six different panels of the HTA program and included representatives across the five cycles of research management (see Table 1). We undertook a review of the HTA's documentation, both its policy documents and standard operating procedures, as well as working documents including minutes of the Public Involvement Steering Group (PISG) from 1999–2009. Internal reports were also used as well as external reviews evaluating the HTA program's progress (conducted in 2003–04) (10). We noted patient and public involvement strategies across the five stages of research management and recorded what procedures were used and with what aim. We also conducted a brief literature search to ascertain where the HTA program's story stood in relation to studies of public involvement in other funders of health research. We searched Medline, Embase, the UK Centre for Reviews and Dissemination (UK CRD), and the Health Management Information Consortium (HMIC).

RESULTS

The program allocates funded staff time to public involvement. Of approximately fifty HTA staff, two posts at the level of senior program manager (the Patient and Public Involvement Manager referred to above) and program manager facilitate the implementation and development of the public involvement strategy. More widely, HTA program staff implement public involvement practices where appropriate, such as engaging referees (see below). This section outlines public involvement strategies at each stage of research management.

Identification

There are two main opportunities for public involvement in the identification of topics for research, the first by means of the program's Web site, open to all to input an idea for research using a permanently available online form.

The second strategy is a more proactive activity on the part of the program which builds networks and forums with other research organizations, such as the James Lind Alliance (see www.lindalliance.org) and with user groups and charities. Members of the groups are invited to submit suggestions for research, based on their experiences of particular health issues or conditions.

Prioritization

The HTA program has two main avenues enabling public contributions: as reviewers of briefs and as voting committee members. Potential topics for research are expanded into written briefs to inform decision making. The brief is sent to experts in the relevant field for comments, including at least one public reviewer. The reviewer comments on the relevance and importance of the evidence gap, the knowledge that may potentially be generated, and indicates their support or otherwise for the research. The brief is then discussed by one of six advisory prioritization committees which votes on the importance of the topic to the NHS. A committee has up to fifteen members, and three of these are normally public representatives, supported through an induction and training program and a mentoring scheme. To help alleviate specialized knowledge gaps when reading HTA documentation, public committee members are provided with a medical dictionary. High value is placed on sustained support for public committee members by the program. During a themed call (see below) in 2005, two public members joined the funding committee to support its second function of prioritizing the topics as well as assessing methods, and they were able to usefully advise on methods. Their feedback resulted in the development of user-friendly documentation and highlighted a need for a clearer discussion on the importance of proposals to the NHS, rather than just on scientific quality. This insight helped to strengthen the working practices of the committee.

Commissioning

At this third stage of research management, the HTA commissioning (funding) boards undertake academic peer-review (by academic health professionals and methodologists) and funds the most competitive proposals. The commissioning boards focus on the methodology of proposed research projects. At this stage, there is no public membership on the board as proposals that get this far have undergone public involvement at the prioritization stage. The only occasion where public advisors may be present at commissioning boards is when the prioritization and commissioning stages are combined during annual themed calls (funded sets of studies round a single theme, e.g., obesity). Public referees are invited to submit written comments on full proposals, considered by the commissioning board as part of the peer-review process. While appraising the research methods of proposals, the commissioning board also makes an assessment of whether the public involvement proposed in the trial is adequate, with the advice of the public reviewer.

Monitoring

At this stage, the HTA program expects public involvement to be led by the research team conducting the study. The program requires public involvement to be considered on project proposals, supports costs, and does six-monthly monitoring of researchers' public involvement achievements.

Publication and Dissemination

In our review period, this process had no public involvement. It is concerned with disseminating findings to the NHS, guideline authors and policy makers to indirectly improve the cost-effectiveness of healthcare and thereby improve patient health. The HTA program publishes research in its monograph series Health Technology Assessment which is available on the Web site.

DISCUSSION

Table 2 summarizes the spread of public involvement strategies through the UK HTA program up to 2009. Public advisors are most involved in prioritization of research topics, with contributions to the identification and funding processes. There are opportunities to strengthen parts of the research management cycle so that the organization remains a leader in public involvement practice. We found our more limited findings supported Royle and Oliver's robust assessment (12) that inclusive and reproducible methods for supporting public involvement require dedicated staff time and regular feedback. This is the lesson we would articulate to other funding organizations interested in implementing and improving public involvement. Although we hope this study will contribute to the publication gap on public involvement and its influence on the commissioning of research (14), our conclusions are drawn from an unfunded internal informal review and should be read as such. External scrutiny was preferable but not possible at the time. This study aimed to offer a transparent review of the HTA program's public involvement. This exercise helped highlight the gaps in public involvement within the organization and encouraged continual discussion on the appropriate use of public involvement in health research.

Table 2. Public Involvement in the Five Stages of Research Management in the UK HTA Programme

The HTA program is in a strong position to facilitate public involvement uptake at different stages, yet the appropriate level (or amount) of public involvement may vary. Public involvement done properly requires training, support, management, and payment of public reviewers. Currently, resource is put where there is greatest opportunity to shape research: at its earliest stage of development (identification, but mainly prioritization). However, based on the lessons learned in the 10 years reviewed, the program hopes to explore future opportunities in 2011 to broaden the distribution of public input. It may be important for later, more robust research, to investigate if public involvement at an early stage (e.g., prioritization of research) unintentionally reduces its significance later on (e.g., among researchers) (see O'Donnell and Entwistle) (8).

Even within areas perceived to require public involvement, the program has not necessarily reached its full potential. Although one route for topic identification is by means of the HTA Web site, people may not know of this opportunity for input without pre-existing connections. Our review of HTA strategies demonstrates that public involvement is possible as a matter of procedure, it does shape research processes, and it does require a sustained effort. Research funders may also identify a responsibility to shape acceptance of public involvement among the research community. One study in Australia suggested that making public involvement compulsory for research funding was a significant catalyst for uptake of involvement strategies by researchers (13). Although public involvement is not currently mandatory for research funding in the HTA program, it will become strongly encouraged in program policy from 2011 (see www.hta.ac.uk/public). Practice among other UK funders is mixed. Cancer Research UK has public members on its main trials grant board (CTAAC), whereas the Medical Research Council does not on its Population and Systems Medicine Board. The HTA's sister program, the Efficacy and Mechanism Evaluation program, is piloting public members on its funding board. HTA will learn from this and anticipates following suit in due course.

CONCLUSION

Public involvement strategies feature mainly in the prioritization of research and are being followed by similar activity in the commissioning and conduct of research. Evidence of the value of public involvement is not yet clear. Further research is needed to contribute to the literature around the effectiveness of public involvement on research management; and to explore the impact public involvement can have on those who participate. A joint call for research is being funded by the NIHR through INVOLVE and delivered by the HTA's sister program the Health Services Research program (www.hsr.nihr.ac.uk). The HTA program is implementing a new public involvement framework in 2011 to continue to explore best practice.

CONTACT INFORMATION

Rebecca Moran, PhD (rebecca.moran@pcmd.ac.uk), Associate Research Fellow, European Centre for Environment and Human Health, Peninsula College of Medicine and Dentistry, Knowledge Spa, Royal Cornwall Hospital, TR1 3HD Truro, UK

Peter Davidson, MBBS, MSc, MFPH (p.davidson@southampton.ac.uk), Director of Health Technology Assessment, NETSCC, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK

CONFLICT OF INTEREST

As employees of Southampton University, Peter Davidson coordinates the research program described in this article and Rebecca Moran worked as a research fellow until April 2011.