Crossref Citations
This article has been cited by the following publications. This list is generated based on data provided by
Crossref.
Abelson, Julia
Bombard, Yvonne
Gauvin, François-Pierre
Simeonov, Dorina
and
Boesveld, Sarah
2013.
ASSESSING THE IMPACTS OF CITIZEN DELIBERATIONS ON THE HEALTH TECHNOLOGY PROCESS.
International Journal of Technology Assessment in Health Care,
Vol. 29,
Issue. 3,
p.
282.
Lander, Jonas
Hainz, Tobias
Hirschberg, Irene
Strech, Daniel
and
Virolle, Marie-Joelle
2014.
Current Practice of Public Involvement Activities in Biomedical Research and Innovation: A Systematic Qualitative Review.
PLoS ONE,
Vol. 9,
Issue. 12,
p.
e113274.
Kleme, Jenni
Pohjanoksa-Mäntylä, Marika
Airaksinen, Marja
Enlund, Hannes
Kastarinen, Helena
Peura, Piia
and
Hämeen-Anttila, Katri
2014.
PATIENT PERSPECTIVE IN HEALTH TECHNOLOGY ASSESSMENT OF PHARMACEUTICALS IN FINLAND.
International Journal of Technology Assessment in Health Care,
Vol. 30,
Issue. 3,
p.
306.
Gagnon, Marie-Pierre
Candas, Bernard
Desmartis, Marie
Gagnon, Johanne
Roche, Daniel La
Rhainds, Marc
Coulombe, Martin
Dipankui, Mylène Tantchou
and
Légaré, France
2014.
Involving patient in the early stages of health technology assessment (HTA): a study protocol.
BMC Health Services Research,
Vol. 14,
Issue. 1,
Brice, Anne
Price, Amy
and
Burls, Amanda
2015.
Creating a database of internet-based clinical trials to support a public-led research programme: A descriptive analysis.
DIGITAL HEALTH,
Vol. 1,
Issue. ,
Tricco, Andrea C.
Antony, Jesmin
Zarin, Wasifa
Strifler, Lisa
Ghassemi, Marco
Ivory, John
Perrier, Laure
Hutton, Brian
Moher, David
and
Straus, Sharon E.
2015.
A scoping review of rapid review methods.
BMC Medicine,
Vol. 13,
Issue. 1,
Abelson, Julia
Wagner, Frank
DeJean, Deirdre
Boesveld, Sarah
Gauvin, Franςois-Pierre
Bean, Sally
Axler, Renata
Petersen, Stephen
Baidoobonso, Shamara
Pron, Gaylene
Giacomini, Mita
and
Lavis, John
2016.
PUBLIC AND PATIENT INVOLVEMENT IN HEALTH TECHNOLOGY ASSESSMENT: A FRAMEWORK FOR ACTION.
International Journal of Technology Assessment in Health Care,
Vol. 32,
Issue. 4,
p.
256.
Miller, Fiona A.
Patton, Sarah
Nam, Austin
Bielecki, Joanna
and
Krahn, Murray
2016.
Addressing Patient Needs and Public Commitments in Health Technology Innovation: A Review of Patient and Public Engagement in Technology Development and Assessment.
SSRN Electronic Journal,
Miller, Fiona Alice
Patton, Sarah J.
Dobrow, Mark
and
Berta, Whitney
2018.
Public involvement in health research systems: a governance framework.
Health Research Policy and Systems,
Vol. 16,
Issue. 1,
Abelson, Julia
2018.
Patient engagement in health technology assessment: what constitutes ‘meaningful’ and how we might get there.
Journal of Health Services Research & Policy,
Vol. 23,
Issue. 2,
p.
69.
Toledo-Chávarri, Ana
Alvarez-Perez, Yolanda
Triñanes, Yolanda
Perestelo-Pérez, Lilisbeth
Espallargues, Mireia
Palma, Matilde
and
Serrano-Aguilar, Pedro
2019.
Toward a Strategy to Involve Patients in Health Technology Assessment in Spain.
International Journal of Technology Assessment in Health Care,
Vol. 35,
Issue. 2,
p.
92.
Tembo, Doreen
Morrow, Elizabeth
Worswick, Louise
and
Lennard, Debby
2019.
Is Co-production Just a Pipe Dream for Applied Health Research Commissioning? An Exploratory Literature Review.
Frontiers in Sociology,
Vol. 4,
Issue. ,
Gaspar, Fernanda Drummond Ruas
Abbad, Gardênia da Silva
and
Lima, Marcelo Nunes de
2020.
Importância das habilidades de ensino em saúde atribuída por estudantes e professores universitários.
Trabalho, Educação e Saúde,
Vol. 18,
Issue. 3,
Gagnon, Marie-Pierre
Tantchou Dipankui, Mylène
Poder, Thomas G.
Payne-Gagnon, Julie
Mbemba, Gisèle
and
Beretta, Valentina
2021.
Patient and public involvement in health technology assessment: update of a systematic review of international experiences.
International Journal of Technology Assessment in Health Care,
Vol. 37,
Issue. 1,
Cluley, Victoria
Ziemann, Alexandra
Feeley, Claire
Olander, Ellinor K.
Shamah, Shani
and
Stavropoulou, Charitini
2022.
Mapping the role of patient and public involvement during the different stages of healthcare innovation: A scoping review.
Health Expectations,
Vol. 25,
Issue. 3,
p.
840.
Sathe, Nila
Ovelman, Colleen
Ospina, Naykky Singh
Dewidar, Omar
Terhune, Elizabeth
Francis, Damian
Welch, Vivian
Heyn, Patricia
Duque, Tiffany
and
Viswanathan, Meera
2024.
Centering Racial Health Equity in Systematic Reviews Paper 6: Engaging racially and ethnically diverse stakeholders in evidence syntheses.
SSRN Electronic Journal,
In recent years there has been growing interest and concern with the relationship between the “world of research” and the “real world” it seeks to inform (Reference Staley14). In the United Kingdom, the Health Technology Assessment (HTA) program is part of the National Institute of Health Research (NIHR) and managed by the NIHR's Evaluation, Trials, and Studies Coordinating Centre (NETSCC) (Reference Hanney, Kuruvilla, Soper and Mays5;Reference Stevens and Milne16). Unlike some other HTA programs internationally, it commissions primary research (mainly randomized trials) as well as evidence synthesis reports. The program produces independent research about the effectiveness of different healthcare interventions, identifies the most important questions that the UK National Health Service (NHS) needs answering, and commissions research it thinks is most important.
Discussions around public involvement are moving beyond justifying the need for involvement and beginning to explore when, how, and to what value public involvement is deployed (Reference Boote, Baird and Beecroft1;Reference Mitton, Smith and Peacock6;Reference Staniszewska, Herron-Marx and Mockford15;Reference Telford, Boote and Cooper17;Reference Thompson, Barber and Ward18). Our case study of the NIHR HTA program seeks to contribute to the first two questions, and consider the latter. Others have evaluated the efficacy of public involvement in influencing research commissioned by the NIHR HTA program (Reference Oliver, Armes and Gyte10). This article will explore how public involvement is spread throughout the management of the program, and articulate broader lessons of interest to others. It is important to note that we offer here a summary at the end of a 10-year period.
BACKGROUND
The program funds in excess of £50 m research activity per year (up from around £Reference Oliver, Armes and Gyte10 m five years ago) and aims to be both “needs-led” and scientifically excellent. It serves the needs of patients and the NHS through careful prioritization of research to answer important questions. Rigorous scrutiny and peer-review of research proposals, monitoring of research in progress, and the editorial function of a successful journal series ensure the validity of the research. It now has three main funding streams: commissioned research where the program specifies a research question and investigators apply for funds to carry out the required research, researcher-led where researchers submit unsolicited funding applications to the program, and a program of contracted research for the National Institute for Health and Clinical Excellence (NICE). The third of these is beyond the scope of this study because the public is involved through NICE's topic selection appraisal and processes rather than those of the HTA program. The NIHR HTA program operates along five functional stages of management. These are identification, prioritization, commissioning, monitoring, and publications and dissemination, outlined in Table 1.
Table 1. The Five Stages of Research Management in the UK HTA Programme
Public involvement was first piloted in the program in 1997 and strategies readjusted in light of the pilot's feedback from participants (Reference Oliver, Milne and Bradburn9). An evaluation of public involvement in the HTA program was conducted in 2003–04 and found public involvement increased when recruitment effort from the program was greater (Reference Oliver, Armes and Gyte10). Evidence demonstrating the efficacy of public involvement on research is only slowly emerging (3;Reference Oliver, Clarke-Jones and Rees11;Reference Staley14). A previous article (in this journal) reviewed the HTA program's public involvement strategies over a four year period from 1998 (Reference Royle and Oliver12). Since then, the HTA program has expanded alongside a growing landscape of health research, which has been matched by increasing emphasis on public involvement to reflect these changes (Reference Staley14), in line with the policy of the NIHR (7). In the period reviewed, public involvement strategies were steered by the Public Involvement Strategy Group, made up of public members. The HTA program also worked closely with INVOLVE, the NIHR funded national advisory group for public involvement in health research. These groups helped strengthen public involvement in quantity and quality of engagement.
A Note on Terminology
Public involvement has been conceptualized as an ethical obligation with emphasis in the UK on democratic citizenship and stake-holding as part-owners of the NHS (Reference Boote, Telford and Cooper2), captured in the earlier terminology of “consumer involvement,” now “patient and public involvement” (Reference Hanley, Bradburn and Barnes4). The organization INVOLVE promotes and supports public involvement in health research in the UK (Reference Hanley, Bradburn and Barnes4;Reference Staley14). Adopting INVOLVE's terminology, the HTA program defined “consumers” as “patients, carers, long-term users of services, organizations representing consumers' interests, members of the public who are potential recipients of health promotion programs, and groups asking for research because they believe they have been exposed to potentially harmful circumstances, products or services” (Reference Royle and Oliver12). This definition was then transferred to the new terminology “the public.” For brevity this article will refer to “public involvement” (rather than “consumer involvement” or “patient and public involvement”) incorporating the definition above.
METHODS
Our review focuses on the spread of public involvement strategies within the HTA program by the end of 2009, it is a cross-sectional review rather than a historical longitudinal evaluation from the 1997 pilot. Narrative analysis of semi-structured interviews of key HTA managers and public contributors to the program conducted by the Patient and Public Involvement Manager constructed the story of the development of public involvement in the HTA program. The managers interviewed were from the six different panels of the HTA program and included representatives across the five cycles of research management (see Table 1). We undertook a review of the HTA's documentation, both its policy documents and standard operating procedures, as well as working documents including minutes of the Public Involvement Steering Group (PISG) from 1999–2009. Internal reports were also used as well as external reviews evaluating the HTA program's progress (conducted in 2003–04) (Reference Oliver, Armes and Gyte10). We noted patient and public involvement strategies across the five stages of research management and recorded what procedures were used and with what aim. We also conducted a brief literature search to ascertain where the HTA program's story stood in relation to studies of public involvement in other funders of health research. We searched Medline, Embase, the UK Centre for Reviews and Dissemination (UK CRD), and the Health Management Information Consortium (HMIC).
RESULTS
The program allocates funded staff time to public involvement. Of approximately fifty HTA staff, two posts at the level of senior program manager (the Patient and Public Involvement Manager referred to above) and program manager facilitate the implementation and development of the public involvement strategy. More widely, HTA program staff implement public involvement practices where appropriate, such as engaging referees (see below). This section outlines public involvement strategies at each stage of research management.
Identification
There are two main opportunities for public involvement in the identification of topics for research, the first by means of the program's Web site, open to all to input an idea for research using a permanently available online form.
The second strategy is a more proactive activity on the part of the program which builds networks and forums with other research organizations, such as the James Lind Alliance (see www.lindalliance.org) and with user groups and charities. Members of the groups are invited to submit suggestions for research, based on their experiences of particular health issues or conditions.
Prioritization
The HTA program has two main avenues enabling public contributions: as reviewers of briefs and as voting committee members. Potential topics for research are expanded into written briefs to inform decision making. The brief is sent to experts in the relevant field for comments, including at least one public reviewer. The reviewer comments on the relevance and importance of the evidence gap, the knowledge that may potentially be generated, and indicates their support or otherwise for the research. The brief is then discussed by one of six advisory prioritization committees which votes on the importance of the topic to the NHS. A committee has up to fifteen members, and three of these are normally public representatives, supported through an induction and training program and a mentoring scheme. To help alleviate specialized knowledge gaps when reading HTA documentation, public committee members are provided with a medical dictionary. High value is placed on sustained support for public committee members by the program. During a themed call (see below) in 2005, two public members joined the funding committee to support its second function of prioritizing the topics as well as assessing methods, and they were able to usefully advise on methods. Their feedback resulted in the development of user-friendly documentation and highlighted a need for a clearer discussion on the importance of proposals to the NHS, rather than just on scientific quality. This insight helped to strengthen the working practices of the committee.
Commissioning
At this third stage of research management, the HTA commissioning (funding) boards undertake academic peer-review (by academic health professionals and methodologists) and funds the most competitive proposals. The commissioning boards focus on the methodology of proposed research projects. At this stage, there is no public membership on the board as proposals that get this far have undergone public involvement at the prioritization stage. The only occasion where public advisors may be present at commissioning boards is when the prioritization and commissioning stages are combined during annual themed calls (funded sets of studies round a single theme, e.g., obesity). Public referees are invited to submit written comments on full proposals, considered by the commissioning board as part of the peer-review process. While appraising the research methods of proposals, the commissioning board also makes an assessment of whether the public involvement proposed in the trial is adequate, with the advice of the public reviewer.
Monitoring
At this stage, the HTA program expects public involvement to be led by the research team conducting the study. The program requires public involvement to be considered on project proposals, supports costs, and does six-monthly monitoring of researchers' public involvement achievements.
Publication and Dissemination
In our review period, this process had no public involvement. It is concerned with disseminating findings to the NHS, guideline authors and policy makers to indirectly improve the cost-effectiveness of healthcare and thereby improve patient health. The HTA program publishes research in its monograph series Health Technology Assessment which is available on the Web site.
DISCUSSION
Table 2 summarizes the spread of public involvement strategies through the UK HTA program up to 2009. Public advisors are most involved in prioritization of research topics, with contributions to the identification and funding processes. There are opportunities to strengthen parts of the research management cycle so that the organization remains a leader in public involvement practice. We found our more limited findings supported Royle and Oliver's robust assessment (Reference Royle and Oliver12) that inclusive and reproducible methods for supporting public involvement require dedicated staff time and regular feedback. This is the lesson we would articulate to other funding organizations interested in implementing and improving public involvement. Although we hope this study will contribute to the publication gap on public involvement and its influence on the commissioning of research (Reference Staley14), our conclusions are drawn from an unfunded internal informal review and should be read as such. External scrutiny was preferable but not possible at the time. This study aimed to offer a transparent review of the HTA program's public involvement. This exercise helped highlight the gaps in public involvement within the organization and encouraged continual discussion on the appropriate use of public involvement in health research.
Table 2. Public Involvement in the Five Stages of Research Management in the UK HTA Programme
The HTA program is in a strong position to facilitate public involvement uptake at different stages, yet the appropriate level (or amount) of public involvement may vary. Public involvement done properly requires training, support, management, and payment of public reviewers. Currently, resource is put where there is greatest opportunity to shape research: at its earliest stage of development (identification, but mainly prioritization). However, based on the lessons learned in the 10 years reviewed, the program hopes to explore future opportunities in 2011 to broaden the distribution of public input. It may be important for later, more robust research, to investigate if public involvement at an early stage (e.g., prioritization of research) unintentionally reduces its significance later on (e.g., among researchers) (see O'Donnell and Entwistle) (Reference O'Donnell and Entwistle8).
Even within areas perceived to require public involvement, the program has not necessarily reached its full potential. Although one route for topic identification is by means of the HTA Web site, people may not know of this opportunity for input without pre-existing connections. Our review of HTA strategies demonstrates that public involvement is possible as a matter of procedure, it does shape research processes, and it does require a sustained effort. Research funders may also identify a responsibility to shape acceptance of public involvement among the research community. One study in Australia suggested that making public involvement compulsory for research funding was a significant catalyst for uptake of involvement strategies by researchers (Reference Saunders and Girgis13). Although public involvement is not currently mandatory for research funding in the HTA program, it will become strongly encouraged in program policy from 2011 (see www.hta.ac.uk/public). Practice among other UK funders is mixed. Cancer Research UK has public members on its main trials grant board (CTAAC), whereas the Medical Research Council does not on its Population and Systems Medicine Board. The HTA's sister program, the Efficacy and Mechanism Evaluation program, is piloting public members on its funding board. HTA will learn from this and anticipates following suit in due course.
CONCLUSION
Public involvement strategies feature mainly in the prioritization of research and are being followed by similar activity in the commissioning and conduct of research. Evidence of the value of public involvement is not yet clear. Further research is needed to contribute to the literature around the effectiveness of public involvement on research management; and to explore the impact public involvement can have on those who participate. A joint call for research is being funded by the NIHR through INVOLVE and delivered by the HTA's sister program the Health Services Research program (www.hsr.nihr.ac.uk). The HTA program is implementing a new public involvement framework in 2011 to continue to explore best practice.
CONTACT INFORMATION
Rebecca Moran, PhD (rebecca.moran@pcmd.ac.uk), Associate Research Fellow, European Centre for Environment and Human Health, Peninsula College of Medicine and Dentistry, Knowledge Spa, Royal Cornwall Hospital, TR1 3HD Truro, UK
Peter Davidson, MBBS, MSc, MFPH (p.davidson@southampton.ac.uk), Director of Health Technology Assessment, NETSCC, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK
CONFLICT OF INTEREST
As employees of Southampton University, Peter Davidson coordinates the research program described in this article and Rebecca Moran worked as a research fellow until April 2011.