Internal evaluation of the European network for Health Technology Assessment project

Abstract

Objectives: The internal evaluation studied the development of the European network for Health Technology Assessment (EUnetHTA) Project in achieving the general objective of establishing an effective and a sustainable network of health technology assessment (HTA) in Europe.

Methods: The Work Package 3 group was dedicated to this task and performed the work. Information on activities during the project was collected from three sources. First, three yearly cross-sectional studies surveyed the participants’ opinions. Responses were by individuals or by institutions. The last round included surveys to the Steering Committee, the Stakeholder Forum, and the Secretariat. Second, the Work Package Lead Partners were interviewed bi-annually, five times in total, to update the information on the Project's progress. Third, additional information was sought in available documents.

Results: The organizational structure remained stable. The Project succeeded in developing tools aimed at providing common methodology with intent to establish a standard of conducting and reporting HTA and to facilitate greater collaboration among agencies. The participants/agencies expressed their belief in a network and in maintaining local/national autonomy. The Work Package Leaders expressed a strong belief in the solid base of the Project for a future network on which to build, but were aware of the need for funding and governmental support.

Conclusions: Participants and Work Package Leaders have expressed support for a future network that will improve national and international collaboration in HTA based on the experience from the EUnetHTA project.

The European network for Health Technology Assessment (EUnetHTA), also referred to in this article as the Network, started as a 3-year project in 2006 (4;8;9). It was called for by the European Union (EU) Commission following earlier European collaboration efforts (5;7). The plans of the EUnetHTA Project, also referred to in this article as the Project, included a requirement to perform an internal evaluation (2). The basic structure aimed at producing the necessary tools for the network's functionality and a governance and management structure through using dedicated working groups called Work Packages (WPs) (8;9). The Executive Committee developed a Proposal for the EUnetHTA Collaboration from 2009 and onward (1).

Internal evaluation was the dedicated task of WP 3, which aimed to study the development and use of resources during the 3-year period of the Project. This information was fed back to Work Package Lead Partners (WP LPs), individual participants, and partner organizations, and to the EU Commission.

The objectives of the internal evaluation were threefold: (i) To provide an audit function during the project with regular feedback to the European Commission and the project organization; (ii) To evaluate changes over time during the project period to show development toward the establishment of an effective and sustainable network; and (iii) To summarize lessons learned to support the effectiveness and sustainability of the network in its next phase, from 2009 and onward.

The Work Package 3 (WP3) planned two products: A framework for an external evaluation completed in June 2008 (Deliverable 1) which should include an overview of the Project's progress, documents, and deliverables produced. An Internal Evaluation Report (Deliverable 2) to be completed by end of 2008. This article concerns the results of the internal evaluation, which is fully described in the original WP3 Evaluation Report with a complete description of methods and results (2).

MATERIALS AND METHODS

Data Collection

The evaluation was designed with a prospective intent. Surveys and interviews were designed to give updated yet independent status reports of the Project. Three email surveys sought participants’ views on the objectives, progress, and contribution in the Project. Five rounds of interviews were conducted with WP LPs. Documents and other information were collected to provide additional information and understanding of the processes. WP3 group members discussed all tools and procedures before finalization.

Survey questionnaires, and reminders, were sent by e-mail annually (× 3) to all known participants. In total, the group sent 193 questionnaires in 2006, 181 in 2007, and 243 in 2008. The questionnaires covered views on the work in each WP specifically and on the EUnetHTA Project in general. The questionnaire contained closed questions with 1–3 or 1–5 Likert-scaled answers, open questions, and SWOT (strength, weaknesses, opportunities, threats) analyses of WP work and of the EUnetHTA Project. Additional questions were added to the third questionnaire after revision. Results from the participant surveys were computerized and quantitative data were analyzed using STATA and SPSS (11). The results of the participant surveys and WP LP interviews were published on the EUnetHTA Project intranet to give feedback to the network.

The WP 3 LP conducted the semistructured interviews with three open questions in the first three rounds (2). After the third round of interviews, the WP3 members reviewed the results and added a few more questions. Open questions were chosen to permit a free flow of information around the specific issues. WP LPs validated the text of the interviews before they were published on the EUnetHTA intranet to avoid any possible language inaccuracies or misunderstandings. To what extent the information was used in the work was not studied separately except the information given in the WP LP interviews.

Analysis

Group members (in pairs) analyzed the WP LP interview texts for content (10;12;13). The objective was to identify common themes and/or concerns about the EUnetHTA Project and its progress (6). Content analysis was first performed after the third round of interviews and repeated in later rounds.

Qualitative responses from the questionnaires were analyzed for content in the same manner as the interviews.

The questionnaires were originally sent to individuals by e-mails and reminders were sent. The returned questionnaires were by individuals and, as some agencies chose to do, by Partner agencies. A single response rate cannot therefore be calculated. The numbers of returned questionnaires were forty-five in 2006; forty-one in 2007; and sixty-three in 2008. In terms of the twenty-nine participating European countries, the response was received from sixteen European countries in 2006, thirteen in 2007, and fifteen in 2008. The responses by the total number of sixty-four Partners world wide were twenty-four agencies in 2006, seventeen in 2007, and twenty-two in 2008. These are cross-sectional data. A certain turnover of staff is to be expected in the participating institutions.

The results from the participant survey and WP LP interviews were triangulated using additional information from key documents. Data were judged against a set of criteria.

Criteria of Evaluation

The factors identified in the content analyses were evaluated against the general objective of the EUnetHTA Project as written in the Standard Operation Procedures (SOP) of September 2006 (amended June 2007) (4):

“The general objective of the EUnetHTA is to establish an effective and sustainable European network of Health Technology Assessment to inform policy decisions. The overall strategic objective is to connect public national health technology assessment (HTA) agencies, research institutions and health ministries, enabling an effective exchange of information and support to policy decisions by Member States.”

The criteria used to judge the overall effectiveness of the EUnetHTA Project were sevenfold, and supporting evidence was sought in the available documentation: Production of deliverables in a timely manner, Effective working collaboration among Work Packages, Degree of participation within Work Packages, Effective communication, Sustained commitment to the Project, User and stakeholder satisfaction with new routines and practice, and Perceived added value.

FINDINGS AND DISCUSSION

Survey Response

During the 3 years, survey questionnaires were sent to from 181 to 243 individual participants from sixty-four organizations and only between forty-one to sixty-three were returned. The number of participating individuals varied considerably across Partners, and some of the Partners returned only one questionnaire to reflect the views of their organization. Thus, the survey results should be interpreted with some caution; but they do provide valuable information on a range of participant's views for the evaluation together with the interviews and available documents.

Production of Deliverables in a Timely Manner

The WP LPs were expected to have an overview of the work in their WP and its progress. The combined response from the eight WP LPs and the participants was that their work progressed much as planned, although some problems were reported. These were dealt with and necessary adjustments were made. The participants/agencies supported the notion that the work would improve future HTA reports (Figure 1). Few disagreed and the “strongly agree” group increased from 7.0 percent in 2006, 12.2 percent in 2007, to 14.5 percent in 2008. The “agree” response group decreased during the period and the “neither agree, nor disagree” group increased.

Figure 1. Results of participant survey on the question: “EUnetHTA will improve the quality of HTA reports.”

The Project as such was completed in time with regard to the deliverables and milestones, although a few deliverables were delayed. However, this did not influence the total production of tools within the given time frame.

The project phase included sixty-four Partners representing diverse types of institutions when it ended in 2008. Hence, it is fair to say that the EUnetHTA Project achieved one of its strategic objectives to connect public, national health technology assessment (HTA) agencies, research institutions, and health ministries in its collaboration. This laid the foundation for collaboration and exchange of information.

Effective Working Collaboration among Work Packages

The tools to be developed by the Project were, to some degree, interconnected and required effective collaboration within and between the WPs. In practical terms, participants believed their goals were unclear and the processes somewhat abstract. This reflects the developmental nature of the work and the need to produce new tools. However, all Partners did not contribute equally; many participants were unable to contribute as much as they originally planned, others contributed more than anticipated. On occasion, the large groups, due to their sheer size, caused “production” to slow down. The work depended on the participation of a large number of Partners, working in various settings in different countries. As a consequence, they had a different kind of involvement in HTA. The EUnetHTA consists of different kinds of Partners whose involvement in HTA differs in nature and intensity. Moreover, Partners differ in interests, available resources, organizational structures, and competencies. While they provide a strong pool of resources, the organizational difficulties this poses in running the Network should not be underestimated. In addition, some concern was raised as to the overlap with other networks such as the International Network of Agencies for Health Technology Assessment (INAHTA), the Health Evidence Network (HEN), and the International Information Network on New and Emerging Health Technologies (EuroScan).

Degree of Participation in Work Packages

Participants expressed their expectations of learning from the international cooperation and collaboration. The Network improved communications and the ability to interchange useful information. The project idea generated high expectations because the work would require a range of experiences and skills, and the Partners would bring different perspectives into the work. Methodological issues and learning from experiences of others were viewed as added benefits of joining the WPs. Participants experienced fruitful collaboration with some parties, while others were less active. Dedicated teams took a nonpartisan attitude toward achieving the goals for the Network's success. However, it became evident to participants that a clear work plan was important together with a good leadership, clear objectives, attractive tasks, and adoption of different methods and perspectives when assessing the Project's global value.

Leadership suggested coordinating different groups and the ability to maintain good and frequent communication. The survey responses varied across WPs. Some disagreements and different anticipations were reported regarding how tasks were to be performed. Attaining knowledge and experience in international/European HTA networking, coupled with communication and project network management skills, constituted major added benefits of taking part in WP activities.

Effective Communication

Good communication is a major issue in establishing and running a network as large as the EUnetHTA collaboration. For example, the new HTA Core Model needs to be clearly explained to all the actors that intend to use it. Acceptance of the HTA Core Model among European HTA agencies and its feasibility in daily work still remains to be seen. Translating the intentions of developing a toolkit into something that is genuinely practical and saves time and money is complicated and involves major conceptual issues. Language barriers are real and must be addressed (Figure 2). The proportion who responded “yes” when asked if language is a barrier to networking increased from 9.5 percent in 2006, to 12.2 percent in 2007, to 16.1 percent in 2008. The proportion responding that language is “partly” a problem was greatest in 2007, but decreased in 2008.

Figure 2. Results of participant survey on the question: “Is language a barrier?”

Partners differed in their starting level of HTA knowledge, their goals, and their expectations from the Project. Different backgrounds of the Partners may have led to possible misunderstandings. E-meetings were not as productive as face-to-face meetings. Personnel turnover can be viewed as a threat because new people need an introductory phase. The Clearinghouse had to be reorganized and is now part of the Communication Platform and under further development. Information technology problems existed as well as language problems.

Although e-meetings were the intended format for most WP meetings, this did not always happen. Many participants’ computer systems did not communicate well due to, for example, firewalls.

An unanticipated legal problem arose in one WP concerning the exchange of information between countries, which took time to resolve.

Sustained Commitment to the Project

Financing was an issue throughout the EUnetHTA Project period. The EU Commission reduced the initial budget submitted by each LP. This placed a greater economic and planning burden on LPs. Their commitment to the Project caused the LPs to search for alternative funding, or cut the program, or support the Project financially through their own agency's funding. This lack of resources was most difficult for small units/institutes. Some EUnetHTA funds were reallocated in the last year of the Project to partly compensate this.

Regarding the workload, participants often responded: “Too much workload compared to available resources.” This related to the large volume of documents to read, too many questionnaires, and many e-meetings, totaling a greater time expenditure than originally planned. “It is a continuous challenge to make sure that the work is not becoming too big and diffuse and at the same time useful for the EUnetHTA Project and the future collaboration.”

The difficult financial situation meant that, in some agencies, human resources were probably insufficient to carry the workload. These few comments reflect the situation. However, the willingness from agencies to put in extra resources is an indication of their support for the Project.

User and Stakeholder Satisfaction with New Routines and Practice

Participants expressed belief that the Project would lead to improved HTA reports (Figure 1). SWOT analyses and WP LP interviews supported the positive attitude toward the Project over the 3-year period. Tools and working processes are under development and have not yet been tested in real work settings. It was, therefore, too early to judge the Project's performance against this criterion.

Perceived Added Value

We found support of the view that EUnetHTA Project will add value through improved HTA reports (Figure 1), and that the EUnetHTA collaboration will improve the implementation of HTA reports (Figure 3). The proportion of the combined responses of strongly agree, agree, and neither agree, nor disagree increased slightly through the project period. Likewise, we observed a slightly increasing trend in support of the EUnetHTA collaboration strengthening the national use of HTA as a working method in the health services of the respective countries (Figure 4).

Figure 3. Results of participant survey on the question: “The network will improve the implementation of the HTA reports in the health service of my country.”

Figure 4. Results of participant survey on the question: “The participation of my institution will strengthen the use of HTA as a working method in the health service of my country.”

CONCLUSIONS ACCORDING TO THE SPECIFIED CRITERIA

The results relating to the criteria were extracted from the survey data and the interview text in a systematic manner and were substantiated with information from relevant documents produced during the project period. The evaluation, with respect to the seven specific criteria, concluded:

(i) Production of Deliverables in a Timely Manner

With a few exceptions the deliverables were completed by the scheduled time. The tools that were developed were designed to facilitate networking in preparing HTA reports. There was room for adjustment and development of ideas. However, the HTA tools have not yet been tried under real working conditions. The Network will continue piloting these tools. The EUnetHTA Project was carried out according to plan, maintaining its organizational structure, and the tools were developed within the timeframe.

(ii) Effective Working Collaboration among Work Packages

Collaboration among WPs was necessary as the development of the tools were interconnected. This was demanding in terms of timing and cooperation among parties involved, and should be improved by the Collaboration.

(iii) Degree of Participation within Work Packages

The large number of people representing many different organizations involved in the WPs, coupled with the heavy workload on participants, were perceived to be a cause of delays in deliverables. Working in the WPs was considered to be of benefit in terms of international experience, exchange of knowledge, and developing the tools.

(iv) Effective Communication

Different means of communication were used, with e-mail being the most common. Meetings were organized as face-to-face, telephone, and e-meetings. Language was a complicating and challenging factor. The large number of participating organizations represented a challenge in communication, and participants reported variations in the degree of success regarding communication methods.

(v) Sustained Commitment to the Project

Only one organization left the Network (at the start), and four joined during the project period. Commitment was present throughout. In many instances, greater input was observed from participants and organizations than had been planned. Moreover, some organizations committed extra funding to the Project to help in constructing the tools. However, the contributions of participants varied.

(vi) User and Stakeholder Satisfaction with New Routines and Practice

Participants’ positive attitude toward the Project increased over the period. However, because the tools and processes had not yet been tested in real work settings, it was too early to judge the Project's performance against this criterion.

(vii) Perceived Added Value

The participants expressed belief that their agency's involvement in the EUnetHTA Project was positive. They were in support of the EUnetHTA collaboration remaining a network, and that it should not become a centralized organization, which would undermine local/national autonomy. Dedicated WP LPs and the Secretariat were considered instrumental in helping the EUnetHTA Project achieve its objectives to date and for supporting its future development into a sustainable network from 2009 and beyond.

RECOMMENDATIONS FOR THE FUTURE NETWORK

The following are recommendations going forward: (i) Secure funding and maintain a dedicated coordinating Secretariat; (ii) Ensure efficiency through an organizational structure made up of Work Packages managed by a core of dedicated Partners, with less committed Partners taking part as a wider review group; (iii) Continue developing and evaluating the tools as necessary and in real settings; (iv) Involve people in the work to ensure commitment, a high level of knowledge, and a broad basis for decision-making processes; (v) Encourage collaboration and communication among all parties to ensure coherence within groups and within the EUnetHTA collaboration; (vi) Continue developing the communication platform and clearinghouse functionality to make the EUnetHTA collaboration the central reference point for HTA in Europe; (vii) Arrange face-to-face meetings at the outset of group or committee work to strengthen social coherence and reach a common understanding of the work; (viii) Evaluate the technical communication platform; and (ix) English has been the main language and should continue to be so.

CONTACT INFORMATION

Lise Lund Håheim, DDS, Dr.Phil. (llh@nokc.no), Senior Researcher, Department for Quality Measurement and Patient Safety, Norwegian Knowledge Centre for the Health Services, Pilestredet Park 7, Oslo, N-0130, Norway

Iñaki Imaz, MD, PhD, MPH (imaz@isciii.es), Senior Researcher, Agencia de Evaluación de Tecnologías Sanitarias, Instituto de Salud Carlos III, 6 Sinesio Delgado, Madrid, Spain 28029

Marlène Läubli Loud, DPhil (marlene.laeubil@bag.admin.ch), Head of Evaluation and Research, Health Policy Unit, Federal Office of Public Health, Schwarzenburgstrasse 165, 3097 Liebefeld BE, Switzerland; Lecturer, Department of Social Policy and Social Work, University of Fribourg, Rte des Bonnesfontaines 11, 1700 Fribourg, Switzerland

Teresa Gasparetto, MSc (teresa.gasparetto@regione.veneto.it), Health Care Economist, Health and Social Planning Department, Veneto Region, Palazzo Molin, San Polo 2514, Venice, Italy 30125

Jesús González-Enriquez, MD (jgonza@isciii.es), Senior Researcher, Agencia de Evaluación de Tecnologías Sanitarias, Instituto de Salud Carlos III, 4 Sinesio Delgado, Madrid, Spain 28029

Helena Dahlgren, MBA (dahlgren@sbu.se), Deputy Director, Swedish Council on Technology Assessment in Healh Care, P.O. Box 3657, SE-103 86 Stockholm, Sweden

Igor Trofimovs, MD (igorst@apollo.lv), Head of Department, Department of Strategic Development and Projects, Riga Eastern University Hospital, 2 Hipokrata Street, Riga, LV-1038, Latvia

Elena Berti, MD (eberti@regione.emilia-romagna.it), Assistant, Clinical Governance Area, Agenzia Sanitaria e Sociale Regione Emilia-Romagna, 21 Via Aldo Moro, Bologna, Italy, 40127

Berit Mørland, DDS, Dr. phil. (beritt.morland@nokc.no), Deputy Director General, Norwegian Knowledge Centre for the Health Services, P.O. Box 7004, St. Olavs plass, N-0130 Oslo, Norway