Economic evaluation guidelines in low- and middle-income countries: a systematic review

Abstract

Objectives

To systematically identify the latest versions of official economic evaluation guidelines (EEGs) in low- and middle-income countries (LMICs) and explore similarities and differences in their content.

Methods

We conducted a systematic search in MEDLINE (Ovid), PubMed, EconLit, Embase (Ovid), the Cochrane Library, and the gray literature. Using a predefined checklist, we extracted the key features of economic evaluation and the general characteristics of EEGs. We conducted a comparative analysis, including a summary of similarities and differences across EEGs.

Results

Thirteen EEGs were identified, three pertaining to lower-middle-income countries (Bhutan, Egypt, and Indonesia), nine to upper-middle-income countries (Brazil, China, Colombia, Cuba, Malaysia, Mexico, Russian Federation, South Africa, and Thailand), in addition to Mercosur, and none to low-income countries. The majority (n = 12) considered cost–utility analysis and health-related quality-of-life outcome. Half of the EEGs recommended the societal perspective, whereas the other half recommended the healthcare perspective. Equity considerations were required in ten EEGs. Most EEGs (n = 11) required the incremental cost-effectiveness ratio and recommended sensitivity analysis, as well as the presentation of a budget impact analysis (n = 10). Seven of the identified EEGs were mandatory for pharmacoeconomics submission. Methodological gaps, contradictions, and heterogeneity in terminologies used were identified within the guidelines.

Conclusion

As the importance of health technology assessment is increasing in LMICs, this systematic review could help researchers explore key aspects of existing EEGs in LMICs and explore differences among them. It could also support international organizations in guiding LMICs to develop their own EEGs and improve the methodological framework of existing ones.

Introduction

Given limited healthcare resources, decision makers need to determine which health interventions to adopt for optimal allocation in high-, middle-, and low-income countries. These decisions can be informed by health technology assessment (HTA), where economic evaluation (EE) is essential (1;2). Many countries have included EE as part of their health insurance reimbursement decisions (3;4). To allow transparency and comparison between EE results, economic evaluation guidelines (EEGs) are used as a standard for studies to be included in the application for reimbursement, a guide for designing and conducting a study, or to evaluate the economic study reports (5). Different visions and various reasons underlie the need for country-specific EEGs, such as differences in national contexts, health policies, disease priorities, and availability of data on costs and outcomes (6;7).

EEGs are well established in high-income countries (HICs), which have many years of experience in applying EE tools, and also in Australia, Canada, and the United Kingdom (8), and a large body of the literature has addressed this discipline in these countries (6;9–11). For example, in 2015, the European Network for Health Technology Assessment (EUnetHTA) issued ten main recommendations for EEGs based on a comparative review of EEGs in these European countries (6). On the other hand, little is known regarding the status of EEGs in LMICs, where healthcare resources are even scarcer and their efficient use is an absolute necessity. EEGs detail best practices for conducting economic evaluations; hence, they guide the identification, measurement, and comparison of the value and affordability of technologies in the health system and society (4). International and local organizations have explicitly expressed the need to develop EEGs in LMICs (2;12–15). Specifically, the World Health Organization (WHO) emphasized the need for the development of tools and guidance to support developing countries in the prioritization of health technologies (12;15). Similarly, the Professional Society for Health Economics and Outcomes Research (ISPOR) has committed to supporting health economics and outcomes research advancement in LMICs (13). The Bill and Melinda Gates Foundation funded the International Decision Support Initiative (iDSI) Reference Case, which is a framework for EEs of health interventions with a focus on LMICs (2). Likewise, the International Network of Agencies for Health Technology Assessment (INAHTA) created, in 2015, the Guidelines International Network (GIN) LMICs working group to explore methods to promote guideline development, adaptation, dissemination, implementation, and research within developing countries (14). The Guide to Health Economic Analysis and Research (GEAR) Web site points out that LMICs' governments are increasingly interested in pharmacoeconomics (16).

To date, few published studies addressed EEGs in LMICs (9;17). In 2017, Carapinha published a comparative review of pharmacoeconomic guidelines in South Africa and compared them with other middle- and high-income countries. Nevertheless, the authors did not employ a systematic search and selected only six middle-income countries. Zhao et al. (9) systematically reviewed pharmacoeconomic guidelines (PEGs) in LMICs, but the authors only included studies published in English and failed to include all middle-income countries such as Brazil, Colombia, Cuba, Indonesia, and Mexico. Recently, Sharma et al. conducted a comprehensive search for national EEG without undertaking a systematic search, while including only guidelines published in English without specifically focusing on LMICs (18). Sharma et al. failed to include Bhutan, Brazil, China, Colombia, Cuba, Mexico, and Russian Federation, in addition to Mercosur (18).

Due to the lack of a comprehensive view on the status and content of EEGs in LMICs, the current study aims to systematically identify and review EEGs in LMICs, to investigate which countries have such guidelines, and explore similarities and differences in their content. The ultimate goal of this paper is to assist health economic researchers and guideline developers in LMICs and provide material that support EEG development.

Methods

A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines (19), based on a protocol registered with the Open Science framework platform (https://doi.org/10.17605/OSF.IO/DHRYF).

Literature Search Strategies

We searched MEDLINE (Ovid), PubMed, EconLit, Embase (Ovid), and the Cochrane Library from inception date to 9 February 2020 and performed a literature search update in June 2020.

In the search, three key concepts were combined: “economic evaluation” AND “guidelines,” AND “low and middle-income countries.” For the last concept, we used the Cochrane filter 2012 (https://epoc.cochrane.org/lmic-filters) that we adapted to the 2019–2020 World Bank classification. For the three concepts, Medical Subject Heading (MeSH) and keywords were mapped. The strategy was validated by a medical information specialist. The full search strategies for MEDLINE, PubMed, and Embase are provided in Supplementary material 1.

Moreover, the gray literature was searched, including the Web sites of the World Bank, WHO, ISPOR pharmacoeconomic guidelines around the world, iDSI, GEAR, and Epistemonikos, as a source of systematic reviews. Besides, the Web sites of country-specific HTA agencies or the Ministry of Health were reviewed. A backward citation tracking for relevant systematic reviews and included guidelines was conducted.

The search was limited neither to a specific language nor to a publication date.

Eligibility Criteria

The latest versions of official EEGs, including PEGs and drugs guidelines, developed by the national agencies of relevant LMICs, were included.

Older versions of EEGs in LMICs, publications relating to the development process of EEGs, EEGs from high-income countries, EE studies and reports regarding diseases, and interventions, and other topics not relevant to guidelines were excluded. Nonoriginal documents, including posters, documentaries, meeting abstracts, and studies or expert opinions regarding EEGs, systematic reviews, and unofficial published guidelines, were also excluded.

Study Identification and Screening

Identified records were retrieved. To determine eligibility, titles and abstracts were screened by one reviewer (CDK) and full texts of selected references were then assessed by three pairs of reviewers (CDK/RK, CDK/RR, and CDK/JD). Documents in languages other than English, French, and Arabic were translated via online.doc.translator Web site. Discrepancies were resolved through discussions between each pair of reviewers (CDK/RK, CDK/RR, and CDK/JD), and when needed, a third reviewer, MH or SE, was involved.

Data Extraction

Data related to key features and essential criteria for developing EE studies were extracted based on the comparative table of PEGs developed by ISPOR (https://tools.ispor.org/peguidelines/COMP2.asp). Accordingly, details pertaining to the general characteristics of the studies, methods, presentation of the results, and discussion were retrieved. Three pairs of reviewers (CDK/RK, CDK/RR, and CDK/JD) extracted relevant data from the included studies independently and in duplicate using the predefined form. Disagreements were resolved through discussions.

Data Analysis

Reviewers who independently read texts of eligible documents and collected data into predefined forms formulated a qualitative synthesis of findings. These collected findings were independently refined into key themes such as perspective, time horizon, and preferred method, and presented separately. Discrepancies were resolved between reviewers through discussions or with the help of a third reviewer. Considering the heterogeneity of data, comparative summary tables were provided.

Results

Search Results

Figure 1 represents the PRISMA flow diagram of studies’ selection (19). In total, fifteen records were included, among them, thirteen were EEGs and two were budget impact analysis (BIA) guidelines. Following the literature update on June 2020, one article reporting the 2019 new PEG for Malaysia was included and replaced the 2012 version.

Figure 1. PRISMA diagram of study selection. Abbreviations: BIA, budget impact analysis; EE, economic evaluation; EEG, economic evaluation guideline; ISPOR, professional society for health economics and outcomes research; GEAR, guide to health economic analysis and research; LMICs, low- and middle-income countries.

General Overview

Based on the 2019–2020 World Bank classification, none of the thirty-one low-income countries (LICs) had EEGs. Of the middle-income countries, three out of forty-seven lower-middle-income countries (Bhutan, Egypt, and Indonesia) and nine out of sixty upper-middle-income countries (Brazil, China, Columbia, Cuba, Malaysia, Mexico, Russian Federation, South Africa, and Thailand) had EEGs. Moreover, Mercosur, a union of countries that includes Argentina, Brazil, Paraguay, Venezuela (upper-middle-income countries), Bolivia (lower-middle-income country), and Uruguay (high-income country, which is not covered by this review), has developed a common guideline (20–32).

EEGs General Characteristics

Table 1 summarizes the similarities and differences in the general characteristics of EEGs in LMICs. Country-specific details of each characteristic and key feature in each EEG are presented in Supplementary material 2.

Table 1. Summary of similarities and differences in the general characteristics of EEGs in LMICs

Among the thirteen identified guidelines, seven (20;21;23;28;29;31;32) were mandatory for pharmacoeconomic submission and five (22;24–27;30) were recommended for submitting EE studies. The majority of guidelines (n = 9) (20;21;23;24;26;28–31) were for drugs and other health interventions, whereas three (22;27;32) were specific for drugs, and one (25) had no specific purpose. Decision makers were the target audience for almost all guidelines (n = 12) (20–24;26–32), with six (23;24;28;30–32) including researchers as well, and five (20;22;26;27;29) including healthcare professionals and researchers along with decision makers, and only one (25) did not state the target audience. The majority of guidelines have developed a standard reporting format (n = 11) (20–24;26–32) and required the disclose of funding and authors’ interests (n = 10) (21–24;27–32).

EEGs Key Features: Methods

Table 2 summarizes similarities and differences in the methods in EEGs in LMICs.

Table 2. Summary of similarities and differences in the methods in EEGs in LMICs

Abbreviations: CUA, Cost–Utility Analysis; CEA, Cost-Effectiveness Analysis; CMA, Cost-Minimization Analysis; CBA, Cost–Benefit Analysis; QALYs, Quality-Adjusted Life Years; QoL, Quality of Life; HRQoL, Health-Related Quality of Life; Lys, Life-years; DALYs, Disability-adjusted Life Years; EQ-5D, EuroQol-5 Dimensions; SF-6D, Short-Form-6 Dimensions; SG, Standard Gamble; TTO, Time-Trade-Off; VAS, Visual Analogue Scale; HUI, Health Utility Index; QWB, Quality of Well-Being; SA, Sensitivity Analysis; DSA, Deterministic Sensitivity Analysis; PSA, Probabilistic Sensitivity Analysis.

^a Twelve EEGs were assessed for this key feature.

Preferred Perspective

Four guidelines (Colombia, Egypt, Mercosur, and Mexico) (23;25;28;29) considered the perspective of the healthcare system. Specifically, Colombia justified that this adoption is in line with the mission of the Ministry of Health and is due to the absence of reliable information to carry out studies from the societal perspective (23). Similarly, four guidelines (22;24;26;31) considered the societal perspective. Besides, two EEGs (27;30) preferred the payer or the budget holder perspective, and only one (21) specified both the society and the health system perspective. Finally, two guidelines (20;32) did not specify the preferred perspective.

Indication, Target Population/Subgroup Analysis, and Choice of Comparators

The indication of the health technology to be assessed was a required feature for the EE studies in the majority of included guidelines (n = 11) (20–23;25–27;29–32). All countries (n = 13) required the target population to be stated explicitly (20–32). Regarding the choice of comparators, the majority (n = 9) of the guidelines (20;21;23;26–29;31;32) required selecting the alternatives considered in the national current standard Clinical Practice or the standard treatment guidelines, whereas few (n = 3) (22;24;30) explicitly mentioned selecting the most cost-effective treatment options or the reimbursed healthcare technology (n = 1), along with the standard therapy (25).

Costs and Source of Costs

Over one-third of the EEGs recommended to include both direct and indirect costs (n = 5) (22;24;26;31;32), and four (23;25;29;30) required the inclusion of direct medical costs only. Few guidelines (n = 2) (21;27) recommended including costs that are relevant to the chosen perspective (without giving further specifications), and only one (28) recommended that direct costs related to the public health system be included in the evaluation. Finally, one (20) guideline did not specify the type of costs that should be included.

Almost all guidelines (n = 12) specified the source of costs that should be included in EE studies.

Preferred Analytical Method

One (20) of the thirteen guidelines (n = 1) did not specify the preferred method that should be used for the assessment. The majority of guidelines (n = 12) considered cost–utility analysis (CUA) as one of the preferred EE methods. Three of these twelve guidelines (24;28;30) considered that any method with justification can be used to assess the costs and outcomes of the health intervention under assessment, including CUA, cost-effectiveness analysis (CEA), cost-minimization analysis (CMA), or cost–benefit analysis (CBA). Another four (21;23;25;26) specified CUA as the preferred analytical method for assessment, and two (27;31) stated that CUA or CEA is the preferred analytical method, while requiring the justification of the choice. Similarly, for two EEGs (29;32), CUA, CEA, or CMA were among the preferred methods with justification. Finally, one guideline (22) stated that either CUA or CBA is the preferred method.

Preferred Outcome Measure and Method to Derive Utility

The twelve EEGs that selected CUA as the preferred method to develop EE had chosen the health-related quality of life (HRQoL) as the preferred outcome measure. Among these twelve guidelines, three-quarters (n = 9) (21–23;25–28;30;31) specified the preferred method to derive utility, out of which five (21;23;26;27;30) recommended applying the method on their selected population. Finally, only one guideline (20) preferred the clinical outcome (i.e., to express the outcome in natural units or final health outcomes).

Modeling

From the thirteen guidelines, five (21;22;26;29;31) required the use of modeling if needed to perform the EE, and more than half (n = 7) (23–25;27;28;30;32) recommended modeling when submitting the EE file. Only one EEG (20) did not mention modeling as a key feature.

Time Horizon

The majority of the EEGs (n = 12) (21–32) recommended that the time horizon should be long enough to cover all costs and outcomes. Only one (20) did not mention the time horizon required.

Systematic Review of Evidence and Preference of Effectiveness over Efficacy

Over two-thirds of the guidelines (Brazil, China, Colombia, Egypt, Indonesia, Malaysia, Mexico, South Africa, and Thailand) (n = 9) (21–23;25–27;29–31) preferred a systematic review to derive evidence. Some countries such as Malaysia clearly specified that the lack of local effectiveness data was behind the preference for systematic reviews (27). Four guidelines (20;24;28;32) did not specify the preferred method. More than half (n = 7) (21;22;24;26;28;29;31) preferred effectiveness data over efficacy, whereas six (20;23;25;27;30;32) did not show any preference.

Discounting Costs and Outcomes

The vast majority of included EEGs (n = 12) (21–32) recommended discounting of costs, and eleven recommended the discounting of outcomes. Out of these eleven EEGs, ten required the same discount rate for costs and outcomes. For most of the EEGs, the recommended discount rate ranged between 3 and 5 percent for costs and outcomes. Out of the twelve EEGs that required a discounting rate, nine (21–25;27;29–31) recommended to perform sensitivity analysis (SA) on discounting. In light of the absence of an empirical estimate of the discount rate in Colombia, conducting a SA on discounting was recommended (23).

Assumptions

A reasonable segment of included EEGs (n = 11) (21–31) required specifying all the assumptions presented and applied in the EE studies.

EEGs Key Features: Presentation and Discussion of the Results

Table 3 summarizes similarities and differences in the presentation and discussion of the results in EEGs in LMICs.

Table 3. Summary of similarities and differences in the presentation and discussion of the results in EEGs in LMICs

Abbreviations: ICER, Incremental Cost-Effectiveness Ratio; GDP, Gross Domestic Product.

^a Twelve EEGs were assessed for this key feature.

SA—Methods and SA—Parameters and Range

The majority of included EEGs (n = 11) (21–31) specified the methods required to perform SA, including univariate, multivariate, deterministic, probabilistic, threshold, or best-worst scenario analysis, whereas two (20;32) EEGs did not recommend SA in the economic studies.

The majority (n = 11) of EEGs (21–26;28–32) requested to conduct an SA on uncertain parameters, whereas only one (27) mentioned that the SA parameters and range was not stated as a key feature, even though it requested SA for the discount rate. Additionally, in one EEG (20), conducting SA for parameters was not required.

Results Presentation, Incremental Analysis, and Total Cost/Effectiveness Ratio

Almost all guidelines (n = 12) (21–32) recommended that the results be presented in the EE study, among which eleven (21–31) specified the inclusion of an incremental analysis through the incremental cost-effectiveness ratio (ICER).

The vast majority (n = 11) of the EEGs (21–23;25–32) proposed the calculation of the total cost/effectiveness ratio.

Portability of Results (Generalizability) and Equity Issue

Three-quarters (n = 10) (21–25;27–31) of the guidelines recommended a discussion on study generalizability.

Ten out of the thirteen EEGs (20–26;28;29;31) required EE studies to consider and state equity issues. Among these guidelines, only one (31) clearly described the social and ethical analysis in the EE, whereas the others simply required integrating equity considerations such as sociodemographic, age, and sex factors (25).

Financial or BIA

The majority of included EEGs (n = 10) (20–23;26–29;31;32) proposed that EE studies should include the BIA results. The requirements were provided in an independent guideline or article as Brazil, Russian Federation, and Thailand (21;31;32) or chapter or part as Indonesia, Malaysia, and Mexico (26;27;29), or included as a key feature (20;22;23;28).

Discussion

The inclusion of health economics evidence improves the opportunity to make better decisions on resource allocation and accessibility of healthcare interventions to patients (33). LMICs need to have affordable and accessible health interventions (2). Developing EEGs is an important step to demonstrate the clarity and consistency of EE studies at a national level (2). Similar to HICs, LMICs should tailor their EEGs to meet their health systems bodies’ needs, generate better decisions, and ultimately better health (2). This systematic review identified thirteen EEGs from LMICs, that is, nine upper-middle and three lower-middle-income countries, in addition to Mercosur, compared with ten previously identified by Griffiths in 2016, that is, nine upper-middle and one lower-middle-income countries (7). These results show that, to date, there is no evidence of EEG development in LICs, even though international societies and organizations are exhibiting support. Based on the International support, EEGs in LICs may arise in the near future (2;14;16). In middle-income countries, EEGs are still underdeveloped with only 15 percent of upper-middle-income countries, and Mercosur and 6.4 percent in lower-middle-income countries have developed their guidelines. More EEGs in LMICs are developing, such as Philippines EEGs, that have been identified after the search period of this study. Nevertheless, there seems to be a steady development in this field in some countries, specifically in Malaysia and Thailand, where updated guidelines have been published. Further research is needed to explore the reasons behind the lack of developing EEGs, and especially decisions relating to reimbursement and pricing regulation of pharmaceuticals are highly subject to political decision-making processes with interests beyond national health policy (8). Political dilemmas might delay implementation as well as EEG production (8).

Regarding the characteristics of the EEGs, the data showed that only 54 percent of countries had mandatory EEGs, whereas in most HICs, EEGs were mandatory (17). According to Drummond (3), national guidelines are mandatory to control expenditure for healthcare technologies and to ensure that funds are spent in the best possible way. Consequently, if presenting an EE document was not mandatory, it is believed that few applicants will work to present it. Additionally, most guidelines targeted decision makers and researchers, which is in line with the objective of EEGs for supporting decision makers and the standardization of the economic studies (5). According to Fasseeh (34), the Middle East and North Africa region, which includes several LMICs, is in need to strengthen its HTA capacity building (e.g., decision makers) by improving advanced technical skills and understanding of HTA (34). Similarly, EUnetHTA has developed guidance that can support decision makers to critically assess EEs (35).

Concerning the key features, our findings showed that one-third of the guidelines stated the preference to adopt the societal perspective. These results are in accordance with Zhao, who indicated that 35 percent of EEGs in some high- and middle-income countries were selecting the societal perspective (9). This perspective has been described as the most comprehensive one (35). Furthermore, two countries did not specify which perspective should be adopted for developing their EE studies, although these evaluations should be at conducted at least from a healthcare system perspective (35). To determine which costs are relevant to be included in the EE study, it is fundamental to specify the perspective (36). The most appropriate perspective is the perspective of those who commissioned, or who are intended to be informed by, the analysis (36). In a societal perspective, all costs of technologies should be identified, measured, and valued. Compared with the healthcare perspective, including only costs related to the healthcare sector, the societal perspective also includes informal costs, productivity losses, and costs borne by other sectors of the society (35;37). The societal perspective is the most comprehensive one and may affect the threshold value (37).

The preferred analytical method to be applied in EE studies was the CUA in the majority of the EEGs with the option of using another method with justification. This indicated that LMICs do not differ greatly from the EUnetHTA partners, whose EEGs preferred to present the results in terms of CUA, and CEA, CMA, and CCA can be used with choice justification (37). CUA offers the advantage to provide a similar and generic outcome measure that could be applied to the wide range of interventions and diseases a decision maker may consider, historically, concurrently, and in the future (3). Out of the thirteen-collected official EEGs, 92.3 percent considered the HRQoL as the preferred outcome in the analytical method, whereas 84 percent of EUnetHTA partners preferred QALYs as an outcome measure (6). These results suggest that LMICs’ societies, as other developed societies, have a deep interest in their patients’ quality of life, despite their increasingly scarce healthcare resources and the breakthrough of high-cost medicines. Even though LMICs prioritized HRQoL measures, our study demonstrated a lack in the methodological framework, because out of the twelve EEGs that required HRQoL as a preferred outcome, three did not specify the recommended method for measurement (19;24;27) and nine clearly specified and described it (16–18;20–23;25;26) as presented in Table 2. LMICs started developing their HRQoL instruments using preferences from their local population (21;26;27;30;31).

Most EEGs in LMICs such as Mexico and Thailand required information on equity impact in presenting EEs studies, with sociodemographic, age, sex, social, and ethical considerations, whereas others such as Bhutan required equity without any particular dimension. In recent methodological approaches, equity and efficiency are a key challenge in EE to appraise the equal access of all needed services for all the population (3;38–41). Furthermore, given the importance of health equity in the world policy agenda, the international health economics association (iHEA) has created an Equity Informative Economic Evaluation special interest group to support international decision makers who are facing equity dilemmas (40).

When presenting the results of the incremental analysis, variations in presenting the results between guidelines were detected. Six countries recommended ICER (24;25;27–30), but only five mentioned a threshold, three based on the gross domestic product (GDP) per capita threshold (22;23;26), and two having specific willingness-to-pay (WTP) threshold (21;31). Our results are aligned with those of the Finkelstein study, which showed the differences regarding the WTP threshold adapted in EE studies (42). To assess its value, ICER should be compared with a WTP or a cost-effectiveness threshold (CET) (37). Although a fixed CET cannot be used alone as a criterion for decision making, WHO recommended a transparent context-specific process for decision making (43).

According to the EUnetHTA HTA Core model, uncertainty in the EE results should be explored in a SA (37). Approximately three-fourths of EEGs in LMICs required the SA on parameters and range and the methods that are required to perform SA, whereas all EUnetHTA partners and all countries covered by Zhao recommended SA to explore uncertainty in the EE (9;37). These findings emphasized the lack of EEG methodology in some LMICs. The characterization of uncertainty and its analysis is essential in EE studies (44). SA provides information to decision makers about the robustness of their decision, as well as the need to collect more information before taking a decision (45).

The majority of the EEGs recommended the presentation of the BIA results to check the affordability of adopting a new intervention. Currently, BIA is mandatory to support reimbursement in many developed countries (46). In Australia, the financial impact of a new drug on the Australian budget anticipates the Pharmaceutical Benefits Advisory Committee (PBAC) decision on reimbursement (46). This recommendation is fundamental and represents the need for LMICs to identify the impact of new interventions in their limited budget. If a new intervention is more cost-effective in comparison with another intervention, it may not be affordable for reimbursement. LMICs' health systems are continuously struggling and budget constraints will limit their decisions in adopting new interventions.

This systematic review identified weaknesses and gaps in the included EEGs. Many methodological and essential key features, such as preferred perspective, type of costs, time horizon, and uncertainty/SA, were not addressed in some guidelines (20;32). Additionally, a lot of contradictions and variations in terminologies were detected between EEGs. In one EEG, it is stated in the reference case that SA parameters and range are unavailable; however, this guideline specifies SA values for discounting rate (27). These weaknesses in EEGs could adversely affect the quality of EE studies conducted in these countries and decision makers’ judgment. Moreover, some LMICs lead by example by launching the dynamic process of updating their guidelines with the aim to review their purpose and to address weaknesses identified through implementation in their health system (20;21;27;31). Developing EEGs is, per se, insufficient. It is important to have a formal institutional structure or process in which EEGs are used (7).

Strengths and Limitations

This work followed the recommended methods for conducting systematic reviews (19) and was executed following a protocol that was published a priori (https://doi.org/10.17605/OSF.IO/DHRYF). The search was broad and sensitive, and no restrictions were set for the language and publication date. Full-text screening and data extraction were done following best practices (in duplicate and using predefined forms).

One major limitation to this work is that it dismissed the EEGs published by Iran. A citation of these guidelines was identified on the ISPOR Web site; however, the full guidelines were inaccessible, even after contacting the ISPOR Iranian chapter. Moreover, it is always possible that our search could have missed some guidelines.

Conclusion

This systematic review of EEGs showed that initial work has started in LMICs. International organizations can launch their missions in supporting and guiding the development of EEGs in LMICs (2;13;14). Moreover, an improvement in the methodological framework of these guidelines is required. This work provides relevant material to assist health economic researchers and guideline developers in developing or refining high-quality EEGs that fit in their national health system context. Furthermore, an in-depth research is fundamental to understand the detected variability and to review how these guidelines were developed. Future studies should look at the development process of these guidelines, the expertise involved, and who guided and supported this process. Further research should explore the reasons behind the lack of EEGs in LMICs, the suboptimal requirements in some identified EEGs, the spotted differences between them, the factors that have driven the production of these guidelines, and the barriers and facilitators that developers have faced. Future challenge would be to have more convergence between countries given the large heterogeneity in existing EEGs, an issue arising worldwide and in Europe as well. Further work would be needed to identify the appropriate ways to improve it.