In 2003, the Ministry of Health and Long-Term Care (MOHLTC) in Ontario, Canada's largest province (population 13.6 million), recognized the need for a system to address the uptake and diffusion of nondrug health technologies. The Ontario Health Technology Advisory Committee (OHTAC) was created to serve as a single provincial portal for recommendations on the introduction of nondrug health technologies (Reference Johnson, Sikich and Evans1).
Health technology assessment (HTA) includes several phases: (a) prioritization of the health technology; (b) assessment of relevant evidence; (c) deliberation and recommendation development; and (d) dissemination and elicitation of stakeholder views (2).
Within OHTAC, methods for systematically considering all of the relevant types of evidence in the appraisal process (phase III) have continued to evolve. In January 2007, OHTAC developed a decision-making framework that included four key attributes: (a) overall clinical benefit, (b) consistency with expected societal and ethical values, (c) value for money, and (d) feasibility (system feasibility and economic feasibility) of adoption into the health system (Reference Johnson, Sikich and Evans1). This framework offered a transparent, multi-disciplinary and consistent approach to making decisions in a deliberative manner. The framework recognized the importance of different forms of evidence, and included ethical considerations and social values (Reference Johnson, Sikich and Evans1). OHTAC's approach has been adapted and used by other HTA groups (3;4).
Several operational and conceptual issues prompted a re-evaluation of the framework. Assigning clinical evidence into ordinal categories (e.g., strength of evidence: high versus medium versus low) proved to be difficult in practice due to lack of specific guidance. Systematic collection of evidence related to societal values and ethics was not routinely undertaken; instead colloquial evidence from available experts was used. Feasibility considerations were usually not supported by direct evidence.
Conceptually, the growing importance of patient perspectives and the international prominence of other decision and value frameworks also highlighted the need to reconsider the conceptual foundations of our work. Decision frameworks rooted in clinical evidence (e.g., GRADE) (Reference Alonso-Coello, Oxman and Moberg5) are being widely used in clinical, public health, and policy contexts. Multi-criteria decision frameworks such as EVIDEM (Reference Goetghebeur, Wagner and Khoury6) are increasingly being explored. Value frameworks promulgated by clinical specialty groups (Reference Anderson, Heidenreich and Barnett7;Reference Schnipper, Davidson and Wollins8) are receiving increasing attention, particularly in the United States. Each of these approaches includes a (often implicit) theoretical framework, a set of decision attributes, and a method to integrate them into decision making.
Therefore, OHTAC requested a re-evaluation of our decision-making process. In January 2012, OHTAC convened two subcommittees: (a) a Public Engagement Subcommittee, which began the process of developing a patient and public engagement strategy (Reference Abelson, Wagner and DeJean9); and (b) a Decision Determinants (DD) Subcommittee to develop a decision-making framework, called the “Decision Determinants Framework.”
METHODS
Decision Determinants Subcommittee Objectives
The objectives of the DD Subcommittee were to (a) critically review OHTAC's existing decision determinants framework (2007), and (b) provide advice on potential theoretical and practical improvements.
OHTAC members and relevant experts from bioethics, evidence based medicine, clinical medicine, social science, and health economics, were brought together to provide consensus recommendations for an update of the current decision framework. The committee met monthly for a period of 3 years. Our methods included: (a) Consensus on a set of guiding principles; (b) A scoping literature review, including a review of the gray literature and Web sites of international HTA organizations to determine a comprehensive set of decision attributes used in HTA decision making; (c) Solicited presentations from experts in HTA from a variety of disciplines, such as evidence-based medicine, health economics, decision analysis, bioethics, and health-policy; and (d) Deliberations at monthly meetings between March 2012 and December 2015.
Guiding Principles
A Values-Based Framework
The DD Subcommittee took the view that alignment with social values represents the foundational principle for decision making regarding health technology. The committee believed that all decision criteria, and all decision processes used in the HTA process should be rooted in values about health, and more specifically, about the use of technology. The perspective that social values play a primary role in decision making is consonant with the views of other HTA bodies, which recognize the foundational role of social values (10–Reference Whitty and Littlejohns12).
An inquiry into social values commissioned by OHTAC was taken as a starting point to define the social values that are relevant to the work of the committee (Figure 1) (Reference Giacomini, Wagner and Krahn13;Reference Miller14). Giacomini generated a list of consensus values that included: 1. quality, a meta-concept referring to the achievement of the good and continuous striving for improvement; 2. evidence-informed policy; 3. effectiveness; 4. resource stewardship; 5. resource sufficiency; 6. equity; 7. solidarity; 8. population health; 9. patient-centered care; 10. collaboration; and 11. shared responsibility for health. Values 1–5 are more explicitly linked to traditional HTA practice. These values are discussed further in the Scientific Paradigms section.
Figure 1. Social values relevant to the work of Ontario Health Technology Advisory Committee (OHTAC). EUnetHTA, European Network for Health Technology Assessment; HTA, health technology assessment; IJTAHC, International Journal of Technology Assessment in Health Care; INAHTA, International Nerwork of Agencies for Health Technology Assessment; MOHLTC, Ministry of Health and Long-Term Care.
Simplicity
Supplementary Table 1 identifies over 150 health concepts relevant to decisions about health technology. However, research into human cognition has shown that it is difficult to keep more than five to seven concepts in mind at any time (Reference Miller15;Reference Redelmeier, Rozin and Kahneman16). We, therefore, believed that our framework was likely to be useful in guiding decision making to the extent that it reduced the number of factors considered in decision making.
A Guide to Deliberation, not a Substitute for Deliberation
Deliberation, in the context of HTA, refers to a type of discussion in which there is a careful weighing of reasons for and against some proposition (Reference Abelson, Forest and Eyles17). Collective problem-solving is the critical element of deliberation in which individuals from different backgrounds and experiences are given the opportunity to listen, understand, and potentially persuade. This may lead to more reasoned, informed, and socially legitimate decisions (Reference Stafinski, Menon and Marshall11). We believed that the deliberative process was central to decision making that was fair, wise, reproducible, and grounded in public values.
Decision Attributes, Decision Rules, and Process
Decision frameworks include three components: (a) attributes that define the scope of decision making; (b) decision rules for those attributes, which may be formal and explicit, as in the case of multi-criteria decision analysis, or less formal; (c) the process by which decisions are made. Process includes components, such as the composition of the committee; how information is gathered, synthesized and presented, and by whom; how the deliberation is structured; and how recommendations are formulated and disseminated. Though we understood our remit to include all three components, recommendations about process will be reported elsewhere.
Scientific Paradigms – From Values to Methods
Although there is little literature describing the theoretical foundation of health technology, it does draw from a range of disciplines including evidence based medicine, health economics, bioethics and the interpretive social sciences.
The DD Subcommittee explicitly recognized that these different disciplines differ in terms of the questions they ask, the kind of empirical data they seek, the analytic techniques they use, the academic background and training of researchers within these domains, and the social institutions that disseminate their ideas. For example, within evidence-based medicine: (a) practitioners are often clinicians; (b) clinical outcomes (e.g., death, myocardial infarction) are privileged; and (c) epidemiological methods, such as randomized trials, are regarded as the most reliable source of data. Within health economics: (a) practitioners are often social scientists; (b) cost and preferences for health outcomes, in addition to clinical outcomes are privileged; and (c) model based analyses that integrate varied types of data are regarded as the optimal method to support decision making. In bioethics and social science: (a) practitioners are often philosophers or social scientists; (b) patient experience, power relationships, and ethical principles are privileged; (c) methods include ethical reasoning and qualitative (and occasionally quantitative) studies of experiences, values, and preferences.
We suggest that each of the three major “paradigms” that are attached to HTA differ in key areas: “what is to be observed and scrutinized, the kind of questions that are supposed to be asked, how these questions are to be structured, and how the results of scientific investigations should be interpreted.” These are four key features of paradigms, according to Thomas Kuhn, the philosopher of science with whom the term “paradigm” is most associated. Kuhn says that a paradigm is “an entire constellation of beliefs, values and techniques…shared by the members of a given community” (Reference Kuhn18).
The committee recognized that each of these “scientific paradigms” (Reference Kuhn18;Reference Dogan, Smelser and Baltes19) provide independent, valuable, and complementary ways of thinking about health and health technology. Each has developed rigorous methods to evaluate some component of health technology. Most importantly, however, each of these paradigms can be linked to one or more fundamental social value.
Evidence based medicine is linked most closely to values of effectiveness, evidence informed policy, and quality. Health economics is linked to resource stewardship, resource sufficiency, and population health. By explicitly representing the relationship between constrained resources and health, health economics highlights the importance of making decisions such that the overall population can benefit. Social science/bioethics is most clearly linked to values of equity, solidarity, and patient-centered care. This paradigm reflects the importance of fairness in both process and outcome.
The subcommittee wanted to ensure that these paradigms were appropriately represented within the revised framework. Thus, we divided the work of the committee into three working groups based on their shared epistemological perspective and methodological approach.
LITERATURE REVIEW
A scoping review was conducted to identify an exhaustive list of decision attributes, concepts, or ideas used in HTA decisions. We identified 1–12 sub-attributes or related health concepts for each of the twenty-eight high-level attributes, for a total of 157 health concepts.
Presentations
In the first year, the committee heard a wide variety of presentations from key informants with expertise in the areas of decision analysis, health economics, evidence-based medicine, bioethics, and health policy. Each presented their experience with a specific decision-making framework, including EVIDEM and other multi-criteria decision frameworks (Reference Goetghebeur, Wagner and Khoury6), GRADE (Reference Alonso-Coello, Oxman and Moberg5), and frameworks used by the Pan-Canadian Oncology Drug Review and Ontario Committee to Evaluate Drugs.
Deliberation
The literature review and the solicited presentations provided an opportunity for the subcommittee to discuss how other frameworks might be adapted. Between March 2012 and December 2015, monthly meetings were held by the DD Subcommittee. In addition, the working groups conducted further literature searches into their sub-domains and held their own deliberations. The working groups were responsible for assembling a list of relevant criteria within their domain, as well as describing a suitable methodological approach for generating judgments using those criteria. Each working group also pilot-tested the decision criteria and presented their work at monthly committee meetings.
RESULTS
Multi-criteria Decision Making, but not Multi-criteria Decision Analysis (MCDA)
There was little conflict around the idea that the framework should incorporate discrete decision criteria or attributes. Thus, the committee was committed to a multi-criteria decision-making approach, in which the attributes relevant to decision making serve as the organizing principle for decision making (Reference Devlin and Sussex20). However, the committee did not endorse an MCDA approach, in which explicit attribute weights and decision rules are used to determine the optimal choice.
Some members noted that explicit weights can standardize the decision-making process and potentially reduce bias. Although use of weights can be formulaic and/or prescriptive, in practice, it need not be. Instead, it can support decision making by stimulating reflection and exchange and by making the decision process more explicit. This approach may also mitigate the effects of influential committee members. However, the consensus was that structuring decision making in this way introduces a degree of rigidity into the process that was, on balance, undesirable. One concern was that the score would produce a strong “framing” effect. Another related to independence: in formal MCDA models, criteria should be independent. Some members expressed the view that a quantitative tool may result in placing undue weight on the elements of HTA that are more easily measured. Patient experiences and broader considerations of social value might not receive appropriate consideration and decision weight.
Opportunity Cost in Health and Other Domains
The issue of opportunity cost was vigorously, even hotly debated. The conflict generated by this discussion led to the resignation of several members of the committee. Several of the health economists on the committee believed that opportunity cost was such a central concept that it superseded the role of other decision attributes. They argued that any decision to adopt a particular technology imposed an opportunity cost associated with technologies or interventions that were foregone or displaced. This opportunity cost is usually represented as health foregone, but in a multi-criteria decision framework, opportunity cost might also fall on other domains, such as benefits and risk, or equity. Thus, they argued, the implications of a choice (opportunity cost) were fundamental considerations in arriving at any decision rule, irrespective of which attributes were used.
Other members, while acknowledging the centrality of opportunity cost, also recognized that determining where opportunity cost falls in the Canadian health context is extremely difficult. The extent to which costs imposed on the health budget displaces health, or fall on other budgets is unclear. The committee could not identify any method of operationalizing the concept of opportunity cost in other domains. The committee recognized the pioneering work being done in the United Kingdom to establish an empirical foundation for an estimate of opportunity cost within the health system (Reference Claxton, Martin and Soares21), and strongly endorsed the need for similar work to be done in Canada.
Overall Outline of the Decision Framework
The committee considered that the instrument should have three main categories: (I) context criteria, (II) appraisal criteria, and (III) feasibility criteria.
I. Context Criteria
The committee recognized the importance of framing or contextual considerations in evaluating new technology (Table 1). Understanding who has requested the review and why (i.e., stakeholder issues and pressures), the nature of the technology, and the patients and populations who might benefit, are important. Coverage decisions and availability of the technology in neighboring jurisdictions were also seen as important, as were the scope of the technology appraisal and the affiliation of those who conducted it. The committee did not believe that context criteria should be considered to be primary appraisal criteria.
Table 1. Contextual Factors and Health System Feasibility Components of the Revised Framework
II. Appraisal Criteria
i. Benefits and Harms
The name of the domain was changed to “Benefits and Harms” from “Clinical Evidence,” because the latter term suggested a focus on strictly biomedical interventions and excluded items such as public health interventions (Table 2). The framework now refers to “harms” rather than to “safety.” The subcommittee believed that changing this terminology captures a broader range of types of health interventions and studies.
Table 2. Benefits and Harms Component of the Revised Framework
QALY, quality-adjusted life-year.
Magnitude and Certainty of Benefit and Harm
The new framework explicitly considers both the magnitude of benefit and the certainty regarding this benefit. Magnitude refers to the size of health benefit (e.g. absolute or relative risk reduction, quality-adjusted life-year [QALY] gain). Certainty refers to confidence that benefit will actually be achieved. Uncertainty may be statistical (or random) uncertainty (quantified with p-values or confidence intervals) or bias (systematic uncertainty). The new framework similarly considers the magnitude and certainty of harm.
The selection of the most appropriate measure of benefit should be ascertained at the time of evidence review and will ideally be guided by the availability of data, the opinions of experts, and input from patients (see below).
Representation of estimates of the certainty of benefit and the certainty of harm are related to the GRADE assessment of overall certainty. GRADE assesses uncertainty by explicitly considering both the precision of the estimate and potential biases (Reference Keeney22).
Patients’ Perspectives
The new framework incorporates patients’ perspectives in the Benefits and Harms domain. A frequent criticism of outcomes reported in RCTs is that they may not fully reflect the priorities and values of people who are directly affected (Reference Schoemaker23;Reference von Neumann, Morgenstern and Kuhn24). Patient input is needed to put such outcomes into perspective, particularly when nonmortality related outcomes are being considered, or when the minimum clinically important difference of an outcome measure has not been established.
The inclusion of “patient perspectives” in the Benefits and Harms domain raises questions of double counting (see the Patient Centered Care Domain section, below). This issue was discussed at length. Our final recommendation was that patient perspectives be included here because: (a) the focus on patient perspectives of Benefits and Harms recognizes that such assessments require value judgements, for which patients are perhaps the most authoritative source of information; and (b) patient input is gathered routinely for every assessment, whereas a more detailed assessment is recommended only for selected technologies.
ii. Economics
Cost-Effectiveness Likelihood
The committee suggested that five categories of cost-effectiveness likelihood be implemented as follows (Table 3): (a) highly likely to be cost-effective (80–100 percent), (b) moderately likely to be cost-effective (60–79 percent), (c) uncertain cost-effectiveness (40–59 percent), (d) moderately likely to not be cost-effective (20–39 percent), and (e) highly likely to not be cost-effective (0–19 percent).
Table 3. Economics and Patient-Centered Care Components of the Revised Framework
CE, cost-effectiveness; QALY, quality-adjusted life-year.
Cost-Effectiveness Threshold Values
OHTAC does not have a threshold for determining cost-effectiveness. Until an empirical threshold can be determined, the committee proposed the use of two illustrative ICER thresholds: (a) $50 K/QALY, and (b) $100 K/QALY.
Adequacy of Cost and Outcome Measures
Recognizing that the cost and outcome measures used to derive cost and effect estimates may not capture all relevant information, may be of poorer quality or may have varying levels of relevancy to the Ontario context, the next component of the framework consists of grading of the perceived adequacy of the cost and outcomes measures included in the economic assessment. Specifically, the revised framework includes an input for the user to determine the: (a) comprehensiveness and validity of the cost and outcome valuation/ aggregation; and (b) appropriateness of the cost and outcome measures for Ontario.
Comprehensiveness refers to the completeness with which resource usage estimation was conducted (e.g., were important costs omitted?), or the extent to which important health outcomes were captured (were important health effects omitted?). Validity may refer to either the quality of the decision model, or the appropriateness of the QALY measure in the decision context. For example, the QALY model may not be congruent with social values in circumstances such as end of life care, or “lifestyle” treatments.
Appropriateness refers to the generalizability of the analysis to the Ontario context, including particularly resource use and valuation, and patterns of care.
Summary of Cost-Effectiveness Likelihood
As a final step, the framework combines the quantitative cost-effectiveness results with assessment of comprehensiveness and appropriateness to provide a summary measure for the economic domain.
The summary measure is identical to the previously-mentioned categories but includes a possible “down-grading” of the economic rating if cost, outcomes, or the model itself are considered to be inadequate.
iii. Patient-Centered Care
The previous category of “Social and Ethical Values” was rechristened as “Patient-Centered Care” (Table 3). This new characterization was partly strategic, but it was also recognition of the primacy of the patient's experience in making health decisions.
The Patient-Centered Care subcommittee identified a set of five values of particular salience to OHTAC's commitment to ethics and social values. These include “equity,” “solidarity,” “population health,” “collaboration,” and “shared responsibility for health” (Reference Anderson, Heidenreich and Barnett7).
Trigger Tool
The subcommittee developed a “trigger tool,” adapted from the European Network for Health Technology Assessment (EUnetHTA) model, to determine when a full ethics and social values analysis is warranted. We chose to develop a trigger tool because resources are not available at HQO for full assessments for all technologies. Selected trigger components include: (a) technology decision is identified as being value (or preference) sensitive; (b) use by vulnerable or marginalized populations; and (c) screening interventions; (d) autonomy, privacy, or confidentiality issues especially salient.
With respect to methods, the subcommittee recommended the use of scoping reviews, using a variety of methods. These include: (a) qualitative research synthesis; (b) research synthesis related to health equity; (c) research synthesis for ethics studies; and (d) research synthesis for quantitative evidence of patient preferences (25).
Based on the results of the scoping review, the issues related to alignment with patients’ values and preferences, autonomy, privacy and confidentiality, equity in access or outcomes, and coordination of care (Table 3) should be identified. As with the other domains one of five ordinal categories (strongly supports to strongly discourages use/adoption of the technology) will be selected, based on the body of preference and values evidence identified.
III. Feasibility Criteria
In addition to scientific evidence, the committee considers aggregate budget impact and organizational considerations (Table 1). These include considerations related to impact on human resources (availability of expertise, impacts on work and income of existing personnel) as well as factors related to the organization and administration of health care.
DISCUSSION
Significance of Adopting a Multi-Criteria Framework
In one sense, the idea that health decisions with complicated medical, moral, economic, and organizational consequences involve the consideration of multiple criteria seems too obvious to mention, let alone belabor. It is difficult to identify any decision, in fact, from the most trivial consumer decision, to broad questions affecting societies or the environment, that do not involve considerations of multiple attributes. This was the key insight behind Keeney's extension of expected utility theory to multi-attribute utility theory (Reference Keeney22). What is not so generally appreciated, though, are the potential implications of characterizing social decision making in this way. Thinking about what it is that a society values, and seeking to structure decisions and decision making in a way that is consonant with those values focuses attention on the fact that decisions, and decision making are at the heart of HTA. The theoretical underpinnings of this approach are related to normative decision theories, ideas about how optimal decisions can be made (Reference Keeney22–Reference von Neumann, Morgenstern and Kuhn24).
This is distinct from theoretical perspectives shaped by explicit social objectives. For example, the objective of the National Institute for Health and Care Excellence is the maximization of health, subject to the budget constraint of a fixed NHS budget (25). The theoretical underpinnings of this approach are economic theories of the social good, such as welfarism or extrawelfarism (Reference Brouwer, Culyer and van Exel26). This approach privileges outcomes (societal welfare, health); ours privileges decisions.
Both approaches (making good decisions, and maximizing health or welfare) seem intuitively attractive. And indeed, these approaches should be broadly consonant, if the things we value in decision making are also the social goals our decision making is designed to optimize. Yet, there are important practical consequences of choosing one approach over the other. Seeing decision making about health as a “health maximization problem” places health economic modeling at the center of decision making. Clinical evidence, patient preferences and social values are also relegated to a secondary role. Our committee took the view that a multi-criteria approach is preferable, because it allows each attribute, and the underlying social values to be understood through the prism of the intellectual traditions that have evolved over time to study them.
Significance of a Framework Based on Values
HTA agencies often refer to broad ethical principles which serve as the foundation of their decision making. For example, autonomy, nonmaleficence, beneficence, and distributive justice have been cited as key ethical principles that underlie social value judgments (10). But it is hard to see how clinical evidence review, or cost effectiveness, the practical operational work of HTA, are related to these rather lofty goals. Our work in identifying a broader set of fundamental social values makes it possible to be more explicit about how practical work is linked to values.
Significance of a Framework Based on “Paradigms”
Our work also recognizes that there are intellectual traditions that have developed ways of framing questions, conducting studies, and interpreting results in ways that are distinct and valuable. Our framework organizes attributes by shared epistemological and disciplinary perspectives. For example, some health economists have advocated addressing equity concerns with the use of equity weights in cost effectiveness analysis. The committee's view was that methods and perspectives of interpretive social scientists and bioethicists would likely give equity considerations deeper consideration and greater weight. Similarly, the “health economics paradigm” considers cost and effectiveness together, and, therefore, so do we. In contrast, some frameworks (Reference Goetghebeur, Wagner and Khoury6) explicitly disaggregate these attributes because of concerns related to double counting. The committee's view was that health economics has developed a set of well worked out methods for thinking about value, which would be largely lost if these attributes were separated.
Value Frameworks and Decision Frameworks
An important development in social decision making has been the development and use of “value frameworks” in the United States (Reference Anderson, Heidenreich and Barnett7;Reference Schnipper, Davidson and Wollins8;27). Our proposed framework differs from these in important ways. The most obvious is the concept of “value”. For most of these frameworks, the decision problem is characterized in fundamentally economic terms: value is determined by setting outcomes (variably defined) against cost (Reference Anderson, Heidenreich and Barnett7;Reference Schnipper, Davidson and Wollins8;27). Our approach fundamentally differs by considering value as one key attribute among four, rather than the central principle for decision making.
Portability of This Framework to Other Jurisdictions
We think there are aspects of our framework that will be useful internationally. We hope that thinking about HTA as an exercise in operationalizing social values will resonate elsewhere, and will lead to additional theoretical work that explicitly grounds technology decisions in social values. We also believe that representing appraisal criteria as factors that are embedded in social context and intellectual traditions is portable. We hope the simplicity, breadth, and practicability of this general approach will find it a wide audience.
FUTURE WORK
This decision-making framework is being implemented in Ontario, Canada. Forthcoming publications will describe in greater detail the ongoing development and utility of this framework in real-world decision making.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: https://doi.org/10.1017/S0266462318000235
CONFLICTS OF INTEREST
The authors have nothing to disclose.
