Until the mid-1980s, most economic analyses of healthcare technologies were based on decision theory and used decision-analytic models. The goal was to synthesize all relevant clinical and economic evidence for the purpose of assisting decision makers to efficiently allocate society's scarce resources. This was true of virtually all the early cost-effectiveness evaluations sponsored and/or published by the U.S. Congressional Office of Technology Assessment (OTA) (15), Centers of Disease Control and Prevention (CDC), the National Cancer Institute, other elements of the U.S. Public Health Service, and of healthcare technology assessors in Europe and elsewhere around the world. Methodologists routinely espoused, or at minimum assumed, that these economic analyses were based on decision theory (8;24;25). Since decision theory is rooted in—in fact, an informal application of—Bayesian statistical theory, these analysts were conducting studies to assist healthcare decision making by appealing to a Bayesian rather than a classical, or frequentist, inference approach. But their efforts were not so labeled. Oddly, the statistical training of these decision analysts was invariably classical, not Bayesian. Many were not—and still are not—conversant with Bayesian statistical approaches.

Objectives: The objectives of this paper were: a) to determine what can be learned from conclusions of systematic reviews about the evidence base of medicine; and b) to determine whether two readers draw similar conclusions from the same review, and whether these match the authors' conclusions.

Methods: Three methodologists (two per review) rated 160 Cochrane systematic reviews (issue 1, 1998) using pre-established conclusion categories. Disagreements were resolved by discussion to arrive at a consensual score for each review. Reviews' authors were asked to use the same categories to designate the intended conclusion. Interrater agreements were calculated.

Results: Interrater agreement between two readers was 0.68 and 0.72, and between readers and authors, 0.32. The largest categories assigned by methodologists were “positive effect” (22.5%), “insufficient evidence” (21.3%), and “evidence of no effect” (20.0%). The largest categories assigned by authors were “insufficient evidence” (32.4%), “possibly positive” (28.6%), and “positive effect” (26.7%).

Conclusions: The number of reviews indicating that the modern biomedical interventions show either no effect or insufficient evidence is surprisingly high. Intterrater disagreements suggest a surprising degree of subjective interpretation involved in systematic reviews. Where patterns of disagreement emerged between authors and readers, authors tended to be more optimistic in their conclusions than the readers. Policy implications are discussed.

Objective: The series of papers in this issue was developed to examine the use of health technology assessment in policies toward prevention—specifically toward mass screening—in European countries. The papers actually examined three screening strategies: mammography screening for breast cancer, prostate-specific antigen screening for prostate cancer, and routine ultrasound in normal pregnancy.

Methods: Papers were sought from the member states of the European Union, plus Switzerland. Ultimately, nine acceptable papers were received, and were reviewed, revised, and edited.

Results: Screening is an accepted strategy in many countries for reducing the burden of disease through early detection and intervention. In part, this is because of successful screening programs that have been evaluated and implemented in many countries. At the same time, unevaluated and even useless and harmful screening programs—unjustified medically or economically—are widespread. Health technology assessment could help assure that only effective and cost-effective screening programs are implemented.

Conclusion: The main conclusion is that screening is an important preventive strategy. Any screening program, however, should be carefully assessed before implementation.

Objectives: Assessments of health technologies increasingly include economic evaluations conducted alongside clinical trials. One particular concern with economic evaluations conducted alongside clinical trials is the generalizability of results from one setting to another. Much of the focus relating to this topic has been on the generalizability of results between countries. However, the characteristics of clinical trial design require further consideration of the generalizability of cost data between centers within a single country, which could be important in decisions about adoption of the new technology.

Methods: We used data from a multicenter clinical trial conducted in the United Kingdom to assess the degree of variation in costs between patients and between treatment centers and the determinants of the degree of such variation.

Results: The variation between patients was statistically significant for both the experimental and conventional treatments. However, the degree of variation between centers was only statistically significant for the experimental treatment. Such variation appeared to be a result of hospital practice, such as payment mechanisms for staff and provision of hostel accommodation, rather than variations in physical resource use or substantive differences in cost structure.

Conclusions: Multicenter economic evaluations are necessary for determining the variations in hospital practice and characteristics that can in turn determine the generalizability of study results to other settings. Such analyses can identify issues that may be important in adopting a new health technology. Analysis is required of similar large multicenter trials to confirm these conclusions.

Interest in Bayesian analyses has increased recently, in part as a response to policy makers wanting sound scientific bases for health technology assessments, and associated healthcare funding decisions. This paper provides a brief and simplified description of Bayesian reasoning. Bayes is illustrated in a clinical setting of an expert helping a woman understand the potential risk of passing on an inheritable disease (hemophilia) to her next child, based on disease occurrence in two living children. The illustration describes fundamental concepts and derivations, such as Bayes theorem, likelihood functions, prior probability, and posterior probability. A second illustration shows the use of Bayes for interpreting clinical trial results. The uncertainty in the clinical effect before and after the trial analyses has been completed is characterized by the Bayes prior and posterior probabilities, respectively. Techniques are also shown for estimating the potential loss (e.g., in lives lost) for making the wrong decision with and without knowledge of the trial results, an estimation that cannot be carried out using techniques of hypotheses testing associated with the frequentist school of statistics. Information from Bayes analysis then may be used to help policy makers decide, or justify, whether the analyses provides a sufficient basis for making a treatment recommendation, or whether there remains a need to request more information. Subsequent papers in this volume offer additional examples and clarification of the use of Bayes in clinical practice and in interpretation of clinical studies.

Objective: To review published data regarding the accuracy and effectiveness of three screening tests: mammography, prostate-specific antigen (PSA), and prenatal ultrasound.

Methods: Published evidence regarding the accuracy and effectiveness of the three tests was collected by computerized literature search and supplemented by manual review of relevant bibliographies.

Results: Screening mammograms lower breast cancer mortality by about 20%. Most data come from women aged 50–64 years; women aged 40–49 years may also benefit, but the absolute risk reduction is lower. Up to 1,500 to 2,500 women must undergo screening to prevent one death from breast cancer. Mammograms miss approximately 12% to 37% of cancers, generate false-positive results, and cause anxiety while abnormal results are evaluated. PSA screening can detect 80% to 85% of prostate cancers but has a high false-positive rate. There is little direct evidence that early detection reduces morbidity or mortality. Indirect evidence includes a trend toward earlier stage tumors and steadily declining mortality rates in geographic areas where PSA screening has become common. Potential harms include the morbidity associated with evaluating abnormal results, and complications from treatment (e.g., impotence, incontinence). The overall balance of benefits and harms remains uncertain in the absence of better evidence. Prenatal ultrasound may reduce perinatal mortality, primarily through elective abortions for congenital anomalies, but does not appear to lower live birth rates. Although ultrasound has no proven effect on neonatal morbidity, it provides more accurate estimates of gestational age that prevent unnecessary inductions for post-term pregnancy. Screening detects multiple gestations, congenital anomalies, and intrauterine growth retardation, but direct health benefits from having this knowledge are unproved. Ultrasound has both positive and negative psychological effects on parents. The scans do not appear to harm childhood development.

Conclusions: Even for the most established screening tests, the appropriateness of routine testing depends on subjective value judgments about the quality of supporting evidence and about the trade-offs between benefits and harms. Individuals, clinicians, policy makers, and governments must weigh the evidence in light of these values and the constraints imposed by available resources.

Objective: To determine the extent to which systematic reviews of public health interventions influenced public health decisions and which factors were associated with influencing these decisions.

Methods: This cross-sectional follow-up survey evaluated the use of five systematic reviews in public health decision making. Independent variables included characteristics of the innovation, organization, environment, and individual. Primary data were collected using a telephone survey and a self-administered organizational demographics questionnaire. Public health decision makers in all 41 public health units in Ontario were invited to participate in the study. Multiple linear regression analyses on the five program decisions were conducted.

Results: The systematic reviews were perceived as having the greatest amount of influence on decisions related to program justification and program planning, and the least influence on program evaluation decisions. The greater the perception that one's organization valued the use of research evidence for decision making and that ongoing training in the critical appraisal of research literature was provided, the greater the perception of the influence the systematic review had on public health decisions.

Conclusions: Organizational characteristics are important predictors of the use of systematic reviews in public health decision making. Future dissemination strategies need to promote the value of using systematic reviews for program decision making as well as promote ongoing training in critical appraisal among intended users in Ontario.

Objectives: Assessment of the appropriateness of screening programs involves consideration of the harms as well as the benefits. These harms include the risk of false-negative results, the consequences of which have remained underinvestigated. This paper reports the results of a systematic literature review that aimed to assess the medical psychological, economic, and legal consequences of false-negative results in national screening programs.

Methods: The review included a comprehensive literature search and contact with experts to identify relevant literature. Most studies that were identified presented only anecdotal evidence. However, thirteen studies presented quantitative information on medical consequences of false negatives, eight studies presented information on psychological consequences, and two studies presented information on economic consequences.

Results: The strength of evidence from most of the primary studies was low. There is some evidence, however, that false-negative results may have a large legal impact. There is also a consensus in the literature that false negatives may have a negative impact on public confidence on screening; evidence is however limited.

Conclusions: False negatives are evident even in high-quality screening programs. They may have the potential to delay the detection of breast and cervical cancer, but there is little evidence to help in assessing their psychological consequences. They also may lead to legal action being taken by those affected and may reduce public confidence in screening. Their impact may be reduced by provision of full information to participants about the benefits and limitations of screening programs and by increasing public education on these issues.

The objective of this paper is to illustrate the advantages of the Bayesian approach in quantifying, presenting, and reporting scientific evidence and in assisting decision making. Three basic components in the Bayesian framework are the prior distribution, likelihood function, and posterior distribution. The prior distribution describes analysts' belief a priori; the likelihood function captures how data modify the prior knowledge; and the posterior distribution synthesizes both prior and likelihood information. The Bayesian approach treats the parameters of interest as random variables, uses the entire posterior distribution to quantify the evidence, and reports evidence in a “probabilistic” manner. Two clinical examples are used to demonstrate the value of the Bayesian approach to decision makers. Using either an uninformative or a skeptical prior distribution, these examples show that the Bayesian methods allow calculations of probabilities that are usually of more interest to decision makers, e.g., the probability that treatment A is similar to treatment B, the probability that treatment A is at least 5% better than treatment B, and the probability that treatment A is not within the “similarity region” of treatment B, etc. In addition, the Bayesian approach can deal with multiple endpoints more easily than the classic approach. For example, if decision makers wish to examine mortality and cost jointly, the Bayesian method can report the probability that a treatment achieves at least 2% mortality reduction and less than $20,000 increase in costs. In conclusion, probabilities computed from the Bayesian approach provide more relevant information to decision makers and are easier to interpret.

Objective: This article outlines basic methods for quantifying reproducibility and responsiveness of evaluative outcome measures.

Methods: The background noise in stable patients provides the desired information to quantify the reproducibility. From this, the smallest real difference (SRD) for longitudinal differences can be derived. We propose to use the SRD to define responsiveness: from all patients who change according to an external criterion, we take the percentage that changes at least SRD on the outcome measure. A more complete picture of the responsiveness of the outcome measure arises when the receiver operating characteristic (ROC) is constructed, corresponding to all possible cut-off change scores. The proposed methods are illustrated with an empirical example.

Results: In the illustration the methods appeared to be very useful and complemented each other. We could evaluate whether the observed change score was larger than that expected due to chance. With the methods it was possible to evaluate both the ability of an instrument to detect change if there is a real change in health status (sensitivity to change) and the ability to detect absence of change if there is no real change (specificity to change).

Conclusion: We presented the use of SRDs and ROC curves for quantifying reproducibility and responsiveness. We started with the basic notions and arrived at methods that are both understandable and useful.

Objective: To study cases of screening in Austria to learn about national strategies to handle the health policy challenge of early detection of widespread diseases and about the outcome of those strategies. The article describes three healthcare interventions (mammography, PSA testing, and routine use of ultrasound in pregnancy) and the instruments of Austrian health policy that are used—with or without explicit intention—to enforce or to control the widespread use of (early) diagnoses.

Methods: Data and information collection on healthcare services, their accessibility, rate of use, expert consensus, and official regulations. For all three case studies, expert interviews were carried out with main actors.

Results: Risk-group screening is not a priority in Austrian federal health policy. Although health promotion and prevention is a national task, examinations for early detection of specific diseases (i.e., carcinoma) are left to the health insurance funds, which delegate the decision to offer early diagnoses to their contracted physicians. In this opportunistic screening, general practitioners or specialists are encouraged by their health insurance funds or motivated by professional guidelines to offer certain examinations.

Conclusions: Screening is a coordinated effort to acquire a grasp of a common disease at an early stage in a specified population. To achieve this objective, a culture of coordination and centralization has to be implemented. The collection of data is an essential element in coordination of decentralized medical interventions as much as quality control is an essential task in looking at and comparing the outcome of interventions. In the three case studies, neither of these two essential criteria were met. Evaluations and scientific evidence on the effectiveness of interventions were not used.

Background: There is a considerable gap between randomized clinical trials and implementing the results into practice. This is particularly relevant in the use of broad-spectrum antibiotics in hospitals. Hospital pharmacists can be effective vehicles for bridging this gap and promoting evidence-based medicine. To determine the most effective way of using the pharmacist in this role, a prospective cefotaxime intervention study was conducted with randomization incorporated into the design as well as patient-related therapeutic outcomes.

Methods: A total of 323 patients who were prescribed cefotaxime were randomized into an intervention or nonintervention group where only the former was challenged by pharmacists for inappropriate cefotaxime usage relative to hospital guidelines. The primary outcome was the appropriateness of cefotaxime prescribing between groups. Logistic regression analysis was then used to identify factors that were associated with successful clinical response.

Results: Overall, 94% of orders in the intervention group met cefotaxime dosage criteria compared with 86% in the control group (p = .018). However, there was no impact with respect to promoting cefotaxime use for an appropriate indication (81% vs. 80%; p = .67). There was a trend for improved clinical outcomes in patients who received cefotaxime within hospital guidelines (OR = 1.73; p = .31).

Conclusions: The pharmacist as a vehicle for promoting the appropriate use of broad-spectrum antibiotics in the acute care hospital setting can improve the dosing of such agents. However, several barriers to optimizing the impact of the pharmacist were implied by the data. Removing these barriers could increase the pharmacists' utility as an agent for improved patient care.

Objective: Renal replacement therapy (RRT) is essential for maintenance of life for those with end-stage renal disease. However, there remain many areas of uncertainty about which method of RRT should be chosen. This paper reports an economic model based on a systematic review that attempts to determine which method of dialysis, continuous ambulatory peritoneal dialysis (CAPD) or hemodialysis, a patient should have as the initial method of RRT.

Methods: A systematic review and a costing exercise carried out in a Scottish hospital were used to populate a Markov model. Scenario analysis was used to model plausible variations in variables included in the model.

Results: In 8 of the 16 scenarios developed hemodialysis dominated CAPD. In a further eight scenarios, the cost per life-year displayed considerable variability, ranging from between £5,000 to £51,000. The higher costs per life-year were associated with minimum estimates of additional survival for a patient starting RRT on hospital hemodialysis.

Conclusion: It may be more cost-effective to manage patients starting on RRT with hospital hemodialysis than CAPD. This has particular implications for the United Kingdom, given that up to 50% of new patients receive CAPD.

Objective: To describe how scientific evidence has influenced healthcare policy making in Belgium in the field of sickness prevention for mammography, PSA testing in prostate cancer screening, and use of ultrasound in pregnancy.

Methods: Review of published and gray literature and interviews with stakeholders and experts.

Results: At the end of 1999, a systematic national/regional screening program had not yet been implemented for any of the three screening strategies. A systematic breast cancer screening program is being prepared for implementation only in Flanders. This limited impact can be attributed to the fragmentation in healthcare policy, the different options among the different regions, fragmentation in healthcare practice, the strong impact of healthcare stakeholders (provider groups and sickness funds) on decision making, and limited attention to scientific evidence in health policy and technology assessment.

Conclusions: Health technology assessment has had very little impact on policy and practice in use of mammography, PSA testing, and ultrasound in pregnancy in Belgium.

Current health economic literature does not provide clear guidelines on how uncertainty around cost-effectiveness estimates should be incorporated into economic decision models. Bayesian analysis is a promising alternative to classical statistics for incorporating uncertainty in economic analysis. Estimating a loss function that relates outcomes to societal welfare is a key component of Bayesian decision analysis. Health economists commonly compute the loss function based on the quality-adjusted life-years associated with each outcome. However, if welfare economics is adopted as the theoretical foundation of the analysis, a loss function based in cost-benefit analysis (CBA) may be more appropriate. CBA has not found wide use in health economics due to practical issues associated with estimating such a loss function. In this paper, we present a method based in conjoint analysis for estimating the CBA loss function that can be applied in practice. We illustrate the use of the methodology using data from a pilot study.

Objectives: In quality-adjusted life-years (QALY) models, it is customary to weigh life-years with quality of life via multiplication. As a consequence, for positive health states a longer duration has more QALYs than a shorter duration (i.e., longer is better). However, we have found that for poor health states, many prefer to live only a limited amount of time (i.e., longer is worse). Such preferences are said to be maximum endurable time (MET). In the present contribution, the following questions are asked: a) How low does the utility have to be in order for a MET to arise? and b) Do MET preferences occur when patients judge hypothetical health states?

Methods and Results: We reanalyzed data from 176 students for the hypothetical health states of “living with migraines” and “living with metastasized cancer.” For utilities smaller than 0.7 (ranging from 0 to 1), the MET preference rate was larger than 50%. High MET preference rates were also found in two new studies on migraine and esophageal cancer patients, who evaluated hypothetical health states related to their disease.

Conclusions: We discuss the interpretation of the MET preferences and the preference reversal phenomenon. Standard QALY models imply that longer is better. However, we find that more often, longer is worse for poorly evaluated health states. Consider the following question: are 3 years with a weight of 0.3 equally as valuable as 1 year with a weight of 0.9? Our results suggest that the 3-year period may be less valuable because for poor health, many will prefer a 1-year over a 3-year period.

A framework is presented that distinguishes the conceptually separate decisions of which treatment strategy is optimal from the question of whether more information is required to inform this choice in the future. The authors argue that the choice of treatment strategy should be based on expected utility, and the only valid reason to characterize the uncertainty surrounding outcomes of interest is to establish the value of acquiring additional information. A Bayesian decision theoretic approach is demonstrated through a probabilistic analysis of a published policy model of Alzheimer's disease. The expected value of perfect information is estimated for the decision to adopt a new pharmaceutical for the population of patients with Alzheimer's disease in the United States. This provides an upper bound on the value of additional research. The value of information is also estimated for each of the model inputs. This analysis can focus future research by identifying those parameters where more precise estimates would be most valuable and indicating whether an experimental design would be required. We also discuss how this type of analysis can also be used to design experimental research efficiently (identifying optimal sample size and optimal sample allocation) based on the marginal cost and marginal benefit of sample information. Value-of-information analysis can provide a measure of the expected payoff from proposed research, which can be used to set priorities in research and development. It can also inform an efficient regulatory framework for new healthcare technologies: an analysis of the value of information would define when a claim for a new technology should be deemed substantiated and when evidence should be considered competent and reliable when it is not cost-effective to gather any more information.

Objective: In this article, three preventive strategies—mammography screening for breast cancer, PSA screening for prostate cancer, and routine ultrasound in normal pregnancy—are discussed in the context of German health care.

Methods: Epidemiologic data and German studies evaluating different aspects of these preventive measures were identified and analyzed.

Results: Only a few studies could be identified that investigate these preventive measures. Despite sufficient evidence, in part derived from a German study, there is not yet a mammography screening program. In contrast, ultrasound in pregnancy is offered routinely, although there are controversies regarding the benefit of this practice. PSA screening is not offered as part of the screening program for prostate cancer. However, PSA tests as well as mammographies are done in large numbers in German ambulatory care—a practice that could be considered wild or opportunistic screening.

Conclusions: These case studies show that preventive programs and practices in Germany are not sufficiently based on sound evidence. The paucity of evaluation activities related to prevention in Germany is probably due to the low threshold to introduce new preventive programs into the German healthcare system in the past.

Objectives: As an initiative of the International Network of Agencies for Health Technology Assessment, an approach to assessment of telemedicine applications was prepared to assist decision makers who are considering introduction and use of this technology.

Methods: Review and commentary drawing on published assessment frameworks and reports of primary evaluations of telemedicine, with particular reference to experience in Finland and Canada.

Results: Elements of the approach included development of a business case (considering population and services, personnel and consumers, delivery arrangements, specifications and costs); subsequent evaluation of the telemedicine application; and follow-up (covering the domains of technical assessment, effectiveness, user assessment of the technology, costs of telemedicine, trials, economic evaluation methods, and sensitivity analysis).

Conclusions: Decision makers should link introduction of new and often costly technology to appraisal of its feasibility, followed by evaluation of the application, including longer term consideration of its sustainability and impact on the healthcare system. As the effectiveness and efficiency of telemedicine applications will often be strongly influenced by local issues, results of assessments may not be generalizable.

Objective: To evaluate the total cost of minilaparotomy cholecystectomy (MC) and laparoscopic cholecystectomy (LC) and perform a cost-effectiveness (CE) analysis.

Methods: One hundred adult subjects with painful gallstone disease were randomized: 59 cases for LC and 41 cases for MC. Patients with gallstones shown on ultrasound with normal common bile duct and no history of icterus were included after an informed consent. Cases with acute cholecystitis and raised alkaline phosphatase were excluded. LC was performed using CO2 insufflation and a Storz 2D video camera. MC was done by transverse rectus cutting incision. Outcome was coded as success or failure. Success was defined as operation without injury to bile duct, viscera or vessels, minimal pain and discomfort at 4 weeks, no wound infection up to 4 weeks, and resumption of work within 2 weeks of operation. The total cost of each case included cost of investigations, cost of disposable articles for operation, cost of drugs, cost of hospital stay, and cost of operation including anesthesia. LC and MC were done with reusable instruments. A “societal viewpoint” has been taken in the cost calculations.

Results: There were 50/59 successful outcomes in LC and 15/40 outcomes in MC group. Total cost for LC was 386,769 rupees (Rs) and for MC was Rs 205,041. CE in LC was Rs 7,735 and in MC was Rs 13,669. Incremental CE ratio comparing LC with MC was 3,028.33.

Conclusion: LC is a more cost-effective method for treatment of gallstone disease.