It was an honor to have known and worked with Seymour Perry, MD, MACP, and to have witnessed his many accomplishments and contributions in several fields—cancer research, government administration, the development and promotion of procedures for evaluating health technologies, particularly consensus development, and leadership in fostering national and international collaboration in health technology assessment—representing a sustained effort in which both quantitative and qualitative benefits have been demonstrated.

The profusion of clinical practice guidelines has been remarked on by many (3;6;8) and ascribed to a need to address the wide geographic differences in practice documented in the last quarter century (9;10), as well as to the increasing concern with the cost of health care. The idea is that guidelines, by outlining efficient care strategies, will enhance the quality of care and reduce unnecessary or unproductive expenditures. Others hold that guidelines are simply a means of transferring the results of research from the literature to clinicians (2;5). A darker view of guidelines sees them as instruments of control of medical practice by uncaring administrators concerned solely with cost reduction (4;7).

Through the use of three scenarios, this paper presents the challenges for clinical practice guidelines in the 21st century. Such challenges relate to technological developments to improve the efficiency and pace of the development process, to ensure that clinical practice guidelines are kept up to date, and to facilitate implementation of guidelines in the clinical setting. To improve and ensure the validity of the content of clinical practice guidelines, we need to address the important problem of publication bias, for which researchers, granting agencies, industry, and journal editors share responsibility. This means insisting on registration of trials at their inception, and incentives backed up by rules for funding and peer review publication that would promote behaviors to avoid publication bias. The more difficult challenges for clinical practice guidelines relate to what are referred to as attitudinal factors. To achieve optimal efficiencies in development and maintenance of clinical practice guidelines, we need to promote cooperation among various information resource providers internationally and to stress partnership over leadership. Finally, there needs to be reconciliation of the different stakeholder perspectives of the value and purpose of clinical practice guidelines so that they are used appropriately as aids to decision making and are not abused as tools for controlling clinical practice.

In an effort to control ambulatory care costs, regulatory practice guidelines (références médicales opposables or RMOs) were introduced by law in France in 1993. RMOs are short sentences, negatively formulated (“it is inappropriate to \ldots”), covering medical and surgical topics, diagnosis, and treatment. Since their introduction, physicians who do not comply with RMOs can be fined. The fine is determined by a weighted combination of indices of harm, cost, and the number of violations.

The impact of the RMO policy on physician practice has been questioned, but so far few evaluations had been performed. At the end of 1997, only 121 physicians had been fined (0.1% of French private physicians). The difficulty of controlling physicians, the large number of RMOs, and the lack of a relevant information system limit the credibility of this policy.

The simultaneous development of a clinical guideline program to improve the quality of care and of a program to control medical practice can lead to a misunderstanding among clinicians and health policy makers. Financial incentives or disincentives could be used to change physician behavior, in addition to other measures such as education and organizational changes, if they are simple, well explained, and do not raise any ethical conflict. But these measures are dependent on the structure and financing of the healthcare system and on the socioeconomic and cultural context. More research is needed to assess the impact of interventions using financial incentives and disincentives on physician behavior.

Economic guidelines recommend methods that should be employed in conducting economic evaluations of healthcare programs. The nature of the efficiency or equity goal underpinning economic guidelines is unclear. What is also unclear is how the methods recommended in the guidelines are linked to the underlying efficiency or equity goal being targeted. If it is unclear what efficiency/equity objectives are being pursued, then it is unlikely that even full implementation of economic guidelines will improve resource allocation.

Clinical practice guidelines are expanding their scope of authority from clinical decision making to collective policy making, and promise to gain ground as resource allocation tools in coming years. A close examination of how guidelines approach patient selection criteria offers insight into their ethical implications when used as resource allocation or rationing instruments. The purposes of this paper are: a) to examine the structure of allocative reasoning found in clinical guidelines; b) to identify the ethical principles implied and compare how guidelines enact these principles with how explicit systems-level rationing exercises and health policy analyses have approached them; and c) to offer some preliminary suggestions for how these ethical issues might be addressed in the process of guideline development. The resulting framework can be used by guideline developers and users to understand and address some of the ethical issues raised by guidelines for the use of scarce technologies.

Practice guidelines are rapidly becoming preferred decision-making resources in medicine, as advances in technology and pharmaceutics continue to expand. An evidence-based approach to the development of practice guidelines serves to anchor healthcare policy to scientific documentation, and in conjunction with practitioner opinion can provide a powerful and practical clinical tool. Three sources of information are essential to an evidence-based approach: a) an exhaustive literature synthesis; b) meta-analysis; and c) consensus opinion. The systematic merging of evidence from these sources offers healthcare providers a scientifically supportable document that is flexible enough to deal with clinically complex problems. Evidence-based practice guidelines, in conjunction with practice standards and practice advisories, are invaluable resources for clinical decision making. The judicious use of these documents by practitioners will serve to improve the efficiency and safety of health care well.

Objectives: To identify and examine the methodologic issues related to evaluating the effectiveness of treatment adherence to clinical guidelines. The example of antiretroviral therapy guidelines for human immunodeficiency virus (HIV) disease is used to illustrate the points.

Methods: Regression analysis was applied to observational HIV clinic data for patients with CD4+ cell counts less than 500 per μL and greater than 50 per μL at baseline (n = 704),using Cox proportional hazards time-varying covariates models controlling for baseline risk. The results are compared with simpler models (Cox model [without time-varying covariates] and logistic regression). In addition, the effect of including a measure of exposure to antiretroviral guidelines in the model is explored.

Results: This study has three implications for modeling clinical guideline effectiveness. To capture events that are time-sensitive, a duration model should be used, and covariates that are time-varying should be modeled as time-varying. Thirdly, incorporating a threshold measure of exposure to reflect the minimum period of time for guideline adherence required for a measurable effect on patient outcome should be considered.

Conclusions: The methods proposed in this paper are important to consider if guidelines are to evolve from being a tool for summarizing and transferring the results of research from the literature to clinicians into a practical tool that influences clinical practice patterns. However, the methodology tested in this study needs to be validated using additional data on similar patients and using data on patients with other diseases.

Objective: To identify and compare clinical practice guideline appraisal instruments.

Methods: Appraisal instruments, defined as instruments intended to be used for guideline evaluation, were identified by searching MEDLINE (1966–99) using the Medical Subject Heading (MeSH) practice guidelines, reviewing bibliographies of the retrieved articles, and contacting authors of guideline appraisal instruments. Two reviewers independently examined the questions/statements from all the instruments and thematically grouped them. The 44 groupings were collapsed into 10 guideline attributes. Using the items, two reviewers independently undertook a content analysis of the instruments.

Results: Fifteen instruments were identified, and two were excluded because they were not focused on evaluation. All instruments were developed after 1992 and contained 8 to 142 questions/statements. Of the 44 items used for the content analysis, the number of items covered by each instrument ranged from 6 to 34. Only the instrument by Cluzeau and colleagues included at least one item for each of the 10 attributes, and it addressed 28 of the 44 items. This instrument and that of Shaneyfelt et al. are the only instruments that have so far been validated.

Conclusions: A comprehensive, concise, and valid instrument could help users systematically judge the quality and utility of clinical practice guidelines. The current instruments vary widely in length and comprehensiveness. There is insufficient evidence to support the exclusive use of any one instrument, although the Cluzeau instrument has received the greatest evaluation. More research is required on the reliability and validity of existing guideline appraisal instruments before any one instrument can become widely adopted.

Objectives: To identify major differences and similarities in the development of clinical guidelines in different European countries.

Methods: A collaboration of researchers is funded by the European Commission to compare the approaches to guideline development in collaborators' countries. The program encompasses a series of tasks, the first being to identify and document current guideline procedures in the collaborating countries. A survey gathered information on guideline production, dissemination, and implementation in the 10 European countries involved in the project consortium: Denmark, England and Wales, Finland, France, Germany, Italy, the Netherlands, Scotland, Spain (both the Basque Country and Catalonia), and Switzerland.

Results: Seven countries have a national policy on guideline production, dissemination, and implementation, and three countries are discussing their policies. A majority of guidelines are currently produced at the national level in six of the countries and at the regional or local level in the other four. Central or national funding supports guideline production in six countries. Additional sources of funding include medical societies, pharmaceutical companies, and health insurance companies. Several of the countries have published or are preparing evaluations of their dissemination strategies.

Conclusions: The survey highlighted wide variation in the methods and policies of guideline development in Europe. The Appraisal of Guidelines, Research, and Evaluation in Europe (AGREE) Collaboration research program will identify the characteristics of the “better” guideline programs and will provide the basis for more research-generated policy initiatives in the future, helping to ensure that guidelines play a major role in improving patient care in the millennium.

Clinicians make judgments under conditions of uncertainty. Decision research has shown that in uncertain situations individuals do not always act rationally, coherently, or to maximize their expected utility. Advocates of clinical guidelines believe that these guidelines will eliminate some of the cognitive biases that the practitioner may introduce into the medical decision-making process in an attempt to reduce its uncertainty. Other physicians have grave doubts about guidelines' application in practice. Guideline implementation lags well behind their development. Studies of practicing physicians and a survey of clinicians in one specialty and setting indicate that experienced clinicians may be implementing guidelines selectively. Many clinicians are concerned that guidelines are based on randomized trials and do not reflect the complexity of the real world, in which a decision's context and framework are important. Their reluctance also may be due to the difficulty of applying general guidelines to specific clinical situations. The problem will only increase in the future. The patients of the 21st century will be older and have more complex disease states. Physicians will have more patient-specific therapies and need to exercise more sophisticated clinical judgment. They may be more willing to use guidelines in making those judgments if research can demonstrate guidelines' effectiveness in improving decision making for individual patients.

Clinical practice guidelines have been available to clinicians for almost two decades, but the consistency of their implementation in practice remains highly variable. This paper describes the various processes and mechanisms used by one managed care organization to develop clinical guidelines and promote their adoption. Some of these mechanisms include provision of individual physician report cards, financial incentives, and various documentation tools that serve as reminders of guideline recommendations and provide an easy format to document recommended services.

There have been measurement challenges in evaluating the effectiveness of selected interventions designed to enhance guideline compliance. Most of these challenges relate to reliability and validity concerns regarding the three primary data sources used in the evaluation process: medical records, administrative claims data, and member survey data. Some of the interventions the health plan has implemented to address these measurement challenges include using hybrid methods of data collection and developing collaborative partnerships with outside organizations to enhance the accuracy and completeness of the available data. Outcomes of these efforts are described, as are physician response and recommendations for future enhancement of practice guidelines.

Healthcare policy, medical practice, and cost of care are no longer considered distinct entities. Each is an integral factor in determining not only what, but how, patient care will be delivered. Clinical practice guidelines are the lynchpin that connects them. This paper addresses the various components of the clinical practice guideline—cost alliance.

Objective: To examine the bidirectional influence of choice of care on costs and of cost of care on decision making.

Methods: The literature was used to identify cost-related factors that influence development of guidelines and change in physician practice behavior. In a MEDLINE search with modifiers to the keywords “clinical practice guidelines,” particular attention was paid to identifying surveys of practitioners. An analysis, prompted by a recently published guideline, of treating penetrating intraperitoneal colon injuries by different surgical approaches (primary repair versus diverting colostomy) exemplified how implementation of a guideline can affect the cost of care. Inpatient cost estimates, adjusted for medical inflation and cost-to-charge ratios and reported in 1999 U.S. dollars, were developed using data from 1996 and 1997 discharge databases from California and Massachusetts.

Results: The results showed that a substantial savings in hospital costs was achieved when a primary repair surgical technique, as advocated by the guideline, was used. The effect of cost influences on the development of clinical practice guidelines was established by demonstrating the cyclical effect between usual and customary practices, guideline implementation, changing practice patterns, and the economic considerations influencing the process.

Conclusions: A growing, albeit uneasy, alliance between costs and clinical practice guidelines is evident.

Objective: We reviewed and appraised the methods by which the issue of the learning curve has been addressed during health technology assessment in the past.

Method: We performed a systematic review of papers in clinical databases (BIOSIS, CINAHL, Cochrane Library, EMBASE, HealthSTAR, MEDLINE, Science Citation Index, and Social Science Citation Index) using the search term “learning curve.”

Results: The clinical search retrieved 4,571 abstracts for assessment, of which 559 (12%) published articles were eligible for review. Of these, 272 were judged to have formally assessed a learning curve. The procedures assessed were minimal access (51%), other surgical (41%), and diagnostic (8%). The majority of the studies were case series (95%). Some 47% of studies addressed only individual operator performance and 52% addressed institutional performance. The data were collected prospectively in 40%, retrospectively in 26%, and the method was unclear for 31%. The statistical methods used were simple graphs (44%), splitting the data chronologically and performing a t test or chi-squared test (60%), curve fitting (12%), and other model fitting (5%).

Conclusions: Learning curves are rarely considered formally in health technology assessment. Where they are, the reporting of the studies and the statistical methods used are weak. As a minimum, reporting of learning should include the number and experience of the operators and a detailed description of data collection. Improved statistical methods would enhance the assessment of health technologies that require learning.

Objective: To explore the impact of methodologic issues on the results of meta-analyses. The following issues were examined: the type of literature search strategy used; inclusion or exclusion of non–peer-reviewed studies; the inclusion or exclusion of non-English language publications; the effect of trial quality; and the inclusion or exclusion of non–placebo-controlled studies.

Methods: The International Study of Perioperative Transfusion (ISPOT) meta-analyses were used to evaluate each of the methodologic issues. The 10 meta-analyses consisted of technologies to reduce the need for perioperative red blood cell transfusion. The number of trials for each of the meta-analyses varied from 2 to 45. Both EMBASE and MEDLINE searches were conducted, including the use of systematic search strategies.

Results: MEDLINE identified the vast majority of trials. Alone, MEDLINE would have missed 8 studies compared to 10 for EMBASE. Use of the systematic search strategies greatly reduced the number of articles to be reviewed compared to open searches. Type of publication, country of study origin, inclusion of non-English publications, and trial quality had very little impact on the estimates of effect. The use of placebo versus open-label control affected the magnitude of the odds ratio for two of the meta-analyses. The results of the two meta-analyses were not statistically significant if only placebo-controlled trials were included.

Conclusions: While methodologic issues had very little impact on the ISPOT meta-analyses, further studies are needed in a variety of other clinical settings. Because MEDLINE, coupled with a review of the references in the identified trials, identified the vast majority of trials, one needs to consider the costs and benefits of searching EMBASE and the pursuance of unpublished and unindexed trials.

Objective: Severe to profound hearing impairment affects one-half to three-quarters of a million Americans. To function in a hearing society, hearing-impaired persons require specialized educational, social services, and other resources. The primary purpose of this study is to provide a comprehensive, national, and recent estimate of the economic burden of hearing impairment.

Methods: We constructed a cohort-survival model to estimate the lifetime costs of hearing impairment. Data for the model were derived principally from the analyses of secondary data sources, including the National Health Interview Survey Hearing Loss and Disability Supplements (1990–91 and 1994–95), the Department of Education's National Longitudinal Transition Study (1987), and Gallaudet University's Annual Survey of Deaf and Hard of Hearing Youth (1997–98). These analyses were supplemented by a review of the literature and consultation with a four-member expert panel. Monte Carlo analysis was used for sensitivity testing.

Results: Severe to profound hearing loss is expected to cost society $297,000 over the lifetime of an individual. Most of these losses (67%) are due to reduced work productivity, although the use of special education resources among children contributes an additional 21%. Life time costs for those with prelingual onset exceed $1 million.

Conclusions: Results indicate that an additional $4.6 billion will be spent over the lifetime of persons who acquired their impairment in 1998. The particularly high costs associated with prelingual onset of severe to profound hearing impairment suggest interventions aimed at children, such as early identification and/or aggressive medical intervention, may have a substantial payback.

Objective: This study investigates the influence of different aspects of methodologic quality on the conclusions of a systematic review concerning treatments of acute lateral ankle sprain.

Method: A data set of a systematic review of 44 trials was used, of which 22 trials could be included in this study. Quality assessment of the individual studies was performed using the Delphi list. We calculated effect sizes of the main outcome measure in each study in order to evaluate the relationship between overall quality scores and outcome. Next, we investigated the impact of design attributes on pooled effect sizes by subgroup analysis.

Results: The quality of most studies (82%) was low; only 4 of 22 trials were of high quality. Studies with proper randomization and blinding procedure produce a slightly higher (not statistically significant) effect estimate compared to the other studies.

Conclusion: Previous research has suggested that methodologically poorly designed studies tend to over-estimate the effect estimate. Our study does not confirm these conclusions.

Objectives: Inappropriate hospital admissions are commonly believed to represent a potential for significant cost reductions. However, this presumes that these patients can be identified before the hospital stay. The present study aimed to investigate to what extent this is possible.

Methods: Consecutive admissions to a department of internal medicine were assessed by two expert panels. One panel predicted the appropriateness of the stays from the information available at admission, while final judgments of appropriateness were made after discharge by the other.

Results: The panels correctly classified 88% of the appropriate and 27% of the inappropriate admissions. If the elective admissions predicted to be inappropriate had been excluded, 9% of the costs would have been saved, and 5% of the gain in quality-adjusted life-years lost. The corresponding results for emergency admissions were 14% and 18%.

Conclusions: The savings obtained by excluding admissions predicted to be inappropriate were small relative to the health losses. Programs for reducing inappropriate health care should not be implemented without investigating their effects on both health outcomes and costs.

Objectives: Cost-effectiveness analysts often present cost-effectiveness results by age to help inform decisions about the use of an intervention. Yet it is not known how well studies model the risks and costs associated with age. We reviewed published studies to examine their modeling of age differences.

Methods: MEDLINE searches identified all cost-effectiveness analyses published between 1985 and 1997 that included adults 50 years of age and older, were based on data for developed countries, and compared cost-effectiveness ratios for adults of different ages or for initiation of an intervention at different ages; 36 articles met these criteria. They were reviewed to determine the extent to which they incorporated age-specific data. Studies that justified using the same data for all ages were counted as having varied the data element by age.

Results: All studies varied life expectancy by age. Most also varied the incidence/prevalence of the target condition and the case fatality rate. Only 36% varied the effectiveness rate of the intervention by age. Costs were usually assumed constant: 42% of studies varied the cost of treating adverse effects and 17% varied the cost of treating the target condition. Whether a data element was varied did not appear to be related to the pattern of cost-effectiveness ratios by age.

Conclusions: Many studies have not modeled age differences in sufficient detail to ensure that differences in cost-effectiveness ratios by age are accurate and a sound basis for decisions. As cost-effectiveness analysis becomes more widespread, analysts should strive to incorporate more complete age-specific data.

Objectives: To estimate the potential for cost reduction in the acute care setting and the required investment in the home care setting of implementing an outpatient/early discharge strategy for operable (stages I and II) breast cancer in Canada.

Methods: Data from a community hospital were augmented by expert knowledge and incorporated into the breast cancer submodel of Statistics Canada's Population Health Model. For the estimated 90% of patients for whom this approach was assumed to be appropriate, the resource utilization for outpatient breast-conserving surgery and 2 days of hospitalization for those women undergoing mastectomy was quantified and costed, as were the appropriate home care services. A 5% readmission rate for complications was assumed. Cost per case, total cost burden, investment in home care, savings in acute care, and net savings were calculated. Sensitivity analyses were performed around readmission rates and home care/surgical follow-up costs. All costs were determined in 1995 Canadian dollars.

Results: The cost of initial treatment for the 15,399 women diagnosed with stages I and II breast cancer in 1995 in Canada was estimated to be $127.6 million. Hospitalization made up 53% of these costs. Under the outpatient/early discharge strategy, the acute care cost of initial breast cancer management could be reduced by $47.2 million, with an investment in home care of $14.5 million ($453 per patient), resulting in an overall net saving of $33 million. Under this strategy, hospitalization would contribute only 21% to the total care cost.

Conclusions: If Canadian surgeons and healthcare administrators were to work together to put in place processes to support ambulatory breast cancer surgery and if resources were redirected to the provision of home-based post-operative care, there would be potential for a large net healthcare saving and preservation of high-quality patient care.