Hostname: page-component-78c5997874-8bhkd Total loading time: 0 Render date: 2024-11-15T17:17:56.978Z Has data issue: false hasContentIssue false
  • English
  • Français

Selection and Evaluation of Empirical Research in Technology Assessment

Published online by Cambridge University Press:  10 March 2009

Thomas C. Chalmers ,
Peg Hewett ,
Dinah Reitman  and
Henry S. Sacks
Thomas C. Chalmers
Affiliation:
Harvard School of Public Health; Boston Veterans Administration Medical Center; Mount Sinai School of Medicine
Peg Hewett
Affiliation:
Harvard School of Public Health
Dinah Reitman
Affiliation:
Mount Sinai School of Medicine
Henry S. Sacks
Affiliation:
Mount Sinai School of Medicine
Get access Rights & Permissions [Opens in a new window]

Extract

Technology assessment involves application of the scientific method to the practice of medicine. Finding all of the assessment reports in a given field is not an easy task. Proper evaluation of those assessments requires the conduct of a prospective experiment in which the sources and results are blinded when the choice is made of papers to exclude and to include, and the process should be carried in duplicate. There are several available data bases for carrying out the search, but because of problems they should be supplemented by reference to the bibliographies of pertinent published articles. Clinical trials included in meta-analyses should be graded by quality and thus facilitate sensitivity analyses. Attention must be paid to the possibility of publication bias. Finally, the advent of meta-analysis makes it desirable to begin randomized controlled trials in areas of uncertainty, even when there is no possibility that individual investigators will encounter enough patients to draw valid conclusions.

Type
Special Section: Alternative Methods for Assessing Technology, Part 1
Information
International Journal of Technology Assessment in Health Care , Volume 5 , Issue 4 , October 1989 , pp. 521 - 536
Copyright
Copyright © Cambridge University Press 1989

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Antczak, A. A., Tang, J. & Chalmers, T. C.Quality assessment of randomized control trials in dental research. I. Methods. Journal of Periodontal Research, 1986, 21, 305–14.CrossRefGoogle ScholarPubMed
Barnett, H. J., Sackett, D., Taylor, D. W. et al. , Are the results of the extracranial-intracranial bypass trial generalizable? New England Journal of Medicine, 1987, 316, 820–24.CrossRefGoogle ScholarPubMed
Bhaskar, R., Reitman, D., Sacks, H. S. et al. , Loss of patients in clinical trials that measure long-term survival following myocardial infarction. Controlled Clinical Trials, 1986, 7, 134–48.CrossRefGoogle ScholarPubMed
Byar, D. P.Why data bases should not replace randomized clinical trials. Biometrics, 1980, 36, 337–42.CrossRefGoogle Scholar
Chalmers, I., Hetherington, J., Newdick, M. et al. , The Oxford Database of Perinatal Trials: Developing a register of published reports of controlled trials. Controlled Clinical Trials, 1986, 7, 306–24.CrossRefGoogle ScholarPubMed
Chalmers, I. & Silverman, W. A.Professional and public double standards on clinical experimentation. Controlled Clinical Trials, 1987, 8, 388–91.CrossRefGoogle ScholarPubMed
Chalmers, T. C.Informed consent, clinical research and the practice of medicine. Transactions of the American Clinical and Climatological Association, 1982, 94, 204–12.Google Scholar
Chalmers, T. C.Meta-analysis in clinical medicine. Transactions of the American Clinical and Climatological Association, 1987, 99, 144–50.Google Scholar
Chalmers, T. C.Randomization and coronary artery surgery. Annals of Thoracic Surgery, 1972. 14, 323–27.CrossRefGoogle ScholarPubMed
Chalmers, T. C.Randomization and coronary artery surgery [letter]. Annals of Thoracic Surgery, 1972, 14, 450–51.CrossRefGoogle ScholarPubMed
Chalmers, T. C.Randomization of the first patient. Medical Clinics of North America, 1975, 59, 1035–38.CrossRefGoogle ScholarPubMed
Chalmers, T. C., and discussants. How to turn off an experiment. In Cooper, J. D. (ed.), Ethical safeguards in research on humans, vol. 5. Washington, DC: H. L. Ley, 1976, 119–43.Google Scholar
Chalmers T. C. Concluding session: General discussion. In Chalmers, T. C. & Amacher, P. (eds.), Conference on recent history of randomized clinical trials. Controlled Clinical Trials, 1982, 3, 299–309.CrossRefGoogle Scholar
Chalmers, T. C., Berrier, J., Sacks, H. S. et al. , Meta-analysis of clinical trials as a scientific discipline. II. Replicate variability and comparison of studies that agree and disagree. Statistics in Medicine, 1987, 6, 733-44.CrossRefGoogle ScholarPubMed
Chalmers, T. C., Block, J. B. & Lee, S.Controlled studies in clinical cancer research. New England Journal of Medicine, 1972, 287, 75–78.CrossRefGoogle ScholarPubMed
Chalmers, T. C., Celano, P., Sacks, H. S. & Smith, H. Jr, Bias in treatment assignment in controlled clinical trials. New England Journal of Medicine, 1983, 309, 1358–61.CrossRefGoogle ScholarPubMed
Chalmers, T. C., Levin, H. R., Sacks, H. S. et al. , Meta-analysis of clinical trials as a scientific discipline. I. Control of bias and comparison with large cooperative trials. Statistics in Medicine, 1987, 6, 315–28.CrossRefGoogle Scholar
Chalmers, T. C., Smith, H. Jr, Blackburn, B. et al. , A method for assessing the quality of a randomized control trial. Controlled Clinical Trials, 1981, 2, 31–49.CrossRefGoogle ScholarPubMed
Chalmers, T. C., van, den Noort S., Lockshin, M. D. & Waksman, B. H.Summary of a workshop on the role of third-party payers in clinical trials of new agents. New England Journal of Medicine, 1983, 309, 1334–36.CrossRefGoogle ScholarPubMed
Chassin, M. R., Brook, R. H., Park, R. E. et al. , Variations in the use of medical and surgical services by the Medicare population. New England Journal of Medicine, 1986, 314, 285–90.CrossRefGoogle ScholarPubMed
Conn, H. O. & Blitzer, B. L.Nonassociation of adrenocorticosteroid therapy and peptic ulcer. New England Journal of Medicine, 1976, 294, 473–79.CrossRefGoogle ScholarPubMed
Conn, H. O. & Poynard, T.Adrenocorticosteroid administration and peptic ulcer: A critical analysis. Journal of Chronic Diseases, 1985, 38, 457–68.CrossRefGoogle ScholarPubMed
Conn, H. O. & Poynard, T.Adrenocorticosteroid therapy and peptic-ulcer disease [letter]. New England Journal of Medicine, 1984, 310, 201–02.Google ScholarPubMed
Cooper, L. S., Chalmers, T. C., McCally, M. et al. , The poor quality of early evaluations of magnetic resonance imaging. Journal of the American Medical Association, 1988, 259, 3277–80.CrossRefGoogle ScholarPubMed
Cordray, D. S. Strengthening causal interpretations of nonexperimental data: The role of meta-analysis. Submitted for publication.Google Scholar
Coronary Artery Surgery Study (CASS). A randomized trial of coronary artery bypass surgery. Survival data. Circulation, 1983, 68, 939–50.CrossRefGoogle Scholar
Coronary Artery Surgery Study (CASS). A randomized trial of coronary artery bypass surgery. Comparability of entry characteristics and survival in randomized patients and non-randomized patients meeting randomization criteria. Journal of the American College of Cardiology, 1984, 3, 114–28.Google Scholar
Coronary Artery Surgery Study (CASS). A randomized trial of coronary artery bypass surgery: Quality of life in patients randomly assigned to treatment groups. Circulation, 1983, 68, 951–60.CrossRefGoogle Scholar
Cowley, M. J., Mullin, S. M., Kelsey, S. F. et al. , Sex differences in early and long-term results of coronary angioplasty in the NHLBI PTCA Registry. Circulation, 1985, 71, 90–97.CrossRefGoogle ScholarPubMed
Cuzick, J., Stewart, N., Peto, R. et al. , Overview of randomized trials of postoperative adjuvant radiotherapy in breast cancer. Cancer Treatment Reports, 1987, 71, 15–29.Google ScholarPubMed
Cuzick, J., Stewart, H., Peto, R. et al. , Overview of randomized trials comparing radical mastectomy without radiotherapy against simple mastectomy with radiotherapy in breast cancer. Cancer Treatment Reports, 1987, 71, 7–14.Google ScholarPubMed
Department of Clinical Epidemiology and Biostatistics, McMaster University Medical Centre How to read clinical journals: V. To distinguish useful from useless or even harmful therapy. Canadian Medical Association Journal, 1981, 124, 1156–62.Google Scholar
DerSimonian, R. & Laird, N.Meta-analysis in clinical trials. Controlled Clinical Trials, 1986, 7, 177–88.CrossRefGoogle ScholarPubMed
Detre, K. M., Myler, R. K., Kelsey, S. F. et al. , Baseline characteristics of patients in the National Heart, Lung, and Blood Institute Percutaneous Transluminal Coronary Angioplasty Registry. American Journal of Cardiology, 1984, 53, 7C–11C.CrossRefGoogle ScholarPubMed
Dickersin, K., Chan, S., Chalmers, T. C. et al. , Publication bias and clinical trials. Controlled Clinical Trials, 1987, 8, 343–53.CrossRefGoogle ScholarPubMed
Dickersin, K., Hewitt, P., Mutch, L. et al. , Perusing the literature: Comparison of MEDLINE searching with a perinatal trials database. Controlled Clinical Trials, 1985, 6, 306–17.CrossRefGoogle ScholarPubMed
Diehl, L. F. & Perry, D. J.A comparison of randomized concurrent control groups with matched historical control groups: Are historical controls valid? Journal of Clinical Oncology, 1986, 4, 1114–20.CrossRefGoogle ScholarPubMed
Elwood, J. M.Interpreting clinical trial results: Seven steps to understanding. Canadian Medical Association Journal, 1980, 123, 343–45.Google ScholarPubMed
European Coronary Surgery Study Group. Coronary-artery bypass surgery in stable angina pectoris: Survival at two years. Lancet, 1979, i, 889–93.Google Scholar
European Coronary Surgery Study Group. Long-term results of prospective randomised study of coronary artery bypass surgery in stable angina pectoris. Lancet, 1982, 27, 1173–80.Google Scholar
Extracranial/Intracranial Bypass Study Group. Failure of extracranial-intracranial arterial bypass to reduce the risk of ischemic stroke. Results of an international randomized trial. New England Journal of Medicine, 1985, 313, 1191–200.CrossRefGoogle Scholar
Favaloro, R. G.Saphenous vein autograft replacement of severe segmental coronary artery occlusion: Operative technique. Annals of Thoracic Surgery, 1968, 5, 334–39.CrossRefGoogle ScholarPubMed
Freiman, J. A., Chalmers, T. C., Smith, H. Jr & Kuebler, R. R.The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 “negative” trials. New England Journal of Medicine, 1978, 299, 690–94.CrossRefGoogle ScholarPubMed
Funk, M. E., Reid, C. A. & McGoogan, L. S.Indexing consistency in MEDLINE. Bulletin of the Medical Library Association, 1983, 71, 17683.Google ScholarPubMed
Garceau, A. J., Donaldson, R. M., O'Hara, E. T. et al. , Boston Inter-Hospital Liver Group. A controlled trial of prophylactic portacaval shunt surgery. New England Journal of Medicine, 1964, 270, 496–500.CrossRefGoogle Scholar
Glass, G. V.Primary, secondary and meta-analysis of research. Educational Researcher, 1976, 5, 3–8.CrossRefGoogle Scholar
Goldring, S., Zervas, N. & Langfitt, T.The Extracranial-Intracranial Bypass Study. A report of the committee appointed by the American Association of Neurological Surgeons to examine the study. New England Journal of Medicine, 1987, 316, 817–20.CrossRefGoogle Scholar
Green, S. B. & Byar, D. P.Using observational data from registries to compare treatments: The fallacy of omnimetrics. Statistics in Medicine, 1984, 3, 361–73.CrossRefGoogle ScholarPubMed
Guyatt, G. H., Tugwell, P. X., Feeney, D. H. et al. , A framework for clinical evaluation of diagnostic technologies. Canadian Medical Association Journal, 1986, 134, 587–94.Google ScholarPubMed
Halsted, W. S.The results of operations for the cure of cancer of the breast performed at Johns Hopkins Hospital from June 1889-January 1894. Annals of Surgery, 1894, 20, 497–555.CrossRefGoogle Scholar
Kloster, F. E., Kremkau, E. L., Ritzmann, L. W. et al. , Coronary bypass for stable angina: A prospective randomized study. New England Journal of Medicine, 1979, 300, 149–57.CrossRefGoogle ScholarPubMed
Liberati, A., Himel, H. N. & Chalmers, T. C.A quality assessment of randomized control trials of primary treatment of breast cancer. Journal of Clinical Oncology, 1986, 4, 942–51.CrossRefGoogle ScholarPubMed
Mak, N., Chalmers, T. C., Moskowitz, G. et al. , Good technology and poor clinical science [abstract]. Clinical Research, 1983, 31, 525A.Google Scholar
Mathur, V. S., Guinn, G. A., Anastassiades, L. C. et al. , Surgical treatment for stable angina pectoris. Prospective randomized study. New England Journal of Medicine, 1975, 292, 709-13.CrossRefGoogle ScholarPubMed
Merrick, N. J., Brook, R. H., Fink, A. & Solomon, D. H.Use of carotid endarterectomy in five California Veterans Administration medical centers. Journal of the American Medical Association, 1986, 256, 2531–35.CrossRefGoogle ScholarPubMed
Messer, J., Reitman, D., Sacks, H. S. et al. , Association of adrenocorticosteroid therapy and peptic-ulcer disease. New England Journal of Medicine, 1983, 309, 21–24.CrossRefGoogle ScholarPubMed
Micciolo, R., Valagussa, P. & Marubini, E.The use of historical controls in breast cancer: an assessment in three consecutive trials. Controlled Clinical Trials, 1985, 6, 259–70.CrossRefGoogle ScholarPubMed
Mosteller, F., Gilbert, J. P. & McPeek, B.Reporting standards and research strategies for controlled trials: Agenda for the editor. Controlled Clinical Trials, 1980, 1, 37–58.CrossRefGoogle Scholar
Murphy, M. L., Hultgren, H. N., Detre, K. et al. , Treatment of chronic stable angina. A preliminary report of survival data of the randomized Veterans Administration cooperative study. New England Journal of Medicine, 1977, 297, 621–27.CrossRefGoogle ScholarPubMed
Norris, R. M., Agnew, T. M., Brandt, P. W. et al. , Coronary surgery after recurrent myocardial infarction: Progress of a trial comparing surgical with nonsurgical management for asymptomatic patients with advanced coronary disease. Circulation, 1981, 63, 785–92.CrossRefGoogle ScholarPubMed
Reitman, D., Chalmers, T. C., Nagalingam, R. & Sacks, H. S.Can efficacy of blinding in randomized control trials (RCTs) be documented by meta-analysis [abstract] Controlled Clinical Trials 1988, 9, 250.CrossRefGoogle Scholar
Reitman, D., Sacks, H. S. & Chalmers, T. C.Technical quality assessment of randomized control trials (RCTs) [abstract]. Controlled Clinical Trials, 1987, 8, 282.Google Scholar
Rosati, R. A., Lee, K. L., Califf, R. M. et al. , Problems and advantages of an observational database approach to evaluating the effect of therapy on outcome. Circulation, 1982, 65, 27–32.CrossRefGoogle Scholar
Ruffin, J. M., Grizzle, J. E., Hightower, N. C. et al. , A cooperative double-blind evaluation of gastric “freezing” in the treatment of duodenal ulcer. New England Journal of Medicine, 1969, 281, 16–19.CrossRefGoogle ScholarPubMed
Sacks, H., Chalmers, T. C. & Smith, H. Jr, Randomized versus historical controls for clinical trials. American Journal of Medicine, 1982, 72, 233–40.CrossRefGoogle ScholarPubMed
Sacks, H. S., Berrier, J., Reitman, D. et al. , Meta-analyses of randomized controlled trials. New England Journal of Medicine, 1987, 316, 450–55.CrossRefGoogle ScholarPubMed
Schoolman, H. M., Becktel, J. M., Best, W. R. & Johnson, A. F.Statistics in medical research: Principles versus practices. Journal of Laboratory and Clinical Medicine, 1968, 71, 357–67.Google ScholarPubMed
Sheps, S. B. & Schechter, M. T.The assessment of diagnostic tests: Survey of current medical research. Journal of the American Medical Association, 1984, 252, 2418–22.CrossRefGoogle ScholarPubMed
Simes, R. J.Publication bias: The case for an international registry of clinical trials. Journal of Clinical Oncology, 1986, 4, 1529–41.CrossRefGoogle ScholarPubMed
Spodick, D. H.The surgical mystique and the double standard. Controlled trials of medical and surgical therapy for cardiac disease: Analysis, hypothesis, proposal. American Heart Journal, 1973, 85, 579–83.CrossRefGoogle ScholarPubMed
Stjernsward, J.Can survival be decreased by post-operative irradiation? International Journal of Radiation, Oncology, Biology, Physics, 1977, 2, 1171–75.CrossRefGoogle ScholarPubMed
Sundt, T. M. Jr, Was the international randomized trial of extracranial-intracranial arterial bypass representative of the population at risk? New England Journal of Medicine, 1987, 316, 814–16CrossRefGoogle ScholarPubMed
Swets, J. A., Pickett, R. M., Whitehead, S. F. et al. , Assessment of diagnostic technologies. Science, 1979, 205, 753–59.CrossRefGoogle ScholarPubMed
Tyson, J. E., Furzan, J. A., Reisch, J. S. & Mize, S. G.An evaluation of the quality of therapeutic studies in perinatal medicine. Journal of Pediatrics, 1983, 102, 10–13.CrossRefGoogle ScholarPubMed
Veterans Administration Coronary Artery Bypass Surgery Cooperative Study Group. Eleven-year survival in the Veterans Administration randomized trial of coronary bypass surgery for stable angina. New England Journal of Medicine, 1984, 311, 1333–39.CrossRefGoogle Scholar
Wangensteen, O. H., Peter, E. T., Nicoloff, D. M., et al. Achieving “physiological gastrectomy” by gastric freezing: A preliminary report of an experimental and clinical study. Journal of the American Medical Association, 1962, 180, 439–44.CrossRefGoogle ScholarPubMed
Wangensteen, O. H., Root, H. D., Jensen, C. B. et al. , Depression of gastric secretion and digestion by gastric hypothermia: Clinical use in massive hematemesis. Surgery, 1958, 44, 165–74.Google ScholarPubMed
Wennberg, J. E.Setting outcome-based standards for carotid endarterectomy. Journal of the American Medical Association, 1986, 256, 2566–67.CrossRefGoogle ScholarPubMed
Wennberg, J. E., Mulley, A. G. Jr. Hanley, D. et al. , An assessment of prostatectomy for benign urinary tract obstruction. Geographic variations and the evaluation of medical care outcomes. Journal of the American Medical Association, 1988, 259, 3027–30.CrossRefGoogle ScholarPubMed
Yusuf, S., Collins, R. & Peto, R.Why do we need some large, simple randomised trials? Statistics in Medicine, 1984, 3, 409–22.CrossRefGoogle Scholar