No CrossRef data available.
Article contents
Empirically evaluating the impact of adjudicative tribunals in the health sector: context, challenges and opportunities
Published online by Cambridge University Press: 26 August 2011
Abstract
Adjudicative tribunals are an integral part of health system governance, yet their real-world impact remains largely unknown. Most assessments focus on internal accountability and use anecdotal methodologies; few, studies if any, empirically evaluate their external impact and use these data to test effectiveness, track performance, inform service improvements and ultimately strengthen health systems. Given that such assessments would yield important benefits and have been conducted successfully in similar settings (e.g. specialist courts), their absence is likely attributable to complexity in the health system, methodological difficulties and the legal environment within which tribunals operate. We suggest practical steps for potential evaluators to conduct empirical impact evaluations along with an evaluation matrix template featuring possible target outcomes and corresponding surrogate endpoints, performance indicators and empirical methodologies. Several system-level strategies for supporting such assessments have also been suggested for academics, health system institutions, health planners and research funders. Action is necessary to ensure that policymakers do not continue operating without evidence but can rather pursue data-driven strategies that are more likely to achieve their health system goals in a cost-effective way.
- Type
- Articles
- Information
- Copyright
- Copyright © Cambridge University Press 2011
References
Adler, M.Gulland, J. (2003), Tribunal Users’ Experiences, Perceptions and Expectations: A Literature Review, London: Council on Tribunals, http://www.council-on-tribunals.gov.uk/docs/other_adler(2).pdf [2 February 2009].Google Scholar
Allsop, J.Jones, K. (2008), ‘Withering the citizen managing the consumer: complaints in healthcare settings’, Social Policy and Society, 7(2): 233–243.CrossRefGoogle Scholar
Aston, J., Hill, D.Tackey, N. D. (2006), The Experience of Claimants in Race Discrimination Employment Tribunal Cases. Employment Relations Research Series, No. 55, London: UK Department of Trade and Industry, http://www.dti.gov.uk/files/file27818.pdf [12 March 2009].Google Scholar
Blumenthal, S.Wessely, S. (1993), The Pattern of Delays in Mental Health Review Tribunals, London: Stationery Office.Google Scholar
Blumenthal, S.Wessely, S. (1994), ‘The pattern of delays in mental health review tribunals’, Psychiatric Bulletin, 18(7): 398–400, http://pb.rcpsych.org/cgi/reprint/18/7/398.pdf [12 March 2009].CrossRefGoogle Scholar
Bradley, C., Marshall, M.Gath, D. (1995), ‘Why do so few patients appeal against detention under the mental health act?’, British Medical Journal, 310: 364–367, http://www.bmj.com/cgi/content/full/310/6976/364 [12 March 2009].CrossRefGoogle ScholarPubMed
Campbell, N. C., Murray, E., Darbyshire, J., Emery, J., Farmer, A., Griffiths, F., Guthrie, B., Lester, H., Wilson, P.Kinmonth, A. L. (2007), ‘Designing and evaluating complex interventions to improve health care’, British Medical Journal, 334: 455–459.CrossRefGoogle ScholarPubMed
Cane, P. (2004), ‘Understanding Judicial Review and its Impact’, in M. Hertogh and S. Halliday (eds), Judicial Review and Bureaucratic Impact: International and Interdisciplinary Perspectives, Cambridge, UK: Cambridge University Press, 15–42.CrossRefGoogle Scholar
Carscallen, M. P., Gray, W. K.Pink, J. G. (2003), Regulatory Burden Task Force: Report to the Ontario Securities Commission, Toronto: Ontario Securities Commission, http://www.osc.gov.on.ca/About/Governance/Accountability/ga_20031212_rbtf-rpt.pdf [12 March 2009].Google Scholar
Center for Global Development's Evaluation Gap Working Group (2006), When Will We Ever Learn? Improving Lives Through Impact Evaluation, Washington, DC: Center for Global Development, http://www.cgdev.org/files/7973_file_WillWeEverLearn.pdf [22 June 2009].Google Scholar
Chalkidou, K., Tunis, S., Lopert, R., Rochaix, L., Sawicki, P. T., Nasser, M.Xerri, B. (2009), ‘Comparative effectiveness research and evidence-based health policy: experience from four countries’, Milbank Quarterly, 87(2): 339–367, http://www.milbank.org/quarterly/milq_87_2-final-chalkidou.pdf [5 September 2009].CrossRefGoogle ScholarPubMed
Confederation of British Industry (2005), A Matter of Confidence: Restoring Faith in Employment Tribunals, London: Confederation of British Industry, http://www.cbi.org.uk/ndbs/Press.nsf/0/33f9830ed75f765b8025708800523621/$FILE/TribunalsBrief—CBI.pdf [12 March 2009].Google Scholar
Corsi, J. R.Hurley, T. L. (1979), ‘Pilot study report on the use of the telephone in administrative hearings’, Administrative Law Review, 31: 485.Google Scholar
Dolan, M., Gibb, R.Coorey, P. (1999), ‘Mental health review tribunals: a survey of special hospital patients’ opinions’, Journal of Forensic Psychiatry, 10: 264–275.CrossRefGoogle Scholar
Ellis, R. (2009), ‘Executive Branch Justice: Canada's ‘Official Courts’, PhD Dissertation. Toronto: York University.Google Scholar
Employment Tribunals Service (2005), Employment Tribunals Service User Survey 2005, London: Employment Tribunals Service.Google Scholar
Evidence-Based Medicine Working Group (1992), ‘Evidence-based medicine: a new approach to teaching the practice of medicine’, Journal of the American Medical Association, 268: 2420–2425.CrossRefGoogle Scholar
Genn, H., Partington, M.Wheeler, S. (2006a), Nuffield Inquiry on Empirical Legal Research: Law in the Real World: Improving our Understanding of How Law Works: Final Report and Recommendations, London: Nuffield Foundation, http://www.ucl.ac.uk/laws/socio-legal/empirical/docs/inquiry_report.pdf [27 April 2009].Google Scholar
Genn, H., Lever, B.Gray, L., National Centre for Social Reasearch (2006b), Tribunals for Diverse Users, London: UK Department for Constitutional Affairs, http://www.dca.gov.uk/research/2006/01_2006.pdf [29 March 2009].Google Scholar
George, T. E. (2006), ‘An empirical study of empirical legal scholarship: the top law schools’, Indiana Law Journal, 81(1): 141–161, http://www.law.indiana.edu/ilj/volumes/v81/no1/8_George.pdf [15 January 2009].Google Scholar
GRADE Working Group (2004), ‘Grading quality of evidence and strength of recommendations’, British Medical Journal, 328: 1490, doi:10.1136/bmj.328.7454.1490. http://www.bmj.com/cgi/content/full/328/7454/1490 [10 April 2009].CrossRefGoogle Scholar
Green, B. L., Furrer, C. J., Worcel, S. D., Burrus, S. W. M.Finigan, M. W. (2009), ‘Building the evidence base for family drug treatment courts: results from recent outcome studies’, Drug Court Review, 6(2): 53–82, http://www.ndci.org/sites/default/files/ncdc/DCR%2C%20Vol.%206%2C%20No.%202.pdf [23 August 2009].Google Scholar
Gresenz, C. R.Studdert, D. M. (2005), ‘External review of coverage denials by managed care organizations in California’, Journal of Empirical Legal Studies, 2(3): 449–468.CrossRefGoogle Scholar
Guzzo, G., Baird, J., Grimmett, B., Martiniuk, G.Flaherty, J. (1998), Everyday Justice: Report of the Agency Reform Commission on Ontario's Regulatory and Adjudicative Agencies, Toronto: Government of Ontario, http://www.ccat-ctac.org/downloads/1998_Guzzo-report.pdf [accessed on 12 March 2009].Google Scholar
Hayward, B., Peters, M., Rousseau, N.Seeds, K. (2004), Findings from the Survey of Employment Tribunal Applications 2003. Employment Relations Research Series, No. 33, London: UK Department of Trade and Industry, http://www.dti.gov.uk/files/file11455.pdf [12 March 2009].Google Scholar
Health Professions Appeal and Review Board (2008), Recommendations to Health Professions Regulatory Advisory Council, Toronto: Health Professions Appeal and Review Board, http://www.hprac.org/en/projects/resources/hprac-1457May30HPARB.pdf [28 March 2009].Google Scholar
Health Professions Regulatory Advisory Council (2008), HPRAC: Annual Report April 1, 2007–March 31, 2008, Toronto: Health Professions Regulatory Advisory Council, http://www.hprac.org/en/reports/resources/HPRACAnnualReport2007-2008.pdf [22 June 2009].Google Scholar
Hertogh, M.Halliday, S. (eds) (2004), Judicial Review and Bureaucratic Impact: International and Interdisciplinary Perspectives, Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Hoffman, S. J., Lavis, J. N.Bennett, S. (2009), ‘The use of research evidence in two international organizations’ recommendations about health systems’, Healthcare Policy, 5(1): 66–86.Google ScholarPubMed
Jacobs, L. (2008), Reconciling Tribunal Independence and Expertise – Empirical Observations. Presented at The Future of Administrative Justice Symposium, University of Toronto, 18 January 2008, http://www.law.utoronto.ca/documents/conferences/adminjustice08_Jacobs.pdf [29 November 2008].Google Scholar
Jones, N., Jones, H., Steer, L.Datta, A. (2009), Working Paper 300: Improving Impact Evaluation Production and Use, London: Overseas Development Institute, http://www.odi.org.uk/resources/download/3177.pdf [22 June 2009].Google Scholar
Latreille, P. L., Latreille, J. A.Knight, K. G. (2004), Findings from the 1998 Survey of Representatives in Employment Tribunal Cases. Employment Relations Research Series, No. 35, London: UK Department of Trade and Industry, http://www.berr.gov.uk/files/file11453.pdf [12 March 2009].Google Scholar
Latreille, P. L., Latreille, J. A.Knight, K. G. (2005), ‘Making a difference? Legal representation in employment tribunal cases: evidence from a survey of representatives’, Industrial Law Journal, 34(4): 308–330.CrossRefGoogle Scholar
Lavis, J. N., Becerra-Posada, F., Haines, A.Osei, E. (2004), ‘Use of research to inform public policymaking’, The Lancet, 364(9445): 1615–1621.CrossRefGoogle ScholarPubMed
Lavis, J. N., Davies, H. T. O., Oxman, A., Densi, J. L., Golden-Biddle, K.Ferlie, E. (2005), ‘Towards systematic reviews that inform healthcare management and policymaking’, Journal of Health Services Research and Policy, 10(Supplement 1): 35–48.CrossRefGoogle ScholarPubMed
Leggatt, A. (2001), Tribunals for Users: One System, One Service, London: UK Department for Constitutional Affairs, http://www.tribunals-review.org.uk/index.htm [12 March 2009].Google Scholar
Marin, A. (2008), Ombudsman Ontario: Annual Report 2007–2008, Toronto: Office of the Ombudsman, http://www.ombudsman.on.ca/media/18971/ar08_eng.pdf [22 June 2009].Google Scholar
Maynard, A. (2006), ‘The “long and lonely road”: translating evidence into policy’, Health Economics, Policy and Law, 1(4): 415–421, doi:10.1017/S1744133106006049.CrossRefGoogle ScholarPubMed
Mayne, J. (2001), ‘Addressing attribution through contribution analysis: using performance measures sensibly’, Canadian Journal of Program Evaluation, 16(1): 1–24.CrossRefGoogle Scholar
McCarter, J. (2008), Annual Report 2008: Office of the Auditor-General, Toronto: Queen's Printer for Ontario, http://www.auditor.on.ca/en/reports_en/en08/ar_en08.pdf [22 June 2009].Google Scholar
McRuer, J. C. (1968), Ontario Royal Commission Inquiry Into Civil Rights: Report No. 1, Toronto: Queen's Printer.Google Scholar
Mello, M. M.Zeiler, K. (2008), ‘Empirical health law scholarship: the state of the field’, Georgetown Law Journal, 96: 649–702.Google Scholar
Mello, M. M., Studdert, D. M., Kachalia, A.Brennan, T. A. (2006), ‘ “Health courts” and accountability for patient safety’, Milbank Quarterly, 84: 459–492.CrossRefGoogle ScholarPubMed
Moyer, P. (1997), ‘The regulation of corporate law by securities regulators: a comparison of Ontario and the United States’, University of Toronto Faculty of Law Review, 55(1): 43–76.Google Scholar
National Audit Office (2003a), Progress in Improving the Medical Assessment of Incapacity and Disability Benefits: Report by the Comptroller and Auditor General. HC 1141, Session 2002–2003, 17/10/2003, London: Stationery Office, http://www.nao.org.uk/publications/nao_reports/02-03/02031141.pdf [12 March 2009].Google Scholar
National Audit Office (2003b), Getting It Right, Putting It Right: Improving Decision-Making and Appeals in Social Security Benefits. HC 1142, Session 2002–2003, 07/11/2003, London: Stationery Office, http://www.nao.org.uk/publications/nao_reports/02-03/02031142.pdf [12 March 2009].Google Scholar
OECD Development Assistance Committee (1998), Review of the DAC Principles for Evaluation of Development Assistance, Paris: Organisation for Economic Co-operation and Development, http://www.oecd.org/dataoecd/63/50/2065863.pdf [22 June 2009].Google Scholar
Osborne, C. A., Mullan, D. J.Finlay, B. (2004), Report of the Fairness Committee to David A. Brown, Q.C., Chair of the Ontario Securities Commission, Toronto: Ontario Securities Commission, http://www.osc.gov.on.ca/Regulation/FiveYearReview/fyr_20040818_fairness-committee.pdf [12 January 2009].Google Scholar
Partington, M., Kirton-Darling, E.McClenaghan, F. (2007), Empirical Research on Tribunals: An Annotated Review of Research Published between 1992 and 2007, London: Administrative Justice and Tribunals Council, http://www.ajtc.gov.uk/docs/EmpiricalResearch.pdf [9 March 2009].Google Scholar
Pearson, L. (2007), The Impact of External Administrative Law Review: Tribunals. University of New South Wales Faculty of Law Research Series, Paper 53, Sydney: University of New South Wales, http://law.bepress.com/unswwps/flrps/art53 [10 April 2011].CrossRefGoogle Scholar
Peay, J. (1989), Tribunals on Trial: A Study of Decision Making Under the Mental Health Act 1983, Oxford: Clarendon Press.Google Scholar
Petticrew, M.Roberts, H. (2008), ‘Systematic reviews – do they “work” in informing decision-making around health inequalities?’, Health Economics, Policy and Law, 3(2): 197–211.CrossRefGoogle ScholarPubMed
Pitfield, C.Flood, C. M. (2005), ‘Section 7 “safety valves”: appealing wait times within a one-tier system’, in C. M. Flood, K. Roach and L. Sossin (eds), Access to Care, Access to Justice: The Legal Debate Over Private Health Insurance in Canada, Toronto: University of Toronto Press, 477–501.Google Scholar
Pleasence, P. (2008), ‘Trials and tribulations: conducting randomized experiments in a socio-legal setting’, Journal of Law and Society, 35(1): 8–29.CrossRefGoogle Scholar
Plotnikoff, J.Woolfson, R. (2005), Review of the Effectiveness of Specialist Courts in Other Jurisdictions. DCA Report Series, No. 3/05, London: UK Department for Constitutional Affairs, http://www.dca.gov.uk/research/2005/3_2005.pdf [10 April 2009].Google Scholar
Plsek, P. E.Greenhalgh, T. (2001), ‘Complexity science: the challenge of complexity in health care’, British Medical Journal, 323: 625–628.CrossRefGoogle ScholarPubMed
Rothstein, H., Sutton, A.Borenstein, M. (eds) (2005), Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments, West Sussex, England: John Wiley & Sons.CrossRefGoogle Scholar
Rousseau, S. (2008), The Québec Experience with an Independent Administrative Tribunal Specialized in Securities: A Study of the Bureau de décision et de révision en valeurs mobilières, Ottawa: Expert Panel on Securities Regulation, http://www.groupeexperts.ca/eng/reports/research-studies/quebec-independent-adjudicative-tribunal-rousseau.php [12 March 2009].Google Scholar
Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B.Richardson, W. S. (1996), ‘Evidence based medicine: what it is and what it isn't’, British Medical Journal, 312(7023): 71–72, http://www.bmj.com/cgi/content/full/312/7023/71 [8 January 2009].CrossRefGoogle ScholarPubMed
Siegal, G., Mello, M. M.Studdert, D. M. (2008), ‘Adjudicating severe birth injury claims in Florida and Virginia: the experience of a landmark experiment in personal injury compensation’, American Journal of Law and Medicine, 34: 489–533, http://www.hsph.harvard.edu/faculty/michelle-mello/files/FL-VA_PDF.pdf [2 February 2009].CrossRefGoogle ScholarPubMed
Society of Ontario Adjudicators and Regulators (1995), Towards Maintaining and Improving the Quality of Adjudication: SOAR Recommendations for Performance Management in Ontario's Administrative Justice Tribunals, Toronto: Society of Ontario Adjudicators and Regulators, http://www.soar.on.ca/soar-perf_man.htm [12 March 2009].Google Scholar
Sossin, L. (2007), The Effectiveness and Efficiency of Pension Regulation in Ontario and in Comparative Perspective, Toronto: Expert Commission on Pensions, http://www.pensionreview.on.ca/english/summaries/8Sossin.html [accessed on 29 November 2008].Google Scholar
Sunkin, M. (2004), ‘Conceptual issues in researching the impact of judicial review on government bureaucracies’, in M. Hertogh and S. Halliday (eds), Judicial Review and Bureaucratic Impact: International and Interdisciplinary Perspectives, Cambridge, UK: Cambridge University Press, 43–75.CrossRefGoogle Scholar
Thomas, R. (2005), ‘Evaluating tribunal adjudication: administrative justice and asylum appeals’, Legal Studies, 25(3): 462–498.CrossRefGoogle Scholar
Toubman, A. A., McArdle, T.Rogers-Tomer, L. (1995), ‘Due process implications of telephone hearings: the case for an individualized approach to scheduling telephone hearings’, University of Michigan Journal of Law Reform, 29: 407–474.Google Scholar
United States Government Accountability Office (2005), Adult Drug Courts: Evidence Indicates Recidivism Reductions and Mixed Results for Other Outcomes, Washington, DC: United States Government Accountability Office, http://www.gao.gov/new.items/d05219.pdf [23 August 2009].Google Scholar
Urwin, P., Karuk, V., Latreille, P., Michielsens, E., Page, L., Siara, B., Speckesser, S., Boon, A.Chevalier, P. A. (2010), Evaluating the Use of Judicial Mediation in Employment Tribunals. Ministry of Justice Research Series 7/10, London: Ministry of Justice, http://www.justice.gov.uk/publications/docs/evaluating-judicial-mediation-march10.pdf [10 April 2011].Google Scholar
Wilson, D. B., Mitchell, O.Mackenzie, D. L. (2006), ‘A systematic review of drug court effects on recidivism’, Journal of Experimental Criminology, 2: 459–487, doi:10.1007/s11292-006-9019-4.CrossRefGoogle Scholar
Wilson, D. B., Mitchell, O.MacKenzie, D. L. (2007), Effects of Drug Courts on Criminal Offending and Drug Use: A Campbell Collaboration Systematic Review Protocol, Oslo, Norway: Campbell Library of Systematic Reviews, http://campbellcollaboration.org/lib/download/367 [10 April 2011].Google Scholar
World Bank (2006), Conducting Quality Impact Evaluations under Budget, Time and Data Constraints, Washington, DC: World Bank, http://lnweb90.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/757A5CC0BAE22558852571770059D89C/$file/conduct_qual_impact.pdf [22 June 2009].Google Scholar
World Health Organization (2004), World Report on Knowledge for Better Health, Geneva: World Health Organization, http://www.who.int/rpc/meetings/world_report_on_knowledge_for_better_health.pdf [15 April 2009].Google Scholar
Zimmerman, B., Plsek, P.Lindberg, C. (1998), Edgeware: Insights from Complexity Science for Health Care Leaders, Irving, TX: VHA Inc.Google Scholar
Hoffman Supplementary Appendix A
Online Appendix A: Twenty-Seven Examples of Evaluations of Adjudicative Tribunals
File
60.9 KB
Hoffman Supplementary Appendix B
Online Appendix B: Ninety-One Examples of External Impact Evaluations of Specialist Courts
File
92.2 KB