Hostname: page-component-cd9895bd7-gbm5v Total loading time: 0 Render date: 2024-12-25T20:39:18.953Z Has data issue: false hasContentIssue false
  • English
  • Français

Policy Evaluation in the EU: The Challenges of Linking Ex Ante and Ex Post Appraisal

Published online by Cambridge University Press:  20 January 2017

Stijn Smismans
Stijn Smismans*
Affiliation:
School of Law and Politics, the Centre for European Law and Governance, (Jean Monnet Centre of Excellence), Cardiff University
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

The EU's new approach to policy evaluation is characterised by a focus on closing the policy cycle (linking ex ante and ex post appraisal) and by applying evaluation to all types of policy intervention, whether expenditure or regulatory policy. This article analyses the main features and challenges of this new approach. It first studies the conceptual and interdisciplinary challenge of such an encompassing approach to evaluation. It then assesses the new approach in the light of four key objectives of ex ante and ex post appraisal; ensuring evidence and learning; accountability, transparency and participation; policy coherence; and reducing the regulatory burden.

Type
Symposium on Policy Evaluation in the EU
Information
European Journal of Risk Regulation , Volume 6 , Issue 1 , March 2015 , pp. 6 - 26
Creative Commons
Creative Common License - CCCreative Common License - BY
© Stijn Smismans 2015 This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licences/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
Copyright © Cambridge University Press 2013

References

1 E.g. for an overview of 21 countries and three international organisations, see Furubo, Jan–Eric, Rist, Ray C. and Rolf. Sandahl, (eds.), International atlas of evaluation (New Brunswick: NJ Transaction, 2002).Google Scholar

2 For more detail, see Anne Meuwese in this Special Issue.

3 There are rare exceptions, mainly in the field of structural funds policy, e.g. Stern, Elliot, “Evaluation policy in the European Union and its institutions” New Directions for Evaluation (2009), pp. 67 et sqq. CrossRefGoogle Scholar; Mendez, Carlos and Bachtler, John, “Administrative reform and unintended consequences: an assessment of the EU cohesion policy ‘audit explosion’”, 18 Journal of European Public Policy (2011), pp. 746 et sqq. CrossRefGoogle Scholar; Hoerner, Julian and Stephenson, Paul, “Theoretical Perspectives on Approaches to Policy Evaluation in the EU: The Case of Cohesion Policy” 90 Public Administration (2012), pp. 699 et sqq Google Scholar. Iannacci, Federico, Cornford, Tony, Cordella, Antonio and Grillo, Francesco, “Evaluating monitoring systems in the European social fund context: a sociotechnical approach”, 33 Evaluation Review (2009), pp. 419 et sqq. CrossRefGoogle Scholar; and Højlund, Steven, “Evaluation use in evaluation systems – the case of the European Commission”, 20 Evaluation (2014), pp. 428 et sqq. CrossRefGoogle Scholar

4 The broader historical background of EU policy evaluation is set out further by Højlund in this Special Issue.

5 Mark, Melvin M., Cooksy, Leslie J. and Trochim, William M.K., “Evaluation Policy: an introduction and overview”, 123 New Directions in Evaluation (2009), pp. 3 et sqq..CrossRefGoogle Scholar For a similar critique of the literature on ex ante appraisal missing out on the “policy and politics” of appraisal, see Turnpenny, John, Radaelli, Claudio M., Jordan, Andrew and Jacob, Klaus, “The policy and politics of policy appraisal: emerging trends and new direction”, 16 Journal of European Public Policy (2009), pp. 640 et sqq. CrossRefGoogle Scholar

6 Commission Communication “Strengthening the foundations of Smart Regulation – improving evaluation”, COM(2013) 686 final, at p. 7.

7 With the exception of the specific guidelines for ex ante evaluation on expenditure programmes. Commission, “Ex Ante Evaluation. A practical guide for preparing proposals for expenditure programmes”, DG Budget, 10 December 2001.

8 See also Emanuela Bozzini and Jo Hunt in this Special Issue.

9 Highlighted above, supra note 3.

10 Mark, Melvin M., Cooksy, Leslie J. and Trochim, William M.K., “Evaluation Policy: an introduction and overview”, 123 New Directions in Evaluation (2009), pp. 3 et sqq. CrossRefGoogle Scholar

11 Although the topic is only slowly finding its way to the mainstream of public policy. As Turpnenny et al. note, it is telling that no mainstream public policy textbook covers policy appraisal in much detail. Turnpenny, John, Radaelli, Claudio M., Jordan, Andrew and Jacob, Klaus, “The policy and politics of policy appraisal: emerging trends and new direction”, 16 Journal of European Public Policy (2009), pp. 640 et sqq., at p. 641.CrossRefGoogle Scholar

12 Adelle, Camille, Jordan, Andrew and Turnpenny, John, “Proceeding in parallel or drifting apart? A systematic review of policy appraisal research and practices”, 30 Environment and Planning C: Government and Policy (2012), pp. 401 et sqq. CrossRefGoogle Scholar

13 The four main usages are political, instrumental, communicative and perfunctory. Dunlop, Claire, Maggetti, Martino, Radaelli, Claudio and Russel, Duncan, “The Many Uses of Regulatory Impact Assessment: A Meta–Analysis of EU and UK Cases”, 6 Regulation and Governance (2012), pp. 23 et sqq. CrossRefGoogle Scholar

14 Højlund, Steven, “Evaluation use in evaluation systems – the case of the European Commission”, 20 Evaluation (2014), pp. 428 et sqq. CrossRefGoogle Scholar

15 Schrefler, Lorna, “Reflections on the different roles of expertise in regulatory policy making”, in Ambrus, Monika, Arts, Karin, Hey, Ellen, Raulus, Helena (eds), The Role of ‘Experts’ in International and European Decision–Making ProcessesAdvisors, Decision Makers or Irrelevant Actors?, (Cambridge: Cambridge University Press, 2014), pp. 63 et sqq..CrossRefGoogle Scholar

16 Power, Michael ,The Audit Society: Rituals of Verification, (Oxford: Oxford University Press, 1997).Google Scholar

17 Klabbers, Jan, “The virtues of expertise”, in Ambrus, Monika, Arts, Karin, Hey, Ellen, Raulus, Helena (eds), The Role of ‘Experts’ in International and European Decision–Making Processes. Advisors, Decision Makers or Irrelevant Actors, (Cambridge: Cambridge University Press, 2014), pp. 82 et sqq., at p. 87.CrossRefGoogle Scholar

18 Steven Højlund in this Special Issue, and Mendez and Bachtler, “Administrative reform”, supra note 3.

19 Commission, “Public Consultation on Commission Guidelines for Evaluation”, November 2013, available on the Internet at <http://ec.europa.eu/smart-regulation/evaluation/consultation/index_en.htm>, (last accessed on 20 May 2014), at p. 13.

20 Falkner, Gerda, Treib, Oliver, Hartlapp, Miriam and Leiber, Simone, Complying with Europe. EU harmonisation and soft law in the Member States, (Cambridge: Cambridge University Press, 2005)CrossRefGoogle Scholar; König, Thomas and Luetgert, Brooke, “Troubles with Transposition? Explaining Trends in Member–State Notification and Delayed Transposition of EU Directives”, 39 British Journal of Political Science (2009), pp. 163 et sqq. Google Scholar; Mastenbroek, Ellen and Kaeding, Michael, “Europeanization Beyond the Goodness of Fit: Domestic Politics in the Forefront”, Comparative European Politics (2006), pp. 331 et sqq. CrossRefGoogle Scholar; and Versluis, Esther “Even Rules, Uneven Practices: Opening the ‘Black Box’ of EU law in action”, 30 West European Politics (2007),pp. 50 et sqq..CrossRefGoogle Scholar

21 Featherstone, Keith and Radaelli, Claudio (eds.), The Politics of Europeanisation (Oxford: Oxford University Press, 2003)CrossRefGoogle Scholar; and Graziano, Paolo and Vink, Maarten, Europeanization: new research agendas (Palgrave, 2008)CrossRefGoogle Scholar

22 Højlund in this Special Issue.

23 Commission Communication “Responding to Strategic Needs: Reinforcing the use of evaluation”, SEC (2007) 2013.

24 Commission Communication “Smart Regulation in the European Union”, COM(2010) 543 final.

25 Commission Communication “Strengthening the foundations of Smart Regulation – improving evaluation”, supra note 6.

26 Commission, “Public Consultation on Commission Guidelines for Evaluation”, supra note 19.

27 Commission, “2014 Revision of the European Commission Impact Assessment Guidelines. Public Consultation document”, 1 July 2014 , available at <http://ec.europa.eu/smart-regulation/impact/consultation_2014/index_en.htm> (last accessed on 30 September 2014).

28 Although the guidelines do not provide for any systematic screening of DGs on whether they respect these guidelines, in a way that exists through the Impact Assessment Board.

29 Commission, “Evaluating EU Activities. Practical Guide for the Commission Services”, DG Budget, July 2004.

30 E.g. Commission Communication “Focus on results: strengthening evaluation of Commission activities”, SEC(2000)1051.

31 Commission Communication “EU Regulatory Fitness”, COM (2012) 746 final.

32 Commission Communication “Regulatory Fitness and Performance (REFIT): Results and Next Steps”, COM(2013) 685 final, at p. 2.

33 Commission Communication “Smart Regulation in the European Union”, supra note 25, at p. 4.

34 Commission Communication “Strengthening the foundations of Smart Regulation – improving evaluation”, supra note 6, at p. 5.

35 Stern, “Evaluation policy in the European Union”, supra note 3.

36 European Commission, “Evaluating EU Activities”, at p. 30.

37 Commission Communication “Strengthening the foundations of Smart Regulation – improving evaluation”, supra note 6, at p. 6.

38 Commission, “Evaluation”, 12 November 2014, available on the Internet at <http://ec.europa.eu/smart-regulation/evaluation/index_en.htm> (last accessed on 20 May 2014).

39 Commission, “Search evaluation results”, 24 July 2014, available on the Internet at http://ec.europa.eu/smart-regulation/evaluation/search/search.do (last accessed on 21 January 2015).

40 How representative the database is for the entirety of evaluations is difficult to assess. For sure, the database seems to focus on outsourced evaluations, leaving roughly 20% of internal evaluations uncovered.

41 Commission Communication “Impact Assessment”, COM(2002)276. Not all IIAs include such financial ex ante evaluation as the initiative may not engage the Union budget. Vice versa, ex ante financial evaluation continues to exist as a separate process for expenditure actions for which no IIA is required. However, ex anteevaluation is now predominantly conducted in the context of impact assessments. See Commission, “Public Consultation on Commission Guidelines for Evaluation”, supra note 19, at p. 16.

42 Commission, “Public Consultation on Commission Guidelines for Evaluation”, supra note 19, at p. 7.

43 European Impact Assessment Board, “Annual Report for 2013”, available on the Internet at <http://ec.europa.eu/smart-regulation/impact/key_docs/docs/iab_report_2013_en.pdf> (last accessed on 21 January 2015 at p. 7.

44 European Impact Assessment Board, “Annual Report for 2012”, available on the Internet at <http://ec.europa.eu/smart-regulation/impact/key_docs/docs/iab_report_2012_en_final.pdf> (last accessed on 21 January 2015), at p. 27.

45 Commission, “2014 Revision of the European Commission Impact Assessment Guidelines”, supra note 27, at p. 29.

46 Ibid., at p. 10.

47 Fitness checks constitute just one of the tools of the wider REFIT programme.

48 Commission, “Public Consultation on Commission Guidelines for Evaluation”, supra note 19, at p. 16.

49 Commission Communication “Regulatory Fitness and Performance (REFIT): Results and Next Steps”, COM(2013) 685 final, at p. 7.

50 Commission Communication “Commission Work Programme 2015. A new start”, COM(2014) 910 final, at Annex 3.

51 For a detailed assessment of the new CCA tool, based on analysis of the two pilot exercises, see Lorna Schrefler, Giacomo Luchetta and Felice Simonelli in this Special Issue.

52 Commission Communication “Regulatory Fitness and Performance Programme (REFIT): State of Play and Outlook”, COM(2014) 368 final, at p. 15.

53 Claimed objectives do not necessarily correspond with the (strategic) use of evaluation in practice. See Dunlop et al., “The many uses of RIA”, supra note 13; and Højlund, “Evaluation use”, supra note 14; as well as Dunlop and Radaelli in this Special Issue.

54 Both texts have been chosen as they provide the most comprehensive official definition of objectives for each category. For ex post evaluation I have relied on the 2013 Draft Guidelines, even if not yet in force, because the 2004 guidelines are clearly out of date and no longer in line with the new approach developed since then. For ex ante evaluation I have stuck to the 2009 Guidelines, rather than the 2014 Draft IIA guidelines, as they are not yet in force and do not set out the objectives as clearly as the 2009 guidelines.

55 Bäcklund, Ann–Katrin, “Impact assessment in the European Commission – a system with multiple objectives”, 12 Environmental Science and Policy (2009), pp. 1077 et sqq CrossRefGoogle Scholar, Rowe, Gerard G., “Tools for the control of political and administrative agents: impact assessment and administrative governance in the European Union” in Hofmann, Herwig C.H. and Turk, Alexander H. (eds.), EU Administrative Governance (Cheltenham: Edward Elgar, 2006), pp. 448 et sqq,Google Scholar; Scriven, Michael, “Beyond Formative and Summative Evaluation” in McLauglin, Milbrey and D.C. Philips (eds), Evaluation and Education: At Quarter Century (3 edn, Chicago: University of Chicago Press, 1991)Google Scholar; Van der Meer, Frans–Bauke and Edelenbos, Jurian, “Evaluation in multi–actor policy process: Accountability, learning and co–operation”, 12 Evaluation (2006), pp. 201 et sqq,CrossRefGoogle Scholar; Borrás, Susana and Højlund, Steven, “Evaluation and policy learning: the learners’ perspective”, 54 European Journal of Political Research (2015), pp. 99 et sqq,CrossRefGoogle Scholar.

56 Commission Communication “Strengthening the foundations of smart regulation – improving evaluation”, supra note 6, at p. 2.

57 Radaelli, Claudio M. and Dunlop, Claire A., “Learning in the European Union: theoretical lenses and meta–theory”, 20 Journal of European Public Policy, pp. 923 et sqq, at p. 923.CrossRefGoogle Scholar

58 Dunlop, Claire A. and .Radaelli, Claudio M, “Systematising Policy Learning: From Monolith to Dimensions”, 61 Political Studies (2013), pp. 599 et sqq, at p. 599.CrossRefGoogle Scholar

59 The academic literature clearly relies on a broader conceptualization of policy learning than the official documents, and (ex ante) evidence providing to policy makers is considered part of it.

60 Focusing on the type of evidence available and the use of such evidence in policy–making may also be a less tricky research strategy than trying to identify these processes as learning processes. Although Dunlop and Radaelli suggest several avenues in policy learning research (based on an analysis of the existing literature), they also seem to indicate that research framed in terms of knowledge utilisation may be the most promising one. Dunlop and Radaelli, “Systematising Policy Learning”, supra note 58, at p. 615.

61 For the relationship between different types of evidence and the different goals of ex post programme evaluation see Marielle Berriet–Solliec, Labarthe, Pierre, and Laurent, Catherine, “Goals of evaluation and types of evidence”, 20 Evaluation, (2014), pp. 195 et sqq,Google Scholar

62 Regulation of the European Parliament and the Council on the financial rules applicable to the general budget of the Union and repealing Council of 25 October 2012, Regulation (EC, Euratom) No 1605/2002, at Chapter 7, Article 30.

63 Emanuela Bozzini and Jo Hunt, and Lut Mergaert and Rachel Minto in this Special Issue.

64 Commission, “Impact Assessment Guidelines”, SEC(2009) 92.

65 According to a CEPS database of all IIAs adopted between 2003 and 2009, only about 43% of IIAs included operational objectives. The database created by the Centre for European Policy Studies under the supervision of Andrea Renda is not publicly available, but these data were quoted in Giacomo Luchetta, “Impact assessment and the policy cycle in the EU”, 3 European Journal of Risk Regulation (2012), pp. 561 et sqq, at p. 568.CrossRefGoogle Scholar

66 Luchetta, “IA and the policy cycle”, ibid., at p. 573.

67 Emanuela Bozzini and Jo Hunt, and Lut Mergaert and Rachel Minto in this Special Issue.

68 Commission, “2014 impact assessment (IA) reports/IAB opinions , available on the Internet at http://ec.europa.eu/smart-regulation/impact/ia_carried_out/cia_2014_en.htm (last accessed on 13 January 2015).

69 Commission, “Search evaluation results”, supra note 39.

70 Borrás and Højlund, “Evaluation and policy learning”, supranote 55.

71 Sanderson, Ian, “Evaluation, policy learning, and evidence–based policy–making”, 80 Public Administration (2002), pp. 1 et sqq., at p,2.CrossRefGoogle Scholar

72 Scriven, “Beyond Formative and Summative Evaluation”, supra note 55; Van der Meer and Edelenbos, “Evaluation in multiactor policy process”, supra note 55.

73 See also Steven Højlund in this Special Issue.

74 Commission, “Public Consultation on Commission Guidelines for Evaluation”, supra note 19, see table 1 above.

75 Commission, “Public Consultation on Commission Guidelines for Evaluation”, supra note 19, see table 1 above.

76 Commission, “Public Consultation on Commission Guidelines for Evaluation”, supra note 19.

77 Commission, “Public Consultation on Commission Guidelines for Evaluation”, supra note 19, at p. 25.

78 Radaelli, Claudio M., “Whither better regulation for the Lisbon agenda?”, 14 Journal of European Public Policy (2007), pp. 190 et sqq, at p. 192.CrossRefGoogle Scholar

79 There are sometimes (subtle) differences between the two concepts as they are used in official documents. Yet, due to limits of space I use them here as synonyms, as is the case in many official documents. For a more nuanced view, see Alemanno, Alberto, “Unpacking the principle of openness in EU law: transparency, participation and democracy”, 39 European Law Review (2014), pp. 72 et sqq,Google Scholar; and Smismans, Stijn, “Regulating interest group participation in the European Union: Changing Paradigms between transparency and representation”, 39 European Law Review (2014), pp. 470 et sqq,Google Scholar

80 Claudio M. Radaelli, “Whither better regulation”, supra note 78.

81 Commission Communication “Towards a reinforced culture of consultation and dialogue – General principles and minimum standards for consultation of interested parties by the Commission”, COM(2002)704.

82 Still many IIAs do not make use of online consultations, while not all online consultations are used in the context of IIAs. See Emanuela Bozzini and Stijn Smismans, “More inclusive European governance through impact assessments?”, Comparative European Politics, advance online publication, 9 March, 2015, doi:10.1057/cep. 2015.11.

83 Ibid.

84 See also Dunlop and Radaelli in this Special Issue.

85 European Parliament, “European Parliament Work in the fields of ex ante impact assessment and European added value. Activity Report for June 2012-June 2014”, European Parliamentary Research Service.

86 Mendez and Bachtler, “Administrative reform”, supra note 3.

87 The concept of policy coherence has particularly been used by both the OECD and the EU in relation to development policy. It is used here in a more general way to refer to the objective of ensuring coherence within a policy intervention or sector (internal coherence), or ensuring coherence of a policy intervention with other policy objectives of the polity (external coherence).

88 Luchetta, “IA and the policy cycle”, supra note 65, at p. 564.

89 Commission, “Impact Assessment: Key documents”, available on the Internet at <http://ec.europa.eu/smart-regulation/impact/key_docs/key_docs_en.htm> (last accessed on 21 January 2015).

90 Borrás, Susana and Radaelli, Claudio M., “The politics of governance architectures: creation, change and effects of the EU Lisbon Strategy”, 18 Journal of European Public Policy (2011), pp. 463 et sqq,CrossRefGoogle Scholar

91 Stijn Smismans and Rachel Minto (forthcoming), “Are integrated impact assessments the way forward for mainstreaming in the EU?”. See also Dunlop and Radaelli in this special issue, who point to the potentially normatively disturbing finding that IAs develop narratives about values and identities, which are thus developed within bureaucratic documents instead of within constitutional discussions.

92 Radaelli, “Whither better regulation”, supra note 78.

93 Commission staff working document “Operational guidance for assessing impacts on sectoral competitiveness within the commission impact assessment system. A “Competitiveness Proofing“ Toolkit for use in Impact Assessments”, SEC(2012) 91 final.

94 Commission Communication “Commission Work Programme 2015. A new start”, COM(2014) 910 final, at p. 3.

95 European Voice, “The Companion to the European Commission”, February 2015, at p. 24.

96 Commission Communication “Commission Work Programme 2015”, supra note 94, at Annex 3.

97 Commission, “Public Consultation on Commission Guidelines for Evaluation”, supra note 19, at p. 39.

98 Commission Communication “Smart Regulation in the European Union”, supra note 25, at p. 4.