Public policies and social progress: two evaluation models

Abstract

The article presents two models of public policy evaluation: one named idealistic and the other named pragmatic. In the former, social progress occurs when changes in the form of how society is organised bring us closer to social institutions and public authority conduct considered ideal. In the latter, social progress occurs if the prevailing social state (postchanges) is taken to be better than the previous social state (prechanges), according to certain pre-established judgement criteria. It is argued that the adoption of different models is one of the main obstacles to clarity in the public debate on the implementation of public policies, namely, that of making the points of disagreement explicit. The article presents a defence of the pragmatic model, which is considered more compatible with the use of scientific criteria in order to assess the effectiveness of policies.

Two people could disagree on the implementation of a given public policy for at least three different types of reasons: interests, values and models through which they see how the world works.

It is important to highlight that it is not illegitimate to use one’s personal interests as a means of justifying one’s support for a given policy, but the same argument could not plausibly sustain a public debate in favour of that policy. In the ethics of individual behaviour, it can be argued that it is acceptable for each person to place greater importance on their own interests and principles than on those of others in society. However, a plausible public justification for a given policy requires a justification of a moral nature, arguing that, in some way, the aspirations of all members of society were taken into account. ¹ Therefore, regarding the public support of a given policy, the rational debate comes down to two types of argument: that of a moral nature and that of how the world works.

It must be recognised that this debate takes place in an environment of great uncertainty. The theories (both positive and normative) available to support public policies are incomplete and the degree of confidence we attach to them can be quite varied. In general, we are not able to arrange (from best to worst) all the social states which could, in principle, be reached by the implementation of public policies, nor do we know, with any reasonable degree of confidence, the path to be followed to get to each of these states. In practice, the debate deals with implementing, or not implementing, a certain policy or with choosing one of few alternative policies to achieve a specific objective.

In this article, it is admitted that social progress takes place when the changes which occurred in the form of society organization and/or in the social state are taken to be positive according to a certain criterion of judgement. Hence, the evaluation of a public policy consists of verifying whether the results it brought about can be considered social progress, while taking into account the degree of risk involved for such a policy to effectively produce these results.

This article presents and analyses two models of public policy evaluation, which differ in their definition of social progress. The first, named idealistic, takes as a reference to be pursued, the ideal characteristics of how a society should be organised, in terms of social institutions and modes of conduct of the public authority (e.g. an ideal education system or an ideal health system). Progress is supposed to have taken place if the changes brought about by the policy draw us closer to that ideal. The second, named pragmatic, takes the prevailing social state as its reference and adopts certain improvement criteria. Progress is made if the new social state (postpolicy implementation) is considered better than the previous state (prepolicy implementation).

The debating rules that these models impose are quite different; hence, public debate participants who adopt different models have difficulty in communicating. It is argued that the adoption of different models is one of the main obstacles to clarity in the public debate on the implementation of public policies, namely, that of making the points of disagreement explicit.

The debate about the importance of defining an ideal state to guide political action has a long tradition in political philosophy and, in one way or another, is present in day-to-day discussions on public policy. However, this topic has received little attention in the academic literature on public policies. Thus, one of the main contributions of this article is precisely to bring this theme to the academic discussion about public policies. The article intends to bridge the political philosophy and public policy literatures.

If the purposes of public policies are fixed and clear, then public policy decisions can be seen as problems of an instrumental nature. However, in the public policy debate, ends are often a matter of dispute, as evidenced by the contrast between the two evaluation models presented above. In that case, there is no consensus on the problem to be solved. A conflict of ends cannot be resolved by empirical technique or applied research. For example, the evidence-based policymaking (EBP) movement, like the pragmatic evaluation model presented above, emphasises the rigorous and accurate use of scientific evidence in the evaluation of public policy. However, EBP supporters have been accused of depoliticising the public policy debate, as if the evaluation of public policies was limited to technical aspects (see, among others, Hammersley 2005; Russell et al. 2008; and Wesselink et al. 2014). The EBP approach assumes that public policy objectives are given. Thus, it may be difficult for an EBP supporter to convince a follower of the idealistic evaluation model that a particular policy should be implemented by providing evidence that the policy achieves certain outcomes that the EBP supporter considers desirable. ² All of this assuming that the idealist evaluator thinks that this policy would distance us from the social state he considers ideal.

This article defends the pragmatic model of evaluation, which is considered more compatible with the use of scientific methods or criteria to assess the effectiveness of policies. Ideal theorising can play an important role in the diagnosis of social problems to be faced and in the formulation of public policy proposals, but not in the evaluation of these policies (whether or not they promote social progress).

The adoption of a more pragmatic approach on policy analysis is definitely not new. ³ However, the policy analysis literature has focussed on the process of policy formulation and implementation and has given less importance to the issue of validation criteria. In turn, the policy evaluation discussions have often been restricted to impact evaluation (the identification of the causal effects of the policies) and, thus, end up obscuring the fact that the public policies evaluation also involves a dispute between different social values.

In a democratic society, the quality of the public policy debate is fundamental to determining the functionality of democracy itself. Thus, clarity on the reasons for the different positions in the public policy debate is an essential element. The premise of the article is that the evaluation models discussed here contribute for providing greater clarity about the positions taken in the public policy debate.

Social progress, social state and public policies

The key concept for evaluating a public policy is that of social progress, which differs according to the evaluation model considered. The concept of social progress in the pragmatic evaluation model is looked at first.

Definition 1: Social progress occurs when there is change in the social state and the prevailing social state is considered better than the previous one, according to a pre-established judgement criterion.

In this model, social progress is evaluated by directly comparing the social states which prevail before and after the change. Social state is understood as the result of the way a society is organised (the social contract) and concerns both the fruits of the social organization that people desire (income and wealth; freedoms and opportunities; respect and prestige etc.) and the duties and obligations that people have as a result of being part of society.

Definition 2: The social state is made up of a set of social positions and the rules which govern the occupation of these positions by members of society.

Social position determines the relationship that its occupant maintains with other members of society and, for each social position, there is an associated vector of benefits and rights as well as a vector of burdens and obligations. A social position can be related to occupation, position in the community or in the family. In general, individuals have more than one position and the benefits, rights, burdens and obligations associated with social positions can be either formal or informal. ⁴

Although improving the social state is the ultimate goal of public policies, the followers of the idealistic model do not consider that a comparison between successive social states is the best way to evaluate social progress. They identify ideal characteristics in the organization of society, and these should be pursued. Thus, any change which distances us from these ideal characteristics should be avoided, even if the social state resulting from that change were considered superior to the earlier social state. It would be an improvement which would distance us from an even greater advance in the future. For that model, the definition of social progress is as follows.

Definition 3: Social progress happens when changes occur in how a society is organised and such changes draw us closer to what are considered ideal social institutions and modes of conduct of the public authority.

The ideal characteristics of the form of society organization are taken to be achievable, even if in the long term, and, once achieved, they would generate a preferred social state. Although achievable, there may not be full knowledge of the best way to achieve these ideal characteristics; therefore, the debate on public policies involves both the ideal to be pursued and the appropriate strategy to achieve it.

The social state is altered over time as a result of different forces, which could reflect changes stemming from the environment, culture, religion, economic organization, scientific and technological knowledge, etc. This article is concerned with changes in the social state which derive from the intervention of the public authority, herein called public policies.

Definition 4: Public policy is an action or a set of actions taken by public authority (executive and legislative powers) whose ultimate objective is the promotion or prevention of changes in the social state.

This broad definition of public policy is appropriate for the purposes of this article. It allows for all substantive political debate to be organised around public policies. The other sphere of the political debate deals with the “rules of the political game”, which have to do with the definition of the competences of the different spheres of public authority, electoral rules, duration and reasons for interrupting mandates, the operational rules of the executive and legislative powers, etc. ⁵

This article assumes political debate takes place in a democratic society with stable “political game rules”, in the sense that the participants in the debate firmly adhere to them. In this environment, the political debate revolves around substantive themes for the lives of members of society and is driven by the discussion of public policies.

It should be clear that the interest here is not that of analysing the debate on public policies as it actually occurs in day-to-day political discussions. The focus of the article is on the rational justification for adopting (or not adopting) a certain public policy made by interlocutors genuinely interested in promoting the common good.

Although rational and plausible justifications for the implementation of a given public policy are not the only element taken into account by decisionmakers (public agents, politicians and voters), it is supposed to be very important. The better the debate in this sphere, the better the day-to-day discussion on the implementation of public policies tends to be.

This debate is key to helping those who decide on public policies make fewer mistakes in their evaluations. That is due, for example, to clarifying the likely impacts of the policy and exposing the moral dilemmas that could be involved.

Idealistic and pragmatic models of evaluation of public policies

Evaluating public policies, in some way, involves evaluating the resulting social states. Take the existence of a public policy, P1, capable of transforming social state A into social state B. The question then is: should this policy be implemented?

The evaluator’s answer to this question will depend on elements such as his confidence in the effects of P1 and the criteria used to assess social progress. Suppose he is quite confident that P1 does in fact change the social state from A to B. One possibility would be to use a judgement criterion that is limited to comparing states A and B, and supports the implementation of P1 if he considers that state B is better than state A. Acting in that manner, he would be adopting the pragmatic model of public policy evaluation.

However, he may not support the adoption of P1. Suppose he takes the existence of a social state C, which he considers better than any social state attainable on the basis of A. He may consider it undesirable to adopt P1 if he thinks that, when we move to social state B, we are further away from adopting public policies which draw us closer to C. State C would be reached when certain social institutions and/or public sector modes of conduct were adopted. If he acted in this way, he would be adopting the idealistic model of public policy evaluation. In this evaluation model, it would be possible to support a policy P2 which generates a social state D, even if, in a direct comparison, social state D is considered inferior to social state A. P2 would be supported as it improves certain social institutions and/or forms of conduct of public authorities, and thereby draws us closer to the form of society organization which produces social state C.

Someone could say that that the distinction made here between these two models of evaluation bears a strong resemblance to the distinction that Karl Popper made between utopian and piecemeal social engineering (Popper 1944 and 1966). While the pragmatic model of evaluation seems to be in agreement with what Popper calls piecemeal social engineering, the relationship between the idealistic model of evaluation and utopian social engineering is not so direct.

As in the idealistic evaluation model, utopian social engineering stipulates a social ideal to be followed. The difference is that, for the utopian engineer, the social ideal involves society as a whole and, more importantly still, its implementation requires a detailed implementation plan. ⁶ In the idealistic evaluation model, the ideal could refer to certain social institutions and/or public authority conduct and does not require the existence of a detailed implementation plan. Followers of the idealistic model of evaluation accept the rules of a democratic society where public policies are discussed and implemented gradually.

The essential difference between the pragmatic and idealistic models of evaluation lies in the criterion for validating the policy and not in the gradual or holistic nature of the intervention.

The idealistic model

In this model, the first task would be to identify a set of institutions and/or ideal modes of public authority conduct, which should be followed. For example, when discussing public health policies, the first task would be to characterise the ideal organization of a health system or, at least, certain ideal characteristics which a health system should have. This ideal system could be associated with a broader view of an ideal society (the socialist ideal, the liberal ideal, etc.), but not necessarily.

Having defined the ideal characteristics that the social organization should have, the discussion then centres on the best strategy to achieve this objective. Public policies supposed to draw us closer to the set of institutions and/or public authority conduct considered ideal are supported and those supposed to distance us from that ideal are rejected.

In this model, public policies are evaluated from the perspective of certain ideal characteristics in the form of organization of society, which would function as a telos. This ideal of social organization is considered achievable and, once achieved, it would deliver a social state superior to that which would be obtained under alternative forms organization of society. It is worth noting that what characterises this evaluation model is not conjecturing about ideal public institutions and/or public authority conduct, but rather using these ideals as a way to validate public policies.

So, in the public debate on health policies, for example, there are two fundamental questions to be answered by those who follow the idealistic model of public policy evaluation. What ideal characteristics should a health system have? What public authority interventions can be considered to draw us closer to the ideal organization?

In the idealistic model, public policy should not be evaluated for its direct impact on the social state, but for its contribution to an ideal of social organization. ⁷ It is only when this ideal of society organization is achieved that the resulting social state should be considered. Even so, the partial implementation of this system must be seen as positive, as it draws us closer to the ideal health system which would produce a superior social state.

Suppose, for example, it is considered that the ideal education system should be made up of public schools, with pedagogical autonomy and the principal chosen directly by the school community (parents, teachers and auxiliary school staff). However, student learning has to be monitored by external exams in key subjects. In the case of unsatisfactory performance, the school would lose its autonomy and teachers and auxiliary staff could be relocated or even fired. In this scenario, the implementation of a system for electing the principal by the school community could be seen as progress, even though improvement in student performance could not be expected automatically, as the election by the school of a director without pedagogical autonomy and without a system for monitoring student performance might not lead to an improved performance. On the other hand, the implementation of systems such as vouchers or charter schools would be rejected, because the ideal system assumes that education is offered directly by the government. These measures (voucher and charter schools) would be rejected even if there were evidence that they enhanced student learning without increasing costs.

A comment on the term “idealistic” is needed. This term has been used with several related meanings. In philosophy, idealism refers to metaphysical theories that distinguish the real world from the apparent world – the one that is perceivable by the mind – and, in opposition to materialism, considers the primacy of consciousness over matter. What is real is somehow related to our minds. In ordinary language, idealism often means giving priority to ideals, principles, values and goals over concrete reality.

This article uses an even more restricted definition of the term idealism, relating it to a commitment to certain characteristics of society organization, in the belief that, once this ideal has been reached, the resulting social state would be superior to those obtained under alternative forms of society organization. ⁸

Definition 5: Idealism is the belief that certain characteristics considered ideal of the form of society organization produce a social state superior to that which could be obtained by alternative forms and, for that reason, should be pursued.

Idealism can give rise to reformist or conservative agendas at a political level. Conservatism is understood as a doctrine which identifies certain prevailing characteristics of the form of society organization as ideal and which, for that reason, should be preserved.

The pragmatic model

In this evaluation model, there is no ideal state to be pursued. The starting point is the prevailing situation which one seeks to improve. The first step is to identify those aspects of the social state that we consider undesirable and which, we believe, can be changed. Having identified the “social problems” to be confronted, the next step is a diagnosis of their main causes. Then, the third step would be the proposal of measures (public policies) to remedy or, at least, mitigate the social problems identified. The evaluation then follows. Evaluating public policies is a question of verifying whether they really produce the intended results in the social state (impact evaluation) and of using a moral judgement to argue that the resulting social state is superior to that prevailing before the policy was adopted. ⁹

The pragmatic model places emphasis on what is called impact evaluation: an analysis of the causal effect of the policy on the social state. ¹⁰ The impact evaluation can be carried out before and/or after the implementation of the policy and can be called ex-ante impact and ex-post impact evaluations.

An ex-ante impact evaluation is predictive in nature. It addresses the likely impacts of the policy on the social state. An ex-ante impact evaluation involves two steps: (i) specification of a theory of change and (ii) analysis of its validity. The theory of change describes the causal logic of how and why the public policy under consideration would achieve the desired results. The description of the causal links may or may not be undertaken in a more formal way, by means of a theoretical model, a logical model, a logical framework, etc. The search for validity for the theory of change can be undertaken both by the empirical validation of the causal links that constitute it and by the use of ex-post impact evaluations carried out in similar policies implemented elsewhere.

An ex-post impact evaluation, in turn, seeks to measure the results obtained by the public policy after its implementation. In theory, the aim is to compare the social state in two alternative situations: one that operates after the implementation of the policy and other that would operate if the policy had not been implemented. Once the policy is in place, only the former situation is observable. The latter is a counterfactual. If an evaluation of the social state as a whole is impossible, the impact evaluation, in practice, consists of comparing the value of social indicators of interest in the two scenarios (prevailing and counterfactual) and attributing the difference in values to the public policy being evaluated. The ex-post impact evaluation is that which generally appears in the specialised literature as impact evaluation. The said literature presents a series of methodologies which, under certain conditions, would produce valid causal inferences. ¹¹

As impact evaluations (ex-ante and/or ex-post) involve investigating a causal effect and as the relationship between cause and effect has been one of the main concerns of science, scientific methods and criteria can be used in the impact evaluation of public policies. In doing so, the impact evaluation of public policies can be understood as a particular field of science, but more precisely of the social sciences.

In this sense, the pragmatic evaluation model is in line with the evidence-based policymaking (EBP) literature. However, it is necessary to consider that: (i) in many cases we do not have solid studies (e.g. Randomised Controlled Trial and Meta-Analysis) that can safely guide us about the impact of a given public policy; (ii) studies carried out in other contexts may not be generalisable to the case under consideration, since public policies tend to be context-dependent; and (iii) solid evidence of the impact of the policy may not be sufficient, as it is necessary to recognise that the evaluation of public policies involves, in general, the choice between competing social values. A more appropriate term would perhaps be evidence-informed policymaking rather than EBP. ¹²

The ex-ante impact evaluation is of fundamental importance for the debate prior to the implementation of the public policy. It is the review of the literature on the causal mechanisms present in the theory of change and the results seen in similar public policies elsewhere that form our expectations about the likely results of its implementation and about the risks involved. The ex-ante evaluation plays an important role in convincing the audience that the public policy proposal under consideration is an experiment worth trying. In turn, the role of the ex-post impact evaluation is to verify whether the public policy has produced the expected results. In addition to contributing to the elimination of bad public policies, an ex-post impact evaluation can be used to test a pilot project of a policy before generalising it for the set of potential beneficiaries.

The term “pragmatic” also deserves a brief comment. In philosophy, pragmatism has been used to identify a current of thought which includes authors, such as Charles Sanders Peirce, William James and John Dewey. This current of thought is characterised by the idea that “efficacy in practical application - the issue of ‘which works out most effectively’ – somehow provides a standard for the determination of truth in the case of statements, rightness in the case of actions, and value in the case of appraisals” (Rescher 1995, p. 710).

In this article, the term “pragmatic” takes on a very specific meaning: that the evaluation of a public policy must be carried out on the basis of the policy’s ability to resolve or, at least, mitigate the previously identified problem.

Communication difficulties between followers of different models of evaluation of public policies

The main communication difficulty between supporters of the different evaluation models has to do with the fact that they seek to answer different questions. While supporters of the idealistic model ask themselves, for example, about the ideal public health system and about the interventions which draw them closer to it, supporters of the pragmatic model ask themselves about the deficiencies in the prevailing system and how they can improve it.

The communication difficulties stemming from the fact different questions about public policies are asked should not be underestimated. Let us return to the example of the organization of the education system presented before, where a follower of the idealistic model of evaluation considers that schools should be provided directly by the public sector, should have pedagogical autonomy and the principal should be chosen directly by the school community. He also considers that learning should be monitored by external exams in key subjects and that the school would lose its autonomy (with possible reallocation or dismissal of teachers and auxiliary staff) if performance were unsatisfactory.

Now suppose that a follower of the pragmatic model of evaluation defends the introduction to the system of charter schools (schools financed by the public sector, but with private management). She takes this stance after analysing the theoretical arguments for the establishment of charter schools and after reviewing the great number of experiences of adopting this type of school, and concludes that, as it increases competition between schools, the presence of charter schools would increase the efficiency of the educational system. ¹³ The question to be raised is: what would the debate between these two hypothetical debaters be like?

To simplify the question, let’s imagine that the debaters do not disagree on the objectives that an educational system should present and that the discussion is restricted to how to achieve them. As already pointed out, the follower of the idealistic model will reject the proposal, as charter schools have no place in his ideal education system. In situations like this, it is common for the supporter of the idealistic model to try to infer on the ideal system of their debating opponent. He can infer that the follower of the pragmatic model has an ideal education system formed by private schools and charter schools would only be the beginning of the privatisation. In turn, the follower of the pragmatic model may find it difficult to understand her opponent’s reason for rejecting the proposal. Because she considers that the rejection took place without due regard for her arguments or evidence, the follower of the pragmatic model might think that her debating opponent is not well-intentioned and is hiding inadmissible interests.

In general, people are better prepared to debate with those who adopt the same public policy evaluation model as they do. The debate between two people who adopt different evaluation models may be very unproductive.

A critique of the idealistic evaluation model

The main criticism of the idealistic model for evaluating public policies is that it is less appropriate for using scientific methods and criteria and can lead its followers to adopt dogmatic positions.

In the search for corroboration or refutation, it is much more difficult, if not impossible, to test ideal systems than it is to test specific public policies. How can one gauge whether an ideal health system will achieve the desired results, once it is implemented? The more the ideal system differs from the prevailing system, the more difficult it is to answer that question. ¹⁴

When the ideal system is very different from the prevailing system, there are two problems to be considered. The first concerns the basis underlying the belief that, once implemented, the ideal system will present the desired results. The second has to do with the conduct adopted by the followers of the idealistic model to achieve the system considered ideal.

Considering the first problem, one aspect to be highlighted is that, in general, the claim for the superiority of the system considered ideal is purely theoretical, with no support from any previous experience. If dealing theoretically with small changes is quite difficult, then major changes, which involve several changes being carried out simultaneously, could prove to be intractable.

Sometimes the ideal system of education (health, transport, etc.) of the supporter of the idealistic model of evaluation is a system adopted somewhere else (city, state or country). In that situation, they could say that the impressive performance of students from that other location is evidence of the efficiency of the educational system adopted as a reference. However, that relationship is not direct. The fine performance of students does not solely depend on the design of the educational system. It depends on the value the community places on education, on the formation and involvement of parents in their children’s education, on the existence of people qualified and willing to exercise the profession of teachers, etc. There is no guarantee that the adoption of another country’s educational system will yield similar results. In general, public policies are very dependent on the context in which they are applied.

Given the lack of a solid basis to support the validity of an ideal system, one must analyse the conduct of the followers of the idealistic model in their pursuit of this ideal. However, it is first worth noting that having an ideal system as a reference would not be a problem if the evaluator did not use it as a criterion for validating the public policies being considered at present; if he only considers the impacts of the policy on the social state as the relevant criterion for its adoption. In that case, the ideal system would only play the role of helping the participant in the debate to formulate the public policies that he puts up for discussion. But once formulated, their validation should follow criteria more consistent with those practiced in the sciences.

The problem is that the follower of the idealistic model adopts the ideal system as a criterion for validating the policies which are currently being discussed. Based on an ideal, whose results are quite uncertain, the idealistic evaluator is willing to allow a short-term deterioration in the social state to bring the form of society organization closer to that which they consider ideal. Likewise, he is willing to block policies which would lead to immediate improvements in the social state because they distance us from social institutions and ways of acting of public authorities which he considers ideal. Supporters of the idealistic model of evaluation often take dogmatic positions on public policies.

As already noted, the idealistic evaluator does not need to have the ideal organization of a complete system (education, health, justice, etc.) as a reference. His reference can be limited to certain ideal characteristics of the system or more general principles that can be applied to different systems. For example, propositions such as “the production of goods and services is always more efficient when undertaken by the private sector rather than by governments” or “the state is always more efficient than the private sector in the production of public and/or utility goods and services” are also considered idealistic. These are not statements which can be answered in the realm of science. Efficiency in the production of goods and services by the private sector depends, among other things, on the structure of the market in question, while the efficiency of governments can vary from case to case. The way a government intervenes in the production of goods and services can be understood as a social technology and, like technologies based on the natural sciences, they can be improved. If after an exhaustive empirical study, it was concluded that the telephony sector tends to be more efficient when managed by the private sector, that does not mean that it has to be always like that. ¹⁵ Taking that kind of statement as a non-negotiable truth is dogmatic. It does not in any way help to improve public policies and could act as a deterrent to improvements in the social state.

A defense of the pragmatic model of evaluation

The task of the pragmatic evaluators is more modest and feasible. As in the sciences and technologies, the pragmatic model of evaluation provides accumulated knowledge on the effectiveness of public policies in the most diverse areas of life in society.

One critique that could be made of the pragmatic model of evaluation is that, as it deals with particular public policies only, it would be less likely to produce a more profound transformation in society, the type of transformation which could improve people’s lives more significantly. In the absence of a telos, the pragmatic model could leave one going round and round the present social state without, in fact, producing any significant social progress.

In the pragmatic model of evaluation, there is an implicit reformist doctrine. A diagnosis of social problems is carried out, measures are proposed to remedy them and, finally, an evaluation is made. It is a stance that is contrary both to conservatism, which takes the prevailing social organization as an ideal to be maintained, and to the utopian promises of an ideal future system. That does not mean, however, that the result is only a marginal progress. The pragmatic model has the same notion of progress that is applied to scientific and technological knowledge. And the enormous progress made in these areas is undeniable.

Another key aspect to be highlighted is that it would be much easier to reach consensus under the pragmatic than under the idealistic model of evaluation. Under the idealistic model, people would need to agree on both the ideal system (which involves agreeing with the system’s impacts on the social state and with the social state itself) and on whether a given policy draws us closer to the ideal system. It is, therefore, a fertile field for divergence. In the case of the pragmatic model, on the other hand, the convergence of values could be relatively easy for many public policies. For example, few people would be expected to oppose an educational policy which would enhance student learning without significantly increasing costs. Likewise, since these are particular public policies and because they use scientific methods and criteria to assess their impacts, agreement on the positive aspects (as opposed to the normative) of the policies would also be relatively easier.

The pragmatic evaluation model would allow a significant set of policies to be implemented with little resistance, leaving the controversies for those policies whose impacts on the social state are less clear and where differences in moral values are important.

Ideal and nonideal theory in political philosophy

The discussion about the two models of public policy evaluation (idealistic and pragmatic) leads us to an older and broader issue in political thought: about the importance of defining an ideal state to guide political action. ¹⁶ As highlighted by Gaus (2016, p. 3), “right from the beginning political philosophy has sought to describe the ideal state, which, even if not fully achievable, gives us guidance in constructing a more just social world”. While the discussion of this question is old, it seems correct to say that the most recent debate over ideal and nonideal theory originates with John Rawls.

Rawls (1999) divides his theory of justice into two parts: “the first or ideal part assumes strict compliance and works out the principles that characterise a well-ordered society under favorable circumstances”, while the second part consists of the nonideal theory, where “the parties ask which principles to adopt under less happy conditions (…) the principles for governing adjustments to natural limitations and historical contingencies, and the (….) principles for meeting injustice” (p. 216). In A Theory of Justice, Rawls deals fundamentally with the ideal theory: the principles of justice for the basic structure of society and the institutions that satisfy them, in a perfectly just society. According to Rawls (1999, p. 8), “the reason for beginning with ideal theory is that it provides, I believe, the only basis for the systematic grasp of these more pressing problems”.

The distinction introduced by Rawls between ideal and nonideal theory was not seriously challenged in the three decades following the publication of the first edition of A Theory of Justice in 1971. According to Simmons (2010) “the idea behind the distinction is simple and appealing, no doubt accounting for the widespread acceptance of (or indifference toward) Rawls’s version of it”. This scenario, however, began to change in the mid-2000s.

Sen (2006 and 2009) challenges the need to define an ideal state in order to promote justice. ¹⁷ He calls the approach adopted by Rawls as transcendental, “focusing (…) on identifying perfectly just societal arrangements”. In contrast, Sen argues for a comparative approach, which “would concentrate instead on ranking alternative societal arrangements (whether some arrangement is ‘less just’ or ‘more just’ than another)” (Sen 2006, p. 216).

Assuming that we live in a profoundly unjust world, the institutional arrangements of a perfectly just world (if such a world exists) must be very different from the existing institutional arrangements. In normal situations, the approach to the ideal (transcendental) world should be gradual. This would require a way of comparing different social states with the aim of promoting justice. If the comparison is based on the distance between the current and ideal institutional arrangements, Sen argues that there are several ways to construct a measure of distance and, therefore, the evaluation of whether a given policy produces an advance or a setback in the promotion of justice may depend on the specific measure of distance. On the other hand, he points out that the comparison between two social states, according to the criterion of justice, does not require identifying the ideal social state: “In arguing for a Picasso over a Dali we do not need to get steamed up about identifying the perfect picture in the world”. It is difficult to disagree with Sen’s argument that identification of the ideal state is neither necessary nor sufficient to assess whether moving from state A to state B increases justice (or reduces injustice). ¹⁸ However, I believe, the main question is not whether identification of the ideal state is necessary and/or sufficient, but whether it is useful.

Moral and political philosophy provides ethical principles that help us in the task of evaluating different social states. If the ideal theory is used only to derive such principles, it would not be incompatible with a comparative approach to evaluation, such as the pragmatic model of public policy evaluation discussed here. Likewise, the exercise of identifying ideal institutions (e.g. compatible with a perfectly just society) would not be a problem if the identified institutions were used only in the phase of formulating policy proposals, whose validation would require a comparative approach. The problem is when the ideal theory is used to identify an ideal institutional arrangement that serves as a reference to be pursued by public policies (an orientation condition, to use the term of Gaus (2016)). ¹⁹ In this case, we are acting according to the idealist model of public policy evaluation and all the criticisms made to this approach in the present text would be applicable.

The main criticism of this role of ideal theory is that the results that would be obtained by the institutional arrangement considered ideal would be purely theoretical, projected by a model (formal or informal), and we do not have the knowledge to build models that project, with any reasonable degree of confidence, the realisation of an institutional arrangement very different from the one we have today. The optimism, in the ability to project results to worlds very different from ours, that advocates of the idealist model of evaluation possess is totally unrealistic. ²⁰

It is important to emphasise that adopting the comparative approach of the pragmatic model of public policy evaluation does not mean always looking for an immediate improvement, without concern for the long term. On the one hand, short, medium and long-term impacts are widely recognised in the public policy evaluation literature. There are several examples of impact evaluation (experimental and nonexperimental) whose results are considered decades after the implementation of the policy. On the other hand, the pragmatic model does not prevent the establishment of long-term goals for a policy. For example, we can consider that keeping the planet habitable for future generations is a matter of justice and, therefore, it would be necessary for countries to agree to reduce the emission of greenhouse gases. However, such a reduction is costly from an economic point of view and it may be convenient that it be done in stages, until reaching the target, say, in 50 years. As the policy is being implemented, it is possible to verify whether the measures adopted produced the expected effects, in terms of emission reduction and economic costs. This is entirely different than having a commitment to an institutional arrangement entirely different from the current one, in the belief that once implemented it will produce a much better world than could be achieved by any other institutional arrangement.

The argument defended in this article is that the ideal theory can play an important role in the diagnosis of social problems to be faced and in the formulation of public policy proposals, but not in the evaluation of these policies. We cannot consider that a particular public policy promotes social progress by bringing us closer (by some measure of distance) to the institutional arrangement considered ideal.

Conclusions

This article presents and discusses two models of public policy evaluation: idealistic and pragmatic. In the former, social progress occurs when changes in the form of how society is organised bring us closer to social institutions and public authority conduct considered ideal. In the latter, social progress occurs if the prevailing social state (postchanges) is taken to be better than the previous social state (prechanges), according to certain pre-established judgement criteria. The article presents a defense of the pragmatic model, considered more appropriate for using scientific methods and criteria, and for allowing a significant set of policies to be implemented with little resistance.

It is argued that the adoption of different models is one of the main obstacles to clarity in the public debate on the implementation of public policies, namely, that of making the points of disagreement explicit.

The article deals with the issue of the importance of defining an ideal state to guide public policy. An issue that has long been addressed in the political philosophy literature and that, in one way or another, is present in day-to-day discussions on public policies, but which has received very little attention in the academic literature on public policy.