The Use of Evidence in Evidence-Based Legislation

• A Evidence-Based Legislation and the Ideal of Rational Policymaking

  In this contribution, I will reflect on a number of important issues related to evidence-based legislation that appear from the other contributions to this Special Issue. In particular, I will address two questions: (1) what counts as ‘good’ evidence and (2) how and why evidence is used or not used in legislation and lawmaking. These two questions are crucial for understanding the meaning and role of ‘evidence’ in legislation. Taking a closer look at them may help us to take one step back and reflect on the way evidence is constructed and used in legislation and legislative processes.
  My starting point is that we need to look critically at common depictions of ‘evidence’ and ‘scientific knowledge’ in practices of evidence-based legislation. These common depictions tend to see evidence as a self-standing, objective description of facts about the world. This has two implications. First, in this view, evidence and knowledge are better the more closely they represent the actual state of reality ‘out there’. True knowledge is knowledge that depicts as closely as possible the state of affairs it seeks to describe and explain. Second, this depiction assumes (or is at least often associated with the assumption) that knowledge moves forward in an independent, cumulative way. Because knowledge describes an objective reality that is independent from the observer, and the measuring rod for good knowledge is how well it describes that objective reality, knowledge progresses by coming increasingly closer to describing the ‘real’ state of affairs. This, then, leads to a view of knowledge development as a succession of steps towards greater and deeper understanding.
  This depiction of evidence and scientific knowledge will sound familiar to most people, as it is deeply rooted in the ideas and ideals that led to the creation of modern science from the enlightenment onwards. Together with a number of other characteristics (such as the emphasis on empirical observations and the search for general law-like regularities), it is integral to the standard conception that underpins the modern notion of ‘science’.¹x Cf. the descriptions in James Ladyman, Understanding Philosophy of Science, London and New York: Routledge, 2002, at pp. 95-96; and Alexander Rosenberg, Philosophy of Social Science, 5th ed., Boulder, CO: Westview Press, 2016, at p. 12.
  Evidence-based legislation builds directly on this conception, by making the use of this type of objectively true knowledge the touchstone of good policymaking. The underlying assumption here is straightforward. Policies (and, by extension, legislation) are attempts to intervene in reality by manipulating certain elements in that reality. Whether or not this attempt will be successful depends on whether or not the right elements are addressed. The better the assumptions underlying the policy reflect the actual state of affairs in reality, the more effective the intervention will be. Hence, the better the evidence underlying a policy, the more effective the policy will be.
  This way of reasoning precedes current notions of evidence-based policymaking by several decades. It underlies the more general notion of ‘rational-comprehensive’ policymaking that has been advocated since the Second World War.²x The term ‘rational-comprehensive’ comes from Charles E. Lindblom, ‘The Science of “Muddling Through”’, Public Administration Review, Vol. 19, No. 2, 1959, pp. 79-88, at p. 81. In that sense, evidence-based policymaking follows in the footsteps of a deeply ingrained tradition (and ideal) in Western thinking, which seeks to detach (or perhaps one should say liberate) policymaking from political ideology, interests and superstitions, and base it on objective and rational approaches.³x Cf. Deborah Stone, Policy Paradox. The Art of Political Decision Making, 3rd ed., New York and London: W.W. Norton & Company, 2012, pp. 9-13.
  Evidence-based legislation as it has come to be established within the European Union (in particular in the way of thinking and working of the European Commission) builds directly on this tradition of rational policymaking. The better regulation/better lawmaking programme explicitly seeks to base EU policies and legislation on the best objective evidence available. This, too, is an attempt to detach it from political interests and ideology.
  This approach to policymaking can be (and often has been) criticized as being based on an unrealistic understanding of how policymaking works⁴x E.g. Paul Cairney, The Politics of Evidence-Based Policy Making, London: Palgrave Macmillan, 2016, at p. 5. and promoting technocratic (and thereby undemocratic) ways of policymaking.⁵x Claudio Radaelli, ‘The Public Policy of the European Union: Whither Politics of Expertise?’, Journal of European Public Policy, Vol. 6, No. 5, 1999, pp. 757-774. At the same time, it should be noted that this view on policymaking also has a distinctly idealistic background. By basing policies on (more and better) evidence, proponents of the approach argue, policies will work better, doing more good in society. Moreover, objective evidence may offer a counterpoint against the dominance of self-interests or ill-founded ideology (sometimes to the point of prejudice) in policymaking. The tension between this ideal and its limitations is one of the core themes that will run through this reflection piece.
  All contributions to this Special Issue attest to the difficulties and dilemmas associated with the notion of evidence-based legislation, pointing out unclarities, inconsistencies and loose ends. In this contribution, I seek to make sense of these findings by reflecting on the assumptions about evidence and its use in policymaking that underlie evidence-based legislation. In doing so, I make use of insights from the field of policy studies, my ‘home discipline’. In addition, I draw lines with debates on (the philosophy of) science, so as to highlight the importance of thinking about the nature of scientific evidence in assessing its use in policymaking and legislation. In brief, this contribution tries to understand better what role evidence plays in evidence-based legislation by looking at different conceptions of ‘evidence’ and the way evidence is used in legislation and lawmaking processes.

• B Approach and Key Questions

  An important starting point if one is to understand evidence-based legislation is to acknowledge that what counts as ‘good’ evidence is intimately linked to prior ideas, values and interests. Evidence is not ‘free-standing’, independent from the issues that it addresses. Rather, what evidence is produced, how it is interpreted and how it is used is itself a reflection of societal and political choices. This becomes apparent in several contributions to this Special Issue.
  For example, whether to use a rational consumer model or insights from behavioural science in designing consumer law, as is discussed in the article by Van Schagen, Rueda and Baaij,⁶x Esther Van Schagen, Jaap Baaij and Isabelle Rueda, ‘Designing Information Obligations in EU Consumer and Energy Law: Behavioural Research and Legal Design as “Best Available Evidence”’, this issue. is not merely an academic debate about the best way to describe how people behave but also has to do with political objectives and interests. In that sense, any notion of consumer behaviour is not purely descriptive but also includes a normative assessment. A rational consumer model does not simply state that individuals are rational; it also posits that they should be rational. This is different if, based on insights from behavioural science; certain cognitive biases are seen as inherent in human decision-making. The model used in devising consumer protection is therefore intimately linked to how people are supposed to behave – or in other words, what counts as normal behaviour (and therefore behaviour that is understandable, reasonable, pardonable if you like) and what does not. The rational consumer and behavioural science models offer partly different standards of normality and abnormality in this regard. Such a choice is as much based on one’s normative views on how humans should behave and what a well-working society should look like as on any objective, scientific knowledge.
  Likewise, in the debate on the Green Deal (and sustainability more general) a key question is what impacts count and what do not. As Khadim and Van Eijken show, this question will be answered differently depending on which frame of reference is taken.⁷x Asmaa Khadim and Hanneke Van Eijken, ‘A Citizen-Centric Approach to Evidence-Based Decision-Making under the European Green Deal’, this issue. From a climate science perspective, different facts are relevant than from an economic, cultural or human rights perspective. Conversely, some facts that are important in one of these perspectives are irrelevant in (one of) the others. One cannot simply arrive at a comprehensive understanding of the issue by adding up all facts from the various perspectives and then weighing them against each other. In order to determine what facts are relevant and should therefore be taken into consideration and how they should be weighed against each other, one needs to start from some kind of organizing perspective that tells what to look for and what to ignore. The choice of organizing perspective itself can never be made on the basis of facts. It is a choice that precedes the selection and consideration of facts and is made on other (normative) grounds.
  These general observations about the nature of ‘evidence’ lead to three more specific questions, which I will address in subsequent sections:

  1. Who needs to produce evidence about what?

  2. What counts as (good, proper) evidence?

  3. For what purpose is evidence produced?

• C Who Needs to Produce Evidence about What?

  The first question addresses the issue of the burden of evidence: who is supposed to provide evidence and of what? This question is well known in the context of court proceedings, where assignment of the burden of proof is a key procedural issue. However, it is useful also to look at the role of evidence in legislation from this point of view.
  Rhoda Jennings’ piece is instructive in this regard.⁸x Rhoda Jennings, ‘The Use of Scientific Evidence in Precautionary Decision-Making in EU Environmental Law’, this issue. In it, she shows how the Court of Justice of the European Union (CJEU) has developed evidentiary standards in and under EU nature conservation legislation. Central to these standards is the precautionary principle, which, simply put, states that in case of doubt about the consequences of some kind of decision or behaviour it is better to be safe than sorry. In the context of the Birds and Habitats Directives, the CJEU has translated this into a double standard for evaluating the compatibility of Member State projects with the directives. First, only ecological evidence may be used to assess projects, to the exclusion of economic, social, cultural or other types of evidence. And second, projects may only go ahead if it is certain that they will not have adverse ecological effects in protected areas.
  What these standards do, in terms of policymaking dynamics, is that they allocate a specific burden of evidence in terms of who needs to prove what. In this case, it is Member State governments (and associated with them possibly the private developers of projects) that need to provide evidence, and that evidence needs to relate to the (absence of) adverse ecological effects. This is much more than a legal specification of a vague element in the directives. Rather, this allocation of the burden of evidence arguably forms the key mechanism through which the Birds and Habitats Directives affect decisions on protected natural areas.
  Through its definition of the burden of evidence, the CJEU’s approach restructures debates on projects in protected natural areas. By ruling out certain types of evidence and placing the greater burden of evidence on proponents of projects, the balance between proponents of economic development and proponents of nature conservation is shifted towards the latter. At the same time, shifting the evidentiary burden does not in and of itself predetermine the outcome of domestic decision-making processes. This also depends on other factors, such as the prevailing balance of interests in a Member State and the way the specific requirements of the directives are interpreted.⁹x Cf. the various studies that show substantial differences in the actual implementation and application of the Birds and Habitats Directives across member states. E.g. Raoul Beunen and Martijn Duineveld, ‘Divergence and Convergence in Policy Meanings of European Environmental Policies: The Case of the Birds and Habitats Directives’, International Planning Studies, Vol. 15, No. 4, 2010, pp. 321-333; Lars Borrass, ‘Varying Practices of Implementing the Habitats Directive in German and British Forests,’ Forest Policy and Economics, Vol. 38, 2014, pp. 151-160; Sharon Bryan, ‘Contested Boundaries, Contested Places: The Natura 2000 Network in Ireland,’ Journal of Rural Studies, Vol. 28, 2012, pp. 80-94; Francesca Ferranti, Raoul Beunen and Marie Speranza, ‘Natura 2000 Network: A Comparison of the Italian and Dutch Implementation Experiences,’ Journal of Environmental Policy & Planning, Vol. 12, No. 3, 2010, pp. 293-314.
  As Jennings notes, this emphasis on scientific evidence ‘removes the need for a value judgment’ from the assessment in individual cases. Yet, it is important also to note that the choice on the allocation of the burden of evidence is itself highly value-laden. Value judgements are therefore removed from individual cases by settling them in the underlying rules on the burden of evidence.
  This can be seen as an instance of what legal philosopher Joseph Raz called ‘exclusionary reasons’: general norms that exclude certain considerations from a decision.¹⁰x Joseph Raz, Practical Reason and Norms, Oxford: Oxford University Press, 2002, at p. 35ff. As a result, no balancing between all possible interests and considerations in individual cases is needed (or indeed, even allowed), as the exclusionary reason restricts the range of considerations that may be taken into account. By defining what types of evidence may and may not count in individual cases, the evidentiary rules in the Birds and Habitats Directives act as exclusionary reasons in this sense. From a political or governance perspective, this can also be interpreted in terms of interests: by admitting only certain types of considerations (in this case ecological effects), the standard set by the CJEU provides support for defenders of ecological interests against defenders of other interests (such as economic development or social impacts on local communities).
  This way of reasoning can also be applied to other contributions in this Special Issue. The article by Martin Munu highlights six factors that are important in assessing e-commerce regulation under the WTO:

  facilitating imports and export, addressing tariffs as a form of government revenue, attracting investment, preserving policy space for digital industrialization, providing development assistance, and providing for different rights and obligations according to development levels.¹¹x Martin Luther Munu, ‘Assessing the Adequacy of Existing Multilateral Rules Regulating E-Commerce: Lessons from EU Law,’ this issue.

  These six factors can be seen as different interests that the regulatory regime seeks to address and (ideally) balance. From the point of view of evidence-based legislation, one way of doing so would be to define who needs to provide evidence on what and how that affects the balance between the six factors. Such rules on the burden of evidence could be part of a broader approach to balancing the different interests involved in the regulation of cross-border e-commerce, but they require a prior normative decision on what interests are to prevail in what situations.
  So far, the discussion has focused on the role of evidence in individual decisions that are taken under or in the context of a regulatory regime. However, questions about the burden of evidence can also be asked in relation to the role of evidence in forming such regulatory regimes. The article by Martin Unfried and his co-authors offers a case in point.¹²x Martin Unfried, Pim Mertens, Nina Büttgen and Hildegard Schneider, ‘Cross-Border Impact Assessment for EU’s Border Regions,’ this issue. They call upon EU lawmakers to include cross-border effects for border regions in ex ante impact assessments. As they show, doing so would have avoided problems with, among other things, social security coordination and the General Data Protection Regulation (GDPR).
  This is a valid argument from the point of view of cross-border regions and the problems they encounter. On a systemic level, however, the question is which and how many angles should be included in ex ante impact assessments. The Treaty on the Functioning of the European Union (TFEU) defines a range of considerations that should be taken into account in all policies: gender equality, a number of social policy objectives, discrimination, environmental protection, consumer protection and, in certain fields, animal welfare.¹³x Arts. 8-13 TFEU. In the Better Regulation Guidelines, the minimum requirements for impact assessment reports include consideration for social, economic and environmental impacts, with particular attention to small and medium-sized enterprises (SMEs) and competitiveness, fundamental rights, the objectives of the European Climate Law and the UN’s sustainable development goals.¹⁴x European Commission, Better Regulation Guidelines, November 2021, at p. 35. The Better Regulation Toolbox identifies 35 ‘key impacts to screen’.¹⁵x European Commission, Better Regulation Toolbox, November 2021, at pp. 137-138.
  This list could even be extended by defining more specific types of impact. Yet, the question is how this can reasonably be included in the policymaking process. The more potential impacts are taken into account, the greater the likelihood that they will offer contradictory conclusions and recommendations. How should these be weighed against each other? Moreover, both individual policymakers and policymaking institutions can only process limited amounts of information.¹⁶x Cf. Bryan D. Jones and Frank R. Baumgartner, The Politics of Attention. How Government Prioritizes Problems, Chicago and London: Chicago University Press, 2005; Paul Cairney, The Politics of Evidence-Based Policy Making, London: Palgrave Macmillan, 2016, chapter 2. As a result, there are always more issues that merit attention than can be handled at the same time. Under these conditions, adding ever more issues to impact assessments is likely not to lead to more comprehensive policymaking but to selective choices within the available evidence, based on what is politically salient or taken for granted at a given point in time, or the cognitive shortcuts that policymakers (as all individuals) use to manage large amounts of information.
  Requiring evidence therefore always involves a choice as to what evidence is required. This choice inevitably risks ignoring certain aspects. However, if that choice is made on a reasonable basis and is open to amendment if needed, it increases the usability of evidence. The alternative of seeking comprehensive evidence on all possible aspects is delusionary. It muddles the assessment and, in the end, offers no direction, in particular if evidence on different angles points in different directions.¹⁷x For a classic critique on the notion of policy-making based on a comprehensive analysis of potential impacts, see Charles E. Lindblom, ‘The Science of “Muddling Through”’, Public Administration Review, Vol. 19, No. 2, 1959, pp. 79-88.
  This calls for ‘meta-questions’ on evidence-based legislation: not only how to produce and use evidence in specific cases within a legislative framework, but also how to determine what evidence is needed and relevant to develop that framework. In its Better Regulation Toolbox, the European Commission acknowledges this by offering guidance as to which of the potential impacts to include in an impact assessment. This includes criteria such as the political controversiality of the initiative, the type of legal act and size and complexity of both the policy problem and the expected impacts.¹⁸x European Commission, Better Regulation Toolbox, November 2021, p. 81ff. Although these are reasonable criteria, their use in practice requires further specification.
  For instance, to return to the examples given by Unfried et al: based on these criteria, should an assessment of the impacts on border regions be made for social security coordination and the GDPR? On the face of it, the a priori case seems stronger for the former than for the latter, as the link between border regions and cross-border work is more apparent. Yet, as the authors show, both issues have had effects on border regions and would have benefited from a cross-border impact assessment. However, if one takes a broad perspective and includes all proposals that may have an effect on border regions, it is questionable if there are many examples of EU policy for which such an assessment would not need to be done. This then brings us back to the initial point that it is impossible and indeed undesirable to strive for comprehensiveness. In the end, one can therefore not avoid questions about what is the best way to balance evidentiary requirements in lawmaking processes.

• D What Counts as Evidence?

  A second question on evidence-based legislation is what counts as evidence. In other words, what evidence is seen as valid or proper? Here, too, an evidence-based approach to legislation defines the playing field by allowing certain types and disallowing other types of evidence. This question is brought to the fore in the article by Asmaa Khadim and Hanneke van Eijken, who discuss the role of evidence-based policymaking under the Green Deal.¹⁹x Asmaa Khadim and Hanneke Van Eijken, ‘A Citizen-Centric Approach to Evidence-Based Decision-Making under the European Green Deal’, this issue. In particular, they take issue with the one-sided conception of ‘evidence’ in prevailing impact assessments, which is based on economics and climate science, and argue for the inclusion of social and human rights impact assessments in developing policies under the Green Deal.
  Partly, this ties in with the point made above about the inclusion and exclusion of certain interests in the allocation of the burden of evidence. However, it also highlights a particular issue within evidence-based legislation, i.e. what standards to apply to determine what distinguishes ‘evidence’ from other types of information or opinion. In the philosophy of science, this question is known as the ‘demarcation problem’, as it concerns the demarcation of scientific knowledge from other types of knowledge. If one adheres to the standard conception of science, which was sketched in the first section of this piece, the answer is more or less straightforward. Even though considerable argument may still be devoted to determining whether or not a specific method meets scientific standards, the standards for such a determination are clear: methods are scientific if they produce knowledge that is valid (i.e. which reflects how thing ‘actually’ are) and reliable (not dependent on a specific observer and thereby reproducible) and their results can be independently verified.
  However, these standards are not uncontested, in particular when it comes to knowledge about social (rather than natural) phenomena. Here, it can be argued that social phenomena are not ‘objective’ facts that can and need to be described from an outside position, but that they are inherently human constructions, which require understanding and reasoning rather than explanation and ‘pure’ observation.²⁰x Mark Risjord, Philosophy of Social Science. A Contemporary Introduction, New York and London: Routledge, 2014, p. 34ff.; Alexander Rosenberg, Philosophy of Social Science, 5th ed., Boulder, CO: Westview Press, 2016, p. 23ff. This leads to a different set of criteria for ‘good’ science, which do not stress validity, reliability and objectivity, but a trustworthy and contextualized interpretation of the way people make sense of and give meaning to reality.²¹x For a stylized yet clear juxtaposition of the two perspectives on social science, see Markus Haverland and Dvora Yanow, ‘A Hitchhiker’s Guide to the Public Administration Research Universe: Surviving Conversations on Methodologies and Methods,’ Public Administration Review, Vol. 72, No. 3, 2012, pp. 401-408.
  Khadim and Van Eijken’s argument can be understood in these terms. In their description of existing methodologies for social and human rights impact assessments, they note the importance of selecting the proper baseline indicators and collecting data for them. This fits into the conception of science as a search for objective facts in reality. At the same time, they note the importance of

  ensur[ing] that people’s feelings and concerns are seriously considered, even if such concerns are not substantiated by the evidence, as these can still affect the way that people relate to a project.

  This points towards a more interpretivist approach, which stresses the importance of understanding the meanings that people attach to certain phenomena. Related to this, Khadim and Van Eijken stress the importance of engaging citizens in impact assessments. This moves research away from the outside description of an objective reality to inside engagement with the lived experience of people.
  This is a kind of knowledge that relies on different methods and, eventually, different assumptions about what ‘counts’ in social reality. As a result, it sits uneasily with approaches that rely on ‘objective’ indicators. This may be one of the reasons why, as Khadim and Van Eijken point out, this type of analysis has not yet been used to its full potential in impact assessments by the EU. This may not only be a matter of political priority (although this will probably also play a role) but also of diverging or even incommensurable outlooks on what counts as proper evidence.

• E For What Purpose is Evidence Produced?

  The third and final question is for what purpose evidence is produced. From an orthodox evidence-based policymaking perspective, the answer to this question is obvious: evidence is produced to improve the fit between the assumptions underlying the policy and reality, which should lead to more effective policies. From a broader perspective, however, evidence may also serve other purposes. Besides as a way to improve policies, knowledge may also be used to bestow legitimacy on an organization (attesting to its competence and professionalism) or as a way to bolster one’s position in a political struggle with opponents.²²x Christina Boswell calls these the ‘instrumental’, ‘legitimising’ and ‘substantiating’ functions of expert knowledge; Christina Boswell, ‘The Political Functions of Expert Knowledge: Knowledge and Legitimation in European Union Immigration Policy,’ Journal of European Public Policy, Vol. 15, No. 4, 2008, pp. 471-488, at p. 472. Lorna Schrefler uses a similar typology, labelling these functions ‘technical-instrumental’, ‘symbolic’ and ‘strategic’ respectively; Lorna Schrefler, ‘The Usage of Scientific Knowledge by Independent Regulatory Agencies,’ Governance, Vol. 23, No. 2, 2010, pp. 309-330, at pp. 314-315.
  Both other roles of knowledge can be seen in evidence-based legislation and lawmaking within the EU. Presenting evidence is a way for an actor to convince others of its preferred position. Evidence lends weight to one’s arguments and confers to those arguments a sense of objectiveness that may help to convince others. This is particularly so if evidence ties in with the preferences or interests of those others. For instance, if scientific studies show that an investment programme proposed by the European Commission will benefit the EU economy, this is likely to add to the appeal of the proposal among the Member States.
  In addition, on an institutional level, for the European Commission in particular, using evidence is a way to present itself as an honest broker between Member States, which does not act out of self-interest or political ideology but out of an objective assessment of the common good. For the Commission, then, the use of evidence is not only a strategic move in a specific policy file, but also a more fundamental way to carve out its position and legitimacy vis-à-vis the other institutions.
  These types of use of evidence tread a fine line between objectivity and self-interest. On the one hand, to be convincing, the evidence that is produced and presented needs to be credible. If it is seen to be a purely self-serving fabrication, it is not likely to be taken seriously. On the other hand, there is usually quite some room to mould evidence in one’s direction. This can be done by framing the evidence in such a way that those findings and conclusions are highlighted that fit best with an actor’s interests.²³x See Falk Daviter, ‘Policy Framing in the European Union,’ Journal of European Public Policy, Vol. 14, No. 4, 2007, pp. 654-666. More generally on framing in policy-making: David A. Rochefort and Roger W. Cobb (eds), The Politics of Problem Definition. Shaping the Policy Agenda, Lawrence, KS: University Press of Kansas, 1994; Deborah Stone, Policy Paradox. The Art of Political Decision Making, 3rd ed., New York and London: W.W. Norton & Company, 2012. Moreover, choices need to be made about what kind of evidence to produce, which actors can use strategically to produce the kind of evidence that best fits their agendas. A good example of the latter can be found in a study by Markus Haverland, Minou de Ruiter and Steven Van de Walle, which shows how the inclusion and exclusion of certain public opinion questions in the European Commission’s Eurobarometer survey is affected by strategic considerations.²⁴x Markus Haverland, Minou De Ruiter and Steven Van de Walle, ‘Agenda-Setting by the European Commission. Seeking Public Opinion,’ Journal of European Public Policy, Vol. 25, No. 3, 2018, pp. 327-345.
  The analysis of Ljupcho Grozdanovski and Jérôme De Cooman in this Special Issue can be understood against this background.²⁵x Ljupcho Grozdanovski and Jérôme De Cooman, ‘Of Hypothesis and Facts: The Curious Origins of the EU’s Regulation of High-Risk AI,’ this issue. They show how the European Commission strategically used (and ignored) outcomes from the public consultation on a new Artificial Intelligence (AI) Act so as to uphold its pre-existing policy objectives. From a normative point of view that stresses ‘epistemic validity’, to use Grozdanovksi and De Cooman’s term, this outcome is undesirable. However, from a political perspective on (the use of) evidence, it makes perfect sense, and their analysis shows clearly how evidence may be used selectively to fit pre-existing interests and objectives.
  A similar dynamic seems at play in Jasper Sluijs’ observations about the unbalanced use of evidence in the European Commission’s evaluation of the Vertical Block Exemption Regulation.²⁶x Jasper Sluijs, ‘Evidence-based Legislation in EU Competition Law,’ this issue. As he notes, the Commission was keen to point out the limited relevance of studies that contradicted its preferred approach while claiming broad relevance for studies that supported its approach, even though the scopes of the two types of study were similar. Here, too, the interpretation of evidence was shaped by the purpose it served for the Commission; in this case, to convince others of its preferred policy approach.
  Having said this, it is less easy to distinguish ‘strategic’ and ‘substantive’ uses of evidence than it may seem. Most policies are underpinned by a ‘policy paradigm’, which Peter Hall defined as the

  framework of ideas and standards that specifies […] the goals of policy and the kind of instruments that can be used to attain them [and] the very nature of the problems they are meant to be addressing.²⁷x Peter A. Hall, ‘Policy Paradigms, Social Learning, and the State: The Case of Economic Policymaking in Britain,’ Comparative Politics, Vol. 25, No. 3, 1993, pp. 275-296, at p. 279. The concept builds on the notion of (scientific) ‘paradigm’ in Thomas Kuhn’s seminal work on scientific revolutions; Thomas S. Kuhn, The Structure of Scientific Revolutions, 3rd ed., Chicago: University of Chicago Press, 1996.

  EU competition policy offers a good example of a strongly established policy paradigm, especially with the rise of the ‘More Economic Approach’ as described by Jasper Sluijs, in which both the definition of the underlying problem and the approach to tackling it are based on a specific notion of consumer welfare as well as on assumptions on how agreements between firms affect consumer welfare. Salvo Nicolosi’s article on the use of evidence in EU asylum policy juxtaposes two competing paradigms in this area: humanitarianism, which focuses on the protection of refugees; and securitization, which is aimed at stemming the influx of refugees.²⁸x Salvo Nicolosi, ‘Evidence-based Policy-making in European Union Asylum Law: Potential and Pitfalls,’ this issue. The key distinction here is what asylum policy is, in the end, for: to protect asylum seekers (as the ‘final users’ of the policy, as Nicolosi puts it) or to protect the national and local communities in which they seek refuge.
  By defining the range of relevant issues, desirable goals and appropriate instruments to achieve them, policy paradigms also determine what kind of evidence is relevant in evaluating existing policies and making new policies, and what counts as convincing evidence. The standards are usually set higher in this regard for evidence that contradicts a prevailing paradigm than for evidence that supports it. This makes sense insofar as contradictory evidence challenges deeply held convictions that, in the minds of policymakers, have served well as a basis for understanding reality and making policies. Evaluation of an established policy paradigm does not take place de novo on the basis of the then available evidence but against the baseline assumption that the prevailing policy paradigm is sound. One does not give up something that works well just because there is some evidence against it.
  Still, the flip side to this is that outdated or dysfunctional policy paradigms may outlive their usefulness by quite some time. More generally, the close link between evidence and policy paradigms makes it difficult for opponents that adhere to different paradigms to arrive at a joint understanding of an issue. For Martin Rein and Donald Schön this is the defining difference between ‘policy disagreements’, which can be settled by reference to facts, and ‘policy controversies’, in which participants do not agree on the (interpretation of) available evidence because they start from different assumptions about what counts as relevant evidence to begin with.²⁹x Donald A. Schön and Martin Rein, Frame Reflection: Toward the Resolution of Intractable Policy Controversies, New York: Basic Books, 1994. In their work, Rein and Schön offer an approach to overcoming these types of controversies by reflecting on the underlying (normative) assumptions of each paradigm and identifying a shared core. This approach they call ‘frame reflection’. Most fundamental disputes in strongly value-laden policy areas (such as asylum and migration policy) are more aptly characterized as ‘controversies’ than as ‘disagreements’. In Rein and Schön’s terms, these controversies are ‘intractable’ because they cannot be settled by an appeal to facts alone.
  This intractability is reinforced by the fact that in most cases scientific evidence is not fully unambiguous, but may be interpreted in different ways and is open to methodological criticism. Salvo Nicolosi’s description of the contradictory claims on migration flows by the European Commission and the European Parliament is a case in point. Depending on the time frame chosen, the specific data that are used, and possible ‘correcting’ factors that are taken into account, one can arrive at different conclusions about patterns and trends in migration flows, with different implications for migration policies. Although it is sometimes possible to reach firmer ground by closer scrutiny of existing data, it is almost always possible to point at methodological caveats or rival interpretations that undermine the point made by one’s political opponent.
  Moreover, even if a study offers clear evidence against a given approach, it is often not immediately clear why this is so – does the study deal with an atypical case that happens to show specific peculiarities or does it reveal more fundamental problems with the approach?³⁰x In the philosophy of science, this is known as the Duhem problem, after the French philosopher who pointed out that hypotheses can never be tested in isolation since they are always part of a broader theory. If a finding contradicts a hypothesis, it is unclear which part of the theory is false: the hypothesis itself or some auxiliary assumption in the theory. See James Ladyman, Understanding Philosophy of Science, London and New York: Routledge, pp. 79ff. These ambiguities offer a kind of ‘protective shield’ around established paradigms, which can be upheld even in the face of mounting contradictory evidence. Only when the weight of contradictory evidence becomes too large to ignore and/or new policymakers with different sets of beliefs enter the stage will a firmly established policy paradigm give way to a new approach.
  In practice, the role of political interests and the role of beliefs (such as policy paradigms) in the production and use of evidence cannot easily be separated. The policy preferences that policymakers hold and promote are at least partly informed by the policy paradigms that they rely on (and may sincerely believe in). Conversely, supporters of an existing legal framework have (material and institutional) interests in upholding the policy paradigm that underpins and thereby legitimizes that framework. Evidence can then be used strategically to offer a semblance of impartiality to the pursuit of these interests. Either way, the protective shield around an established policy paradigm offers room to ignore or counter criticism from others. In this way, substantive beliefs, interests and strategic manoeuvring are intricately linked in a double-edged process of simultaneous ‘powering’ and ‘puzzling’ that characterizes much policymaking.³¹x To borrow the often-cited labels coined by policy scholar Hugh Heclo; Hugh Heclo, Modern Social Politics in Britain and Sweden, New Haven, CT: Yale University Press, 1974.

• F Conclusions

  In this contribution, I have reflected on evidence-based legislation on the basis of three questions. The common thread running through the answers to these questions is that ‘evidence’ is not simply an objective depiction of reality that is produced in separation from the practices in which it is used. Rather, what types of evidence are produced and used, how they are constructed and on what basis they are deemed to be valid or invalid is intimately linked to the ideas, values and interests of the people and institutions involved in making and applying legislation. As a result, what counts as good and relevant evidence is determined by the way the burden of evidence is defined, one’s view on the types of knowledge that are best suited to understand (social) reality and the purposes for which evidence is used. In that sense, evidence is never fully objective but always dependent on choices that precede its production and use.
  This is not meant to deny the value and importance of (independent) evidence in legislation and lawmaking. The choice is not between an unconditional belief in objective truth and an anything-goes type of relativism. Evidence offers an important counterpoint to unexamined assumptions and requires participants in legislative processes to subject their positions to systematic and critical examination. However, for evidence to fulfil that function, it is important to acknowledge the choices that underlie the evidence in evidence-based legislation.
  This argument has both normative and factual implications. Normatively, it requires a reflection on the criteria by which evidence is considered to be valid and valuable. Factually, it points to the importance of the way evidence is actually used in legislation and legislative processes. Lawyers and legal scholarship can play important roles in these endeavours, in particular at the crossroads of the normative and factual aspects.
  More concretely, beyond the systematic mapping of existing legal frameworks, in my view three issues are of particular importance in a legal approach to evidence-based legislation. The first issue is how to regulate the production of evidence for use in legislation and lawmaking processes. As was argued above, this evidence is not independent from those processes, but intimately linked to them. Hence, if the law is about defining the rules by which social processes and social conduct are organized, this applies in equal measure to the production of evidence. This requires (re)thinking what types of knowledge are needed and how they can best be produced. In doing so, existing legal notions such as transparency, accountability and checks-and-balances, which were developed in the context of political and administrative processes more broadly, may also be fruitful starting points to think about the way evidence is produced.
  The second issue is how to support the substantive and reduce the strategic use of evidence. If evidence is to fulfil its function as a critical touchstone in legislation and lawmaking, its production and use need to be regulated in such a way that it best performs that function. As in the production of evidence, this does not imply a detailed legal specification of what policymakers and other stakeholders should and should not do. However, law can outline the basic rules of the game that enable a fair and fruitful exchange between policymakers and stakeholders on the evidence before them and the implications it has for legislation.
  The third and final issue, which is a specific aspect of the previous ones, is how to bolster ‘weak’ participants in the evidence-based legislation process. Not all participants in lawmaking and the application of legislation are equally well-equipped to produce, debate and use the kind of evidence that is central to those processes. This goes for citizens but also for certain interest groups. As a result, the production and use of evidence tend to become dominated by specific categories of stakeholders (i.e. experts) who are most skilled at working with the required types of evidence. In doing so, they rely on taken-for-granted assumptions in their areas of expertise, which promote certain types of knowledge and include certain (implicit) priorities and values at the expense of others. Both from a democratic point of view and in light of the inherent fallibility of expert knowledge, it is important also to include evidence from other stakeholders. This, however, requires the active creation of a playing field on which different stakeholders are able to present their evidence and debate it with each other. Legal regulation may play an important role in achieving this.

Noten

• 1 Cf. the descriptions in James Ladyman, Understanding Philosophy of Science, London and New York: Routledge, 2002, at pp. 95-96; and Alexander Rosenberg, Philosophy of Social Science, 5th ed., Boulder, CO: Westview Press, 2016, at p. 12.

• 2 The term ‘rational-comprehensive’ comes from Charles E. Lindblom, ‘The Science of “Muddling Through”’, Public Administration Review, Vol. 19, No. 2, 1959, pp. 79-88, at p. 81.

• 3 Cf. Deborah Stone, Policy Paradox. The Art of Political Decision Making, 3rd ed., New York and London: W.W. Norton & Company, 2012, pp. 9-13.

• 4 E.g. Paul Cairney, The Politics of Evidence-Based Policy Making, London: Palgrave Macmillan, 2016, at p. 5.

• 5 Claudio Radaelli, ‘The Public Policy of the European Union: Whither Politics of Expertise?’, Journal of European Public Policy, Vol. 6, No. 5, 1999, pp. 757-774.

• 6 Esther Van Schagen, Jaap Baaij and Isabelle Rueda, ‘Designing Information Obligations in EU Consumer and Energy Law: Behavioural Research and Legal Design as “Best Available Evidence”’, this issue.

• 7 Asmaa Khadim and Hanneke Van Eijken, ‘A Citizen-Centric Approach to Evidence-Based Decision-Making under the European Green Deal’, this issue.

• 8 Rhoda Jennings, ‘The Use of Scientific Evidence in Precautionary Decision-Making in EU Environmental Law’, this issue.

• 9 Cf. the various studies that show substantial differences in the actual implementation and application of the Birds and Habitats Directives across member states. E.g. Raoul Beunen and Martijn Duineveld, ‘Divergence and Convergence in Policy Meanings of European Environmental Policies: The Case of the Birds and Habitats Directives’, International Planning Studies, Vol. 15, No. 4, 2010, pp. 321-333; Lars Borrass, ‘Varying Practices of Implementing the Habitats Directive in German and British Forests,’ Forest Policy and Economics, Vol. 38, 2014, pp. 151-160; Sharon Bryan, ‘Contested Boundaries, Contested Places: The Natura 2000 Network in Ireland,’ Journal of Rural Studies, Vol. 28, 2012, pp. 80-94; Francesca Ferranti, Raoul Beunen and Marie Speranza, ‘Natura 2000 Network: A Comparison of the Italian and Dutch Implementation Experiences,’ Journal of Environmental Policy & Planning, Vol. 12, No. 3, 2010, pp. 293-314.

• 10 Joseph Raz, Practical Reason and Norms, Oxford: Oxford University Press, 2002, at p. 35ff.

• 11 Martin Luther Munu, ‘Assessing the Adequacy of Existing Multilateral Rules Regulating E-Commerce: Lessons from EU Law,’ this issue.

• 12 Martin Unfried, Pim Mertens, Nina Büttgen and Hildegard Schneider, ‘Cross-Border Impact Assessment for EU’s Border Regions,’ this issue.

• 13 Arts. 8-13 TFEU.

• 14 European Commission, Better Regulation Guidelines, November 2021, at p. 35.

• 15 European Commission, Better Regulation Toolbox, November 2021, at pp. 137-138.

• 16 Cf. Bryan D. Jones and Frank R. Baumgartner, The Politics of Attention. How Government Prioritizes Problems, Chicago and London: Chicago University Press, 2005; Paul Cairney, The Politics of Evidence-Based Policy Making, London: Palgrave Macmillan, 2016, chapter 2.

• 17 For a classic critique on the notion of policy-making based on a comprehensive analysis of potential impacts, see Charles E. Lindblom, ‘The Science of “Muddling Through”’, Public Administration Review, Vol. 19, No. 2, 1959, pp. 79-88.

• 18 European Commission, Better Regulation Toolbox, November 2021, p. 81ff.

• 19 Asmaa Khadim and Hanneke Van Eijken, ‘A Citizen-Centric Approach to Evidence-Based Decision-Making under the European Green Deal’, this issue.

• 20 Mark Risjord, Philosophy of Social Science. A Contemporary Introduction, New York and London: Routledge, 2014, p. 34ff.; Alexander Rosenberg, Philosophy of Social Science, 5th ed., Boulder, CO: Westview Press, 2016, p. 23ff.

• 21 For a stylized yet clear juxtaposition of the two perspectives on social science, see Markus Haverland and Dvora Yanow, ‘A Hitchhiker’s Guide to the Public Administration Research Universe: Surviving Conversations on Methodologies and Methods,’ Public Administration Review, Vol. 72, No. 3, 2012, pp. 401-408.

• 22 Christina Boswell calls these the ‘instrumental’, ‘legitimising’ and ‘substantiating’ functions of expert knowledge; Christina Boswell, ‘The Political Functions of Expert Knowledge: Knowledge and Legitimation in European Union Immigration Policy,’ Journal of European Public Policy, Vol. 15, No. 4, 2008, pp. 471-488, at p. 472. Lorna Schrefler uses a similar typology, labelling these functions ‘technical-instrumental’, ‘symbolic’ and ‘strategic’ respectively; Lorna Schrefler, ‘The Usage of Scientific Knowledge by Independent Regulatory Agencies,’ Governance, Vol. 23, No. 2, 2010, pp. 309-330, at pp. 314-315.

• 23 See Falk Daviter, ‘Policy Framing in the European Union,’ Journal of European Public Policy, Vol. 14, No. 4, 2007, pp. 654-666. More generally on framing in policy-making: David A. Rochefort and Roger W. Cobb (eds), The Politics of Problem Definition. Shaping the Policy Agenda, Lawrence, KS: University Press of Kansas, 1994; Deborah Stone, Policy Paradox. The Art of Political Decision Making, 3rd ed., New York and London: W.W. Norton & Company, 2012.

• 24 Markus Haverland, Minou De Ruiter and Steven Van de Walle, ‘Agenda-Setting by the European Commission. Seeking Public Opinion,’ Journal of European Public Policy, Vol. 25, No. 3, 2018, pp. 327-345.

• 25 Ljupcho Grozdanovski and Jérôme De Cooman, ‘Of Hypothesis and Facts: The Curious Origins of the EU’s Regulation of High-Risk AI,’ this issue.

• 26 Jasper Sluijs, ‘Evidence-based Legislation in EU Competition Law,’ this issue.

• 27 Peter A. Hall, ‘Policy Paradigms, Social Learning, and the State: The Case of Economic Policymaking in Britain,’ Comparative Politics, Vol. 25, No. 3, 1993, pp. 275-296, at p. 279. The concept builds on the notion of (scientific) ‘paradigm’ in Thomas Kuhn’s seminal work on scientific revolutions; Thomas S. Kuhn, The Structure of Scientific Revolutions, 3rd ed., Chicago: University of Chicago Press, 1996.

• 28 Salvo Nicolosi, ‘Evidence-based Policy-making in European Union Asylum Law: Potential and Pitfalls,’ this issue.

• 29 Donald A. Schön and Martin Rein, Frame Reflection: Toward the Resolution of Intractable Policy Controversies, New York: Basic Books, 1994. In their work, Rein and Schön offer an approach to overcoming these types of controversies by reflecting on the underlying (normative) assumptions of each paradigm and identifying a shared core. This approach they call ‘frame reflection’.

• 30 In the philosophy of science, this is known as the Duhem problem, after the French philosopher who pointed out that hypotheses can never be tested in isolation since they are always part of a broader theory. If a finding contradicts a hypothesis, it is unclear which part of the theory is false: the hypothesis itself or some auxiliary assumption in the theory. See James Ladyman, Understanding Philosophy of Science, London and New York: Routledge, pp. 79ff.

• 31 To borrow the often-cited labels coined by policy scholar Hugh Heclo; Hugh Heclo, Modern Social Politics in Britain and Sweden, New Haven, CT: Yale University Press, 1974.