Half-empty or half-full? How public policymakers use scientific evidence

Jelmer Gerritsen

Most governments nowadays claim to pursue ‘evidence-based policymaking’: a technical type of decision-making in which scientific evidence about the performance of policy measures determines policy choices. The electorate, however, often thinks that public policymakers only use the evidence that fits policymakers’ political preferences. But how do policymakers really use evidence? Jelmer Gerritsenargues that political priorities determine how policymakers use evidence and that we should explain evidence-use in two ways: as a glass-half-empty, or a glass-half-full.

Evidence-based policy or policy-based evidence?

Over the last few decades, scientific evidence about the performance of policy measures obtained a central role in policymaking. To improve policy programs and increase the social added value of public tax money, governments worldwide shared their commitment to ‘evidence-based policymaking’ (EBPM). Simply put, EBPM is a technical type of decision-making in which scientific evidence determines whether policymakers implement or stop implementing policy measures.

To illustrate this, imagine the following: a government aims to reduce alcohol use among youngsters and starts an online campaign. After a while, policy evaluations demonstrate that the campaign in fact increased alcohol use among youngsters. According to EBPM, governments then stop the campaign as the evidence shows that the measure (the campaign) did not have the intended effect (reducing alcohol use).

The electorate, however, often does not seem to believe that this is how governments use evidence. Instead of EBPM, they use the pejorative term policy-based evidence making’ (PBEM) to emphasize that policymakers cherry-pick, distort, and search for evidence to legitimize pre-determined political goals. For example, imagine that a cabinet wants to lower the maximum car speed on highways (measure) to decrease CO2 emissions (goal). After a year, a policy evaluation shows that lowering the maximum car speed did not have any effect on reducing CO2 emissions. According to PBEM, this government will search for any type of evidence that suggests that lowering the speed does have a reduced effect on emissions, or they may slightly distort the evidence to use it to their advantage in parliamentary debates.

Why we should interpret evidence-use as a glass-half-empty or a glass-half-full

But which approach is the most accurate to explain how policymakers use scientific evidence? Shortly, my research has demonstrated that the PBEM approach delivers the most accurate description of evidence-use, whereas the EBPM approach does not. Political priorities, first and foremost, determine policy choices and not “objective” science.

However, I argue that we shouldn’t interpret evidence-use solely in terms of EBPM or PBEM, but more so in terms of a pessimistic “half-empty glass”- or optimistic “half-full glass”. EBPM and PBEM are relevant approaches, but they mutually exclude optimistic and pessimistic explanations of evidence-use. Indeed, EBPM solely relies on the optimistic notion that not political ideas determine decisions, but scientific evidence. And PBEM works from the pessimistic notion that only political priorities determine policy choices. I claim that there are pessimistic and optimistic aspects about evidence use.

Glass half-empty

On the one hand, we can speak of a glass-half-empty because policymakers are likely to only use the evidence that aligns with their political priorities. Based on document analyses and several interviews with policymakers from Dutch ministries, my research demonstrates that policymakers distort, frame, and search for evidence to legitimize their political priorities.

Let me give a brief example to underpin this claim. In higher education policy, policymakers pursue three political goals: more accessibility, efficiency, and quality of expenses. However, policymakers have to choose between the goals due to the conflicting nature of the goals. From the coalition agreement and policy documents, we can conclude that the Minister of Higher Education prioritizes accessibility (and well-being of staff and students) over efficiency. Although policy evaluations have shown that universities can better strategically manage the intake of students through implementing instruments for stricter selection, the Minister did not utilize this evidence to change the policy as the Minister explicitly noted that these measures may be positive for the efficiency, but negative for the accessibility of education. Remarkably, the Minister did utilize the evidence demonstrating that the well-being of university staff is precarious to make policy changes, because this is one of her policy priorities. This example shows that the electorate has good reasons to be pessimistic about the use of evidence as policymakers selectively pick evidence.

Glass half-full

On the other hand, I contend that we can interpret evidence-use as a glass-half-full. My analysis has shown that there are at least two reasons for this.

First, policymakers are likely to act upon evidence if it tells them that the implemented measures do not reach their formulated policy goals. In the previous example, we have seen that the Minister prioritized accessibility (and well-being) over the efficiency of expenses, and that evidence about the well-being of scientific staff at universities was an incentive for implementing measures to improve their situation. This demonstrates that Ministers have to make choices in favor of their formulated policy goals as they can’t possibly reach all political goals, but that that they have good intentions to impact people’s lives positively.

Source: Koeman et al. (2019, p.8).

Second, policymakers store evidence in the memory of the organization. Ministers and State Secretaries may not directly act upon evidence, but the evidence has an indirect impact on the undercurrent of the policy process. Many scholars contend that this ‘conceptual’ use, the indirect impact, is the most common form of evidence use. The pessimistic view of the electorate exists because policymakers do not directly implement new measures, which makes people think that they disregard the evidence. Policymakers at Ministries, however, noted that they always “read, consult, and store” evidence to take it off the shelf when the political momentum is there. So, direct evidence-use depends on the political situation, but evidence always informs policymakers’ decisions or preparations for new policy proposals.