The importance of evidence for policy
Good paper by the Prime Minister’s Chief Science advisor on policy – focused strongly on transparent evidence based policy. I can’t disagree with this as a framework!
1) identifying problems; 2) measuring their magnitude and seriousness; 3) reviewing policy options; 4) systematically assessing likely consequences of options; and 5) evaluating results of policy interventions.
When the Chief Scientist discusses evidence “nudging” policy, he is prescribing a Bayesian view of how we update our priors based on evidence. This is all cash.
However, I don’t feel that the first stage is ever covered off particularly well … by anyone really. In this context I just find the term problem uncomfortable – if research is based on understanding “issues” instead of “problems” we can then use data and theory to identify issues where a “problem” exists. This is far from semantic – the very nature of how we decide something is a “problem” will influence the entire process of analysing policy interventions, either for the better of the worse!
The “appropriate interpretation” of data requires a transparent theoretical base as a starting point, a view I didn’t quite get out of this paper on reading.
Now do not take this as a criticism of the paper, I strongly recommend it, the description in it, and the conclusions of it – I agree with this post by Peter Griffin indicating the sheer importance of thinking in terms of evidence based policy! And I would note that Bill Kaye-Blake has been making noises about this for some time!
I just thought I’d add my two-cents about the very hard initial identification step, especially given my view that it is very difficult to get sufficient evidence involved to shift our priors together from disparate starting points for many policy questions. Describing a relationship between a set of variables observed variables is inherently different than identifying than doing welfare analysis of policy relevant issues given:
- That heterogeneous individuals make choices based on values we cannot observe, but implicitly wish to target
- We need an understanding of the way relationships may change when policy changes
All in all implying our conclusion must involve tacit (or preferably explicit) assumptions about value, and structure to help inform the role of policy given the incentives and following choice of individuals 😉
And this view is indeed in the thrust of where the advisory paper is going, I just felt spelling it out in the way social scientists/economists interpret it would be useful 🙂 .
Hi Matt,
You might also be interested in this post about evidence-based policy:
http://judithcurry.com/2013/09/02/professors-politics-and-public-policy/
It’s come at it from a climate science perspective, but it provides a view-point that points out that there’s often a large amount of uncertainty when dealing with complex issues, and the uncertainty is often not considered as part of the evidence (those putting forward the evidence often overestimate their certainty).
Cheers,
Mark
Hi Mark,
Cheers for that – the govt paper discussed uncertainty a lot as well, important stuff.
However, there were two additional points I wanted to raise:
1) The common one from economists, the Lucas Critique – given that policy may have a different impact if we can’t “reduce” the issue of interest down to issues of individual choice (the parameters in a model are not policy invariant in many cases).
2) The more insidious issue that the target variable of policy is unobservable – as a result, our “problem”, “issue”, or “targeted endpoint” is not really something that is that evident as a starting point! With climate science there is an idea about what a GWE is, but when we dig right into social policy the way to view outcomes is a funky issue.
All good fun, and beyond a shadow of doubt economists are pro using the scientific method to come up with an objective, or at least as transparent as possible, way of describing relationships and outcomes. However, how we use these outcomes and define a “problem” offer another important element that requires significant investment.
On the note of the post you attached, this quote was indeed golden:
“To the extent that they want to ground policy making exclusively in evidence and/or values, they misconceive policy making as a search for means to achieve predetermined ends, when in fact it is a dialectical process of identifying and reconciling ends in light of the means which may turn out to be available and acceptable.”
Nice!
Fair criticism, but I do hate it when people make their case with “when in fact”. Another popular one is “the reality is…”
The paragraph that precedes that quote:
“To the extent that they want to base politics on public reason, in
which policies are to be judged against an explicitly logical,
transparent version of some public interest, they reconstruct politics
as a quest for truth and substitute academic for civic judgment.”
This sounds a lot like mass psychoanalysis. I don’t think academics are naive enough that they completely misunderstand politics any more than the average politician who has their own ideas about what politics is, and what it’s for. And politics as “a quest for truth”? Why would they see politics itself as a quest for truth, rather than science as a quest for truth and politics as best undertaken with reference to that quest for truth?
And then this: “…they indulge in an heroic and utopian denial of human character and motivation”. How exactly are they utopian? Doesn’t utopia involve a whole worldview, not just a specific environmental issue? Or are they heroic in a unrealistic way, because they have a utopian idea of the noble quest? Aren’t they usually accused of being catastrophists? This seems like a total mess.
Scientists sometimes have a reputation for being nerdy and not adept at understanding emotional life to the extent of humanities oriented people, but even I give them more credit than this guy. That’s not to say they couldn’t benefit from learning from political strategists like Karl Rove or whoever, and there are plenty of arguments you can level against their strategies, but I’m not sure that ever brings them to the level of “a complete denial of human character and motivation”.
Indeed, I can definitely understand where you are coming from here.
And to top it off, the paper was very clear about the fact that using structured empirical evaluation wasn’t the entire process – and it justified it perfectly with a call to transparency! Oft times, the same people calling scientists naive are in fact engaging in a territorial pissing contest over their own priors rather than a comparison of evidence – which is not cool 🙂
I just picked out that quote as it illustrated the fact that there is not so much a linear process starting from a problem or paradox when we think about policy making. This is another point of interest on top of the ‘Lucas Critique’ view and the ‘unobservable’ view, and one I’d like to give a little thought to.
I thought it was a good paper too. I suspect that when the report refers to “problems” it will be talking about the matters that government ministers describe as problems. The report is talking about the context of a govt department – and in that context its Minister’s who decide what they want advice on.
I think this was a great paper and all, but these issues I’ve chatted about are something pretty specific to social science and policy making – things that need to be taken into account as well. If anything, I was trying to add something complementary to the paper rather than straight critique 🙂
Another view The Bluff of Evidence-Based Policy
Indeed – I would note the areas of interest when identifying a problem I noted above are along the same lines of the criticisms being made there. I have just framed them as things we need to add to such policy.
Don’t get me wrong, if “evidence based policy” is released that bastardises trade-offs I will be against it – just like you guys 😉