Evidenceballs

2 October 2012

The shadow chancellor’s announcement yesterday of a temporary stamp duty holiday for first time buyers to boost the housing market tells us a lot about the way in which evidence is ‘used’ in policy making.

The Blair government made much of its belief in evidence-based policy making. Yet as our report, Policy Making in the Real World showed, 12 years after the original Modernising Government report, both minister and civil servants recognised that learning from evaluation in particular remained a continuing area of relative weakness. Our new report, Evidence and Evaluation in Policy Making: a problem of supply or demand, draws on four seminars we organised in the first half of the year, in collaboration with the Alliance for Useful Evidence and the National Institute of Economic and Social Research to find out why there remained this gap between aspiration and practice.

We found barriers on both the supply and demand side. Those can be summarised as ‘can’t’ and ‘don’t’ and ‘won’t’ barriers.

Can’t use evidence

In some cases there is a real problem with availability of timely evidence – quite often because earlier policies have not been evaluated properly. And too often data was not available in useable form. And some government policies were not necessarily amenable to rigorous testing when implemented – though our panellists thought that there was scope in most cases for evaluating some elements of the approach to a problem. But others recognised that there was an absence of the sort of ‘fast-acting evidence’ that ministers needed.

Don’t use evidence

Our seminars suggested that the bigger barriers lay on the demand side. Some of these were sins of omission – by both ministers and civil servants. Too often civil servants were not on top of the current evidence base and able to present ministers with a synopsis of what that said. Ministers could interpret a proposal to seek out evidence or to proceed through piloting as an attempt to block, particularly relevant in the light of Francis Maude’s comments today. The analytic community was too distant from decision-making – and reluctant to make the case for more rigorous evidence to senior policy makers. And the Treasury failed to link spending decisions to good evidence of what was working. All these were reasons why evidence and evaluation was used less to inform policy than it might be.

Won’t use evidence

Into the final category comes a refusal to use evidence. In some cases politicians are clearly pursuing policies as an end in themselves rather than as a means to achieve an objective. In those cases they need to be clear about what they are doing. So, in an example discussed at our seminars, if ministers want to promote marriage as a ‘good thing’ that is fine. If they want to promote marriage to reduce poverty, that proposition can and should be tested. But there is a second category – where a policy is instrumental, has a clear objective, but evidence says it will not work. That is where the shadow chancellor’s proposition comes in – HMRC evaluated exactly this policy last year and concluded that it led to a 0-2% increase in demand and a cost per additional transaction of a stunning £160,000. In other words, an amazingly ineffective waste of money.

Fewer places to hide

But the reaction to the shadow chancellor’s speech shows another thing about the emerging evidence landscape discussed at our seminars. Within minutes of his making the proposition, Twitter was alive with people tweeting the link to the HMRC evaluation and pointing out that this policy was a proven failure. A new breed of citizen scrutineers can be a powerful force for changing the incentives to base policy on better evidence.

Share this post:

Comments (3)

  1. Walker on 3 October 2012 at 8:10 am

    Did you consider the supply of existing external evidence in policymaking? I’m talking about the evidence presented by campaigns and lobby groups. I appreciate government should be conscious of the policy agenda of such groups, but it seems a shame that all too often their advice is freely given and well intended, but dismissed out of hand.

  2. Jill Rutter on 3 October 2012 at 5:49 pm

    I am not sure it is dismissed out of hand – I think sometimes it is actually treated too uncritically with people not thinking through the motivation of the group. campaign/lobby evidence is of course particularly important for oppositions because they have relatively little resource to do their own detailed research (especially true on quiet technical issues).

    But you are right that depts and Ministers should be open to evidence from this source too –

    one of the issue our report looks at is the need to be open to more qualitative citizen feedback from citizens (and those who organise on their behalf) – some very interesting stuff from Patient Opinion.

  3. John King on 11 October 2012 at 11:29 am

    There is a rather large difference between evidence on the one hand, and knowledge and understanding on the other. All research relies on assumptions and models. Crucially, the details of these assumptions are usually removed before evidence reaches ministers (it is difficult to make a decision when faced with the uncertain nature of the world as it really is). So as information moves up the chain it becomes less and less precise, perhaps even to the point where decision-making on the basis of tacit understanding (i.e., what could be seen as ‘non-evidence-based’ decision making) becomes a logical response.
    We should refrain from uncritically accepting (often disputed) evidence as fact and begin to discuss how we can increase our (as policymakers or members of the public) deep understanding of issues. For example, the information ‘bandwidth’ of putting a minister or civil servant into a housing estate for a week (whilst being a single case study and therefore of uncertain generalisability), might outweigh a thousand pages of written analysis. How can we ensure that ministers and civil servants understand an issue (and the nature of evidence) well enough to choose between competing claims of truth, yet without becoming paralysed by information overload and indecision?

Blog: Permalink