Evidenceballs

The Blair government made much of its belief in evidence-based policy making. Yet as our report, Policy Making in the Real World showed, 12 years after the original Modernising Government report, both minister and civil servants recognised that learning from evaluation in particular remained a continuing area of relative weakness. Our new report, Evidence and Evaluation in Policy Making: a problem of supply or demand, draws on four seminars we organised in the first half of the year, in collaboration with the Alliance for Useful Evidence and the National Institute of Economic and Social Research to find out why there remained this gap between aspiration and practice. We found barriers on both the supply and demand side. Those can be summarised as ‘can’t’ and ‘don’t’ and ‘won’t’ barriers. Can’t use evidence In some cases there is a real problem with availability of timely evidence – quite often because earlier policies have not been evaluated properly. And too often data was not available in useable form. And some government policies were not necessarily amenable to rigorous testing when implemented – though our panellists thought that there was scope in most cases for evaluating some elements of the approach to a problem. But others recognised that there was an absence of the sort of 'fast-acting evidence' that ministers needed. Don’t use evidence Our seminars suggested that the bigger barriers lay on the demand side. Some of these were sins of omission – by both ministers and civil servants. Too often civil servants were not on top of the current evidence base and able to present ministers with a synopsis of what that said. Ministers could interpret a proposal to seek out evidence or to proceed through piloting as an attempt to block, particularly relevant in the light of Francis Maude’s comments today. The analytic community was too distant from decision-making – and reluctant to make the case for more rigorous evidence to senior policy makers. And the Treasury failed to link spending decisions to good evidence of what was working. All these were reasons why evidence and evaluation was used less to inform policy than it might be. Won’t use evidence Into the final category comes a refusal to use evidence. In some cases politicians are clearly pursuing policies as an end in themselves rather than as a means to achieve an objective. In those cases they need to be clear about what they are doing. So, in an example discussed at our seminars, if ministers want to promote marriage as a ‘good thing’ that is fine. If they want to promote marriage to reduce poverty, that proposition can and should be tested. But there is a second category – where a policy is instrumental, has a clear objective, but evidence says it will not work. That is where the shadow chancellor’s proposition comes in – HMRC evaluated exactly this policy last year and concluded that it led to a 0-2% increase in demand and a cost per additional transaction of a stunning £160,000. In other words, an amazingly ineffective waste of money. Fewer places to hide But the reaction to the shadow chancellor’s speech shows another thing about the emerging evidence landscape discussed at our seminars. Within minutes of his making the proposition, Twitter was alive with people tweeting the link to the HMRC evaluation and pointing out that this policy was a proven failure. A new breed of citizen scrutineers can be a powerful force for changing the incentives to base policy on better evidence.