09 June 2016

A report by Sir Stephen Sedley for Sense about Science revealed the slapdash approach departments take to the research they (and, by extension, we) pay for. Jill Rutter argues they need to learn from the good practice out there and up their game.

Missing Evidence, published this month, reviews how much government-commissioned research is publicly available. Sir Stephen went in suspecting conspiracy – and, surprising no-one who has worked inside government, found cock-up. A few high-profile cases of delayed publication suggested there might be a deliberate strategy within government to suppress externally commissioned research. Indeed, there were some cases when publication of inconvenient findings had been managed.

But the far more mundane finding was simply that departments were not in control. They did not routinely know what they had commissioned. They did not have a clear view of their forward programme of research (or indeed easy access to their back catalogue). Material was not easily accessible – either within or between departments. No one thought the gov.uk website was a useful place to try to find government-funded research. Excessive staff turnover within departments meant that staff on the receiving end of research were often the successors of those who had originally commissioned it and were not confident about handling publication, which led to delays.

Rules and protocols said the right thing – but were variably applied and there was no common understanding of how they were meant to operate. This variability of intent and practice mirrors many of the findings of the National Audit Office when it looked at the patchy track record of departments on evaluation. The good news was that Sir Stephen found exceptions, as the diagram below shows. Defra, DfID, the Department of Health and the Department of Transport all maintained up to date, comprehensive and searchable databases of commissioned research – others could link to lists of published reports, or at least could assemble the information. But seven departments – including the Cabinet Office, FCO and HMT, could not provide the information within FOI cost limits.

sense-about-science-report-chart

Sir Stephen noted this commitment to transparency had not chilled their enthusiasm for commissioning more research (though Defra and DH did feature in a number of the case studies documenting delay). He also pointed out that the What Works initiative was fostering a more sophisticated approach to trialling, evaluation, publication and dissemination.

The report makes sensible recommendations – a centralised searchable database, common standards and a commitment to transparency as well as an investment in staff capability. In terms of efficiency and effectiveness of government, the first is a no-brainer – departments who do not know what research they have already commissioned, waste time and money and fail to learn.

Other departments and agencies – as well as the outside world – need to have access to studies commissioned in another part of government. A clear database of commissioned research (with due protection for a few really sensitive issues) would also help external scrutiny of what questions the Government was failing to ask about its policy – and help guide independent research to fill those gaps. That, of course, will only work if the Government is committed to making its data available to external researchers. The report highlights the benefits of HMRC’s Datalab and the National Pupil Database – which allow approved researchers to use sensitive data – and the way the Educational Endowment Foundation publishes to allow reanalysis of its findings.

One of the big issues is whether research should be published as soon as it is available – or whether it is acceptable to hold it back to produce alongside a policy announcement. Many argue that earlier publication allows proper scrutiny of the research and helps inform thinking – the Health Select Committee, for example, pressed for early release of the Public Health England study into sugar consumption, which allowed it to inform its inquiry in advance of the Government’s own childhood obesity strategy – while evidence on the (positive) impact of minimum alcohol pricing was published at the same time as the announcement that the Government was not planning on proceeding.

What is vital is that the Government makes plain the evidence base on which it is working – and allows the public access to the research it is drawing on – as we called on government to show your workings. The UK Statistics Authority noted to the Sedley Inquiry the number of times it had to upbraid Ministers for failing to make research they cited available to the public. The benchmarking exercise Sense about Science are now leading (with IfG and the Alliance for Useful Evidence) to follow up the report will see the extent to which research – whether commissioned by government or not – is being used effectively to make policy. Hopefully that will provide a further incentive for departments to get their collective evidence act together.