“There is a question about our demand for evidence and strategy in politics”

Thursday’s seminar on How Can Government Make the Best Use of Evidence? was the latest in a series run by the Alliance for Useful Evidence which has been working in Northern Ireland for two and a half years. Peter O’Neill introduced the session and gave some context to A4UE’s purpose. He suggested that evidence, politics and delivery all need to intersect if civil servants are to be successful in rolling out a policy.

Evidence to back up coherent policymaking can be distilled from many sources: the history of a sector; lessons learned from previous policies as well as evaluations and academic research; parallel initiatives in other places; internal and external expertise; the legal and legislative context; as well as what can be learned from accountability processes, the media and interested parties.

Some politicians instinctively want to be well informed; others take an opposite view. In April 2011 former Prime Minister David Cameron said that “politics shouldn’t be some mind-bending exercise: it’s about what you feel in your gut”.

While there are countless research tools available – and some go through their own cycles of trendy adoption before being displaced – the public mood today is often to distrust experts, creating a challenge for researchers to effectively use and promote research.

Peter was honest about experts sometimes getting it wrong: Dr Spock’s advice ignored other contradictory research findings and led to thousands of cot deaths. And sometimes politicians use less-than-trustworthy single sources of evidence like Wikipedia.

There’s sometimes a challenge for civil servants and advisors to distil everything down to a two page briefing. Independent What Works Centres across the UK – but still not set up in Northern Ireland [Ed: though it is a policy that can be found in DUP’s 2016 manifesto] – offer ‘rapid evidence assessments’ to help wade through the copious amounts of research and highlight high quality evidence to inform decision-makers. NESTA have published an Evidence Practice Guide as well as a clickable/searchable ‘three sausage diagram’ of the UK Evidence Ecosystem for Social Policy.

The main keynote address was delivered by Siobhan Carey. She’s the chief executive and registrar general of NISRA (the Northern Ireland Statistics and Research Agency) and offered some observations on the interface between analysts, academia and policy.

She began by asking why it was important that government make the “right” use of evidence?

“I think everyone agrees that we want government to make the best decisions and we want them to make good choices. We recognise that sometimes those decisions and choices will be ideologically based. But the general expectation is that those choices and decisions will be made on the best available evidence.”

She acknowledged that some people are scared of numbers and perhaps put off by her ‘scarey’ email address!

Data shouldn’t be collected for the sake of collecting it. “Our job is not done when we have collected the research, published it and moved onto our next juicy project. If it’s not being used there’s no point.”

While people think that collecting data is expensive, Siobhan retorted by asking “how expensive is it not to have the evidence?” if you, say, are spending a lot of money on a big policy intervention and you don’t have any evidence and it doesn’t deliver what you thought?

She challenged the myth that “data grows on trees”: policy delivery programmes need to plan to collect data and not leave it as an afterthought. “Data only exists because someone has gone to the bother of collecting it, harvesting it, curating it” and – particularly as services move online – attention has been paid to using a fit-for-purpose data model.

“I absolutely love data. I’ve spent three and a half decades collecting data and being at the interface between data and research and policy choices but I don’t care about it [data] because I don’t need it.

“I’m not the person who has to make the choice. I’m not the person who is deciding whether this million is being spent on HIV retrovirals or a demographic and health survey. I’m not the person who chooses policy intervention A over policy intervention B.

“But if I was that person, I’d want some data to make those decisions on because in God we trust but could all others please bring some data to the policy choices!”

Siobhan identified some issues:

  • People spend too much time locating research, and not enough time developing insight from it.
  • Different sources of data may look like similar measures but can tell conflicting stories, and there is a temptation to use the set of data that suits best.

“If we want people to use research and evidence we need to make it easy to find, easy to understand, highlight the big key messages, get them drawn into it, make sure it’s as comprehensive as possible and written in a really accessible way.”

What can government analysts do? They been to be involved from the start; have a strong curiosity shared with the policy team; and well communicated.

What can policy makers do? Know what data you have and how to access it. Take responsibility for making sure measurement will happen, and finish every conversation with ‘how am I going to measure that?’

What can the wider research community do? Make sure your research is relevant, timely, on the topics of the day in society and in the economy. They should ask themselves who will be interested in their results, why, and what decisions does it support? Statisticians “should not be the people looking in the rear view mirror telling you what happened yesterday or probably what happened three years ago”.

What’s the opportunity?

“The Programme for Government very much has data at its heart … high level outcomes, indicators … it’s setting a clear direction of travel in terms of the approach, so very much outcome based accountability, monitoring and measuring … underpinned by a collaborative delivery model with an opportunity to join up the evidence at the beginning of the process with the monitoring and evaluation at the end.”

That should lead to better policy decisions; better targeted service delivery; lower cost operations; reduce fraud error and debt; a more innovative economy; and better informed public debate.

Siobhan finished with the quote: “Without data you are just another person with an opinion.”

Two former permanent secretaries responded to Siobhan Carey’s talk.

First up was Aideen McGinley who admitted she was once an “I don’t do numbers” person. She said that inside government there was a hunger for good evidence. And outside government people want evidence to be genuine and reflect the lives they live. She has been heavily involved with the Carnegie Roundtable on Wellbeing and highlighted a report detailing The Scottish [Government] Approach to Evidence. Aideen commended NISRA’s approach in seeding statisticians within NI government departments “who became colleagues with other public servants to support and sustain them”.

Aideen recalled her time in Derry working on ILEX. “There were 80 strategic plans for the city and no one sense of what is the vision of what the city needs.” The local politicians knew they needed a sound evidence base to get central government attention for their definitive plan for the city. Aideen’s ‘magic formula’ is”

“extensive engagement plus intensive analysis gives you good policy.”

It gives politicians assurance and people recognise their voice in the plans.

She summed up with an old American proverb: “If you tell me I’ll forget. If you show me I may not remember. If you involve me I will understand.”

Will Haire suggested that there wasn’t a lack of evidence or a lack of ability to work on that process.

“My office when I [retired] two years ago was full of very good documents, evidence right across, wall to wall Northern Ireland strategies, too many of which I was involved with writing over the last twenty years.

“And they were basically ignored by us all. We wrote them. We are great at writing strategies and we launch more strategies than we launch boats …

“But there is a question about our demand for evidence and strategy in politics. And the complexity that we are facing in the twenty years since devolution of our confidence that we can do it. I think that’s the really big issue for us.”

He added:

“We have to collectively agree that we all have a role in building a sense of anger and worry about the sheer size of the issues we have to deal with.”

NI has an evidence base around childhood obesity. But it’s accompanied by a societal complacency.

“We have change the dynamics and the demand that we can do things … and we have to create a politics that is about people’s needs. Understandably we have constitutional obsessions and issues [but once those are addressed] we have to deal with need.”

It was Will’s opinion that the public “are fairly awful to our politicians – we give them hell on lots of our constituency issues but we don’t stand back and give them the space when they have to deal with the wider difficult issues. We all look at our self-interest and we fight our own bit. We’re not good at giving that broader support.”

He commented on the role of the ‘third sector’ (community and voluntary organisations).

“The third sector always frustrates me. So many in Northern Ireland. We have great charities, and lots of them have policy officials, and I saw damn all coming out of them. I saw very little stuff really other than lobbying often on behalf of the interests of their charity, really For the amount of resource we put in there, I don’t think we produce the results.”

Will characterised the notion of a What Works Centre for Northern Ireland as “an interesting idea” but said that “one institution is not going to solve the process”.

Returning to the idea that “we need to build a political culture that wants to use evidence”, Will asked “what is the evidence that really does make political change?”

His experience was that “constituents that had really got under my ministers skin at the weekend” gave politicians a story and an issue they could understand. Excellent local examples of people solving problems could be powerful.

“Often I found that my job was could I find good stories and could I raise the profile on them because that was what could make change. Could I get my minister to see [a particular] group project? … That was often the evidence. It wasn’t some academic document or paper.”

– – –

I also doubt NICVA will recognise Will’s comments about the “damn all” coming out of the third sector, given the number of reports and evidence-based interventions and policy coproduction.

As chair of FactCheckNI’s advisory group, I’ve been impressed with the attitude and input of staff from NISRA. It’s an organisation that wants people – whether they’re public sector or not – to understand the data they produce and understand how to interpret it.

But listening to the two former permanent secretaries, and in particular to Will Haire, I began to understand why there is a lot of work to be done within and without government departments to embed the value of data and the evidence it shows. Compelling stories seem to trump evidence. They’re fast to understand, quick to retell, and make an emotional impact.

Politicians – who in the NI Assembly situation nearly all seem to have the potential to become Executive ministers – need to be (re)trained to be inspired by compelling stories but to doggedly pursue evidence to back up the projects that they’re being shown.

It’s not a matter of ignoring gut instinct or common sense. There seems to be a gap in quantifying the size of need and the size of the best investment to fill it, and designing the evaluation up front. (The fixed term numeracy and literacy intervention by OFMdFM – and championed by Emma Little Pengelly as special advisor – came to an end and despite evidence from schools that pointed to its success the project seemed not be renewed due to the need to pause to evaluate.)

An independent What Works Centre for Northern Ireland, aimed directly at supporting policy makers could ease the burden of finding relevant evidence and make up for the lack of local think tanks pumping out timely reports.

, , , ,

  • Sharpie

    There is still precious little ability to prove that an intervention works. Statistics are great for saying how bad we are doing – and that is what people don’t want to hear. When they roam into the area of policy though – the amount of “credit capture” by different initiatives and interventions is galling. Any given conference has a powerpoint slide with some grand conclusion postulated in a statistic alongside a supposed magic intervention by some far sighted civil servant or politician.

    Statistics and measurements cannot be useful if used in isolation – outside of the context of who was involved in what was being measured and why it was being measured and whether or not there is someone involved who can use the information to change decisions. In other words Aideen McGinley had it right – only engagement along with the right measurements is helpful and useful.

    Oh, and longitudinal studies are really essential too and no-one likes to fund those.

    Will Haire’s attack on the third sector is a bit mental – especially when they have a different reporting regime for every funder (especially Gov’t and EU) and mostly dreary information that means nothing and cannot be usefully used.

    I also heard that 83.6% of statistics are made up.

  • Skibo

    Collecting data is always useful to assist in decision making. The problem i see is when decisions are made for political reasons and then the data is manipulated to confirm the suitability of the decision.
    The shocking question for next year will be how many deaths can be accrued and cost savings for reduced building regulations still make economic sense!

  • Barneyt

    Data has to be transformed and in isolation is meaningless. This transformation process creates information and this is where the devil lies. As you suggest you can create whatever message you want be it truth or designed to steer in the wrong direction.

    I saw this article and for a brief moment i hoped it would address the truth, meaning and accountability with regards to agreements and government acts. I’m sure I read on slugger that the DUP by definition cannot vote on the queens speech as a result of the fixed term act. Seems this might be bad data, information or plain old fake news