#Soapbox: The sorry tale of ‘Outcome-Based Performance Management’

Dr. Toby Lowe is a Senior Research Associate at the University of Newcastle. Here he lays out in detail some of the flaws in a branded system ostensibly constructed to ensure delivery of government policy which the current NI Executive parties have agreed to implement at Stormont. 

The Northern Ireland Executive has just published its Draft Programme for Government Framework. Many people will no doubt want to comment on the priorities it contains. But there is actually a more fundamental problem – the Framework is based on “an outcomes approach”.

This approach will undermine the effective delivery of any policy it contains.

Northern Ireland is about to join the back of a long and dismal queue – places which are still waiting to see real improvements in the social interventions which make people’s lives better, despite have spent a huge amount of scarce resources, and wasted lots of precious time, implementing an outcomes-approach.

Let’s talk about outcomes

The Government has stated that its approach “will focus on outcomes”, and that “Outcome-based approaches have gained currency internationally in recent years”. It even references the high-priest of outcomes himself, Mark Friedman, and his well-known book “Trying hard is not enough”.

What does all this mean? At the policy level, thinking about outcomes can be a very sensible thing to do.  Political activity is supposed to enable people to have a better life. Thinking about the outcomes that political activity could help create is a good tool for building a shared vision about the kind of society we want.

Turning those desired outcomes into indicators, and measuring progress against those indicators, can also be helpful for policy discussions. Using those measurements to inform rigorous debate about policy priorities is a good thing to do, as it helps us to understand which issues are most pressing, and what the trends in those issues are.

So, yes, let’s talk about outcomes. But we cannot do is seek to manage the performance of our public services, and other social interventions, using Outcome-Based Performance Management approaches.

Let’s not use outcomes to performance manage social interventions

Why shouldn’t we do that? Because the evidence is clear – Outcomes-Based Performance Management (OBPM) doesn’t work. The Executive is right, many people and organisations internationally have tried to implement them in the last few years, and here’s the summary of what they found:

The overall conclusion from international experience of implementing an outcomes approach is that the journey is long and the results are disappointing.” (Wimbush 2011)

Every study of the impact of using OBPM says pretty much the same thing (Perrin 1998; Van Thiel and Leeuw 2002; Mayne 2007; Chamberlain et al 2010; Soss et al 2011; Keevers et al 2012; Newton 2012). Taking all this research together, we see that the good aspect of using such approaches were that they enabled people to talk about social goals together.

But as soon ‘outcomes’ or ‘results’ are used to performance manage the people and organisations who seek to deliver those results, they create perverse incentives, which get in the way of people doing their jobs well.

Just this week, a systematic review of the studies of outcome-based contracting was published – a study of all the studies into the effectiveness of outcomes approaches. Want to guess what it found?

“The findings indicate that while outcome-based contracts deliver the measures of outcome for which they pay, these measures do not always reflect the intention of the contract designers, or desirable outcomes for the end-client. Measures of outcome that were not related to payment did not improve and sometimes worsened.

“Some outcome payments created incentives for service providers contrary to the achievement of desired outcomes. For example, employment services contracts that were meant to increase tailoring and flexibility had the opposite effect” (Tomkinson 2016)

Why do Outcomes-Approaches Fail?

Previously, people who have studied the implementation of OBPM have thought that such approaches fail to work because there has been some technical problem with their implementation: they weren’t measuring the right things, or they weren’t measuring them properly.

This would be good, because technical problems can be fixed. But it’s not a technical problem – OBPM fails because it’s conceptually flawed. It has two problems which cannot be fixed:

The Measurement Problem

The idea of outcomes approaches is that they improve the lives of the people who experience those interventions. Unfortunately, measuring the impact of an intervention in someone’s life is a very expensive and research intensive thing to do.

For example, Robert Schalock, who wrote the book on Outcomes-Based Evaluation (2011), says that if you want to understand the impact of an intervention on someone’s life you need to do both qualitative interviews and quantitative questionnaires (which themselves have been designed by people who have experienced the intervention), and he recommends a minimum period of 22 months research after the programme has finished.

So, people never do that. Instead, they measure what can be measured and call that an outcome. They use what are called ‘proxy indicators’ – things that are easy and quick to measure, rather than a real-life outcome in all its complexity. 

And here’s the problem, if you use these proxy ‘outcomes’ as measures to hold organisations accountable to – you’re holding them accountable for what is easily measurable, not what is important.

If you hold an organisation accountable for making changes in simplified proxy indicator, the difference between that proxy and a real outcome in someone’s life is exactly a measure for the way in which that target distorts the practice of those who are being held accountable. You’re making them work to the measures, not what matters, because what matters is always too complex to fit into a simple measure.

Proxy indicators turn everyone into a caricature, and then waste time and money trying to provide inappropriate services to that caricature, rather than helping the real person in front of them. Want to see what happens when services use proxy measures to understand people? Here’s a great example…

They tell champion fitness instructors to take more exercise, and go on a calorie-controlled diet, because the proxy indicator (in this case, Body Mass Index) says they’re overweight.

We all know that what matters in our own lives are too complex to be summed up in a simple measure. But somehow we think such simplification is good enough for others.

The Attribution Problem

The measurement problem is significant, but it’s not the worst flaw that bedevils OBPM.  The most significant problem is that the premise which underpins OBPM – that outcomes can be attributed to particular programmes or organisations –  is simply wrong.

OBPM seeks to make organisations (or people, teams or programmes) accountable for outcomes, because they need to know whether those organisations have achieved the results that they are supposed to.  Should we recommission this organisation to deliver this intervention? OBPM says ‘look at the results they achieved’.

Payment by Results, which is the most extreme form of OBPM, takes this one-step further – it says that organisations should only be paid if they achieve certain results. To hold programmes or organisations accountable for producing results, you must be able to identify who has been responsible for producing which outcomes.

The trouble is, that’s impossible. Outcomes are not produced by organisations (or programmes, teams etc). Real-life outcomes are produced by a huge range of factors and interventions working together (in technical terms – outcomes are emergent properties of complex systems).

To understand this, all anyone needs to do is look at this systems map of obesity. It was produced by the Government Office for Science in 2007, and it is an amazing piece of work.

It shows all the factors which contribute to whether a person is obese, and how they interact. At a glance, we can all see that the outcome – whether someone is obese or not – is the result of a huge range of factors which are beyond the control (or influence) of any intervention, or even combination of interventions.

Screen Shot 2016-07-05 at 16.35.20

To hold organisations accountable for outcomes is to make them accountable for things they do not control. Even Mark Friedman acknowledges this, but his response is striking:

““Don’t accept lack of control as an excuse… If control were the overriding criteria for performance measures then there would be no performance measures at all. ” (Mark Friedman, Results-Based Accountability Implementation Guide)

Making organisations accountable for results which are beyond their control means that organisations learn instead to manage what they can control – the production of data. This is exactly what the evidence says happens , and why outcomes approaches often produce good-looking data. It’s because the organisations are concentrating on data production at the expense of actually doing the complex reality of the job that clients need.

And this is the most profound danger of all from OBPM – the data starts to tell policy makers things are improving when they’re not. If people are being made responsible for things they don’t control, and are being measured using indicators that they know don’t reflect people’s real lives, then they start to lie. This is the account provided by one staff member who has worked under OBPM:

“It [an outcomes orientated system] wouldn’t be honest. Because everybody manipulates the outcomes it just wouldn’t be honest, it wouldn’t be the same. You’d be frightened to speak because you might have put an outcome for a client that’s not a true reflection so how could you sit there in the team and say, “I feel like I’ve failed with this client,” when you’ve ticked a positive outcome.” (Lowe, Wilson and Boobis 2016)

It’s no coincidence that Outcome-Based Accountability talks about ”turning the curve” – OBPM is supposed to make the numbers look better, to make changes in the shape of the graph.  But the problem is, even when it achieves this, the numbers cannot be trusted because the numbers are not real. They’ve been produced by people whose job it is to produce good-looking data.

What’s the most frustrating thing about this? We’ve known that this is exactly what happens since 1976. This is the year that Donald Campbell, an American social scientist wrote his famous essay “Assessing the impact of planned social change”. In it he formulated Campbell’s Law:

“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

Northern Ireland can only hope against hope that, magically, it escapes the dismal path which is laid out before it. But those taking it down that path can have no excuse that they were not warned about what lies ahead.

, ,

  • Sharpie

    Thank you for bursting this balloon of group think that has landed in Government and so many others. The attribution issue is so big, so seismic and yet so denied. There are a fair few civil servants who run a mile from this stuff and I suppose it is why PWC et al have remained so popular in NI Government – they go linear and complicated but the pill of outcomes measurements has certainly been swallowed here. To be fair -the intent is fine, its just not thought through.

    Having people understand complexity, what it means, how it needs to be dealt with is the first essential step in coming up with effective solutions – of which there are many many many.

  • D99

    Question is: Who is accountable when it comes to the outcomes or results based accountability approach taken by the NI PfG?
    The answer might be: Everyone generally and no one specifically.

    If it’s possible to identify a set of actions that (if delivered or implemented appropriately) will lead to progress toward a specific outcome then:
    If the actions outlined in the PfG fail to achieve the desired outcomes, those who chose the actions may be responsible and accountable because they chose the wrong actions.
    But if the PfG takes a whole of government, cross-departmental approach and is co-designed with civic society and interested citizens through a prolonged process of consultation and engagement, does that not mean that everyone involved (generally) is responsible and no one (specifically) can be held accountable?

    If it’s not possible to identify a set of actions that (if delivered or implemented appropriately) will lead to progress toward a specific outcome, then we’re in even bigger trouble than I thought. Because then we would have to accept that the variables involved are too complex to understand and handle with any certainty and that most policy involves mainly trial and error.

    Either way, the one thing that can be measured, and the one thing that specific people can be responsible and accountable for is the appropriate implementation and delivery of the agreed actions. So, it’s probably best to choose what we think are the right actions, monitor the delivery of those actions to ensure their delivery, hold people to account if the actions are not delivered appropriately and change the actions if they don’t result in progress toward the desired outcome.

  • D99

    To put this a little more concisely: you monitor the actions to ensure they’re being implemented; and then, once you’re sure that’s the case, you measure the progress towards the outcomes to see if the actions are working.

    I’d be interested to hear your thoughts on this Toby.

  • Sharpie

    What is helpful here is to ensure that policy and expenditure is based on principles – if those are simple, agreed, concrete, and just about sufficient then any action can be measured against it to see if it is fitting. If all the actions across the breadth of the implementers are roughly coherent based on principles then you could safely assume each is working in the collective interest. Of course getting that level of buy-in and comprehension is the trick but should certainly be achievable for geographically based measures for instance on LGD poverty or mental health.

  • CB

    So what does Toby suggest as an alternative?

  • Toby Lowe

    We can start by asking a different question. Instead of ‘how do we make people/organisations accountable for producing X result?’ (which is impossible), we can ask ‘how can we help people to make better judgements in situations of uncertainty?’ And there’s good work been done on how this can be done, for example by creating ‘positive error cultures’ (organisational cultures in which it’s ok to say ‘i’ve made a mistake’, or ‘i don’t know what to do now’. And by promoting horizontal accountability – i.e. creating structures in which peers hold each other to account for the quality of their practice.

    We need to let go of the idea we can control outcomes – we can’t – it’s a fantasy. We need to focus on what we can improve, which is helping people to do the right job, well.

  • Toby Lowe

    the problem is, no one can know whether it is those actions that are producing outcomes, or something else entirely from another part of the system.

    I think we should all be engaged in the classic Plan-Do-Check-Act cycle (which is kind of what you’re saying, i think.) But the trick is to keep the plan (and therefore what counts as a ‘desired outcome’) dynamic, so that when the context changes (which it will), the plan, and the actions which follow, can change too.

    This means NOT using outcome-based contracts – which fix desired outcomes, along with the proxy measures used to monitor progress. Because these set particular goals and measures in stone – measures which become useless when the situation changes. (see the famous ‘Peterborough Social Impact Bond Pilot for a classic example of this).

  • D99

    I get that everything is relative, interconnected and complex; and that just because it rains doesn’t mean the ground is wet. A butterfly flapping its wings in the UK probably impacted on the Arab Spring…
    But let’s not use complexity to avoid accountability or excuse inaction.
    Intentional action can help produce an intended outcome.
    That’s not to say that other non-intentional actions and other environmental factors don’t influence the outcome too.
    Or to suggest that we can fully control the outcome.
    Or that actions will exclusively guarantee the outcome. This was the kind of intellectual dishonesty that allowed government to fool us with the payments by results model that is so open to gaming and manipulation, but so easy to sell through the notion that it would save public money if we only paid for results.

    But if actions can’t help achieve the desired impact, then policy making is irrelevant. From a philosophical point of view, what we do (action) to a large extent determines who we are (outcome); and from a policy point of view the actions we choose largely determine the kind of society we become, or at least would like to see.

Join us for the Slugger End of Year Review Show, Wed 14th Dec 2016
Get your tickets