Ask It Backwards: The hidden framing in Northern Ireland’s unity polls

Patrick Harkness asks what if we reverse the NILT question on the constitutional question and keep the format identical? He explores how question framing sets the political weather in Northern Ireland, and asks whether should we level the field?

Opinion polling sets the weather. That’s why there has long been an interest in making things poll better than they really should. There is a famous sketchin ‘Yes Prime Minister’ which explains how a series of leading questions can be used to do this. The point is so well-made that pollsters like IPSOS use it as a teaching tool.

A slightly different approach is called framing. This technique can generate headlines like “Poll finds more than 60% would vote for united Ireland” in last month’s Irish News. To their credit, that team were more than transparent about their approach. They went so far as to point out that deliberate framing of the question was driving the unexpected result. Nobody can have any complaints about that.

However, we would like to think that the framing is strictly neutral in ‘gold standard’ polling series, such as the Northern Ireland Life and Times Survey (NILTS). These respected polling series make the weather for everyone.

The question NILTS asks is:

“Suppose there was a referendum tomorrow on the future of Northern Ireland and you were being asked to vote on whether Northern Ireland should unify with the Republic of Ireland. Would you vote ‘yes’ to unify with the Republic or ‘no’?”

Seems fair enough? And with recent results showing only 42% backing the United Kingdom compared to 36% going for the Republic of Ireland, it seems that nationalism has fair grounds for optimism.

But, just to make sure that there are no framing effects going on, let’s do an experiment. Let’s ask the question the other way around.

Let’s give the UK the ‘yes’ proposition, instead of the ‘no’. Let’s explicitly spell out the option for remaining part of the UK, and fail to spell out the alternative. Let’s even repeat our chosen framing, by only asking whether we should ‘continue to exist as part the UK’, rather than suggesting the alternative: that of ‘unifying with the Republic’. In fact, let’s not mention the Republic at all.

If you felt that the NILTS question was a fair proposition, then you have to agree that these changes would be equally fair. This is an equal-and-opposite switch. Nothing more.

Our inverted question now becomes:

“Suppose there was a referendum tomorrow on the future of Northern Ireland and you were being asked to vote on whether Northern Ireland should continue to exist as part of the United Kingdom. Would you vote ‘yes’ to continue to exist as part of the United Kingdom, or ‘no’?”

When this inverted formulation is polled by the Institute of Irish Studies, support for remaining in the UK jumps to more than 61%. The change proposition slides to just 28%.

This is hugely significant. Unionism’s lead of just 6% in the NILTS can be increased to 33% – that is, it can be multiplied by a factor of more than five – simply by reversing the assumptions implicit within the question being asked. This suggests that there is much more framing baked into the NILTS approach than we might perhaps have expected.

Of course, this does not mean that the inverted question is any fairer. In truth, both formulations exploit the same biases. The power of repeated suggestion, the omission of alternatives, and the positivity of the word ‘yes’ all drive up support for whatever proposition might be doing the asking. These biases, like so many others, only become clear when you reverse them.

For the NILTS, there is an easy fix that would cancel out most of the bias while preserving the value of the historical dataset: simply start polling both formulations and take the average. But for the country at large, this will not work. Nationalism, which is doing the asking in Northern Ireland, must propose a question that unionism can accept as fair. Unionism, meanwhile, should apply a simple acceptance test: ask the question backwards, and see if you can get the same result. That levels the field.

That would be fair as well.

Patrick is an Engineer who works at the James Watt School of Engineering, University of Glasgow as Professor of Exploration Technology. He’s writing here in his own personal capacity. He is member of the Ulster Unionist Party.