We need to talk about the ARINS survey on Ireland’s constitutional futures (Part One)…

Colin Coulter is Professor of Sociology in Maynooth University. Here he asks further searching questions about the “gold standard” ARINS survey, and what happens to the outcome if you adjust for weighting issues in the original sample.

The latest instalment of the ARINS/Irish Times survey points to a sharp increase in support among the Northern Irish public for the project of reunification. In the space of just two years, there has, apparently, been a net shift toward the nationalist cause of some 9%, with almost all of that, 7%, taking place in the last twelve months alone (see Table 1, below).

Table 1: Headline Data, ARINS/Irish Times Survey 2022-2024

Those findings were, predictably, cause for celebration among proponents of a border poll. Among the first to comment was the pressure group Ireland’s Future who observed that the patterns revealed the ARINS project mean that supporters of a united Ireland will outnumber supporters of the United Kingdom as soon as 2028. And that sense of optimism would be heightened further, of course, when the latest LucidTalk poll also disclosed a rise in public support for a united Ireland, albeit on a much smaller scale.

The euphoria that greeted the latest instalment of the ARINS/Irish Times survey was entirely understandable. At a time when those making the case for constitutional change appeared to be flagging, this was, after all, the first poll for quite some time that at least appears to indicate the tide of popular opinion is turning in their favour. As is so often the case in Northern Ireland, however, appearances can be deceptive.

A snapshot of Northern Irish society…in 2008

There is at the heart of the ARINS/Irish Times survey a fundamental design flaw. As I pointed out in a previous article for Slugger O’Toole, the editions of the opinion poll that have emerged to date are all based on very different profiles of the Northern Irish public. The scale of those differences can be seen quite clearly in the table below.

Table 2: Weightings by ‘community background’, ARINS/Irish Times Survey, 2022-24

In any survey series, it is, needless to say, especially critical that the sample population in the very first edition is weighted in a way that reflects closely the wider body of people it is intended to represent. When we take a closer look at the 2022 instalment of the ARINS/Irish Times poll, however, we discover that the weightings deployed are clearly askew.

Noticing that this was the case, I approached the contact person designated on the survey website who informed me that the profile of respondents according to ‘community background’ that year was actually based on the 2011 Census. It now transpires that I was misinformed.

In a response to a recent letter to the Irish Times that remains, naturally, unpublished, members of the ARINS teams clarified that the weightings for religion in the 2022 survey had, in fact, a different basis to the one I had been told. Although the data from the 2021 Census were already available, those at the helm of the project decided to base their inaugural poll on the results of the 2022 Assembly elections instead. That is a genuinely inexplicable decision, but one that would inevitably have quite profound consequences.

In particular, it meant that the inaugural edition of the ARINS/Irish Times survey was based not on the one data source (the Census) that provides a virtually complete profile of Northern Ireland’s population but rather on another (electoral trends) that excludes a large minority of people in the region, namely the two-fifths who routinely feel disinclined to vote.

In addition, the decision to overlook the latest Census data and turn to the latest election results instead ensured that the baseline poll enlisted one key variable (electoral preference) to stand in for another key variable (religious affiliation) to which it bears a strong resemblance, but with which it is far from synonymous.

Those critical errors of design would be compounded by critical errors of detail documented in the table below. While it has been claimed that religious weightings in the 2022 ARINS/Irish Times poll were meant to reflect voting patterns in the Assembly elections that year, the numbers simply don’t match up (see Table 3 below).

Most importantly, perhaps, there are 5% more Protestants in the sample than there should be if one were accurately weighting ‘community background’ according to prevailing electoral trends. And that overstatement could only have produced one outcome, to which we will return shortly.

Table 3: Discrepancies between 2022 ARINS/Irish Times Survey ‘Community Background’

 Weighting and 2022 Assembly Election Results

Regardless of where the error originated, the fact remains that the first edition of the ARINS/Irish Times survey clearly overestimates the size of the Protestant community. Data from the 2022 instalment were released shortly after the latest Census results revealed that those who identify, or were raised, as Catholics outnumber those who identify, or were raised, as Protestants by a margin of 2.2%. And yet the inaugural version of the ARINS/Irish Times poll requires us to believe that Northern Ireland remains a place where Protestants are still the largest community.

In the 2022 edition of the survey, it is worth repeating, the sample population includes 5% more Protestants than Catholics. The last time Northern Ireland looked like that was some time before the 2011 Census. It’s hard to be precise here, but 2008 seems as plausible a year to choose as any.

Third time lucky

The inaugural edition of the ARINS/Irish Times survey was, therefore, based upon a cultural profile of Northern Irish society that was already more than a decade – perhaps even a decade and a half – out of date. That critical, entirely needless, error would inevitably contaminate the entire series. In particular, it ensured that if subsequent editions were to introduce more accurate profiles of Northern Ireland’s population, that would give the impression of opinions shifting even if that were not, in fact, the case. And that is precisely what has happened.

The inaccurate weighting of ‘community background’ that was such an anomalous feature of the first version ARINS/Irish Times survey would be resolved in the twin editions that have appeared since (see Table 2 above). While the 2023 poll substantially increased the number of Catholics in the sample population, the 2024 edition sifted out a large body of Protestants.

Those twin adjustments have ensured that the ‘community background’ of respondents in the latest survey now accords with the data from the 2021 Census. It is unclear why it has taken three attempts to arrive at that outcome. But it seems safe to say that the balance between the communal blocs that features in the current ARINS/Irish Times survey now mirrors much more closely the composition of Northern Irish society as a whole.

The recalibration of the ‘community background’ variable required to correct the original weighting errors was, of course, a very substantial one. In the space of just two years, those who design the ARINS/Irish Times survey were required to reduce the proportion of Protestant respondents by 3.5% and increase the proportion of Catholic respondents by 3.7%, a net shift of some 7.2%.

Let’s place that recalculation in its appropriate context. In the decade between the last two instalments of the Northern Ireland Census, the balance between the Protestant and Catholic communities tilted 5.5% in favour of the latter. In effect, then, the academics behind the ARINS project have compressed rather more demographic change into the last two years than actually occurred in the previous ten.

The consequences of that compression are not hard to predict. A net shift of 7.2% in the balance between the ‘two communities’ was always going to be reflected in the constitutional aspirations stated by respondents. More specifically, a reweighting of the population sample on that scale, and in that direction, could only have resulted in a sharp increase in support for a united Ireland. And that would have been true even if nothing else had actually changed.

It seems plausible, therefore, to conclude that those steering the ARINS project are, in fact, producers of much of the change they claim merely to describe. In noting that the dramatic reweighting of the ‘community background’ variable will have had an impact, we are not, however, suggesting that all of the shift in public opinion documented in the series can be attributed purely to that recalibration. There may well be ‘real’ changes in the hearts and minds of respondents happening at the same time as well.

Sifting out the ‘real’ change

We need, then, to work out how much of the net increase in support for a united Ireland is an artefact of design flaws and changes in the survey itself and how much is an expression of a genuine turn in public opinion. The best way to do that would be to be to conduct some reweighting procedures on the original data. Unfortunately, however, those are unavailable to us.

While the data sets for 2022 and 2023 were initially posted on the ARINS website, they were subsequently withdrawn, for reasons yet to be explained. That is at odds with the most elementary academic protocols concerning the presentation and dissemination of data. Such poor practice hardly inspires confidence and means we will have to improvise here.

A straightforward way of filtering out the changes that have arisen specifically out of the research design is to standardise the weighting of the religious identity variable across all three years of the survey. The table below summarises what happens when you do just that. Given that the 2024 edition is the one with an accurate profile of ‘community background’, we have simply applied its weightings to the previous two years.

The distribution of constitutional preferences (the proportion of Protestants wanting to stay in the Union, the percentage of Catholics wishing to leave etc.) in each year has, however, remained the same. That specific recalibration allows us to observe what would have happened to the survey outcomes had the appropriate weightings for the three communal blocs been applied from the outset.

Table 4: All editions of ARINS/Irish Times Survey, Standardised With 2024 Weightings

Once the distortions that stem from inconsistent weighting are removed, we encounter a rather different trajectory to the one mapped out in the pages of the Irish Times. The critical difference, predictably, is to be found in the first year of the survey. With the number of Protestant respondents now reduced to an appropriate level, the numerical advantage enjoyed by those who wish to remain in the UK falls accordingly, from 23% to 18%.

And that different starting point means, of course, that the shift in opinion that was so stark in the original survey becomes rather less so in this rendition. As we noted at the outset, the 2024 ARINS/Irish Times poll indicated that the gap between support for the Union and support for Irish unification had fallen by 9% in just two years. Once we apply the appropriate weightings across the board, however, that figure shrinks to 4%.

The difference between those percentages reveals something really quite critical that will not be apparent in the headlines greeting the latest ARINS/Irish Times survey. Once we filter out the anomalies in the sample population that featured in previous editions, most of the net shift towards the reunification project that grabbed so much attention simply evaporates.

That recalculation discloses that most of the apparent change documented in the series, 5% to be exact, is a product not of a real tilt in the public mood but rather of the methodological alterations that were introduced along the way. And that does rather bear out the suspicion that the ARINS project is, at times, the author of certain trends and not simply their narrator.