Why the Critics of the ARINS•Irish Times Surveys are Wrong: A response to Professor Coulter

By Professor John Garry, Queen’s University Belfast. Professor Brendan O’Leary, University of Pennsylvania. Dr Jamie Pow, Queen’s University Belfast & Dr Dawn Walsh, University College Dublin.  

Professor Coulter has written a number of pieces (here and here) on Slugger O’Toole that are critical of the ARINS/Irish Times surveys. These criticisms are based on false premises and are wrong.

Comparison with other academic surveys

We will shortly address Professor Coulter’s unfounded methodological criticisms, but let us first confirm the substance of our findings. Data collection for the 2024 ARINS/Irish Times surveys took place at approximately the same time as another major academic survey, the 2024 Northern Ireland General Election Study led by the University of Liverpool. The two independent surveys found a strikingly similar percentage distribution of voting intentions in a hypothetical referendum on the constitutional question. See the table below.

	2024 ARINS• Irish Times	2024 NI General Election Study (University of Liverpool)
Stay in UK	48	49
United Ireland	34	34
Would not vote/refused	3	4
Don’t know	14	14

To examine possible changes over time, the Liverpool 2024 General Election Study can be compared to its predecessor: the 2019 General Election Study. In the 2019 study 52 percent of respondents said they would vote for Northern Ireland to stay in the UK, 29 percent said they would vote for a united Ireland, and 17 percent said they didn’t know if or how they would vote. Therefore, the Liverpool General Election Study finds a recent trend over time towards support for Irish unity: support for the Union was 23 points ahead of unity in 2019 but only 15 points ahead of unity in 2024. This trend is very similar to the trend picked up by the ARINS•Irish Times survey which found the Union ahead by 23 points in 2022 but only 14 points ahead in 2024.

In no sense, therefore, are the ARINS•Irish Times results outliers, certainly when compared to the Liverpool General Election Study. Professor Pete Shirlow, who has also made repeated public criticisms of the ARINS•Irish Times surveys in recent weeks, is part of the research team on that Liverpool project.

A trend in the same pro-Irish unity direction is picked up by another academic study: the Northern Ireland Life and Times Survey. In 2019 the Northern Ireland Life and Times Survey started asking the referendum voting intentions question on an annual basis. In that year support for the Union was at 51 percent and support for a united Ireland was at 25 percent. In the most recent (2023) survey, support for the Union was at 47 percent (a fall of 4 percentage points) and support for a united Ireland was at 35 percent (a rise of 10 points). So, the NILT researchers also find a strong trend in a pro-Irish unity direction: support for the Union was 26 points ahead in 2019 but only 12 points ahead in 2023.

In short, the ARINS•Irish Times findings are consistent with those of other independent academic surveys, including the broad trends in a pro-Irish unity direction identified to date. All survey results based on representative samples are necessarily estimates of public opinion, subject to a margin of error of +/-3 percentage points in these examples.

Looking below the surface, we can better understand the nature of these changes. Professor Coulter claims that “(t)he distribution of constitutional preferences (the proportion of Protestants wanting to stay in the Union, the percentage of Catholics wishing to leave etc.) in each year has … remained the same.” This claim, however, is simply not true. In the ARINS•Irish Times surveys the table below shows that in 2022, 55 percent of people from a Catholic background said they would vote for a united Ireland while in 2024 this proportion had risen to 63 percent. Note that the figures in brackets are the unweighted results (we will turn to the issue of weighting shortly).

	Catholic		Protestant		Other
	2022	2024	2022	2024	2022	2024
Stay in UK	21 (22)	18 (18)	79 (78)	82 (81)	35 (33)	29 (30)
United Ireland	55 (54)	63 (63)	4 (4)	7 (7)	20 (23)	30 (28)
Would not vote	3 (3)	4 (3)	5 (5)	2 (2)	14 (13)	8 (9)
Don’t know	21 (22)	15 (16)	13 (13)	9 (9)	31 (32)	32 (34)

It is also important that the share of Catholics responding “don’t know” declined between these two surveys, from 21 percent to 15 percent. Moreover, there has been a rise in support for a united Ireland and a slight decline in support for maintaining the Union among the smaller group of ‘Others’, but there is much more stability in voting intentions among people from a Protestant background.

In short, these changes within groups help explain the differences that we observe in the aggregate results between 2022 and 2024.

Sampling and weighting

The sampling procedure has been consistent across each of the three sets of ARINS•Irish Times survey data (2022, 2023 and 2024). In each year Ipsos NI, the independent survey company commissioned to collect the survey data in Northern Ireland, randomly selected 100 sampling points from across Northern Ireland and conducted at least ten face-to-face interviews at each location, with overall quotas based on geography, age, gender, and class.

As is standard practice in survey research, weights are applied to ensure that the sample mirrors known quantities of the target population. For the 2022 ARINS•Irish Times survey we adopted the following approach. At the time of planning the research in spring 2022, it was by no means clear that the complete 2021 Census results would be available at the time of data collection or analysis. Thus, for some demographic factors (geography, region, age, and class) evidence from the 2011 Census was used. But we did not use evidence on religion breakdown from the 2011 Census because of our concern that the distribution on this particularly relevant variable for political attitudes would be out of date by 2022. Instead, we ensured that the 2022 survey results were weighted to account for the political landscape of Northern Ireland, as manifested in the real-world Assembly election of 2022 which occurred only a short time before data collection.

Professor Coulter describes as “inexplicable” the fact that we did not use the 2021 Census breakdown on religion to weight the 2022 survey. But, in keeping with established considerations of research design, we had developed the above plan for analysing our data (including weighting) before any fieldwork took place and we implemented that plan.

When planning the subsequent rounds of the survey in 2023 and 2024 we did not use recalled vote choice in the 2022 Assembly election in our weightings, because of the risk that this particular variable would be less reliable with the passage of time, and because the complete 2021 Census results were by then available.

Instead in both the 2023 and 2024 surveys, the weighting factor accounted for five demographic variables: gender, age group, social class, geographical region, and religion/religion-brought-up-in. All targets were derived from the 2021 Census. For religion/religion-brought-up-in, this meant the following distribution: 45.7 percent Catholic, 43.5 percent Protestant, 1.5 percent other, and 9.3 percent none.

In the 2023 and 2024 ARINS•Irish Times surveys the weights applied to the results, reported in the Irish Times, were identical. We agree that it is reasonable to examine whether the 2022 results would have been any different if they had been weighted in the way the subsequent 2023 and 2024 sets of results were weighted. A completely identical weighting factor cannot be constructed for 2022 because a constituency variable was not included in the 2022 dataset, but we can construct a fresh weight based on the four remaining variables (gender, age group, social class, and religion/religion-brought-up-in), all of which are from the 2021 Census.

Referendum voting intention results from 2022 are reported in the table below, using both the original weighting factor and the alternative four-factor weighting variable. Professor Coulter claims our adjustment in weighting had “profound consequences”, but the table below shows that this claim is not true: the differences are between one quarter and one half of a percentage point.

	2022 % (original five- factor weighting)	2022 % (identical four-factor weighting)	Difference
Stay in UK	49.55	49.29	0.26
United Ireland	27.11	27.64	0.53
Would not vote	5.39	4.88	0.51
Don’t know	17.95	18.20	0.25

For completeness, in the tables below we replicate the 2023 and 2024 results for this question using the new four-factor weight based on gender, age, social class, and religion/religion-brought-up-in (i.e., excluding constituency).

	2023 % (original five- factor weighting)	2023 % (identical four-factor weighting)	Difference
Stay in UK	50.67	50.71	0.04
United Ireland	30.29	30.60	0.31
Would not vote	4.52	4.51	0.01
Don’t know	14.62	14.18	0.44

 

	2024 % (original five- factor weighting)	2024 % (identical four-factor weighting)	Difference
Stay in UK	47.80	47.90	0.10
United Ireland	34.37	34.15	0.22
Would not vote	3.45	3.49	0.04
Don’t know	14.39	14.46	0.07

The differences, as shown in the last column of each table, are miniscule.

When an identical four-factor demographic weight based on the 2021 Census is applied consistently to the 2022, 2023 and 2024 sets of results, the substantive findings of the surveys remain strikingly similar. We are therefore very confident that we can robustly report meaningful comparison across each of the time points.

To be clear, the results based on the updated weightings reflect the 2021 Census distributions on religion/religion-brought-up-in and the three other standard demographic variables.

Professor Coulter, however, performs his own analysis using the 2022 data, but appears to use the wrong variable in constructing his own weights. Specifically, he appears to measure religion by responses to the question “Which of the following best describes your community background?” rather than responses to the two questions on current religion and religion-brought-up-in that were identical to those used in the actual Census. The self-described community background and religion/religion-brought-up-in variables are of course very similar, but they are not the same, and it matters for the analysis.

Party support

On the composition of the 2024 sample itself, Professor Coulter suggests that there are more nationalist party supporters than unionist party supporters, and that this result is out of line with recent electoral results. Is he correct?

We released a brief description of the sample composition and methodology on the Royal Irish Academy website at the launch of the most recent ARINS/Irish Times North and South series. This description included a breakdown of the sample according to party support – as a measure of voting intention in an Assembly election held tomorrow. In our survey, we also asked respondents whether they voted in the 2022 Northern Ireland Assembly election and, if so, to which party they gave their first preference vote.

Among the 66 percent of respondents who reported voting in that election, the distribution of recalled vote choice is reported below for the 2024 survey, as well as that in last year’s survey, and they are then compared to the actual 2022 results, which can be seen on the second column in the table below.

	2022 Assembly election (FPV %)	2023 ARINS•Irish Times survey (%)	2024 ARINS•Irish Times survey (%)
Sinn Féin	29	30	28
DUP	21	20	21
Alliance	14	14	16
UUP	11	9	7
SDLP	9	8	9
TUV	8	5	3
Other party/ind.	8	7	6
Can’t recall	NA	6	9

The distribution of recalled vote choice in the 2024 survey is very similar to that observed in the actual 2022 results. Given the passage of time, however, there were some (9 percent) who reported voting but cannot recall to which party they gave their first preference. It is reasonable to suggest that many of these are TUV voters – the only category under-represented by more than 4 percentage points. Many 2022 TUV voters will have cast a first preference vote for the party for the first time, and most respondents in this “can’t recall” group (51 percent) are from a Protestant community background.

Of those who said they voted in 2022, the voting intentions in a future Assembly election are as follows:

Sinn Féin	26
DUP	13
Alliance	14
UUP	8
SDLP	9
TUV	6
Other party/ind.	6
Don’t know	14
Would not vote	4
Refused	1

Clearly support for the DUP appears to be much lower than in recent elections. But the relatively large “don’t know” category (14 percent) is also notable, a factor completely neglected by Professor Coulter.

Among 2022 voters of Sinn Féin, Alliance, the UUP, the SDLP and the TUV, between 74 percent and 95 percent indicated in 2024 that they intended to vote for that same party again. But, among 2022 DUP voters, only 56 percent affirmed that they will vote for the DUP in a hypothetical Assembly election tomorrow. The remainder indicated that they would vote for a different party (19 percent), that they don’t know (17 percent), or that they would not vote (8 percent).

We are therefore confident that we have a representative proportion of 2022 DUP voters in our sample. And the reason that future party support for the DUP is lower, and thus total unionist party support, is because these voters are less likely to commit to the DUP in a hypothetical election held tomorrow than are other party voters. There is likely no mystery here. The recently highly turbulent leadership turnover that the DUP has experienced is but one factor that has affected support for the party. Of course, many of these undecideds at the time of the survey may well vote for the party again in an actual election.

These measures of retrospective or prospective party support are not the only ways of evaluating the distribution of unionist and nationalist respondents in our sample. Indeed, given the propensity of some respondents to answer “can’t recall” or “don’t know” to such questions, it is important to consider other relevant variables to establish a more comprehensive understanding of the sample. Further questions in the ARINS•Irish Times survey asked respondents on a scale from 1-7 the extent to which they would be opposed (1) or in favour (7) of Northern Ireland staying in the United Kingdom. On the same scale they were also asked the extent to which they would be opposed or in favour of Northern Ireland leaving the United Kingdom to unify with the Republic of Ireland.

Among those who reported voting in the 2022 Assembly election, a significantly higher proportion (46 percent) scored Northern Ireland staying in the UK more favourably than the alternative scenario (38 percent). When 2022 Alliance voters are excluded, this basic ratio also holds (47 percent more pro-Union; 39 percent more pro-united Ireland). Similarly, excluding those who say they would not vote in a future Assembly election, 47 percent give a more favourable score to Northern Ireland staying in the United Kingdom than to Northern Ireland leaving to join a united Ireland, with 38 percent scoring in the opposite direction. Again, this basic ratio holds when prospective Alliance voters are excluded (49 percent more pro-Union; 39 percent more pro-united Ireland).

In short, a more comprehensive analysis of the sample than that performed by Professor Coulter dispels the notion that the ARINS•Irish Times surveys and reports are out of step with the actual electoral landscape.

Concluding observations

We would like to conclude by highlighting three key points.

The first is that high quality survey research necessarily depends on robust sampling, which we are fully confident Ipsos NI conducted, as well as standard weighting adjustments in analysis. Both aspects of research design require judgments. Anyone is welcome to scrutinise these judgments. We have explained in detail these key decisions which we hope any fair-minded observer will find reasonable.

Secondly, we completely agree that it is best practice in academic research to make data publicly available. That is why the 2022 and 2023 results have already been published on the ARINS website. The website itself was refreshed during the summer of 2024, and the relevant pages were temporarily unavailable during the update, but the links have since been fully restored. To ensure the data are as accessible as possible to researchers, the Royal Irish Academy has kindly created a new page where all relevant files can be found in a single location.

As stated at the launch of the 2024 results, the 2024 datasets for both Northern Ireland and the Republic of Ireland will be published on the ARINS website on 15 June 2025. We are additionally in the process of submitting the data and details to the Harvard Dataverse, one of the leading social science repositories of data, to make it easily and permanently accessible for international academics.

Thirdly, the goal of our research is emphatically to understand public attitudes better on aspects of an important debate, whatever those attitudes may be. The suggestion that we have constructed trends rather than found them is risible. We plan to conduct the 2025 survey in Northern Ireland using precisely the same sampling procedures used in the past three years. The results will be weighted using exactly the same 2021 Census benchmarks that have been available and applied in the last two years, and the data will ultimately be available for any researcher to scrutinise.

To sum up, Professor Coulter’s criticisms are based on false premises and are simply empirically wrong, as we have sought to explain above. The ARINS•Irish Times findings are fully consistent with other independent academic analyses: the University of Liverpool’s General Election studies, and the Northern Ireland Life and Times Surveys. Our sampling and weighting procedures are appropriate, and robust, and our distributions on party support reflect the actual political landscape.

If, from Professor Coulter’s perspective, or from any other academic’s perspective, there are any remaining ambiguities or issues to be clarified, we would welcome being contacted directly on our university email addresses: a normal professional academic courtesy that Professor Coulter did not follow before composing his articles. We are happy to meet, in person or online, to discuss all matters methodological. Many academic journals are interested in methodological issues, and debates between academics occur frequently in such journals. Professor Coulter could consider submitting an academic critique subject to anonymous peer reviewing in such a journal.

Either of these suggestions would have been preferable to Professor Coulter’s decision inaccurately, and inappropriately to attack the integrity of the ARINS research team in blogposts on Slugger O’Toole. Shooting the messenger is all too common in politics; it should not be a feature of social science communications, especially on Northern Ireland.