General election seat forecasts are no longer useful

For much of the last century, general elections in Great Britain have been largely predictable affairs. Aside from the odd Liberal, SNP and Plaid Cymru MP returned here and there, Westminster elections were a battle between Labour and the Conservatives.

The relationship between the opinion polls and election results was straightforward; most constituencies were either safe Labour or Tory seats, and it was well established that a few points gained or lost in the opinion polls could predict the winner in bellwether constituencies such as Harlow or Dartford, and therefore the overall election.

No longer. The chaos that has descended on politics in Britain since the referendum on EU membership in 2016 means that the straightforward relationship between national opinion polls and the election result has been broken. Polls such as this from YouGov, which had the resurgent Liberal Democrats topping the popular vote with 24%, the newly formed Brexit Party on second with 22%, with Labour and the Conservatives tied on third on 19%, mean that seats that have reliably voted Labour or Conservative for generations are now highly competitive.

In a first-past-the-post election, there are three categories of votes. There are wasted votes, or votes for a candidate that fail to win a seat. Then there are useful votes, the votes that got a candidate elected (put another way, the number of votes received by the candidate that came second, plus one). Then there are votes that are not needed, votes for a winning candidate that increases the size of his or her majority. For example, if Candidate X received 20,000 votes and Candidate Y received 30,001 votes, then Candidate X received 20,000 wasted votes, Candidate Y received 20,001 useful votes and 10,000 votes that weren’t needed.

A party will want to maximise the proportion of votes that are “useful”, in order to maximise the number of MPs won from the national popular vote received. The SNP are efficient at converting popular votes into MPs, whilst Ukip in 2015 would be an example of an inefficient vote, as they received 3.9m votes but returned just one MP.

The chart at the top of the post shows how the vote was broken down for the Conservatives in 2017. Of the votes received, 38% (dark blue) were “useful” votes which were enough to get their winning MPs a majority of one, 34.5% (light blue) of their votes were “wasted” in seats that they lost, and 27.5% (grey) of their votes were unnecessary votes that increased the size of their majority in seats that they won. The black lines show how far away the Conservatives were from winning in the seats that they lost.

The chart shows that the 2017 general election was, in England and Wales, for the most part a traditional two-party election. The Tories had a substantial number of safe seats (there were 187 where their majority was 20% or higher), and their majority was lost because they lost by narrow margins in traditional bellwether constituencies such as Ipswich and Stroud.

The chart below shows Labour’s distribution of votes at the 2017 election. This shows a similar pattern to the Conservatives (obviously in reverse). It is worth noting that Labour have more seats with absurdly large majorities than the Tories, such as Liverpool, Walton (majority: 77.1%). Their vote is less efficient than the Conservatives; 30.7% votes were “useful”, compared with the Tories’ 38%. The flipside of this is that their vote is more resilient; they are more likely to hold on to their seats in the context of an overall nationwide collapse in their vote. This might turn out to be quite useful.

The chart below shows how the Liberal Democrats performed in 2017. Even though their share of the popular vote has plummeted from before they went into coalition with the Conservatives in 2010, their vote is still mostly wasted in seats where they didn’t win.

By contrast, a majority (54%) of the SNP vote was actually useful in terms of getting an MP elected. There were no seats where they ran up a supermajority, and only in two constituencies (Berwickshire, Roxburgh and Selkirk; Edinburgh South) did they lose by more than 20%.

Recent opinion polls suggest that forecasting the number of seats that each party will receive is much more difficult than it used to be. For a demonstration of this, I took the YouGov poll and adjusted the 2017 results according to the movement in voting intention from the 2017 election. For example, the poll says that 38% of Tory voters intend to vote for the Brexit Party, 45% of Tory voters intend to vote for the Conservatives again, and 11% intend to vote Lib Dem. The results, which obviously shouldn’t be taken very seriously, are as follows:

  • Brexit Party: 270
  • Labour: 188
  • Liberal Democrats: 111
  • SNP: 55
  • Plaid Cymru: 5
  • Greens, Speaker, Independent: 1 each
  • Conservatives: 0 (zero)

Of course, trying to forecast MP totals in such an environment is a fool’s errand. Take, for instance, Labour’s projected vote total using this methodology, as shown in the chart below.

If Labour really do poll 18% nationally then, like almost everyone else, they will face an environment where they have no safe seats. However their existing supermajorities mean that they are quite resilient to the prospect of being wiped out. But their winning majorities would be very narrow. For the 188 seats where they are projected to win, their average winning majority would only be 3%. Clearly, if their actual vote received dips much below 18%, they are in serious trouble. But if they really did win 188 seats, and the Liberal Democrats won over 100, then it would be likely that Labour would lead the next government. The margins between victory and annihilation could be miniscule.

The Conservatives’ problems are greater still. They have two problems that Labour don’t; the fact that most of the support for the Brexit Party are former Tory voters, and the fact that a relative lack of ultra-safe seats means that their vote lacks resilience. The Tory projection can be shown in the chart below.

Whilst this projection would give the Conservatives no seats, there would be 368 (enough for a comfortable majority) where they would be 10% or less away from winning the seat. Essentially, the Conservatives could end up with a comfortable majority, be entirely wiped out, or somewhere in between. This obviously isn’t a useful prediction! It’s like saying “eyewitnesses report that the snake was between 1 millimeter and 1 kilometer long.”

The prognosis for the Lib Dems is equally uncertain. The poll put the Lib Dems in the lead in terms of the popular vote, but a lack of constituencies where they came close in 2017 could well mean that they end up with fewer seats than Labour even if they win the popular vote. But they mightn’t be far off the lead in many places. There are 558(!) constituencies where they would be forecast to win or be within 10% of winning.

Finally, current polling suggests that the Brexit Party could inherit a large amount of formerly safe Tory seats. Again, there is no certainty at all about how they might perform. There are 432 seats where they would be projected to either win or come within 10 points of victory. Also, they have no safe seats anywhere.

It could well be the case that current polling shouldn’t be taken too seriously. Polling around the last European election used to show Ukip with high shares of the vote, and the Lib Dems had polling leads in the lead up to the 2010 election, and polling reverted back to more “normal” levels soon afterwards.

However, support for traditional parties of the centre left and centre right has crumbled across Europe, and there is no reason that similar things can’t happen in Britain. Things that are seen as “impossible”, such as Labour dying out in Scotland or Alliance winning a seat in the European Parliament, can and do happen.

Conveniently, there will be a by-election in Peterborough that could serve as an indicator as to whether the mooted realignment in British politics is real. The simple model outlined above predicts that the Brexit Party will win the seat with 25%, with the Lib Dems second on 24%, Labour on 23% and the Tories on 21%. Should all four parties be within touching distance then it could serve as an indicator that the next general election could be an unpredictable free-for-all.

The first-past-the-post system has been rightly criticised for the fact that voters in safe seats are effectively disenfranchised. However, the nature of holding multi-party elections under such a system means that every vote would count, and the difference between, say, a Liberal Democrat majority government and a Brexit Party majority could come down to a small number of votes in a number of seats.

Predicting elections is hard. Models such as Nate Silver’s model at FiveThirtyEight (usually) work for US Presidential elections because there is a simple two-party system, and the outcome is determined by the winner in a small number of intensively-polled swing states. Even if there was a glut of constituency-level polling it would be an all but impossible task to accurately forecast the outcome in individual seats if the next election goes along the lines of what current opinion polling suggests.

In summary, nobody knows what will happen at the next general election. But if the next Conservative Party leader has any sense, he or she will be willing to do almost anything to avoid one any time soon.