Opinion polling on Scottish independence and EU membership referendums

Pollsters have had a good independence referendum.  However, improved explanations about why voters have swung to ‘Yes’ would be helpful.  This blog discusses public attitude change and how it might be better explained during a future EU referendum campaign.

Opinion pollsters have added insight and direction to an engaging campaign.  Without the polls we would be unaware just how close the contest had become, lost in a fog on punditry.  We would be subject to endless claims and counter-claims from each lobby about who was really speaking for the people.  Polling keeps politicians honest, telling us about the evolving credibility of each side’s message.  Interviews and focus groups with voters may have their own value, but they leave us little wiser about the overall outcome of the event.  By contrast, polling forecasts have moved financial markets, changed campaign strategies, perhaps even caused the No lobby to concede ‘Devo Max’ to the Scots.  Given the increasing demand for polling predictions, and the internet technologies that make the supply of polls cheaper, we should expect increased polling activity if and when an EU Referendum is called.  But the product itself can be improved.      

Whilst the Scottish experience has been great news for the pollsters, it has also given them a couple of headaches.  Firstly, it may go horribly wrong by Friday morning.  Any celebrations of their impact could prove premature if the actual result reveals an industry-wide miscalculation.  Given that the independence ballot is a one-off, methods cannot be tweaked from previous experiences to correct for non-random (systemic) error.  Polling companies are therefore less sure than usual about the accuracy of their surveys, particularly those using online panels that rely upon heavy weighting to the initial samples.  The +/- 3 point ‘margin of random sampling error’ may not tell the full story of polling reliability.  To the pollsters’ credit, they have been both honest and humble about their methods, taking every opportunity to raise their own self-doubts about their figures.  An excellent discussion of referendum polling accuracy can be heard on the BBC Radio 4 More or Less programme, here, and the matter is well reviewed in a blog by Stephen Fisher here and Anthony Wells here.  The general feeling seems to be that, if anything, pollsters are more likely to be overestimating rather than underestimating actual support for Yes.  Even if they get it wrong this time, they deserve credit for sticking their stake in the ground, being open about their methodology so it can be improved upon in future.

Assuming the pollsters have told us correctly what will happen, to be confirmed tomorrow, they are less impressive at telling us why it is happening.  Ultimately, what really matters is change in public attitudes because it’s this change which could split the United Kingdom apart.  Understanding why support has changed requires a fuller range of survey indicators than the simple and less reliable self-reports by voters of their own motivations – the questions which are asked currently.  The public need to be probed in more detail about their national identities, partisanship, economic evaluations and levels of political trust that are all key causal variables.  Also such indicators need to be analysed over time rather than as one off measurements, to pick up on trends.  Very little of this over time analysis has been conducted on attitudes towards independence, in part through lack of data.  This may explain why it took a shock poll from YouGov, showing Yes in front for the first time, to jolt public interest and the No side into serious action, less than two weeks prior to the ballot.  Whilst traditional polling based on individual snapshots of opinion did not see a close race coming, it was in fact predicted by researchers at Southampton University, using a rare (Bayesian) model of vote intention based on trends and probabilities.  Better partnerships between public facing polling companies and academics versed in time-series methodologies could improve the offering and its overall impact.

Understanding underlying trends in support is crucial to framing the right campaigning strategies on both sides, shedding light on what is working and what is not.  Misunderstanding of why support has moved, could explain the poor performance of the No campaign.  In a matter of weeks, opinion is now split close to 50/50 between Yes and No, having been 60/40 against, a shift described by Anthony Wells of YouGov as a ‘real and sustained large change’.  Polls may be overestimating or underestimating support at any one time, but on the question of over time trends they are unequivocal.  As the ‘poll of polls’ below shows, this growth in support for independence has actually been going on for months rather than weeks:

Poll of Polls – 12/09/14  (ScotCen SocialResearch)

Pollsters and political scientists who specialise in public opinion tend to be conservative about shifts in attitudes, particularly short-term ones.  As the Polling Observatory make clear, these can often be nothing more than random noise, artefacts of measurement instruments rather than a significant signal of attitude change.  However, long-term change as evident in Scotland cannot be so easily dismissed. 

The question of explaining attitude change is also a theoretical one about how voters form their opinions, extending beyond practical questions of data and polling methods.  The problem here is a lack of clear underlying theory why individual voters have actually shifted their attitudes.  Within some quarters of academia, there is outright denial that voters possess the capacity to change their views in response to campaign information, at least in a meaningful way.  Dr Rob Johns, writing on the British Election Study website, warned in July – just before the No campaign started to implode: ‘do not expect major shifts in opinion in the run-up to 18 September and that ‘the millions to be spent on persuading voters between now and polling day will be largely wasted’.

What appears in hindsight as irresponsible advice, reflects a gloomy appraisal of the voter.  Rather than behaving in a largely economic (or ‘rational’) fashion by consciously weighing up the costs and benefits of Yes and No, Johns argues voters develop their preferences in an irrational or psychological manner, working against change.  Preferences are deeply rooted in ‘early upbringing and personality’.  This idea dates back to the seminal ‘Michigan School’ studies of the American Voter in the 1950s, an account of electoral behaviour that stressed the role of ‘enduring partisan commitments in shaping attitudes’, formed early in life and remaining stable through adulthood (Campbell et al. 1960).   Despite concerted attempts in voter studies to revise the Michigan School approach with more dynamic ‘rational choice’ type models, the older approach has never really gone away.  Indeed national identity attachment is ingrained in academic literature as the dominant variable for explaining support for the EU (Hooghe and Marks, 2009).  The question is, when significant opinion change does happen, as it did during the Eurozone crisis in respect of attitudes towards EU membership, and now again during the Scottish Referendum campaign, is the Michigan School theory up to the task of explaining it?

Under the Michigan model, when the voter is confronted by new information, he is likely to become polarised in his existing opinions because he assimilates news through the prism of his long-standing biases: ‘the individual is more a rationaliser (of his existing prejudices) than a rational decision maker’ (Lodge and Taber, 2013:26).  For example, in the Scottish case, the voter who watched the televised referendum debates, if a nationalist, would tend to dismiss Alistair Darling’s arguments whilst agreeing with Alex Salmond, without judging the competing arguments on their merits.  The unionist would behave similarly, blindly agreeing with Darling and rejecting the words of Salmond.  Hence, not much change in the polls would result.  If one side enjoys a short-term bounce, it is likely to be superficial, dissipating as voters powerful long-term attachments kick back in. 

There is another specific reason why this psychological, non-rational model theory of the voter is popular in explaining opinion on the Scottish referendum.  Independence is an issue akin to the question of EU membership, long considered difficult for the public to appreciate in a thoughtful way.  Academics like to argue that because the economic and constitutional questions of EU membership (or Scottish independence) are so complex, an essentially lazy and ill-informed public cannot engage with them.  Instead, EU membership or Independence stirs passions that are not reconcilable with rational discourse – so the argument goes.  These passions are based on national identities and populist distrust of the governing elite, which combine in a heady cocktail of emotion.  Whenever we hear the media framing the debate as ‘hearts versus heads’ this line of thinking is in play.  The assumption is the public cannot reconcile competing (affective-cognitive) dimensions in their support, meaning their evolving attitudes are unstable, and usually constructed externally of them.  In the simplest terms, rather than the voter consciously and reasonably weighing up different aspects of the debate and making informed choices himself, the different dimensions in his attitudes become mobilised by the media and politicians instead.  And this becomes the explanation for changing preferences to Yes – the Yes campaign could effectively mobilise voters key underlying attachments against Westminster, together with Scottish nationalism.  The No campaign was always likely to struggle to mobilise Scottish voter’s economic calculus against separation by highlighting its risks.  If the Yes campaign wins, it will be said that hearts have triumphed over minds.  The result was somewhat inevitable and No’s big mistake was allowing the referendum to happen in the first place.

There is an alternative explanation of why attitudes have changed in Scotland which gives more credit to the voter.  It does not deny that Scottish voters’ beliefs are strongly influenced by their national identity and that many have a passionate distrust of the British political establishment, nor does it take a view on the worthiness of these attitudes.  It simply maintains that these emotional aspects of their support are more closely related to their economic evaluations than the psychological theory of voting insists.  Rather than voters’ support for independence being structured along two unrelated dimensions, attitudes are simply a mix of the voters own prior beliefs and new evaluations.  Voters update their prior beliefs as a ‘running-tally’ of their new evaluations of information, changing these prior beliefs when they consider the new information to be sufficiently valuable to affect their interests – and ignoring it if they consider it invalid (Fiorina, 1981).  The voter isn’t so conditioned by his background and psychological processes, rather he has subjective views of his own and is capable of making choices.  The problem with the No campaign wasn’t necessarily the dry emphasis on economic and political costs and benefits, but that the public either didn’t receive sufficient information early enough; they judged the message too disorganised and confused; or discounted the information because they distrusted the source.  If No loses, it was not inevitable, their arguments weren’t necessarily wrong but their presentation was.  In particular it needed to come from more trusted sources than the declining Westminster mainstream elite – perhaps the Scottish voters themselves via a more grass-roots orientated campaign.  (The importance of grass-roots campaigning and the weakness of mainstream political parties in influencing their voters is a major lesson for EU referendum campaigners.)

The running-tally explanation of changing attitudes demands that voter attitudes are modelled over time, to reflect the probability calculation that voters make between the value of their previous beliefs and the worth of new information.  It requires long-running time-series data with the same questions asked in each poll, such as by the monthly British Election Study Continuous Monitoring Survey.  New data, new methods and revisionist theories about the capabilities of voters to make informed choices could all contribute to make polling on a future EU referendum even more impactful than the polling on the Scottish independence referendum.

References:

Campbell, A, Converse, P, Miller, W and Stokes, D.  (1960).  The American Voter.  University of Chicago Press.

Fiorina, M. (1981).  Retrospective Voting in American National Elections. New Haven: Yale University Press.

Hooghe, L and Marks, G. (2009).  A Postfunctionalist Theory of European Integration: From Permissive Consensus to Constraining Dissensus.  British Journal of Political Science, Volume 39, Issue 1, pp 1-23.

Lodge, M and Taber, C. (2013). The Rationalizing Voter.  New York: Cambridge University Press.
My planned PhD thesis is summarised here.  It uses the ‘running-tally’ theory and time-series methods to explain why public attitudes have moved so sharply both for and against EU membership in the last ten years, in response to record EU immigration and financial crisis. 

A note of caution about political betting markets being our guide to British politics

In Mike Smithson's recent post, he draws conclusions on the relative public interest levels in different political events, derived from betting market volumes published on Betfair.  Being a former trader of political markets at the Sporting Index spread betting firm (albeit a very unsuccessful one) and an ex-employee of other betting firms for many years, I've set out below a couple of concerns about the noisy claims about British politics that flow from analysing 'punting activity':

1.  As with any betting market, a 'Pareto rule' applies.  A very high percentage of turnover is likely to be the responsibility of a handful of customers.  For this reason, using turnover as an indicator of wider public salience is fraught with danger.

The Pareto effect varies between gambling products.  In an online casino as much as 95 per cent of the turnover may be generated by just 5 per cent of the customer base (a fact that tells you much about the appeal to hardened gamblers of online roulette and blackjack - and why low staking limits on fixed odds betting terminals are so heavily resisted by the operators).  In the slightly more egalitarian world of sports betting there might be an 85-15 proportion; maybe with political betting the ratio is more equal still.  Betting companies see quite a few 'political geeks' open accounts solely to bet on politics, more out of that passion than a wider interest in gambling.  They are generally low-staking, risk-adverse types who enjoy picking off the big firms in esoteric events like by-elections.
Consequently, political markets see a majority of small wagers - but the important point is that the overall turnover-level (as in nearly all betting markets) is still largely determined by a very small proportion of high-stakers. Turnover is therefore often an unreliable measure of mass-interest in the political event.  

These dominant high-end players are more interested in the attractiveness of the betting proposition than a fascination with the political event itself, although they may share that too.  This is probably why the Scottish Independence Referendum is a relatively high-turnover contest (relative to other political betting heats - but not betting markets generally which I come to below).  It is a two-horse race with a clear (but drifting) long odds-on favourite (No) for chunky players to get 'stuck into'.  There is anecdotal evidence of big odds-on punters refusing to countenance the wild and shocking proposition that Scotland will vote to become a separate country and are lured by the 20-30 per cent return on their stake that it won't happen.  How much this is irrational 'heart not head' betting is the truly interesting aspect of the market.  I remember during the 1997 General Election a similar phenomenon, when I was involved in managing that market for Sporting Index.  The (then) largely Tory supporting stock-broking client base of the city spread betting firms struggled to conceive of a 100+ Blair majority.  It caused arbitrage situations between the ultimately victorious IG Index layers - who took a bold and vindicated position with Labour (their market-maker was Patrick Jay, grandson of Labour premier James Callaghan), and the rest of the industry who moved their prices largely in response to customer demand - diverging all the time from the opinion poll predictions.

By comparison to the Scottish Referendum, the Euro Election for serious gamblers is an all too trappy three-horse affair, fraught with uncertainty and a lack of reliable form.  If it were a horse race, it would be one of those muddling early season handicaps on heavy ground, with all the runners 'first time out'.  Then throw in the UKIP runner, a classic 'dark horse' if ever there was one.  Formbook defying UKIP, with disputed performance on the gallop polls (sorry ed) - is a betting proposition for the brave-hearted only.   

2.  Too many wider inferences about British politics are drawn from low volume, low liquidity political markets that exist for the purposes of bookmaker public relations rather than their direct profit - and also the advantage of other political interests eager to create a bandwagon for their runner. 
   
Let's be absolutely clear here.  Most political markets are tiny economic events relative to sports markets and horse races.  Politicalbetting.com accurately report that over £250,000 has been matched on Betfair on the Scottish Independence contest, and claim that by the time of the vote it could be millions, which I don't doubt.  However this is a fraction of what would usually be matched on an individual football or cricket match covered on TV.  What is more, the 'matched bets' measure refers to the amount of bets that have been agreed on a betting exchange, where many punters open and close positions again and again to lock in a small profit or loss, inflating an aggregate figure that isn't turnover as you would imagine it in a high street betting shop.  Secondly, one or two large wagers struck with the traditional bookmakers may well have been 'laid off' on the exchanges, inflating the matched bets figure.
  
With low-liquidity markets, very small volumes drive relatively large changes in price.  Too often we hear on Twitter from politicians, pundits and the betting companies themselves that there has been a 'significant move' for a political runner.  Yet in reality this price change is often next to meaningless.  It is likely to be either a nervous response from an uncertain bookie to a small volume trade in a market that itself is low on volume and confidence.  Then it becomes a stoked up revelation from a eager PR man, either a bookmaker seeking publicity or a political interest hyping their own success. The markets which are particularly prone to this are the weakest ones of all - such as 'next Cabinet Minister to leave', the 'next leader of a political party' or by-election results.  Again and again we get this spin effect, linked up with some recently occurred political event, rather like the reporting of suspect short-term opinion poll 'bounces' that don't really demonstrate any significant long term trend but merely 'noise' in the system.  There seems to be an unhealthy lack of checks and balances among all those involved - the bookmaker, the pundit, and the political interest - without any input from someone who knows about the weakness of political markets, to blow the whistle. 

In the world of opinion polling, the likes of the Polling Observatory and Anthony Wells provide some corrective rigour to those who might misuse the numbers, but as for the public debate involving political betting, well it resembles the Wild West where the loser is the uninitiated layman.  Too often the conclusions become the contorted plaything of those with an axe to grind. 

So why has yesterday's article by Mike Smithson got my heckles up?  Well it's a good example of how the operation of political betting markets are abused by pundits in making a political point - in this case the inference that the wider public don't care much about EU matters.  The contention is this: the betting volumes on the Indy referendum dwarf those on the Euro elections which, according to Mike Smithson "really does suggest that the people of Scotland, where I am at the moment, are taking a huge interest in September’s vote. Their future is at stake."  No doubt the Scots do care deeply about this big decision (look at the betting on referendum turnout), but the conclusion made from the betting market volumes on Betfair is utterly invalid.  Then the cheeky political point follows (from a self-confessed Lib Dem and probable Europhile).  With somewhat mock surprise, he wonders why there is not more interest in the Euro elections, "despite a possibility that the purples could come out on top". Once again, through subtle suggestion we are being reminded, via a bogus contention that bet volumes relate to wider political interest, that public salience for anything Euro is low, perhaps even among UKIP supporters. (Remember also how UKIP voters are more interested in immigration than the unrelated? EU question).

So how can political betting inform us about our politics? 

There is much to be said for the idea that 'money talks'.  Political betting could be a reliable guide, informing on changing public narratives towards political outcomes, but only when the markets are sufficiently liquid and recognising that key high-end punters are not representative of the general public.  Divergences between political markets and opinion poll trend lines raise interesting questions of each predictor.  General Election markets have the greatest volumes, particularly closer to the event, and therefore greater reliability and usefulness.

In respect of judging political interest in events, 'number of bets' rather than 'volume of bets' might be a slightly better indicator, but I don't think it ever gets published by any betting company.  And there is a good reason for this.  It would reveal just how low salience political markets are at the mass-level, belying claims made in press releases.  When vague references to the number of political bets taken is revealed by bookies such as 'we have been inundated with wagers' or 'flooded with cash' - I feel a burning sense of scepticism.  If only we could interrogate their backend bet capture systems.  It would be nice to know just how many Scottish punters have engaged as Mike S suggests they have, but how will we really ever know?  I would hazard a guess of a hundreds across the board via traditional online betting sites - at the very most - not thousands.  I would love to be proved wrong and I don't really know these days, but don't be fooled into thinking that for the Scots the Referendum is the great betting event to rival even a Scottish Grand National at a soggy Ayr.