How Wrong Would the Polls Have to Be?
Two months ago, it seemed unlikely (to me anyway) that a Democrat would have the lead in a runoff for governor in Louisiana. The standard predictors of outcomes in typical elections (the fundamentals) just do not bode well for a Democrat’s success. The probabilities favor Republican wins in a state like Louisiana. But, that’s the beauty of probability. It helps us get a sense of what to expect most of the time, but it also reminds us that the unlikely can happen.
Poll after poll in the final three weeks of the runoff has consistently shown Democrat John Bel Edwards leading Republican David Vitter. The range of the lead varies across polls from 6 points to 22 points. [The head-to-head polls from September and October fall in this same range.]
Polls this late in the campaign usually predict outcomes of gubernatorial contests well – or, at least, polling averages from late in the campaign generally predict well. Using information from multiple polls generally gets you closer to the actual outcome than just focusing on a single poll. 538’s averages for polls from the final three weeks of each 2014 gubernatorial election had an average absolute error (the absolute value of the difference between the average margin in the polls and the actual margin of victory) of 3.8 points. This means that the polls last year tended to miss the actual margin by a little under four points. That’s pretty much in line with how polls have done in gubernatorial elections for over a decade. So, although we should never dismiss the power of the fundamentals, the polls also cannot be ignored.
There are several different ways of calculating a polling average. You have to make choices about which pollsters to include, how far back in the campaign to go, how much to favor more recent polls, how to account for differences in sample size and likely voter models, and so on. 538 uses a sophisticated methodology to compute their averages. So, too does HuffPost’s Pollster. Regardless of the approach, for the Louisiana runoff polls it is hard to escape a double digit lead for Edwards. HuffPost has it at 11.2 points. A more naïve approach uses the polls from the final three weeks of the runoff and accounts for nothing but sample size (ignoring differences in methodology or how recent the polls are) yields an average of 13.2 points. [It should be noted that both of these estimates include the three polls with the smallest margins and did not call cellphones.]
Whereas the fundamentals did not bode well for Edwards, the polling does not bode well for Vitter. For Vitter to win, one of two scenarios must occur.
Scenario #1: A Late Surge
Vitter has performed worse than the typical southern Republican among Republican voters and among white voters generally. He’s winning the majority of these groups, but by a much smaller margin than the average Republican candidate running against a Democrat. Vitter gets about 60% of Republicans in most polls. As I have noted before, it is extremely rare for a candidate to fare so poorly among his own partisans in contemporary American gubernatorial elections. It happens only about 7% of the time.
If Republicans specifically and whites generally come around to Vitter, then we will see the race narrow – perhaps to the point where Vitter wins. The problem is that we have not seen signs of this yet in the publicly-released polls. Vitter has been hitting the Syrian refugee message pretty hard this week, likely in hopes of winning back these groups of voters. We only have one publicly-released poll in the field after Monday of this week, but it generally falls in line with earlier polls.
Scenario #2: Polling Failure
Barring a late surge, the Vitter camp has to hope that the polls are wrong. Not just wrong, catastrophically wrong. Encouragement for that hope seemed to come from a state a bit to the north of us – Kentucky.
Just this month, the polls missed the mark by a wide margin in the Kentucky governor’s race. There were three publicly-released, non-candidate-sponsored polls during the final three weeks of that election. The simple average weighting only by sample size had Republican Matt Bevin down by 3.5 points. The HuffPost average had him down by 2.1 points. Instead, Bevin won by 8.7 points. That’s a double digit miss by the polls. Based on the HuffPost average, the polls missed by 10.8 points. Vitter needs the polls to be off by just a bit more, by 11 to 13 points depending on how the average is calculated (and, again, assuming no actual late surge).
If it happened in Kentucky, why not here? There are a few features that make Louisiana less likely to go the way of Kentucky. First, the Louisiana runoff has more than twice the number of polls in the final three weeks than Kentucky did. Numbers and timing matter. The more quality polls available late in the campaign, the better their average will predict the outcome.
Second, it looks like a significant part of the problem in Kentucky were the likely voter/turnout models. They overestimated the Democrats’ share of the electorate. Kentucky pollsters are not the only ones to make these kinds of mistakes. It is difficult to model the likely electorate in a low turnout election like Kentucky’s gubernatorial election. Louisiana’s gubernatorial election will also have relatively low turnout – relative, that is, to a presidential election where it is much easier to model the electorate. On the other hand, several of the polls in the HuffPost model for the Louisiana runoff (all three of the MRI polls, for example) model the black share of the electorate at 26%, which is almost certainly lower than what the actual share will be. Because black voters overwhelmingly support Democrats, it is unlikely that these polls are overestimating the share of likely Democrat voters. If anything, these polls are underestimating it.
Third, the primary polls in Louisiana generally did okay in estimating the share of votes won by the Republican candidates. Their weakest performance was on Edwards’ vote share.
With all that said, what happened in Kentucky remains possible…but how probable?
Let’s be generous and take the smallest estimate of the average gap between Edwards and Vitter at 11.2 points and use this as a benchmark for how wrong the polls have to be for Vitter to win (and, again, assuming there is no actual Republican surge in the final days of the campaign.) That’s a bigger error than what happened in Kentucky.
Despite the growing despair about election polling, polls rarely perform as poorly as they did in Kentucky. To see this, it is helpful to look at the average absolute error across different elections. The site 538 is a great source for historical data on these errors. [To be fair, what I am about to do really is not a one-to-one comparison because HuffPost computes their polling averages in a different way than 538, but the differences in their estimates are not drastic.]
There were 35 gubernatorial elections in 2014. The polling averages from the final three weeks of the campaign missed by 11.2 points or more in just four of those contests. Two of the four misses occurred in states with just one poll available, an insufficient number for computing an average. If we only consider races that had at least three polls (like in Kentucky), the polls missed by 11.2 in just two states out of 26: South Dakota and Maryland. Only in Maryland did the double digit error mean that the polls predicted the wrong winner. The South Dakota polls predicted the winner correctly, they just missed the amount of votes he won by.
If we add in the 34 U.S. Senate races from 2014 we get a similar picture. There were four misses of 11.2 or more. Again, two of those were based on a single poll. The polls missed by 11.2 or more in just two of 20 contests that had three or more polls during the final three weeks: Arkansas and Tennessee. In neither case did the error cause the polls to incorrectly predict the winner.
If we add Kentucky to the mix (again, being generous here because Vitter actually needs the polls to do slightly worse than they did in Kentucky), we have just two out of 47 statewide elections for either governor or U.S. Senate that had at least three publicly-released, non-candidate-sponsored polls during the final three weeks of the campaign in which the polls missed by a double digit margin and picked the wrong winner as a result. Without an actual eleventh hour surge in support, this is how wrong Vitter needs the polls to be.
The problem is that the polls are only that wrong about 4% of the time. To get a sense of what this means take an analogy from college sports. The college basketball season just started. The season will culminate in the NCAA Men’s Basketball Tournament next spring. The polls being as wrong as Vitter needs them to be happens about as often as a #15 seed beats a #2 seed in the tournament.
Sure, it happens – much to the chagrin of the 2012 Duke Blue Devils – but very rarely.
So, which unlikely scenario seems more plausible to you? That a Republican candidate fails to consolidate support of Republican voters in a state as red as Louisiana? Or, that the polls are as wrong as Vitter would need them to be?
A Guess (Sort Of)
To be clear, I think we’re in for an election that is much closer than the polls show now, but I think something closer to scenario #1 is more likely. I think this based on the partisan tendencies of voters. Whether it will be enough for Vitter to win…I'm skeptical because that bar is getting harder and harder to reach, but it would foolish, I think, to say it is out of reach. Still, I am puzzled why we have not seen some of this in the polling yet. Perhaps there are systematic errors in the polls. It could very well happen that the polls are wrong, but for them to be even worse than in Kentucky? It certainly is not a likely scenario.