The poll crisis and what we’re doing about it

By usual standards, last year’s midterm polls were among the most accurate ever.

But in harder-to-measure ways, it’s possible that those same polls were extremely bad.

Poll after poll seemed to tell a clear story before the election: Voters were driven more by the economy, immigration and crime than by abortion and democracy, raising the specter of a “red wave.”

In the end, the final results looked something like the last polls, but they told a very different story about the election: When abortion and democracy were on the line, Democrats excelled. And while the polls sometimes or even often showed Democrats excelling, they almost always failed to convincingly explain why they were ahead — making Democratic poll results seem fragile and weak.

Take our own Times/Siena polls. Our results in states like Pennsylvania and Arizona were very close to the final results and showed Democrats in the lead. By all accounts, abortion and democracy were major factors that helped explain democratic strength in these states, especially in the face of election deniers like Doug Mastriano or Kari Lake.

But while these polls performed well, they simply did not explain what was happening. In any case, the polls showed the conditions for a Republican victory. They showed that voters wanted Republican control of the Senate. They showed that a majority of voters didn’t really care whether a candidate thought Joe Biden won the 2020 election, even though election deniers were clearly punished at the ballot box in the end. Voters said they cared more about the economy than issues like abortion or democracy and so on.

The Times/Siena polls were not alone in this regard. Virtually all the major public pollsters told the same basic story, and it’s the opposite of the story we told after the election. If we assess these questions on these issues in the same way we assess the key election results – a comparison between the pre-election polls and what we believe to be true after the election, with the benefit of the results – I think we would have to say this was a complete failure.

If you do this exercise for previous elections, poll failures seem to be the norm rather than the exception. There just aren’t many elections when you can read the story of a pre-election poll, line up the story with the post-election story, and say that the pre-election poll captures the key dynamics of the election. For example, the latest CBS/NYT, Pew Research, and ABC/Washington Post polls from the 2016 election shed little light on the strength of Donald J. Trump. They contributed essentially nothing to the decade-long debate over whether the economy, racial resentment, immigration or anything else helped explain Trump’s success among white working-class voters in that election.

With such a poor track record, there is a case where the ‘issue’ polls are facing a much more serious crisis than the ‘horse race’ polls. I imagine many public opinion pollsters would recoil at this claim, but they can’t prove it wrong either. The polls confronting the crisis are almost entirely unfalsifiable – just like the polls themselves. The fact that the problems with polling are so difficult to quantify is likely why the problems have been allowed to fester. Most pollsters probably assume that they are good at polling; Unlike horse racing polls, they are almost never demonstrably wrong.

In fairness to pollsters, the problem is not just that the usual questions are unlikely to fully capture the attitudes of the electorate. It’s also true that pollsters try to figure out what drives voters’ behavior, which is a different and more challenging question than simply measuring who they will vote for or what they believe. These causal questions go beyond what a single poll of “problem” questions can realistically expect to answer. The worlds of political campaigns and social science research, with everything from experimental designs to message testing, probably have more relevant tools than public pollsters.

Over the coming year we will try to include some of these tools in our polls. We will focus more on analyzing what factors predict whether voters have “flipped” since 2020, rather than looking at what attitudes prevail among a majority of the electorate. We will try new question formats. We can even try an experiment or two.

We already tried such an experiment in our last Times/Siena battleground state poll. We divided the sample into two halves: one half was asked whether they would vote for a typical Democrat versus a Republican who expressed moderate views on abortion or democracy; the other half pitted the same Democrat against a Republican who expressed more conservative or MAGA views on abortion or democracy.

In the next newsletter I will tell you about the results of that experiment. I think it was promising.