Two polls collide: a look back at a tumultuous decade
It’s been almost a decade since I first attended the annual conference of pollsters known as AAPOR.
Then it was a completely different place. It was dominated by traditional pollsters who knew change was inevitable but who seemed uncomfortable with the sacrifices required to accommodate new people, methods and ideas.
At the time, that gathering reminded me of the Republican Party, then grappling with demographic change and Hispanic voters in the wake of Barack Obama’s re-election. There are clear differences, but the AAPOR crowd’s talk of reaching out to new groups and ideas was animated by similar feelings of threat that Republicans then faced – the concern posed by long-term trends, the status threat of new entrants, and the sense that traditional values would be threatened by the accommodation of new ideas.
But if Donald J. Trump showed that Republicans didn’t have to support immigration reform to win, he certainly showed pollsters that they needed to innovate. A decade and two historically significant bad cycles later, AAPOR is a very different place. The old guard is still there, but presentation after presentation uses methods that would have been scorned a decade ago. This year’s Innovators Award went to someone who referred to AAPOR as an association of “Buggy-Whip Manufacturers” in 2014, the year I attended for the first time.
The innovative turn in the polling community is very real, including in public political polls. Today, virtually no pollsters use the methods they did ten years ago. The ABC/Post survey is perhaps the one major exception, with its live-interview, random-digit-dialing telephone surveys. But so far, innovation and change have not been enough to solve the industry’s problems. It’s been enough to stay afloat, if it’s still struggling to stay afloat.
Heading into 2024, pollsters still don’t know if they can successfully reach Trump’s voters. They are still struggling with rising costs. And they’ve really lost something they had a decade ago: the belief that a well-designed survey would produce a representative sample. Today, a well-designed survey is not enough: the most theoretically sound surveys often produced the worst results of 2020.
So far, innovation in polls has been on two parallel tracks: one to find new ways to sample voters in an era of low response rates; another was designed to improve unrepresentative samples through statistical adjustments. If there’s an underlying theory to the Times/Siena survey, it’s to try to get the best of both worlds: high-quality sampling with advanced statistical adjustments. There are surprisingly few public opinion polls that can make a similar case: there’s a bad sample with nice statistical modeling, and there’s a good sample with simple demographic adjustment, but not much of either.
Due to the pandemic, it’s been a few years since I personally visited AAPO. But from my point of view, this was the first time that these two parallel tracks seemed to be moving closer together. They haven’t merged – the old guard remains reluctant to make some of the sacrifices necessary to improve its methods of adaptation; cost will prevent the upstarts from matching the expensive sampling of the old guard. But they’re getting closer, as researchers on both tracks realize their own efforts are insufficient and dive a little more into the other side’s ideas.
For example, an early theme was the recognition that even the most sophisticated survey designs still struggle to reach less engaged voters, who are generally less educated and likely to support Trump as well. This problem may never be addressed perfectly, which is why it takes advantage of both the best traditional and non-traditional methods.
For our part, I promise we’ll have more in the weeks to come on our Wisconsin experiment — which had parallel high-incentive phone and mail surveys ahead of the 2022 election. Over the past week or so, we’ve seen the received final data needed to begin this analysis, and I’ve started digging over the past two days. It’s early in the analysis, but there are some interesting things. Stay tuned.