If the pollsters and handicappers end up being spectacularly wrong on Election Night, there’s one group that won’t be too surprised: the pollsters and handicappers themselves.
The 2022 midterms could go exactly as modeled—a 20-some-odd-seat pickup for Republicans in the House and maybe a 51-49 GOP Senate—but the people who watch these races the closest are also warning they might be wrong in decisive ways. In either direction.
No one really knows because, like every election, pollsters are extrapolating their best guess based on a set of assumptions. But unlike previous elections, the assumptions are getting bigger.
ADVERTISEMENT
According to Dave Wasserman, the U.S. House editor at the Cook Political Report, the big problem is that “response rates suck.”
“We’re down to 1 percent of people on a good day who are willing to talk to a pollster for free,” he told The Daily Beast.
Wasserman, perhaps the top handicapper of U.S. House races, said everyone was trying different ways to solve for “partisan non-response bias”—essentially a measure of how a poll isn’t representative of the actual population—but that means every pollster was making “a different assumption about who’s going to show up on Nov. 8 that may or may not be accurate.”
Part of the issue is that Republicans seem less and less inclined to answer poll questions. And another part is that pollsters are being less transparent about their methodologies to correct for those types of difficulties.
“We are, in many respects, stumbling through the dark with headlamps and flashlights,” Wasserman said. “And we have a vague understanding of where these races stand, but there are bound to be surprises.”
Nate Silver, the founder and editor in chief of the data-driven news site FiveThirtyEight, expressed many of the same concerns as Wasserman.
“The quick version is that polling is getting harder because fewer and fewer people answer phone calls from unknown numbers, and among those who do, it’s still a fairly big ask to have them complete a long survey at a time of declining civic trust,” Silver told The Daily Beast. “So those people who do respond are unusual in some respects, in ways that you may or may not be able to correct for—and there may also be the risk of overcorrecting.”
Silver noted that online polls could avoid some of these problems, but they introduce others, “namely that it’s hard to get a truly random sample online the way you can with phone numbers.”
“I don’t think this means that polling is irrevocably broken,” he continued. “But we shouldn’t expect pinpoint accuracy and there is not necessarily a correct, ‘gold standard’ way to conduct polling anymore.”
No one really knows if the polls will have systemic problems this year. While 2020 polls undercounted GOP support, the 2018 midterm polls were pretty much dead on. “But I think it’s sunk in more after 2020 that polling may be hard to fix,” Silver said, “and even the numbers nerds like me are approaching polling with a bit more skepticism.”
One of the reasons for that skepticism is partisan polling. A flood of new firms this cycle has raised questions about how handicappers should treat these polls that are conducted by partisan operatives—and what these polls (most of them GOP-affiliated) are doing to polling averages.
Silver addressed some of these questions over the weekend, when he tweeted that he didn’t think the complaint that Republican-leaning pollsters were “flooding the zone” was very convincing. He said his news site adjusted the polls to some extent and that the polling averages were a “free market.” Democrat-leaning pollsters could release polls if they wanted to. “That they don’t says something,” he tweeted.
But many Democrats have expressed frustration that some of these polls are being used for political ends, such as fundraising, voter outreach, and even shaping or manufacturing a narrative. In their minds, these polls aren’t measuring public opinion so much as they’re trying to inform it. That isn’t a new phenomenon, but seasoned observers are noting an uptick this cycle.
Stephen Fowler, a political reporter for Georgia Public Broadcasting and the host of the Battleground Ballot Box podcast, has tracked the trend in Georgia this year.
Fowler told The Daily Beast that voters need to know who is behind a poll and why—that not all “snapshots” of a race are equal, that some polls this year are “not realistic,” and that on the whole, the media and pundits could do a better job filtering the signal from the noise.
“There have been partisan polls released in Georgia that have Republican candidates ahead by 14 points in a state that was decided by 12,000 votes in 2020, which is not realistic,” Fowler said, “while nonpartisan polls essentially have the governor and Senate races as neck-and-neck tossups that will be determined by who shows up more before polls close.”
Part of the problem is human nature. Polls are often an “easy shorthand” that can reinforce preconceived notions of who is winning, Fowler said. But the devil is in the details.
“Plenty of the more recent Senate polls have unrealistic crosstabs that show Republicans having record Black support but cratering preference among white voters—which is not something real-life reporting has borne out in either case,” he observed. “Still, others don’t share many details about how they conducted the poll, who they talked to, and why, making the skepticism meter run even higher.”
Jacob Perry, the co-founder of Center Street PAC and a former Republican campaign manager for six U.S. House races before he turned against former President Donald Trump, noted that his group does online polling, which is much cheaper to conduct but frowned upon by most handicappers because of the self-selection bias.
But Perry said he thinks all polling has flaws. People like to treat polls like an accurate snapshot of a race, he said, “but you have no fucking idea what’s going to happen.”
The thing no one can really predict in any poll, Perry said, is turnout. He noted that, in 2016, pollsters had real difficulty forecasting how Trump’s voters would show up.
“Here’s a guy who breaks all the rules, does everything wrong, tells you to fuck off, does what he wants to do, and it worked for him,” Perry said of Trump.
“All the rules are over,” he added.
Jim Hobart, a partner at the prominent GOP polling firm Public Opinion Strategies, observed a paradox at work—polling is getting both harder and easier.
“The irony of polling now is that, even though polling has gotten much more challenging, the barrier of entry is low. For instance, text message and autodial is pretty cost-effective, so if some new firm is looking to build their name ID and they think it’s gonna be a good night for Republicans, I can see the logic in releasing a bunch of polls that show that,” Hobart said.
Hobart told The Daily Beast that, while polling has gotten more challenging, baked-in partisan polarization has determined a lot of the playing field. He put it like this: Major upsets in college football are rarer than in the NFL. Louisiana-Monroe is extremely unlikely to beat Alabama, but, given the more equal distribution of talent among NFL teams, it’s possible the Jets beat the Bills, as they did on Sunday. The political races drawing most of the attention are ones where the outcome is in play.
Democratic pollster Ethan Winter made a similar point about the limit of how wrong polling could be given what we know about partisan divides among voters. Winter, the lead analyst at Data for Progress, said he was a “glass half-full guy” on the accuracy of polling. He noted that, even with the challenges everyone faced with non-response rates in 2020, Data for Progress predicted the winner in more than 70 percent of close races that year.
“It’s basically a miracle and I think a testament to the sorts of innovations that the polling industry has been able to deploy in a relatively short amount of time,” he said.
Those innovations are a blending of different kinds of polling—live caller, texting people to fill out a survey online, and sending people postcards to convince them to fill out online surveys. And it’s the aggregate averages, as well as the consistency of many of those averages, that gives handicappers some degree of confidence that they’re not completely off. While pollsters have to make some assumptions, they’re not starting from zero. The partisan breakdown among different constituencies determines part of the playing field for any candidate, and history is a helpful guide on what sorts of assumptions are solid.
Pollsters could be off by five points—a huge but theoretically possible miss—but the majority of congressional races will be determined by double digits. Even if pollsters undercount GOP support by five points, a Republican running in a district with a Partisan Voting Index that benefits Democrats by 30 points won’t be turning red.
However, a systemic undercount of one party could mean there are a handful of surprises in close House and Senate races. It happened to some degree in 2016 and 2020, and pollsters are worried it could happen again, in either party’s favor.
On Saturday, The New York Times’ chief political analyst, Nate Cohn, sent out his newsletter with the headline “Polling Averages Can Be Useful, but What’s Underneath Has Changed.”
“There has been a wave of polls by firms like the Trafalgar Group, Rasmussen Reports, Insider Advantage and others that have tended to produce much more Republican-friendly results than the traditional pollsters,” Cohn wrote. “None adhere to industry standards for transparency or data collection. In some states, nearly all of the recent polls were conducted by Republican-leaning firms.”
If that sounds like a potential recipe for overstating the GOP’s standing in polls, it could be. But Cohn sent out a different newsletter on Nov. 1. That one was titled “A Worrisome Pattern Re-emerges in Seeking Response From Republicans.”
“In our final wave of Senate and House polls in the last few days, that hallmark of nonresponse bias looks as if it’s back,” Cohn wrote.
“Overall, white registered Democrats were 28 percent likelier to respond to our Senate polls than Republicans—a disparity exceeding that from our pre-election polling in 2020,” he continued. And Cohn’s point is that, if their polling was seeing a big partisan non-response bias, it’s not much of a stretch to think other polls are experiencing that, too.
He concluded it was “entirely possible” polls on the whole wouldn’t have the same problems as two years ago, and that non-response bias was just one factor among many that could cut either way.
But the simple truth is that, no one knows for sure.
