Why We Can’t Quit Our Toxic, Insatiable Appetite for Polls
Frank Bruni, a contributing Opinion writer, hosted a written online conversation with Amy Walter of The Cook Political Report and Patrick Ruffini, a Republican pollster, to discuss the state of polling and of Democratic anxiety about polls ahead of the midterms.
Frank Bruni: Amy, Patrick, as if the people over at Politico knew that the three of us would be huddling to discuss polling, it just published a long article about the midterms with the gloomy, spooky headline “Pollsters Fear They’re Blowing It Again in 2022.”
Do you two fear that pollsters are blowing it again in 2022?
Patrick Ruffini: It’s certainly possible that they could. The best evidence we have so far that something might be afoot comes from The Times’s own Nate Cohn, who finds that some of the Democratic overperformances seem to be coming in states that saw large polling errors in 2016 and 2020.
Amy Walter: I do worry that we are asking more from polling than it is able to provide. Many competitive Senate races are in states — like Arizona, Georgia, Pennsylvania and Wisconsin — that Joe Biden won by supernarrow margins in 2020. The reality is that they are going to be very close again. And so an error of just three to four points is the difference between Democratic and Republican control of the Senate.
Ruffini: This also doesn’t mean we can predict that polls will miss in any given direction. But it does suggest taking polls in states like Ohio, which Donald Trump won comfortably but where the Republican J.D. Vance is tied or slightly behind, with a grain of salt.
Bruni: So what would you say specifically to Democrats? Are they getting their hopes up — again — in a reckless fashion?
Walter: Democrats are definitely suffering from political PTSD. After 2016 and 2020, I don’t think Democrats are getting their hopes up. In fact, the ones I talk with are hoping for the best but not expecting such.
Ruffini: In any election, you have the polls themselves, and then you have the polls as filtered through the partisan media environment. Those aren’t necessarily the same thing. On Twitter, there’s a huge incentive to hype individual polling results that are good for your side while ignoring the average. I don’t expect this to let up, because maintaining this hype is important for low-dollar fund-raising. But I do think this has led to a perhaps exaggerated sense of Democratic optimism.
Bruni: Great point, Patrick — in these fractured and hyperpartisan times of information curation, polls aren’t so much sets of numbers as they are Rorschachs.
But I want to pick up on something else that you said — “polls will miss in any given direction” — to ask why the worry seems only to be about overstatement of Democratic support and prospects. Is it possible that the error could be in the other direction and we are understating Republican problems and worries?
Ruffini: In politics, we always tend to fight the last war. Historically, polling misses have been pretty random, happening about equally on both sides. But the last big example of them missing in a pro-Republican direction was 2012. The more recent examples stick in our minds, 2020 specifically, which was actually worse in percentage terms than 2016.
Walter: Patrick’s point about the last war is so important. This is especially true when we are living in a time when we have little overlap with people from different political tribes. The two sides have very little appreciation for what motivates, interests or worries the other side, so the two sides over- or underestimate each other a lot.
As our politics continue to break along educational attainment — those who have a college degree are increasingly more Democratic-leaning, those with less education increasingly more Republican-leaning — polls are likely to overstate the Democratic advantage, since we know that there’s a really clear connection between civic voting behavior and education levels.
Ruffini: And we may be missing a certain kind of Trump voter, who may not be answering polls out of a distrust for the media, polling and institutions generally.
Bruni: Regarding 2016 and 2020, Trump was on the ballot both of those years. He’s not — um, technically — this time around. So is there a greater possibility of accuracy, of a repeat of 2018, when polling came closer to the mark?
Ruffini: The frustrating thing about all of this is that we just don’t have a very good sample size to answer this. In polls, that’s called an n size, like n = 1,000 registered voters. There have been n = 2 elections where Trump has been on the ballot and n = 1 midterm election in the Trump era. That’s not a lot.
Bruni: We’ve mentioned 2016 and 2020 versus 2018. Are there reasons to believe that none of those points of reference are all that illuminating — that 2022 is entirely its own cat, with its own inimitable wrinkles? There are cats that have wrinkles, right? I’m a dog guy, but I feel certain that I’ve seen shar-pei-style cats in pictures.
Walter: First, let’s be clear. Dogs are the best. So let’s change this to “Is this an entirely different breed?”
I’m a big believer in the aphorism that history doesn’t repeat, but it does rhyme.
Ruffini: Right. Every election is different, and seeing each new election through the lens of the previous election is usually a bad analytical strategy.
Walter: But there are important fundamentals that can’t be dismissed. Midterms are about the party in charge. It is hard to make a midterm election about the out-party — the party not in charge — especially when Democrats control not just the White House but the House and Senate as well.
However, the combination of overturning Roe v. Wade plus the ubiquitous presence of Trump has indeed made the out-party — the G.O.P. — a key element of this election. To me, the question is whether that focus on the stuff the Republicans are doing and have done is enough to counter frustration with the Democrats.
Ruffini: 2022 is unique in that it’s a midterm cycle where both sides have reasons to be energized — Republicans by running against an unpopular president in a time of high economic uncertainty and Democrats by the Dobbs v. Jackson Women’s Health Organization decision overturning Roe. It’s really unique in the sweep of midterm elections historically. To the extent there is still an energized Republican base, polls could miss if they aren’t capturing this new kind of non-college, low-turnout voter that Trump brought into the process.
Bruni: Patrick, this one’s for you, as you’re the one among us who’s actually in the polling business. In the context of Amy’s terrific observation about education levels and the Democratic Party and who’s more readily responsive to pollsters, what are you and what is your firm doing to make sure you reach and sample enough Republican and Trump-inclined voters?
Ruffini: That’s a great question. Nearly all of our polls are off the voter file, which means we have a much larger set of variables — like voting history and partisan primary participation — to weight on than you might typically see in a media poll (with the exception of the Times/Siena polls, which do a great job in this regard). We’ve developed targets for the right number of college or non-college voters among likely voters in each congressional district. We’re also making sure that our samples have the right proportions of people who have registered with either party or have participated in a specific party’s primary before.
But none of this is a silver bullet. After 2016, pollsters figured out we needed to weight on education. In 2020 we weighted on education — and we got a worse polling error. All the correct weighting decisions won’t matter if the non-college or low-turnout voter you’re getting to take surveys isn’t representative of those people who will actually show up to vote.
Bruni: Does the taking of polls and the reporting on polls and the consciousness of polls inevitably queer what would have happened in their absence? I will go to my grave believing that if so many voters hadn’t thought that Hillary Clinton had victory in the bag, she would have won. Some 77,000 votes in Pennsylvania, Michigan and Wisconsin — the margin of her Electoral College loss — are easily accounted for by overconfident, complacent Clinton supporters.
Walter: In 2016, there were two key groups of people that determined the election. Those who never liked Clinton and those who disliked Trump and Clinton equally. At the end, those who disliked both equally broke overwhelmingly for Trump. And, those Democratic-leaning voters who didn’t like her at all were never fully convinced that she was a worthy candidate.
Ruffini: I don’t worry about this too much since the people most likely to be paying attention to the daily movement of the polls are people who are 100 percent sure to vote. It can also work in the other direction. If the polls are showing a race in a red or blue state is close, that can motivate a majority of the party’s voters to get out and vote, and that might be why close races in those states usually resolve to the state fundamentals.
Bruni: Evaluate the news media in all of this, and be brutal if you like. For as long as I’ve been a reporter, I’ve listened to news leaders say our political coverage should be less attentive to polls. It remains plenty attentive to polls. Should we reform? Is there any hope of that? Does it matter?
Ruffini: I don’t think there’s any hope of this getting better, and that’s not the media’s fault. It’s the fault of readers (sorry, readers!) who have an insatiable appetite for staring at the scoreboard.
Walter: We do pay too much attention to polls, but polls are the tool we have to capture the opinions of an incredibly diverse society. A reporter could go knock on 3,000 doors and miss a lot because they weren’t able to get the kind of cross-section of voters a poll does.
Ruffini: Where I do hope the media gets better is in conducting more polls the way campaigns conduct them, which are not mostly about who is winning but showing a candidate how to win.
In those polls, we test the impact of messages on the electorate and show how their standing moved as a result. It’s possible to do this in a balanced way, and it would be illuminating for readers to see, starting with “Here’s where the race stands today, but here’s the impact of this Democratic attack or this Republican response,” etc.
Bruni: Let’s finish with a lightning round. Please answer these quickly and in a sentence or less, starting with this: Which issue will ultimately have greater effect, even if just by a bit, in the outcome of the midterms — abortion or gas prices?
Walter: Abortion. Only because gas prices are linked to overall economic worries.
Ruffini: Gas prices, because they’re a microcosm about concerns about inflation. When we asked voters a head-to-head about what’s more important to their vote, reducing inflation comes out ahead of protecting abortion rights by 67 to 29 percent.
Bruni: Which of the competitive Senate races will have an outcome that’s most tightly tethered to — and thus most indicative of — the country’s mood and leanings right now?
Walter: Arizona and Georgia were the two closest races for Senate and president in 2020. They should both be indicative. But Georgia is much closer because the G.O.P. candidate, Herschel Walker, while he’s still got some problems, has much less baggage and much better name recognition than the G.O.P. candidate in Arizona, Blake Masters.
Ruffini: If Republicans are going to flip the Senate, Georgia is most likely to be the tipping-point state.
Bruni: If there’s a Senate upset, which race is it? Who’s the unpredicted victor?
Walter: For Republicans, it would be Don Bolduc in New Hampshire. They’ve argued that the incumbent, Senator Maggie Hassan, has low approval ratings and is very weak. It would be an upset because Bolduc is a flawed candidate with very little money or history of strong fund-raising.
Ruffini: I’d agree about New Hampshire. The polling has shown a single-digit race. Republicans are also hoping they can execute a bit of a sneak attack in Colorado with Joe O’Dea, though the state fundamentals look more challenging.
Bruni: You (hypothetically) have to place a bet with serious money on the line. Is the Republican presidential nominee in 2024 Donald Trump, Ron DeSantis or “other”?
Walter: It’s always a safer bet to pick “other.” One of the most difficult things to do in politics is what DeSantis is trying to do: not just to upend someone like Trump but to remain a front-runner for another year-plus.
Ruffini: I’d place some money on DeSantis and some on “other.” DeSantis is in a strong position right now, relative to the other non-Trumps, but he hasn’t taken many punches. And Trump’s position is soft for a former president who’s supposedly loved by the base and who has remained in the fray. Time has not been his friend. About as many Republicans in the ABC/Washington Post poll this weekend said they didn’t want him to run as did.
Bruni: Same deal with the Democratic presidential nominee — but don’t be safe. Live large. To the daredevil go the spoils. Joe Biden, Kamala Harris or “other”?
Walter: History tells us that Biden will run. If he doesn’t, history tells us that it will be Harris. But I feel very uncomfortable with either answer right now.
Ruffini: “Other.” Our own polling shows Biden in a weaker position for renomination than Trump and Democrats less sure about who the alternative would be if he doesn’t run. I also think we’re underestimating the possibility that he doesn’t run at the age of 81.
Bruni: OK, final question. Name a politician, on either side of the aisle, who has not yet been mentioned in our conversation but whose future is much brighter than most people realize.
Walter: If you talk to Republicans, Representative Patrick McHenry is someone they see as perhaps the next leader for the party. There’s a lot of focus on Kevin McCarthy now, but many people see McHenry as a speaker in waiting.
Ruffini: He’s stayed out of the presidential conversation (probably wisely until Trump has passed from the scene), but I think Dan Crenshaw remains an enormously compelling future leader for the G.O.P. Also in Texas, should we see Republicans capitalize on their gains with Hispanic voters and take at least one seat in the Rio Grande Valley, one of those candidates — Mayra Flores, Monica De La Cruz or Cassy Garcia — will easily be in the conversation for statewide office.
Bruni: Thank you, both. I just took a poll, and 90 percent of respondents said they’d want to read your thoughts at twice this length. Then again, the margin of error was plus or minus 50 percent, and I’m not sure I sampled enough rural voters in the West.
Frank Bruni (@FrankBruni) is a professor of public policy and journalism at Duke, the author of the book “The Beauty of Dusk” and a contributing Opinion writer. He writes a weekly email newsletter and can be found on Twitter, Instagram and Facebook. Patrick Ruffini (@PatrickRuffini) is a co-founder of the Republican research firm Echelon Insights. Amy Walter (@amyewalter) is the publisher and editor in chief of The Cook Political Report.
The Times is committed to publishing a diversity of letters to the editor. We’d like to hear what you think about this or any of our articles. Here are some tips. And here’s our email: [email protected].
Follow The New York Times Opinion section on Facebook, Twitter (@NYTopinion) and Instagram.