7 ways to read polls like a superhero
Public opinion surveys have value, if you read them right
I’ve frequently criticized polls here, but they do serve a vital purpose.
They can be a gut check on our beliefs. Is our anecdotal evidence widespread, or just something we’re paying more attention to?
I’ll never forget reading Geraldine Ferraro’s autobiography. The 1984 Vice Presidential candidate was convinced she and Mondale would win, because her rallies were filled with thousands of people and everyone she knew told her she was headed to victory. They were trounced by Ronald Reagan in an electoral college landslide of 525 to 13.
After Nixon’s 1972 comfortable win, a New Yorker writer famously said that she only knew one person who voted for him.
You and I can be just as mistaken. A person who watches Fox News all day will have a very, very different worldview than a regular CNN watcher.
So polls are important … but. You really need to have a strong BS detector. I studied this in graduate school and have delved into hundreds of polls in my life. Here are the best questions to ask yourself to determine whether you’re getting the real picture.
1. Who paid for it?
It’s the most important question to ask because pollsters can skew things in minor ways we can’t detect.
And many – perhaps most – polls aren’t trying to make you smarter. They’re trying to convince you of something.
I guarantee that the National Vegan Society’s poll will show the unpopularity of beef, and I can predict what the Hawaiian Tourism Board’s poll showing America’s favorite vacation destination will show.
In political campaigns, polls often are sponsored by candidates or a PAC with a hidden agenda. Most large media outlets usually do their best to tell the truth – even when they’re spectacularly wrong – because those numbers actually get verified by election day numbers. They don’t want to look like fools.
Overall, media polls are generally better. But hopefully, you’ve read me long enough not to trust them too much.
2. What do the questions actually say?
If an article about the poll doesn’t link to the actual questionnaire, that’s a red flag. When it does, read the questions and even answer them in your head. One small word can change the meaning completely.
Recently, there’ve been a number of polls on transgenders, for instance. And while media coverage of trans issues is almost always glowing, I’ve been interested to know how much acceptance there’s been of different trans policies across the country.
NPR touted some surprising numbers a few months ago, with a subheading that read, “Republicans vulnerable on transgender rights.” But here was their actual question:
First of all, the word “criminalizes” skews things, compared to “forbids” or even “outlaws”.
More importantly, “gender transition-related medical care” can mean a whole lot of things. It’s broad and squishy enough that even 35% of Republicans don’t want to criminalize it.
If you added “that could result in the sterilization of children under 18”, which is the crucial point, support would nosedive.
The same (crappy) “NPR/PBS NewsHour/Marist” poll also asked about business practices, so that NPR could swipe at Republicans in the poll write-up like this: “Many in the GOP have made ESG (Environmental, Social and Governance) and DEI (Diversity, Equity and Inclusion) practices in companies bogey men.” Here’s what they asked:
Which would you choose? I’m a hard-nosed investor, but of course I want companies to be mindful of their impact on society or avoid pumping poison into a river. Yet this question was then used as proof that Republicans are scared of companies trying to do what’s best for society.
3. If they’re comparing numbers, did they ask the same questions?
If the poll is comparing the current numbers of something to past ones, did they keep everything about the polling the same? A change in wording, a different group of people answering, and you could skew the results.
4. Did the poll’s timing affect it?
Polls about Biden taken shortly after the Afghanistan debacle were horrible, and if you compare them to the boost after his surprise Ukraine visit, you’re seeing differences, but not necessarily a real trend.
Similarly, Barack Obama’s poll numbers went up 6% after the death of Osama bin Laden. There’s even a name for when presidents’ poll numbers go up after a major international event, the “rally ‘round the flag” effect.
5. Is it something people might lie about?
People get fired for saying crazy things like “girls have vaginas” these days, so they may not be completely honest with a stranger calling them on the phone. They may be wary of announcing they’re voting for Trump. They may feel weird saying they don’t support black reparations if the interviewer sounds black.
People also don’t like to look stupid, even to a stranger on a phone, and may answer something like “Uh, yeah, I’ve heard of Sen. Filibuster and I don’t like her one bit!”
6. Who did they ask?
If it’s an online-only poll, dismiss it out of hand. Those are for generating publicity, not knowledge.
If people can choose to be included in a poll, there’s zero scientific credibility, and real pollsters try hard to get an accurate sample of the population they’re sampling.
Even then, the “curmudgeon factor” is real … people who hang up on pollsters think differently than those who talk. Heck, even people who naturally answer their phones are a different demographic than those who are used to texting. Also, know that registered voters skew more Republican than the overall population.
7. Is it an outlier?
Between sampling, questions, and all the other factors, many polls that aim for accuracy are just wrong. So, how does the poll you’re reading stack up?
You can look at the list of the latest Trump vs. DeSantis polls and, while Trump is leading in all of them, there’s a wide range of numbers showing how far ahead he is. People trying to persuade you have a tendency to cherry-pick the polls that support their cause.
As we get farther into election season, polls are going to gain more prominence. They’re not a perfect solution, but if you read them carefully and skeptically, you can glean some insight into how things are going for the candidates.
I might add one word of caution to viewers: polls are make-news, not real news. They are often lazy journalism and given far more weight than is merited. A poll represents a snapshot in time that is framed by the parameters Ken outlined. Keep your salt shaker nearby.
Loved the article on polling --- reminded me that one of the best books I ever read was "How To Lie With Statistics"