That’s it, I’m stat shaming
This month in wack public policy polling: A worse-than-useless abortion survey
My high school statistics teacher was a patient man named Mr. Brinson. He had spent most of his career in some private industry or another (we never asked) and had decided, as he approached retirement age, to teach 16-year-olds a thing or two about linear regression analysis.
I was part of the Advanced Placement class, an overachieving cohort of cocksure Überdorks at my South Carolina public high school. I am certain we made him regret coming into the classroom some days.
When he wasn’t rolling his eyes at our juvenile puns about p-values, he did manage to teach us well. To this day, I can spot junk statistical analysis by asking questions about sampling methodology, survey questions, and confidence intervals.
Beyond the formulas and graphs, he trained us to look at statistics with basic skepticism. He showed us how to lie with statistics and how to tell when someone else is lying. Alarm bells still go off in my head when I read about a study with a non-random sample, a tiny sample size, or a large margin of error. When I was a news reporter, I would often get PR pitches about some survey or another and reply with a detailed breakdown of why I wouldn’t use their findings in any of my reporting.
You have to keep your head on a swivel when a pollster makes some claims about the world, especially in a state like South Carolina. As the first Southern state to hold a primary in each presidential election cycle, we are on the cutting edge of push polling, a tactic in which campaign workers present political narratives or disinformation under the guise of a poll.
Infamously, in the run-up to the 2000 Republican presidential primary, thousands of South Carolina Republicans received phone calls (rumored to be funded by the George W. Bush campaign) from self-described pollsters who asked, “Would you be more likely or less likely to vote for John McCain for president if you knew he had fathered an illegitimate black child?" The people running these campaigns are not spiritually capable of shame.
This brings me to one of the lousiest public opinion polls I’ve seen all year.
Lies, damned lies, and abortion
Many of the Republicans who control South Carolina’s legislature have been dead set on forcing women and children to give birth, even when their lives or health are in danger. Since the U.S. Supreme Court deleted the constitutional right to a legal abortion in June, our state lawmakers have been hashing out some narrow exceptions to an all-but-total abortion ban. People will die and face incalculable harm because of the decisions they are making. They know this and are pushing forward anyway.1
Against that grim backdrop, a company called the Trafalgar Group announced on Aug. 30 that they had conducted some relevant polling of South Carolina voters. Here’s a bar graph from their report:
For starters, the wording of this poll obfuscates some basic medical facts. “Fetal heartbeat” is a Republican talking point to sell extreme abortion bans to voters, not a medically meaningful distinction. At 6 weeks of pregnancy (4 weeks after a missed period), the cutoff frequently used by “fetal heartbeat” abortion ban authors, an embryo has a tube that generates electrical impulses, but not a heart.
“Partial-birth” is, similarly, a political distinction and not a medical one. A pro-abortion voter might reasonably rule out the poll option “Legal up until the moment of birth, including partial birth” because it is couched in anti-abortion language.
Next, I downloaded the full report from the Trafalgar Group website to get an idea of how the survey was conducted. The company says it surveyed 1,071 respondents from a population of “Likely General Election Voters.”
Trafalgar reports a response rate of 1.43%, and while I’m not a professional pollster, that rate seems … shockingly low? Low survey response rates have been the bane of pollsters for the past two decades — particularly as a flood of deregulated robocalls made Americans less likely to pick up the phone — but even phone surveys got an average response rate of 6% in recent studies by the Pew Research Center. So what gives?
The Trafalgar report is light on its explanation of methodology. It links to a general-purpose Polling Methodology page on the company website that says:
The Trafalgar Group delivers our polling questionnaires utilizing a mix of six different methods:
Integrated voice response
Two other proprietary digital methods we don’t share publicly.
That last bullet point is, in technical terms, a “just trust me, bro” clause. Proprietary digital methods! Literally incredible.
Whatever black-box polling techniques and statistical analysis these pollsters are using, they claim they can account for “social desirability bias,” or the tendency of respondents to say what they think people want to hear. They steadfastly refuse to explain their methods on the Methodology page.2
“This allows us to obtain a poll participant’s true feelings in situations where we believe some individuals are not likely to reveal their actual preferences,” the page states. The Trafalgar Group’s founder has said in interviews that he was able to accurately predict Donald Trump’s 2016 presidential election victory because he tapped into the true feelings of the “shy Trump supporter."3
How did Trafalgar pollsters account for social desirability bias and suss out the hidden feelings of South Carolinians on the subject of abortion? They won’t tell us. They also won’t tell us who funded the survey, in nose-thumbing defiance of the American Association of Public Opinion Research’s Code of Professional Ethics and Practices.
The Trafalgar Group was founded in 2006 by a Republican political consultant named Robert Cahaly, a South Carolina native who cites the legendarily cruel George H.W. Bush strategist Lee Atwater as an inspiration.
“Lee Atwater drilled into everyone around me that you have to get out of the head of politicos and into the head of Joe Six-Pack,” Cahaly told The New York Times in November 2020, when he predicted that President Donald Trump would win re-election. “What do the average people think? And to do that I like to talk to average people. I like to follow up polling calls and chat with people for 30 minutes.”
At the time in 2020, Trafalgar was described as a one-man operation. I'm not sure if Cahaly has hired any employees since then. Here’s another fun detail from that Times profile:
In 2010, Mr. Cahaly was arrested and taken to court for violating a law against using automatic calling machines — known as robocalling — to conduct polls. The charges against him were eventually dropped, and he later successfully sued a state law enforcement agency, causing South Carolina’s prohibition on robocalls to be declared unconstitutional.
That does help answer why survey responses are so low in South Carolina, actually. Thanks for absolutely nothing, Robert.
For Banned Books Week, I’ll be hosting a free panel discussion on the state of censorship in South Carolina. If you’re in the area, come out to Itinerant Literate Books in North Charleston on Thursday, Sept. 22 at 7 p.m.!
It’s a free event, but please RSVP here so my friends at the bookstore know how many guests to expect. Here are the panelists:
AJ Davis (@Anjene1976)
Aj Davis is a passionate community advocate who is focused on quality, equitable public education, economic and social justice and building inclusive communities.
Loni Lewis (@djlanatron)
Loni Lewis is a teacher and librarian at Sangaree Middle and is currently the chair of the Diversity, Equity, and Inclusion Committee for the South Carolina Association of School Librarians.
Marjory Wentworth (@MarjWentworth)
Marjory Wentworth is a College of Charleston English instructor who teaches courses on banned books and a National Coalition Against Censorship Free Speech Is for Me Advocate. She was South Carolina’s poet laureate from 2003-2020.
One last thing: The essayist Phil Christman is looking for volunteers to review manuscripts for the Michigan Review of Prisoner Creative Writing. Click over to his short Twitter thread for more details on what he needs and how to get involved.
I signed up to read a few pieces, and I know I have some writer friends who might want to join in. Let’s take notes and see if we can’t start something similar in our own backyards.
Brutal South is a free newsletter about education, class struggle, and religion in the American South. If you would like to support my work, get some cool stickers in the mail, and read / listen to some subscriber-only content, paid subscriptions are $5 a month. No worries if you can’t afford that; just tell your friends about the newsletter!
If you’d like to fight back, I’d encourage you to support the Palmetto State Abortion Fund and the South Carolina Women’s Rights and Empowerment Network (WREN), who have been organizing relentlessly and taking the fight to the Statehouse week after week. All we have is each other.
For some actual scholarly research on social influence in surveys and how to account for it, check out this November 2020 piece in The Conversation, “Election polls are more accurate if they ask participants how others will vote.” Stick around for the line about “Bayesian truth serum!”
Hi I’m from South Carolina and what the hell is a shy Trump supporter?