That’s it, I’m stat shaming
This month in wack public policy polling: A worse-than-useless abortion survey
My high school statistics teacher was a patient man named Mr. Brinson. He had spent most of his career in some private industry or another (we never asked) and had decided, as he approached retirement age, to teach 16-year-olds a thing or two about linear regression analysis.
I was part of the Advanced Placement class, an overachieving cohort of cocksure Überdorks at my South Carolina public high school. I am certain we made him regret coming into the classroom some days.
When he wasn’t rolling his eyes at our juvenile puns about p-values, he did manage to teach us well. To this day, I can spot junk statistical analysis by asking questions about sampling methodology, survey questions, and confidence intervals.
Beyond the formulas and graphs, he trained us to look at statistics with basic skepticism. He showed us how to lie with statistics and how to tell when someone else is lying. Alarm bells still go off in my head when I read about a study with a non-random sample, a tiny sample size, or a large margin of error. When I was a news reporter, I would often get PR pitches about some survey or another and reply with a detailed breakdown of why I wouldn’t use their findings in any of my reporting.
You have to keep your head on a swivel when a pollster makes some claims about the world, especially in a state like South Carolina. As the first Southern state to hold a primary in each presidential election cycle, we are on the cutting edge of push polling, a tactic in which campaign workers present political narratives or disinformation under the guise of a poll.
Infamously, in the run-up to the 2000 Republican presidential primary, thousands of South Carolina Republicans received phone calls (rumored to be funded by the George W. Bush campaign) from self-described pollsters who asked, “Would you be more likely or less likely to vote for John McCain for president if you knew he had fathered an illegitimate black child?" The people running these campaigns are not spiritually capable of shame.
This brings me to one of the lousiest public opinion polls I’ve seen all year.
Lies, damned lies, and abortion
Many of the Republicans who control South Carolina’s legislature have been dead set on forcing women and children to give birth, even when their lives or health are in danger. Since the U.S. Supreme Court deleted the constitutional right to a legal abortion in June, our state lawmakers have been hashing out some narrow exceptions to an all-but-total abortion ban. People will die and face incalculable harm because of the decisions they are making. They know this and are pushing forward anyway.1
Against that grim backdrop, a company called the Trafalgar Group announced on Aug. 30 that they had conducted some relevant polling of South Carolina voters. Here’s a bar graph from their report:
For starters, the wording of this poll obfuscates some basic medical facts. “Fetal heartbeat” is a Republican talking point to sell extreme abortion bans to voters, not a medically meaningful distinction. At 6 weeks of pregnancy (4 weeks after a missed period), the cutoff frequently used by “fetal heartbeat” abortion ban authors, an embryo has a tube that generates electrical impulses, but not a heart.
“Partial-birth” is, similarly, a political distinction and not a medical one. A pro-abortion voter might reasonably rule out the poll option “Legal up until the moment of birth, including partial birth” because it is couched in anti-abortion language.
Next, I downloaded the full report from the Trafalgar Group website to get an idea of how the survey was conducted. The company says it surveyed 1,071 respondents from a population of “Likely General Election Voters.”
Trafalgar reports a response rate of 1.43%, and while I’m not a professional pollster, that rate seems … shockingly low? Low survey response rates have been the bane of pollsters for the past two decades — particularly as a flood of deregulated robocalls made Americans less likely to pick up the phone — but even phone surveys got an average response rate of 6% in recent studies by the Pew Research Center. So what gives?
The Trafalgar report is light on its explanation of methodology. It links to a general-purpose Polling Methodology page on the company website that says:
The Trafalgar Group delivers our polling questionnaires utilizing a mix of six different methods:
Live callers
Integrated voice response
Text messages
Emails
Two other proprietary digital methods we don’t share publicly.
That last bullet point is, in technical terms, a “just trust me, bro” clause. Proprietary digital methods! Literally incredible.
Whatever black-box polling techniques and statistical analysis these pollsters are using, they claim they can account for “social desirability bias,” or the tendency of respondents to say what they think people want to hear. They steadfastly refuse to explain their methods on the Methodology page.2
“This allows us to obtain a poll participant’s true feelings in situations where we believe some individuals are not likely to reveal their actual preferences,” the page states. The Trafalgar Group’s founder has said in interviews that he was able to accurately predict Donald Trump’s 2016 presidential election victory because he tapped into the true feelings of the “shy Trump supporter."3
How did Trafalgar pollsters account for social desirability bias and suss out the hidden feelings of South Carolinians on the subject of abortion? They won’t tell us. They also won’t tell us who funded the survey, in nose-thumbing defiance of the American Association of Public Opinion Research’s Code of Professional Ethics and Practices.
The Trafalgar Group was founded in 2006 by a Republican political consultant named Robert Cahaly, a South Carolina native who cites the legendarily cruel George H.W. Bush strategist Lee Atwater as an inspiration.
“Lee Atwater drilled into everyone around me that you have to get out of the head of politicos and into the head of Joe Six-Pack,” Cahaly told The New York Times in November 2020, when he predicted that President Donald Trump would win re-election. “What do the average people think? And to do that I like to talk to average people. I like to follow up polling calls and chat with people for 30 minutes.”
At the time in 2020, Trafalgar was described as a one-man operation. I'm not sure if Cahaly has hired any employees since then. Here’s another fun detail from that Times profile:
In 2010, Mr. Cahaly was arrested and taken to court for violating a law against using automatic calling machines — known as robocalling — to conduct polls. The charges against him were eventually dropped, and he later successfully sued a state law enforcement agency, causing South Carolina’s prohibition on robocalls to be declared unconstitutional.
That does help answer why survey responses are so low in South Carolina, actually. Thanks for absolutely nothing, Robert.
Some announcements
For Banned Books Week, I’ll be hosting a free panel discussion on the state of censorship in South Carolina. If you’re in the area, come out to Itinerant Literate Books in North Charleston on Thursday, Sept. 22 at 7 p.m.!
It’s a free event, but please RSVP here so my friends at the bookstore know how many guests to expect. Here are the panelists:
AJ Davis (@Anjene1976)
Aj Davis is a passionate community advocate who is focused on quality, equitable public education, economic and social justice and building inclusive communities.
Loni Lewis (@djlanatron)
Loni Lewis is a teacher and librarian at Sangaree Middle and is currently the chair of the Diversity, Equity, and Inclusion Committee for the South Carolina Association of School Librarians.
Marjory Wentworth (@MarjWentworth)
Marjory Wentworth is a College of Charleston English instructor who teaches courses on banned books and a National Coalition Against Censorship Free Speech Is for Me Advocate. She was South Carolina’s poet laureate from 2003-2020.
One last thing: The essayist Phil Christman is looking for volunteers to review manuscripts for the Michigan Review of Prisoner Creative Writing. Click over to his short Twitter thread for more details on what he needs and how to get involved.
I signed up to read a few pieces, and I know I have some writer friends who might want to join in. Let’s take notes and see if we can’t start something similar in our own backyards.
***
Brutal South is a free newsletter about education, class struggle, and religion in the American South. If you would like to support my work, get some cool stickers in the mail, and read / listen to some subscriber-only content, paid subscriptions are $5 a month. No worries if you can’t afford that; just tell your friends about the newsletter!
Bookshop // Twitter // Bandcamp // Apple Podcasts // Spotify Podcasts
If you’d like to fight back, I’d encourage you to support the Palmetto State Abortion Fund and the South Carolina Women’s Rights and Empowerment Network (WREN), who have been organizing relentlessly and taking the fight to the Statehouse week after week. All we have is each other.
For some actual scholarly research on social influence in surveys and how to account for it, check out this November 2020 piece in The Conversation, “Election polls are more accurate if they ask participants how others will vote.” Stick around for the line about “Bayesian truth serum!”
Hi I’m from South Carolina and what the hell is a shy Trump supporter?
I can't handle any more subscriptions it is taking up too much time and energy but sometimes you read something that begs response.
I live in Quebec and last time I was in Columbia SC the fountains were frozen and yesterday we had a tropical storm here in the frozen North.
I am 74 and done statistical analysis with the best of them for 72 years and better than the best than the best of them. In fact when I see the pundits lie with statistics I usually start haranguing my poor wife as to the ignorance of her countrymen and their propaganda.
In Quebec abortion is not a right it is a medical procedure whose efficacy is decided by the woman and her moral compass. Citizenship begins at birth and ends at death and doctors do doctoring and priests do whatever priests do but they only have the power of persuasion. The Quebec language press says only journalists can censor what they write and most Quebec journalists are censored by their own moral compass except those working for English Language corporate media or Quebec language corporate media.
I read latribune.ca every morning and I subscribe and pay into a fund setup by OUR government to make sure we have a media we can trust so we can continue being a Secular Humanist liberal democracy.
I was never an advanced placement student which is not achievable for an autistic entity with a genius in logic and linguistics and who at 74 uses one finger and a multicoloured keyboard to type but still can't sign his name and can't draw the facsimile of straight line with a pencil and a straight edge.
I do however talk a great deal because I can no longer tolerate all the BS. Voltaire was correct perception is reality and if you don't understand what is being said all that is left is perception.
South Carolina lives on perception and Clyburn, Scott and Graham are America; all sizzle no steak. Fries straight out of the freezer. Fried battered texturized protein straight out of Brave New World with just a hint of chicken and lots of salt and fat.
I live in Appalachia just north of Vermont in an agricultural and tourist region world famous for its maple syrup, poutine and its Fall colours.
I know what real fried chicken tastes like: first you start with God's own chickens not hybrid Cornish cross. Real chickens grow protein they don't just convert water and imitation feed into fat and water and fragile bone. In Quebec James Clyburn would be a right wing conservative but but when your choice runs from right wing conservative to Monty Python's Very Silly Party James Clyburn seems almost liberal. Liberale means FREE to adapt to reality in the language of Edmund Burke and Benjamin Franklin. I belong to Thomas More Institute of Montreal. It is devoted to life long learning. The Thomas More Society in America is a society for legal pedants devoted to maintaining perpetual stupidity. The Same Thomas More who lost his head to Henry the Eighth's axe handler.
My wife is cursed with a PhD and was a public intellectual before she retired and I am under the care of our Bureau of Autistic affairs. In South Carolina Autistic means intellectually challenged in Quebec autre means the other and autistic means strange and exotic.
I live 100 metres from the Appalachian trail. You know a Quebec metre is about a million light years longer than an American epiphany. Of course light can slowed down and even stopped and the speed of the energy is now an unknown. It is an x rather than a constant.
I know the Bible. It is the myth legend, poetry and understanding of my people. It is the story of evolution written in the vernacular of the time where almost no one read and scribes scribed Kings and story tellers. They didn't even know how to make magnifying glass. Imagine an electron microscope made out of volcanic glass.
My Brother Jesus believed in Darwin. He said the meek shall inherit the Earth. Did he mean peasants like me or cockroaches like congress people and Senators.
I always wanted to be an advanced placement student and the the status of being wise informed but I failed shop and only differently gendered hominids could take Home Economics.
I make no pretenses about my mental stability. This week I stood in our local supermarket offering logic lessons in exchange for help filling my cart. My eyesight is that of an old man who lost sight in one eye. I always shop with my wife but she grew up in the American Midwest and understands little about the way people are supposed to eat even if she taught teachers who reach home ECONOMICS. My wife is a scientist not a priest and doesn't hocus pocus but she never taught logic which is metaphysics and makes things simple not complex. If you can do logic shopping is a breeze and inflation is for other people to figure out. I don't have a Ouija board.
This can of of beans costs x pennies per gram of protein, fat and carbohydrates and that bag of refried beans costs x pennies per gram of protein, fat and carbohydrates and the Canada food guide suggests you consume x amount.
i don't understand why people who have gone to college can't read nutritional labels. They are more important in understanding their universe than FOX News.
Home economics is science American economics is Voodoo and witchcraft and crystal balls.
No wonder Mr Graham is a leader in the land of the blind a a blind drunk leads the parade.