All Episodes
Previous Next
Plain Text
Aug. 23, 2016 - Skeptoid
16:56
Skeptoid #533: Don't Try It Before You Knock It

Try it before you knock it… unless you want to know if it really works. Learn about your ad choices: dovetail.prx.org/ad-choices

Transcriber: nvidia/parakeet-tdt-0.6b-v2, sat-12l-sm, and large-v3-turbo
|

Time Text
Why Personal Experience Fails 00:07:05
Try it before you knock it sounds like almost perfect common sense, doesn't it?
Who has standing to make negative statements about something unless they've personally tried it for themselves, had a bad experience, and can then speak with first-hand knowledge.
But forget all about that.
Today, I'm going to explain why doctors and scientists never do this and why you shouldn't either.
Don't try it before you knock it is today on Skeptoid.
Hi, I'm Alex Goldman.
You may know me as the host of Reply All, but I'm done with that.
I'm doing something else now.
I've started a new podcast called Hyperfixed.
On every episode of HyperFixed, listeners write in with their problems and I try to solve them.
Some massive and life-altering, and some so minuscule it'll boggle your mind.
No matter the problem, no matter the size, I'm here for you.
That's HyperFixed, the new podcast from Radiotopia.
Find it wherever you listen to podcasts or at hyperfixedpod.com.
You're listening to Skeptoid.
I'm Brian Dunning from Skeptoid.com.
Don't try it before you knock it.
One of the biggest challenges in science writing when discussing unproven or implausible therapies and products is that people tend to trust their own personal perceptions more than any other source of information.
We tend to go by what we've experienced ourselves rather than by what other people say they've experienced.
Consequently, people are rarely moved by the results of testing and experimentation if the results contradict their own experience.
I know it works, they tend to say, because it worked for me.
And so often this gets projected back onto me because I've not tried the product myself, but only reported the results of testing, I should not comment on it.
Don't knock it until you've tried it, I am told.
Today, I'm going to explain exactly why it is not only appropriate to avoid a personal experience, as the best experimenters do, but it's actually a better way to learn about something.
Today, I say, don't try it before you knock it.
Here's the main reason we have that rule, and it's the reason that experimenters seldom include a personal experience when they evaluate something.
Human beings are wonderful at appraising experiences, but we're terrible at collecting data.
Our senses are faulty, prone to error, and everyone's are calibrated differently.
We all bring preconceived notions and expectations.
We have personal biases.
We are subject to all manner of perceptual errors.
We make interpretations of our experiences, and we all interpret them differently.
We have moods, up days, down days, personalities, tendencies.
To expect any one random person's assessment of an experience, filtered through his own prejudices and biases and preferences, to be a truly objective and factual representation, is a fool's errand.
The scientist knows this.
He knows his own experience is a worthless indicator, and he knows that an objective evaluation should not be tainted with it.
So, what do we do instead of trying it?
We rely on controlled testing.
When we want to know if something works as advertised, we design a test.
A properly designed test employs controls and randomization that cancel out all of the biases and other weaknesses we've discussed.
If you want to know whether, for example, listening to a binaural beat audiophile will make you fall asleep, a true science fan knows not to bother trying it to see.
She knows her sleepiness varies throughout the day anyway.
She knows that the expectation that it's supposed to make her sleepy skews her perception.
Instead, she looks at properly controlled testing that's been done.
Those subjects didn't know what they were listening to.
They didn't know what it was supposed to do to them.
And some of them unknowingly listened to a placebo recording.
She knows the difference between real, statistically sound data and one person's anecdotal experience.
So whenever we hear, Don't knock it until you've tried it, we know a personal experience is about the worst way to learn about something, for all the reasons discussed above.
But even more than that, having tried it personally skews your ability to objectively interpret the data.
And this doesn't just go for products like new medical treatments, but also for experiences, like watching a psychic give supposed readings to audience members, or seeing an apparition believed to be a ghost.
In cases like these, the personal experience can be very powerful, especially for those not familiar with the underlying science, and can make it extraordinarily difficult to objectively understand what the data say.
If you truly want to learn about a phenomenon by following the scientific method, there are precious few cases where you should ever include a personal experience in your analysis.
I was a skeptic until I tried it.
No, the skeptic would have known not to try it for the reasons we just covered.
The scientifically literate skeptic would have chosen a far more disciplined protocol for learning about this thing.
Simply handing over money to obtain a worthless personal anecdote means that you are gullible, not skeptical.
I know it works because it worked for me.
This is perhaps the single most common misunderstanding, misinterpreting a personal experience as a universal one.
One person's subjective assessment of their own feeling at one moment certainly doesn't mean others will make the same assessment, or even that that same person would experience the same thing at a different time under different circumstances.
If we look at the 2016 Rio Olympics, many athletes from the USA, a world leader in unscientific alternative treatments, and even a few from other countries, proudly sported cupping bruises or brightly colored elastic kinesio tape.
Two alternative therapies completely lacking in either evidence or plausible theoretical foundation.
Yet despite a solid evidential basis showing these things do not help in any way, world-class athletes use them and their professional world-class trainers administer them.
Why?
Because their personal experiences told them they worked.
Sailing Into Skepticism 00:02:16
In the Olympics, an athlete is given excellent nutrition, regimented rest, massage, icing, and all manner of professional attention to every detail.
In such an environment, any non-functional addition to this regimen is going to be correlated with recovery and maximum performance.
Michael Phelps would have still won 23 gold medals if I danced around him and shook a rattle.
And if his trainer told him it was part of his therapy, he might have become a believer in that too.
Hey everyone, I want to remind you about a truly unique and once-in-a-lifetime adventure.
Join me and Mediterranean archaeologist Dr. Flint Dibble for a skeptoid sailing adventure through the Mediterranean Sea aboard the SV Royal Clipper, the world's largest full-rigged sailing ship.
This is also the only opportunity you'll have to hear Flint and I talk about our experiences when we both went on Joe Rogan to represent the causes of science and reality against whatever it is that you get when you're thrown into that lion pit.
We set sail from Malaga, Spain on April 18th, 2026 and finish the adventure in Nice, France on April 25th.
You'll enjoy a fascinating skeptical mini-conference at sea.
You'll visit amazing ports along the Spanish and French coasts and Flint will be our exclusive onboard expert sharing the real archaeology and history about every stop.
We've got special side quests and extra skeptical content planned at each port.
This is a true sailing ship.
You can climb the rat lines to the crow's nest, handle the sails.
You can even take the helm and steer.
This is a real bucket list adventure you don't want to miss.
But cabins are selling fast and this ship does always sell out.
Act now or you'll miss this once-in-a-lifetime opportunity.
Get the full details and book your cabin at skeptoid.com slash adventures.
Hope to see you on board.
That's skeptoid.com slash adventures.
Debunking the Closed Mind Myth 00:05:31
Crediting something with efficacy because it appeared to work when you tried it is a perfectly rational conclusion for an intelligent person to make.
It just happens to also be unscientific and no more likely to be true than your mood being the cause of today's weather.
Here's another example of a popular way that many people place personal experience above empirical evidence.
Have you ever heard a young Earth creationist ask of a scientist who says the Earth is billions of years old, Were you there?
The obvious implication being that if the scientist was not there personally observing the Earth a billion years ago, he couldn't know anything about it.
This also shows ignorance of how science works.
In fact, scientific conclusions are never based simply on personal accounts, but upon direct measurements of testable evidence.
Nobody's been to the sun either, but we know a great deal about it because we can directly measure and analyze the various types of radiation it puts out.
Challenging a scientist with, were you there, is like assessing his expertise on thermodynamics based on whether he's watched some YouTube video made by a perpetual motion crank.
The science of thermodynamics exists independently of YouTube.
The geological evidence of the age of the Earth exists independently of what observers were or weren't there.
Here's a question I get nearly every time I do an episode on some spin the wheel and invent a new alternative therapy.
What's the harm if it makes someone feel better?
The harm is that the new thing probably isn't what made them feel better.
But now that they think it is, they'll spend their time and money on it instead of on something that might actually help.
This is the basic danger of anecdotal thinking.
It encourages us to not only embrace the unreal, but to abandon the real.
I know what I saw.
No, you don't.
You know your brain's current interpretation of whatever part of the experience it abstracted and stored away.
We know for a fact that all our memories change dramatically over time and were incomplete to begin with.
And who knows how good was the data that your brain had to work from?
Lighting conditions came into play, perhaps movement, distractions, backgrounds, expectations of what should be seen, possible misidentifications, and perceptual errors all had a part in building your brain's experience.
Science conclusions are never based on a person's visual sighting.
It is anecdotal evidence, and the value of anecdotal evidence is to suggest a direction for research.
If you know for certain that you saw the Loch Ness monster in Urquahard Bay, apologies, but the science community is not going to call the mystery solved based on that alone.
However, it might make sense to then go look in Urquah Bay to see if we can find anything that can be collected and tested.
Then we'll know for sure what we've got, regardless of how close it may or may not match what you know you saw.
The scientific method's unwillingness to accept anecdotes as evidence often turns into a mudslinging contest of who's open-minded and who's closed-minded.
Thus, science is closed-minded.
You wouldn't try it because your mind is closed, or it didn't work for you because your mind is closed.
This popular charge against scientists has always boggled me.
Closed-mindedness is not just an unwillingness to consider new ideas.
It's more importantly, the stubborn refusal to change your mind no matter how much evidence piles up proving you wrong.
Open-mindedness should be the willingness to change your mind when you discover you're wrong.
Yet, take virtually any alternative belief that has mountains of absolutely conclusive disproof, like homeopathy or the flat earth or the vaccine autism link.
And adherents who reject the disproving evidence, shutting out all but their preferred belief, always accuse the science-based perspectives of being closed-minded.
It's a bizarre charge.
People who say this generally regard science not as a process, but as a rigid set of assertions, dogmatic edicts from on high, from which no departure is tolerated.
Or as one astrologist recently put it, knowledge rubber-stamped by some Orwellian ministry of truth.
It's a straw man caricature of science, of course.
It's hard to criticize a process that emphasizes thoroughness and verifiability.
But it's very easy to criticize an authoritarian set of doctrines.
Such a set would be closed-minded.
But a process that encourages constant change and improvement, like science, is by its very nature, open-minded.
It is a stubborn insistence on remaining open-minded that compels the followers of the scientific method to emphasize only the best data and to avoid our personal interpretations and preferences.
Science vs. Pseudoscience 00:01:59
This is why we do not try it before we knock it.
We'll test it before we knock it, or embrace it, or react however the test results prescribe.
But there's a huge difference between scientific testing and a personal dalliance.
Most assuredly, we need not personally sample a pseudoscience in order to form a well-informed opinion of it.
Become a supporter of the show to get your USB drive filled with all Skeptoid media content from the past decade and get a premium subscription to get the entire show ad-free.
You're listening to Skeptoid.
I'm Brian Dunning from Skeptoid.com.
Hello everyone, this is Adrienne Hill from Skookum Studios in Calgary, Canada, the land of maple syrup and mousse.
And I'm here to ask you to consider becoming a premium member of Skeptoid for as little as five US dollars per month.
And that's only the cost of a couple of Tim Horton's double doubles.
And that's Canadian for coffee with double cream and sugar.
Why support Skeptoid?
If you are like me and don't like ads, but like extended versions of each episode, Premium is for you.
If you want to support a worthwhile nonprofit that combats pseudoscience, promotes critical thinking, and provides free access to teachers to use the podcast in the classroom via the Teacher's Toolkit, then sign up today.
Remember that skepticism is the best medicine.
Next to giggling, of course.
Until next time, this is Adrienne Hill.
From PRX.
Export Selection