All Episodes
Previous Next
Plain Text
Dec. 11, 2006 - Skeptoid
13:29
Skeptoid #13: A Primer on Scientific Testing

Understand the basics of scientific testing. Learn about your ad choices: dovetail.prx.org/ad-choices

Transcriber: nvidia/parakeet-tdt-0.6b-v2, sat-12l-sm, and large-v3-turbo
|

Time Text
The Science of Blinding 00:08:08
In a world where we're always hearing about studies and clinical trials and published research papers, we often find ourselves mired in a sea of science-y-sounding buzzwords describing the many different types of scientific studies.
Today we're going to boil it all down and lay out what the basic study types are and how they work.
A primer on scientific testing is coming up next on Skeptoid.
Hi, I'm Alex Goldman.
You may know me as the host of Reply All, but I'm done with that.
I'm doing something else now.
I've started a new podcast called Hyperfixed.
On every episode of Hyperfixed, listeners write in with their problems and I try to solve them.
Some massive and life-altering, and some so minuscule it'll boggle your mind.
No matter the problem, no matter the size, I'm here for you.
That's Hyperfixed, the new podcast from Radiotopia.
Find it wherever you listen to podcasts or at hyperfixedpod.com.
You're listening to Skeptoid.
I'm Brian Dunning from Skeptoid.com.
A primer on scientific testing.
Today we're going to hide our crib notes in our hats, pull our sleeve down to cover the notes written on our arms, and dive into the world of testing.
Much of the feedback I received on the wheatgrass juice episode concerned claims that wheatgrass juice has already been tested and been proven to cure many different diseases and promote many types of well-being.
And if I had a dollar for every email I received accusing me of being in the employ of big evil corporations who are frightened of wheatgrass juice, I would be able to afford a shot of this quack beverage every single day.
But has it truly been tested, according to what scientific testing really means?
So I hereby present this primer on scientific testing to better equip the layperson with the ability to determine the validity of claims being made or the published results of supposed research.
Valid claims and real research will follow the whole process that I'm about to outline and they'll tell you about it too.
If the poster you read in Jamba Juice doesn't detail the testing procedure used to substantiate its claims or if the testing procedure is not similar to that outlined herein, you have very good cause to be skeptical of any claims that it makes.
If something works, its makers should be happy to prove it to you.
Testing of something in medicine, for example, is done by what we in the Brotherhood call a clinical trial, more formally known as a randomized controlled trial.
The same general principles apply to any kind of scientific testing.
The aspect of randomization refers to the random distribution of subjects into similarly sized groups.
When done thoroughly and responsibly, statistical methods are used to remove any sort of bias for the assignment of subjects and to ensure that the assignments are not known to the participants or the administrators.
Make no mistake, even this apparently simple first step of testing is a thorough one, and it's this kind of comprehensive attention to detail that separates a real test from the typical anecdotal testing claimed by supporters of most pseudoscientific phenomena.
One of the most important characteristics of a valid test is the control.
Let's say your wrist hurts and you try acupuncture and your wrist feels better.
You're likely to consider that you've just tested acupuncture and it worked, thus proving its efficacy.
But in fact, this was not a valid test because there was no control.
Your wrist may have healed naturally.
Your wrist may have been healed by a psychic in the next room.
There is no way to know what effect, if any, the acupuncture had.
It may have even slowed the healing for all you really know.
The most basic kind of control would have been to have at least two people with similar injuries where one received the acupuncture and the other received a control procedure, and all else would have had to be equal.
Even if you take an aspirin for a headache and someone asks you how it worked, you truthfully can't say, ever, because you had no alternate version of yourself who didn't take the aspirin as a control to compare against.
With a control, you have the beginnings of a valid test.
Blinding is another fundamental of trials.
Blinding means keeping the test participants in the blind.
If people know what they're being given, know what results they're expected to report, or know what kind of result to look for, the results are untrustworthy.
Everyone is a human being, and if you're not blinded, you may unknowingly skew the results, or you may have opportunity to wield some agenda that you might have.
Everyone has expectations, preconceived notions, biases, and perceptual errors.
Researchers know this, and they know unblinded subjects to be virtually useless in most kinds of tests.
Blinding can be single, double, or even triple.
In a single blind test, the participants in the experiment don't know any information that might skew the results.
If they're testing a drug taken orally, the participants must not know whether they're taking the real drug or the control placebo.
If they're receiving acupuncture, they must not know whether they're receiving traditional acupuncture or sham acupuncture.
So the experience would have to be designed to be the same for both groups.
For example, having the needles in sleeves that make it impossible to tell whether they were just poked or if the needle actually went in.
If they're taking wheatgrass juice, they must not be able to tell whether they're drinking real wheatgrass juice or a placebo.
So it would have to be administered in some form where they couldn't tell.
The purpose of blinding the participants is to prevent them from either knowingly or unknowingly manipulating the results of the test by reporting or reacting differently.
Hey everyone, I want to remind you about a truly unique and once-in-a-lifetime adventure.
Join me and Mediterranean archaeologist Dr. Flint Dibble for a skeptoid sailing adventure through the Mediterranean Sea aboard the SV Royal Clipper, the world's largest full-rigged sailing ship.
This is also the only opportunity you'll have to hear Flint and I talk about our experiences when we both went on Joe Rogan to represent the causes of science and reality against whatever it is that you get when you're thrown into that lion pit.
We set sail from Málaga, Spain on April 18th, 2026 and finish the adventure in Nice, France on April 25th.
You'll enjoy a fascinating skeptical mini-conference at sea.
You'll visit amazing ports along the Spanish and French coasts and Flint will be our exclusive onboard expert sharing the real archaeology and history about every stop.
We've got special side quests and extra skeptical content planned at each port.
This is a true sailing ship.
You can climb the rat lines to the crow's nest, handle the sails.
You can even take the helm and steer.
This is a real bucket list adventure you don't want to miss.
But cabins are selling fast and this ship does always sell out.
Act now or you'll miss this once-in-a-lifetime opportunity.
Get the full details and book your cabin at skeptoid.com slash adventures.
Hope to see you on board at skeptoid.com slash adventures.
Single-blind tests are good, but double-blind tests are better.
Navigating Peer Review 00:03:17
In a double-blind test, neither the subjects nor the people administering the tests know what group any given subject is in.
They also don't know whether they're giving the real substance being tested or a placebo.
A double-blind test removes the chance that a test administrator might skew the results by acting differently, either knowingly or unknowingly, and thus providing information to the test subject.
Triple-blind tests take it to the furthest extreme.
A triple-blind test is just like a double-blind test, but with the additional element of the statisticians also being blinded.
For the people tabulating and analyzing the results of the test to be blinded, the data is presented to them in a coded form so that they're not able to know anything about any given subject or administrator.
They'll see data like subject A was given substance B by Administrator C and had a 13% improvement.
They don't know if subject A was in a control group or a test group, and they don't know what substance B is, and they don't know who Administrator C is.
In this way, they're able to present detailed results of the test that are completely unbiased, because even the statisticians themselves don't know what the data mean.
Once your testing is done, your results are ready for publication.
If you want your report to be taken seriously, it needs to be subjected to and survive the process of peer review.
Peer review means having your research submitted to experts in the field.
So who are these experts and who chooses them?
That usually depends on who's publishing the research.
If it's a scientific journal, the editorial staff will usually maintain a stable of referees in the community.
These referees are typically well-regarded professionals in the field, all of whom have been through this ringer themselves and been published.
If your research was responsibly conducted and your conclusions are well supported by the evidence, then the referees will typically give it a passing grade for publication.
Let's say a UFO researcher writes a paper that says UFOs come from another dimension and he has some of his fellow UFologists, whom he considers his peers, to endorse his paper.
Does that make it peer-reviewed?
No, because he chose the referees himself.
What if the editor of an underground UFO pamphlet chooses a panel of UFologists who endorse the paper?
Does that make it peer-reviewed?
No, because these referees are clearly biased, and their scientific acumen would not survive any type of scrutiny from the general scientific community.
Typically, the publication must be one with a long-standing reputation and strict requirement of thorough peer review.
The process of peer review is not perfect, as it relies on individuals who, though they've been scrutinized by a committee themselves, are still human beings who can make mistakes and get lazy, have agendas, or just bad hair days.
But peer review succeeds far more often than it fails.
And if you want anyone to take your research seriously, it must be peer-reviewed.
Support Skeptoid Today 00:01:59
So that's the basics of it.
Good experimental design, proper handling of data, and a thorough and transparent independent review process.
Remember, articles that report reliable results will always detail the testing that was done and the methods used.
If the claim is far-fetched and the supporting documentation of testing that the claimants are willing to share is inadequate, you have very good reason to be skeptical.
Follow me on Twitter at Brian Dunning and find both Skeptoid Podcast and myself on Facebook.
You're listening to Skeptoid, a listener-supported program.
I'm Brian Dunning from Skeptoid.com.
Hello everyone, this is Adrian Hill from Skookum Studios in Calgary, Canada, the land of maple syrup and mousse.
And I'm here to ask you to consider becoming a premium member of Skeptoid for as little as five US dollars per month.
And that's only the cost of a couple of Tim Horton's double doubles.
And that's Canadian for coffee with double cream and sugar.
Why support Skeptoid?
If you are like me and don't like ads, but like extended versions of each episode, Premium is for you.
If you want to support a worthwhile nonprofit that combats pseudoscience, promotes critical thinking, and provides free access to teachers to use the podcast in the classroom via the Teacher's Toolkit, then sign up today.
Remember that skepticism is the best medicine.
Next to giggling, of course.
Until next time, this is Adrienne Hill.
From PRX.
Export Selection