Skeptoid - Skeptoid #381: An Enthusiast's Primer on Study Types
Aired: 2013-09-24
Duration: 16:25


=== A Primer on Study Types (01:38) ===

[00:00:03]  Here on Skeptoid, as you know, we back everything up with published scientific studies, whenever they're available, and they're listed on the transcript pages on the website.
[00:00:13]  But not everyone's familiar with scientific studies, and so today we've got a really fun little primer on what the basic study types are that you're likely to encounter, and how to interpret them.
[00:00:26]  That's today, on Skeptoid.
[00:00:32]  Hi, I'm Alex Goldman.
[00:00:34]  You may know me as the host of Reply All, but I'm done with that.
[00:00:38]  I'm doing something else now.
[00:00:40]  I've started a new podcast called Hyperfixed.
[00:00:43]  On every episode of Hyperfixed, listeners write in with their problems, and I try to solve them.
[00:00:47]  Some massive and life-altering, and some so minuscule it'll boggle your mind.
[00:00:52]  No matter the problem, no matter the size, I'm here for you.
[00:00:55]  That's HyperFixed, the new podcast from Radiotopia.
[00:00:58]  Find it wherever you listen to podcasts or at hyperfixedpod.com.
[00:01:08]  You're listening to Skeptoid.
[00:01:10]  I'm Brian Dunning from Skeptoid.com, an enthusiast's primer on study types.
[00:01:17]  We don't all work professionally in the hard sciences.
[00:01:20]  In fact, most of us don't.
[00:01:22]  Yet those of us who are fans of science and its applications are often confronted with evidence and articles citing studies and other research, and we can sometimes feel at a bit of a loss to understand some of the lingo.
[00:01:35]  One thing I've had to do a lot of in my years as Skeptoid is reading studies.
[00:01:39]  There are all kinds of different studies.

=== Navigating the Sea of Studies (12:52) ===

[00:01:42]  Cohort studies, controlled trials, uncontrolled trials, longitudinal studies, retrospective studies, and many more.
[00:01:50]  The uninitiated student of studies quickly becomes awash in a sea of study types.
[00:01:58]  Today we're going to take a quick helicopter view of study types.
[00:02:02]  As with most lists, this one is not exhaustive, but it does cover the basics that you're ever likely to encounter in your reading.
[00:02:09]  People have actually published taxonomies of study types.
[00:02:13]  The subject can get that complicated.
[00:02:15]  Just about every type of study has various subtypes, each of which may have its own variants, and so on.
[00:02:22]  And because this subject has the appearance of potentially being incredibly boring, we shall use the example of aliens from planet zero studying the earth.
[00:02:32]  In broad strokes, there are two basic kinds of studies, observational and experimental.
[00:02:38]  In an observational study, the zeroids just watch the earthlings to see what happens without intervening in anything.
[00:02:45]  In an experimental study, they watch to see the results of an intervention, such as medical probing or planetary destruction.
[00:02:54]  Experimental and observational studies are sometimes called interventional and non-interventional.
[00:03:04]  Observational studies.
[00:03:07]  A common observational study is the cohort study.
[00:03:14]  Cohorts refer to a group of people who have some trait in common and are thus chosen for observation.
[00:03:20]  The zeroids may be interested in studying a cohort of earthlings consisting of astronauts.
[00:03:26]  They may choose this particular group because they might, for example, be interested in seeing how the astronauts respond to the increased radiation of the space environment.
[00:03:35]  Since they choose the group first and then sit back to watch what happens to them, this would be a prospective cohort study, which means they're going to watch the cohorts in real time with no idea of what might happen to them in the future.
[00:03:53]  Another way to go would be if they beamed down into the Earth astronaut headquarters and secretly stole the record tapes showing archival history of astronauts and how well they responded to radiation.
[00:04:04]  Then it would be a retrospective cohort study, where the outcomes are already known, and it's a case of working backwards to see their histories.
[00:04:16]  If the Planet Zero scientists wish to compare the astronauts to another group, for example, people in Ecuador who are well shielded from radiation by the atmosphere, they might choose to make it a concurrent cohort study, which is when two or more groups of cohorts are studied to compare their results to one another.
[00:04:38]  This might allow the zeroids to learn that people in Ecuador suffer fewer ill effects from radiation than astronauts.
[00:04:46]  Now suppose that one of the Planet Zero researchers points out that evolution in spaceship design over the decades might be giving the astronauts more or less protection.
[00:04:56]  They might elect to perform a non-concurrent cohort study, in which they look at astronaut data from their stolen tapes to compare astronauts from the 1970s with astronauts from the 2000s.
[00:05:12]  As it may turn out, the zeroids might discover that some humans spontaneously sublimate from a solid to a gaseous cloud of molecules, a phenomenon that they've seen before on some of their other planetary travels.
[00:05:24]  They might like to find out the cause of this, because the ability to induce such a reaction might come in very handy for their invasion plans.
[00:05:32]  So the zeroids might well take one cohort of humans who spontaneously sublimated and a second cohort who did not.
[00:05:41]  Then they would perform a case control study, in which they'd analyze the histories of the two cohorts to see what the sublimators had in common that the non-sublimators did not.
[00:05:55]  To improve the quality of their results, they might choose to make it a nested case control study, where a particular variable is added to focus both cohorts.
[00:06:08]  Perhaps the invasion coordinators are most concerned with guys from Montana, so to make the study more efficient for their purpose, they don't bother to study dancers from Ireland or weavers from India, but narrow both groups to just the target variables.
[00:06:25]  There are two final types of observational studies that we're going to gloss over superficially, as we're doing with all of these.
[00:06:32]  They have to do with time.
[00:06:34]  The ship from Planet Zero may have limited time or lots of time, or they may be interested in one type of data or another.
[00:06:41]  These two form acrosshairs in time.
[00:06:48]  A cross-sectional study looks at only a single snapshot in time across all subjects.
[00:06:54]  This would give the zeroids a data set showing how many humans are astronauts at the given moment, how many are from Ecuador, how many are not yet sublimated, and how much radiation each shows signs of having been exposed to.
[00:07:08]  Perpendicular in time to the cross-sectional study is the longitudinal study, in which multiple observations are made over time of the same subject or subjects.
[00:07:23]  By running all of these different observational studies, the zeroids were able to make and confirm a number of conclusions.
[00:07:29]  It turned out that being an astronaut and being exposed to extraterrestrial radiation, especially in the earlier less protected space capsule designs, greatly increase humans' chances of suddenly bursting into disassociated molecules at some future point in their life.
[00:07:45]  But what nobody knew was that as the zeroids were making these quiet, non-interventional observational studies, another more terrible species was making experimental studies.
[00:07:56]  They were the Nulloton.
[00:08:02]  In a world that can feel overwhelming, spreading thoughtful, evidence-based content is one of the best ways to make a positive impact.
[00:08:10]  Ask your local public radio station to air the Skeptoid Files, a 30-minute radio-friendly version of Skeptoid that pairs two related episodes promoting real science, true history, and critical thinking.
[00:08:23]  And in these challenging times for public media, we're offering these broadcasts for free to radio stations, available on the PRX Exchange or directly from Skeptoid Media.
[00:08:34]  It's an easy ask.
[00:08:36]  Just send a quick message to your station's programming director.
[00:08:39]  By helping to bring the Skeptoid files to the airwaves, you'll help promote the essential skills we all need to tell fact from fiction.
[00:08:47]  Just go to your local station's website, find the programming director's email address, or just their general email address.
[00:08:53]  You can even use the telephone.
[00:08:56]  I know that might sound crazy.
[00:08:57]  It's an old legacy device that allows real-time voice communication.
[00:09:02]  I know that's weird, but hey, it's an option.
[00:09:05]  The world can feel chaotic, but you're not powerless.
[00:09:08]  When you promote critical thinking, you can help your community tell fact from fiction.
[00:09:13]  And that's how we shape a better future.
[00:09:15]  In uncertain times, spreading good ideas can make you feel helpful, not helpless.
[00:09:22]  Let's stand up for reason, truth, and understanding.
[00:09:25]  Together, get them to air the Skeptoid files from Skeptoid Media, available on the PRX Exchange, and they'll know what that is.
[00:09:43]  Experimental studies, in which we don't merely observe, we intervene to see what happens.
[00:09:50]  The Nulliton were far less patient than the Xeroids.
[00:09:53]  They were not content to merely observe, and their quasi-dimensional starships allowed them to bring more resources to their study.
[00:10:01]  They had the means and the desire to perform direct interventions, not only on cattle, but on humans as well.
[00:10:10]  The Nulloton wanted a quick and easy way to destroy all life on Earth.
[00:10:14]  Based on their many experiences on other worlds, there were in a number of ways to accomplish this.
[00:10:19]  Weapons could be used, but that's expensive and time-consuming, even with their newest Mobius matter translators.
[00:10:26]  Better if it were possible to catalyze natural processes that may produce conditions under which life could not endure.
[00:10:34]  To find out which would work best on humans, they contrived to test.
[00:10:40]  The most familiar experimental study, and the gold standard, is the randomized controlled trial.
[00:10:49]  In this, the Nulloton gathered humans from all over the Earth, selected randomly, and brought them into their saucer's lab by lifting them in a bright beam of light.
[00:10:57]  They decided to create three test groups, humans left alone, humans warped into non-existence with the matter translator, and humans exposed to conditions similar to a supernova.
[00:11:08]  Subjects were randomly assigned into one of the three groups, and everything was blinded so that the statisticians didn't know anything about each subject's age, background, etc.
[00:11:19]  A variation on this might be the non-randomized comparative trial.
[00:11:27]  This is a way to see what happens when multiple interventions are performed, and the idea is to compare the results of one intervention with subsequent ones.
[00:11:36]  The first trial is likely to be a randomized controlled trial, and then follow-up interventions would be non-randomized comparatives.
[00:11:46]  These nulloton elected to perform a slight variation on the gold standard.
[00:11:53]  The before and after study, in which the precise condition of each human subject was measured before and after the intervention.
[00:12:02]  In a regular controlled trial, some of the humans may have already been sick or about to explode for some other reason.
[00:12:08]  The before and after study allowed the nulloton to better measure exactly how much damage was inflicted by their experiment itself.
[00:12:16]  If one particular subject had zero damage before and another had plus five damage, then having performed the before and after, let the nulloton know how much damage was pre-existing.
[00:12:28]  The study could have been controlled or uncontrolled.
[00:12:38]  The xeroids had found that astronauts were more likely to be damaged than Ecuadorians.
[00:12:44]  The Nulloton could have controlled for this factor by counting astronauts and Ecuadorians separately, or their analysis could have been uncontrolled for this variable, by lumping everyone into a group and only looking at the cumulative data.
[00:12:57]  Either a controlled or uncontrolled study might have been best for the Nulloton, depending on whether they intended to try and inflict extra damage on those who were least susceptible, or whether they wanted a single attack type to take care of everyone.
[00:13:12]  Using historical controls would have been another option.
[00:13:20]  Say the Nulloton had discovered through observation that astronauts typically explode one-tenth of the time after a spaceflight.
[00:13:27]  They can use this as a historical control, against which they can compare their experimental group, all of whom get exposed to the matter translator, in which 90% of astronauts explode.
[00:13:39]  Since they have the historical data, there's no need to run a current control group.
[00:13:45]  They could also run this using external controls, which is the employee of controls that are outside of the study.
[00:13:55]  For example, they could look at astronauts currently flying on missions and watching how many of them explode afterward.
[00:14:01]  Their choice might be based on the availability of the historical records, or based on the expense and resources needed to make the external observations, but either way, that control group is compared to their experimental group.
[00:14:14]  Either way, the Nulloton discover that their weapons systems should have no trouble disabling the bulk of the human race.
[00:14:21]  But anyway, what ended up happening is that the Xeroids and the Nulloton both arrived at about the same time from opposite sides of the Earth, and as soon as they detected one another, they opened fire and fell victim to mutually assured destruction.

=== Listener Supported Skeptoid (01:50) ===

[00:14:34]  The few survivors who floated to Earth in their escape capsules quickly died of viral infections, and thus the Earth was saved.
[00:14:43]  It all goes to show that even with the best study design, there's no guarantee that your intervention is going to work, or that your observational studies will end up matching the eventual outcome.
[00:14:55]  Had either the Zeroids or the Nulloton listened to Skeptoid, both might have been saved.
[00:15:06]  You're listening to Skeptoid, a listener-supported program.
[00:15:10]  I'm Brian Dunning from Skeptoid.com.
[00:15:18]  Hello everyone, this is Adrian Hill from Skookum Studios in Calgary, Canada, the land of maple syrup and moose.
[00:15:28]  And I'm here to ask you to consider becoming a premium member of Skeptoid for as little as $5 per month.
[00:15:36]  And that's only the cost of a couple of Tim Horton's double doubles.
[00:15:40]  And that's Canadian for coffee with double cream and sugar.
[00:15:45]  Why support Skeptoid?
[00:15:47]  If you are like me and don't like ads, but like extended versions of each episode, Premium is for you.
[00:15:53]  If you want to support a worthwhile nonprofit that combats pseudoscience, promotes critical thinking, and provides free access to teachers to use the podcast in the classroom via the Teacher's Toolkit, then sign up today.
[00:16:07]  Remember that skepticism is the best medicine.
[00:16:11]  Next to giggling, of course.
[00:16:13]  Until next time, this is Adrienne Hill.
[00:16:24]  From PRX