All Episodes
Previous Next
Plain Text
June 15, 2007 - Skeptoid
14:43
Skeptoid #50: How to Identify a "Good" Scientific Journal

Here is how to tell whose "scientific journal" article is actually more reputable. Learn about your ad choices: dovetail.prx.org/ad-choices

Transcriber: nvidia/parakeet-tdt-0.6b-v2, sat-12l-sm, and large-v3-turbo
|

Time Text
Evaluating Scientific Journal Quality 00:13:22
You'll find complete bibliographic references and further reading suggestions at the bottom of the webpage on every transcript for every Skeptoid episode.
When I hear from detractors, they often provide alternative references that they claim are just as valid.
Well, today, we're going to try and separate what makes a good scientific journal, references from which I would be glad to use, from a useless rag, references from which I would certainly not.
How to identify a good scientific journal is coming up right now on Skeptoid.
A quick reminder for everyone, you're listening to Skeptoid, revealing the true science and true history behind urban legends every week since 2006.
With over a thousand episodes, we're celebrating 20 years of keeping it focused and keeping it brief.
And we couldn't have done it without your curiosity leading the way.
And now we're even offering a little bit more.
If you become a premium member, supporting the show with a monthly micropayment of as little as $5, you get more Skeptoid.
The premium version of the show is not only ad-free, it has extended content.
These episodes are a few minutes longer.
We get rid of the ads and replace them with more Skeptoid.
The Extended Premium Show available now.
Come to skeptoid.com and click Go Premium.
You're listening to Skeptoid.
I'm Brian Dunning from Skeptoid.com.
How to identify a good scientific journal.
Today's episode is about a subject that's as old as debating itself, and it was prompted by the following email.
I would love to look a lot of this up on my own, but I'm unsure about what sources can be trusted.
I know you talked about how scientists are not created equal, but as an average person without the background to fully understand the primary sources or the ability to synthesize a consensus without reading meta-analyses, where can I go for reliable information?
This listener brings up a great point.
If you follow any of the conversations on the Skeptalk email discussion list, you've probably heard us banter this back and forth.
One guy says, hey, leprechauns are real.
Here's an article in a peer-reviewed scientific journal that says so.
And then someone else replies, no, they're not, because there haven't been any such articles in my peer-reviewed scientific journals.
Almost any debate can degrade into, my peer-reviewed scientific journal is better than yours.
Now, a really satisfying answer to this question would be, here, go to www.legitimatescientificjournals.com, and you can see at a glance if your source is a credible one.
Surely there must be some register like that, right?
I will dash your hopes with a simple answer.
No.
There is no such thing as an authoritative list of reputable scientific journals.
There can't be.
And the reason is that word, authoritative.
Who is qualified to be the authority?
No one is.
No one must be.
The moment that any one group is anointed with the ability to declare a source to be legitimate or not is the moment that we lose objectivity and impartiality.
It is very important to be aware that there is any number of bodies who do presume to be such an authority.
Approach with extreme skepticism.
The only reason anyone would compile such a list is to promote an agenda.
Someone once commented on one of my episodes and they tried to shoot me down by pointing out that one of my sources was discredited on a website called sourcewatch.org.
Sourcewatch.org sounds pretty legitimate.
Sounds like they do good work.
Sounds like they're out there looking out for our best interests by rubber stamping some sources and discrediting others.
But according to critics, SourceWatch is a two-man operation that endorses only publications following their own narrow political bias.
This is a perfect example of what you should expect from any source that attempts to identify itself as a rubber stamping authority.
Be skeptical of any group you find whose purpose is to identify reputable journals.
As long as we're throwing around the word reputable, I might as well give the somewhat disappointing answer to the listener's question and tell where you can find a reliable journal.
Scientific journals achieve their status only through long histories and good reputations.
To be broadly accepted within the mainstream scientific community, a journal must have established a long history of responsible reporting, good quality articles detailing well-performed research, and exhaustive peer review.
Long-standing reputation among the scientists who matter the most.
If you're not one of those scientists, it can be difficult to know which research is good, which editors and referees are good, and which journals have a long history of publishing them in good standing.
For this reputation to have any meaning, it must stand on its own and not be supported by appearing on some simple list.
Unfortunately, listener, you just have to know.
But I will give you a starting point in a moment.
While it is essential that good journals be peer-reviewed, you should be aware that almost every publication hoping for prominence describes itself as peer-reviewed.
When you hear someone defend their source by stating that it's peer-reviewed, be skeptical.
By itself, that's meaningless.
Think back to our old example of the guy writing a UFO newsletter in his basement who has a couple of his UFologist buddies endorse his writing.
Suddenly, he's peer-reviewed.
This is not the type of peer review that carries any meaning within the mainstream scientific community, since the peers have a clear agenda and have not established long histories of scientific acumen by the legitimate community at large.
This is an extreme example, but it does accurately portray a lot of what's out there.
When you don't know anything else about a journal, the fact that it calls itself peer-reviewed cannot, must not, be allowed to carry any weight.
In a world that can feel overwhelming, spreading thoughtful, evidence-based content is one of the best ways to make a positive impact.
Ask your local public radio station to air the Skeptoid Files, a 30-minute radio-friendly version of Skeptoid that pairs two related episodes promoting real science, true history, and critical thinking.
And in these challenging times for public media, we're offering these broadcasts for free to radio stations, available on the PRX Exchange or directly from Skeptoid Media.
It's an easy ask.
Just send a quick message to your station's programming director.
By helping to bring the Skeptoid files to the airwaves, you'll help promote the essential skills we all need to tell fact from fiction.
Just go to your local station's website, find the programming director's email address, or just their general email address.
You can even use the telephone.
I know that might sound crazy.
It's an old legacy device that allows real-time voice communication.
I know that's weird, but hey, it's an option.
The world can feel chaotic, but you're not powerless.
When you promote critical thinking, you can help your community tell fact from fiction.
And that's how we shape a better future.
In uncertain times, spreading good ideas can make you feel helpful, not helpless.
Let's stand up for reason, truth, and understanding.
Together.
Get them to air the Skeptoid files from Skeptoid Media, available on the PRX Exchange, and they'll know what that is.
One source that a lot of lay people are turning to these days is Wikipedia.
But what about Wikipedia?
It's new and it's a very different animal from anything previously available and is something of a paradigm shift.
Wikipedia is not perfect, but it is generally very good.
Its principal weakness, so often pointed out by critics, is also its principal strength.
Critics of Wikipedia love to charge that any old Joe Blow can go in there and edit any article to say whatever the heck he wants.
And this is true to a point, but they do have multiple layers of redundant checks and balances in place.
Every topic has editors, and every edit eventually makes it past several sets of eyeballs.
Every article is read by untold zillions of eyeballs and tempered with suggested edits by many of them.
Most of these suggestions are good, and some of them are bad.
The volunteer editors include some of the foremost authorities on the subject, and they include crank nitwits and everyone in between.
Wikipedia has tens of thousands of regular editors, over a thousand administrators who enforce the behavior of those editors, eventually weeding out the crank nitwits, and even a judicial committee which resolves any disputes that are not otherwise handled by the process.
The underlying software provides a powerful nerve center from which the editors and administrators can track history and changes.
This open source process leads to an inevitable effect.
Many Wikipedia articles end up being the closest thing to an authoritative consensus that we have on a given subject.
Each article continually improves over time until it becomes what Wikipedia describes as the ideal article.
Balanced, neutral, and encyclopedic, containing notable, verifiable knowledge.
When Wikipedia was first conceived, it was a brand new idea that had never been tried on such a scale.
No doubt it had plenty of growing pains.
But they've had years to improve the system.
They've been dragged through the media more than once over high-profile errors resulting from vandalism.
And every day since inception, they've worked to address those loopholes.
The process still isn't perfect, but it's one of the most amazing compendiums in human history.
So I'm going to give our listener a simple answer to a simple question.
Start with Wikipedia or any other encyclopedic resource with a good reputation like Britannica or MSN's Incarta.
Nearly always, good articles will include links to additional references, especially in Wikipedia.
But these links are of tremendously varying quality.
Be careful of their external links and carefully note why each external reference is being cited.
Good articles will often include a section on criticism or skepticism of the subject.
Read it.
Note that I'm no doubt going to be criticized for pointing lay people toward Wikipedia as a starting point for research, mainly due to the usual criticisms of Wikipedia.
But as I said before, Wikipedia's weakness is also its strength, and I do stand by this recommendation, especially for lay people of a given subject who don't otherwise have the experience to choose a good starting point.
What about identifying which scientific journals are reliable?
Since we're not all scientists in the chosen field with the education and experience to know which are the most reputable publications in our field, we need some kind of list.
But as we've discussed, lists are bad things when they come from a source with an agenda.
So we turn again to our source with no agenda, Wikipedia.
Search Wikipedia for list of scientific journals, and you'll find that they have a page listing a few hundred reputable journals in most scientific fields.
Generally, this is an excellent list.
The fact that it comes from Wikipedia and is constantly being revised for objectivity and quality by experts in each field is its strongest recommendation.
Some fields are not listed and most subsections are partial.
You can drill down to find more.
But beware, the further you drill down, the broader the quality control becomes and the more journals of lower repute are included and more non-scientific fields are listed.
If you use this list to gauge the reliability of a source that you see referenced, you are in as good of hands as are available to the inexperienced journeyman.
But you must use the list wisely.
Stay at the top level or as close to it as you can.
With each click that you drill down, reputation for the listed journals is generally lower.
Again, this recommendation will no doubt be criticized, and the criticism is generally valid.
But I maintain that for the average guy off the street, this is the best way to gain a good enough grasp of a journal's quality and to find good research that truly represents the current scientific consensus.
Supporting Skeptoid with Tim Horton's 00:01:16
You're listening to Skeptoid.
I'm Brian Dunning from Skeptoid.com.
Hello everyone, this is Adrienne Hill from Skookum Studios in Calgary, Canada, the land of maple syrup and mousse.
And I'm here to ask you to consider becoming a premium member of Skeptoid for as little as $5 per month.
And that's only the cost of a couple of Tim Horton's double doubles.
And that's Canadian for coffee with double cream and sugar.
Why support Skeptoid?
If you are like me and don't like ads, but like extended versions of each episode, Premium is for you.
If you want to support a worthwhile non-profit that combats pseudoscience, promotes critical thinking, and provides free access to teachers to use the podcast in the classroom via the Teacher's Toolkit, then sign up today.
Remember that skepticism is the best medicine.
Next to giggling, of course.
Until next time, this is Adrienne Hill.
From PRX.
Export Selection