All Episodes
Previous Next
Plain Text
Aug. 18, 2020 - Skeptoid
18:16
Skeptoid #741: The Wisdom and Stupidity of Crowds

How the wisdom of crowds can sometimes allow groups of non-experts to reach an expert conclusion. Learn about your ad choices: dovetail.prx.org/ad-choices

Transcriber: nvidia/parakeet-tdt-0.6b-v2, sat-12l-sm, and large-v3-turbo
|

Time Text
Wisdom of Crowds in Prediction Markets 00:09:50
The wisdom of crowds is an idea backed by experimental validation that a big group of random people can collectively guess at something and do better than trained experts.
A very frustrating proposition for those experts.
But while this concept sounds like it might be the miracle solution everyone's looking for to solve any complicated problem, the reality is very different.
The wisdom of crowds is coming right up on Skeptoid.
Hi, I'm Alex Goldman.
You may know me as the host of Reply All, but I'm done with that.
I'm doing something else now.
I've started a new podcast called Hyperfixed.
On every episode of Hyperfixed, listeners write in with their problems and I try to solve them.
Some massive and life-altering, and some so minuscule it'll boggle your mind.
No matter the problem, no matter the size, I'm here for you.
That's Hyperfixed, the new podcast from Radiotopia.
Find it wherever you listen to podcasts or at hyperfixedpod.com.
You're listening to Skeptoid.
I'm Brian Dunning from skeptoid.com.
The wisdom and stupidity of crowds.
There's an old saying that a person is smart, people are stupid.
And while we might see examples that seem to support that frequently, it turns out that under certain conditions, the reverse can be true.
Surprisingly, crowds consisting of non-experts who have no tools at their disposal other than guesswork can come up with a better answer to certain questions than even an expert in that field when certain conditions are satisfied.
Today we're going to look at how and why this works and also how and when you should and shouldn't rely on it.
The actual term wisdom of crowds comes from the title of the 2004 book The Wisdom of Crowds, Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations by journalist James Surawicki.
He opened the book by telling a famous anecdote.
It's one of the best-known stories about the British polymath Francis Galton.
In 1906, Galton was at a livestock fair where he encountered a very popular type of contest being run.
Guess the weight of this ox, and you'd win a cash prize.
Some 800 people submitted their guesses, and Galton noted that the group was quite diverse.
People who might reasonably know what an ox weighed, such as butchers and farmers, were certainly there.
But also present were plenty of people who would have no idea and who made guesses based on their own assorted experiences of what things weighed.
The ox turned out to weigh 1,198 pounds, and the person with the closest guess was awarded the prize.
Galton obtained the list of guesses and got to work, expecting to find that the average person had no idea what an ox weighed.
But when he took the median guess, he found that it was 1,208 pounds, wrong by less than 1%.
In fact, it was later discovered that Galton had made an error, and the actual median was 1,207, even closer to the actual weight.
But the real surprise was when he later took the average of all the guesses.
1,197 pounds.
Only one pound off.
The diverse crowd was smarter than even the very best expert guess.
Galton published this account the following year in the journal Nature under the title Vox Populi, Latin for the voice of the people.
And with that 1907 article, the wisdom of crowds was born.
This basic pattern of Galton's Vox Populi turns out to be surprisingly repeatable.
One application where we see the wisdom of crowds borne out time and time again is in prediction markets.
These are just like stock markets, but instead of trading stocks, you trade probabilities of future events.
You may remember the short-lived so-called terrorism futures market set up by the U.S. government in an attempt to learn what kind of defense-related events were likely to take place in the world.
But it was politically unpalatable and was shut down.
Its actual name was the Policy Analysis Market.
But many other prediction markets are around now, and the stock prices, or the probabilities being traded, turn out to be eerily accurate at predicting world events like election outcomes, currency trends, government policy decisions, and much more.
The ability to make actual money trading these probabilities keeps people coming, and they are as diverse as Galton's ox weight diviners.
Some may have actual knowledge of these events, most have somewhat informed guesses, and some are purely speculating based on general life experience.
Although individually they may buy and sell all over the map, in the aggregate, these crowds turn out to be wise too.
Why?
Sorowicki proposed a number of characteristics of a wise crowd that separate it from an unwise crowd.
We're going to focus on two of the most important.
First, diversity is key.
Diversity of background, experience, opinion, including having people with little knowledge of the question being explored.
Second, the crowd must remain independent.
They have to come up with their own independent estimates and not copy from those around them.
This independence is crucial.
If someone at the Ox contest had shouted out a guess, some of the people who had no idea whatsoever would have gone with that, as human nature compels us to presume, rightly or wrongly, that such a person probably knows better than we do.
Had that happened, it would have dramatically skewed the guesses to one side or other of the actual weight, and resulted in an average that was more wrong than the average that Galton actually got.
This particular type of breakdown in the process is what economists call an information cascade.
We discussed these in skeptoid number 676 on glyphosate, the active ingredient in Roundup.
Glyphosate is one of the safest and most effective herbicides ever developed, and yet today, a huge number of people believe that it is poisonous or carcinogenic, claims that are at odds with all the scientific data.
The process by which this misinformation spread was a great example of an information cascade.
This is when one person, faced with a decision that's beyond their ability to make based on their knowledge, instead simply adopts the position they've observed someone else take.
The average person lacks the expertise to make an informed decision about whether Roundup is safe or not, and so they simply go with whatever their friend says, exactly like how the man at the ox contest who shouted out a weight triggers many others in the crowd to adopt his guess.
Information cascades are like dominoes toppling over in a great big expanding fan.
When an information cascade takes place, both the diversity of the crowd and the independence of its members are thrown out the window.
People are simply copying from each other, they're no longer independent, and whatever value their diversity may have brought is lost.
So this crowd turns out to be a stupid one, following along with whatever random choice one member made once.
Hey everyone, I want to remind you about a truly unique and once-in-a-lifetime adventure.
Join me and Mediterranean archaeologist Dr. Flint Dibble for a skeptoid sailing adventure through the Mediterranean Sea aboard the SV Royal Clipper, the world's largest full-rigged sailing ship.
This is also the only opportunity you'll have to hear Flint and I talk about our experiences when we both went on Joe Rogan to represent the causes of science and reality against whatever it is that you get when you're thrown into that lion pit.
We set sail from Malaga, Spain on April 18th, 2026 and finish the adventure in Nice, France on April 25th.
You'll enjoy a fascinating skeptical mini-conference at sea.
You'll visit amazing ports along the Spanish and French coasts and Flint will be our exclusive onboard expert sharing the real archaeology and history about every stop.
We've got special side quests and extra skeptical content planned at each port.
This is a true sailing ship.
You can climb the rat lines to the crow's nest, handle the sails.
You can even take the helm and steer.
This is a real bucket list adventure you don't want to miss.
But cabins are selling fast and this ship does always sell out.
Act now or you'll miss this once-in-a-lifetime opportunity.
Get the full details and book your cabin at skeptoid.com slash adventures.
Hope to see you on board.
That's skeptoid.com slash adventures.
When Crowd Wisdom Fails Experts 00:05:52
These pitfalls become more myriad in prediction markets.
Take the example of people betting on the outcomes of a political election.
All else being equal, the bettors generally have excellent diversity of opinion that works to its favor.
But with an election that's often mentioned in the news, polling results may be released.
Candidates may perform well or badly in primaries or in debates.
This news can skew opinion.
News sources and popular pundits can also spread biased information or can make influential predictions of varying quality.
And these can reduce both the diversity of opinion among the bettors and their independence from one another.
In the best of circumstances, again using Galton's Ox contest as an example, a crowd consisting largely of non-experts can often make a prediction that is more accurate than any of the actual experts in the group can come up with.
I have seen this fact spun and misrepresented by certain groups and individuals as evidence supporting their belief that a lack of expertise is just as valid as actual expertise, if not more so.
This can be irrationally applied to issues like vaccination and climate change.
So, fair warning, that the wisdom of crowds gives accurate predictions in certain specific circumstances is not to be misconstrued as lay people are smarter than experts in their own field.
In a wise crowd, a non-expert's own prediction is almost always very wrong.
It is only an aggregate of many such wrong predictions from not only non-experts but actual experts as well that can hope to match or beat the accuracy of expert opinion.
So please don't misunderstand this phenomenon to mean that your own preferred stance on vaccines or climate change is wiser than what the experts in those fields have concluded.
There is one more case I'd like to cover.
If you saw the title of this episode and have read or listened to anything on the wisdom of crowds before, you are almost certainly expecting me to cover the 1968 search for the lost nuclear submarine USS Scorpion.
This ocean drama is one of the most popularly cited examples of a real-world problem being solved by the wisdom of crowds.
The way the story is usually told, Dr. John Craven, the engineer put in charge of finding the wreck, took the few facts they had and asked a diverse group of people to guess the sub's location on a bathymetric map.
He then averaged the locations and miraculously discovered the wreck only 220 yards from the predicted spot.
If that makes the wisdom of crowds sound a little too magical and a little too good to be true, well, you're onto something, because it turns out that this story is usually terribly and inaccurately reported.
The reality of Craven's search had little to do with the wisdom of crowds.
Craven had pounded the pavement to collect as much sonar data as he could from military and private sources and discovered that the sub had made a strange 180 degree turn before imploding.
Experienced submariners advised him that this was a standard procedure to disarm an accidentally armed torpedo, an explanation that the Navy doubted.
Craven then assembled a small team of submarine and salvage experts to bet on the likelihood of this scenario, using bottles of Shiva's Regal whiskey, as per their tradition.
Based on most of the group betting on the accidentally armed torpedo scenario, Craven decreed it to be the assumption.
So for round two, they started from there and then bet on the speed at which the sub struck the bottom.
They took the most popularly bet speed and made that the second assumption.
Then for round three, they bet on the slope of the sub's glide path downward.
With these figures in hand, Craven hired Daniel H. Wagner Associates, mathematicians, to help him devise probabilities for various points on the ocean floor.
It was the same firm he'd used two years earlier to help him find a lost hydrogen bomb.
When the search ship went out, Craven and his mathematicians used Bayesian search theory to constantly refine their probabilities.
And when the sub was ultimately found, it was a mere 220 yards from where the model said was most likely.
The bottles of scotch were duly awarded to the winning experts.
Why was this not a very good case of the wisdom of crowds?
Mainly because Craven's Scotch betting experts were well-informed, not diversely so.
They were not independent of one another's guesses, and because the final most probable crash site was the product of Bayesian statistics refined by actual searching, not the result of guesses by members of a large crowd.
In fact, a few years later, one of the mathematicians Craven had hired, Lawrence Stone, would write a book about their methodology titled Theory of Optimal Search.
It's heavy on math, light on Vox populi.
But for cases that do conform to Surowicki's criteria, the wisdom of crowds turns out to be a real and fascinating phenomenon, if not a universally applicable one.
But if you've got a problem that has an unknown but simple and quantifiable solution, you could do a lot worse than collecting a bunch of randos off the street and extracting a guess from each one.
Skeptoid Supporters and Curiosity Principles 00:02:30
This episode is dedicated to the memory of Skeptoid family member Allison Hudson.
Allison wrote and guest-hosted several episodes of Skeptoid back in 2015.
She was a prolific author on the Skeptoid blog.
and as a career educator also wrote the outstanding free lesson plans provided with our film Principles of Curiosity.
Her talent, her humor, and her friendship will be sorely missed.
A shout out to Skeptoid Premium members Gail Knapp, Abra, Josh Koch, and Mad Brad, Virginia's only skeptic.
Thanks to you all so much for being supporting members in keeping Skeptoid afloat and making it possible for us to produce projects like our 2017 film Principles of Curiosity, for which Allison Hudson did such great work.
Please do check it out.
Enjoy it, share it, and have fun with the journey it takes you on from the California deserts all the way up into space.
It's free at principlesofcuriosity.com.
You're listening to Skeptoid, a listener-supported program.
I'm Brian Dunning from skeptoid.com.
Hello, everyone.
This is Adrienne Hill from Skookum Studios in Calgary, Canada, the land of maple syrup and mousse.
And I'm here to ask you to consider becoming a premium member of Skeptoid for as little as $5 per month.
And that's only the cost of a couple of Tim Horton's double-doubles.
And that's Canadian for coffee with double cream and sugar.
Why support Skeptoid?
If you are like me and don't like ads, but like extended versions of each episode, Premium is for you.
If you want to support a worthwhile nonprofit that combats pseudoscience, promotes critical thinking, and provides free access to teachers to use the podcast in the classroom via the Teacher's Toolkit, then sign up today.
Remember that skepticism is the best medicine.
Next to giggling, of course.
Until next time, this is Adrienne Hill.
From PRX.
Export Selection