Autism, Microbiomes, & Mice Burying Marbles with Kevin Mitchell
This week, we are joined by Kevin Mitchell, Associate Professor of Genetics and Neuroscience at Trinity College Dublin, who has committed the unforgivable sin of pointing out that an entire academic and media hype cycle might be built on… well, very little actually. His new co-authored paper in Neuron politely dismantles the highly promoted link between the gut microbiome and autism, which turns out to rest on flawed studies, contradictory findings, creative statistics, and a touching faith in mice burying marbles.Kevin walks us through the joys of observational studies that don’t replicate, mouse experiments that don't make sense, and clinical trials where there is no blinding and no control wing, and shockingly, everyone reports feeling better. Meanwhile, journalists and wellness gurus eagerly report each new “breakthrough”, unburdened by any concerns about the strength of evidence or methodological robustness.In the end, the microbiome–autism connection looks less like a sturdy scientific stool and more like three damp twigs taped together by optimism and marketing departments.We finish, naturally, by dragging Matt back out of his panpsychism phase and asking whether consciousness is really fundamental to the universe or just something that happens in podcasters who haven’t slept enough.LinksMitchell, K. J., Dahly, D. L., & Bishop, D. V. (2025). Conceptual and methodological flaws undermine claims of a link between the gut microbiome and autism. Neuron.Kevin Mitchell's Website
Hello and welcome to the Coding The Guru's interview edition.
As per usual, Matt, the psychologist of sorts, is there and Chris, the anthropologist of sorts, is here, me.
But today we have returning guest, man also from Ireland, Kevin Mitchell, who is associated with Trinity College in Dublin and associate professor of genetics and no, wait, genetics and microbiology institute of neuroscience.
That's your affiliation, right?
But maybe your title is slightly different.
It's much of a muchness.
Yeah, so we got genetics and neuroscience and microbiology.
Those are the important things there.
And much as we look forward to later in the episode discussing panpsychism and Matt's recent adoption of that philosophy, the reason that we have you on today is that you have a forthcoming paper along with Darren Daly and Dorothy Bishop, the title of which is Conceptual and Methodological Flaws Undermining Claims of a Link Between the Gut Microbiome and Autism.
So that's what we're going to talk about.
But in general, Kevin, thanks for coming on.
And it's good to see you.
Yeah.
Yeah, it's my pleasure.
And thanks so much for having me on.
It's great to see some interest in this paper, which is sort of a null finding in a sense.
And those kind of papers don't often get the same attention that the positive ones do.
So yeah, I'm really happy to be on to be able to chat about it.
So what's the background to this, Kevin?
So we've got this connection between, purported connection, between the gut microbiome and autism.
And it's got a popular discourse aspect to it.
And it's also an active area of investigation in the academic literature.
So how would you describe it?
Yeah, well, I think you hit both nails on the head there, right?
So it is within science.
It's a very active area.
It attracts a lot of funding.
There's loads of papers, there are loads of reviews and so on.
And then it's also the kind of topic that makes its way into the New York Times and The Guardian.
I mean, there was even a Netflix special on it recently and tons and tons of online discourse about it in various groups.
And so, you know, I think the rationale here for studying this, right, for thinking that maybe something about the gut microbiome has something to do with autism or contributes to risk comes from a few areas.
One is just the idea that there's an epidemic of autism, right?
So what we've seen over recent years is rates of autism diagnoses have gone up and up.
And so people point to that as As evidence that there must be something in the environment that's causing that, something that's changing, that's causing this increase in the rates of autism.
And you'll hear RFK Jr., for example, point to that all the time, right, as evidence that there must be environmental factors at play.
It turns out that while there's an increase in the rate of autism diagnoses, that doesn't mean there's any difference in the actual underlying biology or symptoms that people have.
And in fact, there's really, really good evidence that it's just that the diagnostic criteria have expanded and loosened so that it's quite possible to get an autism diagnosis with much less severe symptoms than it would have been 10 years ago or much, much less than 20 years ago, and so on.
So there's a ton of research actually showing that the apparent rise in diagnoses doesn't reflect any change in underlying biology.
It's just convention.
And actually, psychiatric diagnoses are a matter of convention at some point, right?
Yes.
So that's the first, but nevertheless, that's the impression that people get.
There must be some environmental cause.
So let's look for one, right?
And then the second sort of line of evidence is that many children, in particular with autism, also have some gastrointestinal symptoms, right?
So there's some GI disturbances that are more common in people with autism.
And so there's a sort of a link there.
It's like something's going on in the gut, something's going on in, you know, behaviorally.
Maybe those two things are linked together.
And of course, there's lots of ways to think about that.
And one of them is just to say, well, okay, maybe there's a link that goes from a disturbance in the gut causes behavioral symptoms, which is sort of the narrative that's being proposed.
But it could also be maybe something in the behavioral symptoms causes a gut disturbance, right?
So maybe having a really restricted diet and being a very picky eater affects your diet and affects your causes of constipation or diarrhea or other concerns, right?
And then the other possibility is just that they're completely unrelated, right?
So yes, it's true that children with autism have higher rates of gastrointestinal symptoms, but so do children with every neurodevelopmental disorder.
I mean, so do children with Down syndrome, right?
Nobody thinks that the gut disturbances in Down syndrome are causing everything else, right?
They're not causing intellectual disability or anything else, right?
And in fact, you can also say, well, look, you know, many children with autism also have seizures, right?
You know, much higher than the general population.
Again, nobody thinks the seizures are causing the autism.
They're just two manifestations of a syndrome, basically.
And so, you know, I don't think either of those lines of evidence is actually really strong.
You know, they're fine.
It's fine to say, okay, we can have this hypothesis because of these ideas.
I don't think there's a very strong, just sort of preliminary grounding for the hypothesis.
But nevertheless, those are the two main lines of evidence.
And then the third is sort of a negative, which is to say we've really, really good evidence that autism is very strongly genetic.
If you look across the population at the variation and who gets it and who doesn't, about 80 to 90% of that is explained by genetics.
And so what that means is that there should be genetic differences between people that are that are causal for a risk of autism.
And we should be able to find them if we look for mutations.
Now, finding them turns out to be really difficult, right?
You need very large numbers of people because we all carry loads and loads of genetic variants.
It's hard to pinpoint which ones are which.
So we've made tremendous progress in identifying lots and lots of high-risk genes where a mutation can confer a high risk of autism or other neurodevelopmental conditions.
But there's still loads and loads of cases unexplained.
So there's two ways of looking at that.
One is to say, okay, we're just getting started.
We need to do, you know, figure that out more.
We need better ways of finding these things.
And the other is to say, we've hit the end of the road there.
There's something unexplained.
Some other factor must be at play.
So those are the sort of the three lines of evidence that prompted, I think, people to start looking at this hypothesis.
And Kevin, so the paper, just for, you know, obviously the people listening won't have read it.
And I would classify it as a critical review of the literature, right?
So people may or may not be familiar with meta-analyses, but in general, meta-analyses are collecting the statistical results of a selection of studies and trying to combine them together with ways to qualify how rigorous they are and then looking at whether they can find overall effects or not.
And your paper is actually above that kind of level because it's looking critically at the various different meta-analyses that have been conducted.
So kind of taking a top-down look across the whole literature and the quality of the literature therein, right?
And you cover three lines of evidence, which I think we should get into.
But I just want to mention, given that that is such a like a kind of bird's eye view of it.
So you, as we mentioned, you know, associate professor of genetics and Euroscience, your co-author, Dorothy Bishop, I recognize from a well-known figure within the replication crisis, our open science methodological reform movement.
Your other co-author, Darren Dully, I'm not familiar with, but I'm wondering how the three of you ended up deciding that, obviously, the things that you just mentioned are all like the contextual factors, but how did you guys decide that you're going to take it on and look at this literature?
Yeah, so maybe I should just briefly say what the literature sort of entails, because this was what we were looking at, right?
Is that these reports started coming out?
So people had these kinds of hypotheses to test.
And then they started using these technologies that allow you to basically sample the gut microbiome from basically from stool samples, right?
So you can take someone's stool, you can sequence the DNA that's in it, and it will contain loads of the microorganisms that are in your gut.
And then you can use this sort of bioinformatics techniques to figure out which sequence belongs to which species.
And you can determine the relative abundance of different species and phyla and so on from one person to the next, right?
So this tool became available that people could do these kinds of analyses.
And then it was, you know, then you could just ask, well, do people with autism have a different profile in their microbiome from people without?
And when we started to see these studies, like the first one is about, you know, 15 years ago, I think, in 2010, I was interested because I work on the, you know, the genetics of autism and neurodevelopmental disorders more generally.
So always interested at possible factors that could be contributing to it.
But when I looked at those studies, I found they were just really small studies.
They didn't seem to be well done, statistically speaking.
There were loads of what we now know are research practices that basically will generate false positive findings, right?
And so we know that from our history in genetics, our history in neuroimaging, history in psychology.
You guys have talked about it before, right?
All these things that have led to the replication crisis, where we've identified methodological practices that just are not going to be robust, right?
And those were the kinds of things that I was seeing in these papers.
So it made me highly skeptical of the findings.
And Dorothy also has been well known, as you said, in the field of reproducibility and the replication crisis and so on.
She's written lots and lots about what those factors are, what these bad research practices.
But she's also an expert in autism and neurodevelopmental disorders, speech and language disorders, and so on.
So she was quite interested.
And then Darren is a colleague here in Ireland who also is very interested in the replication, reproducibility sorts of things, an advocate of open science and good robust science practices, and also a particular expert in the design of clinical trials.
And, you know, what we started to see in the field was that people were doing these studies, claiming to see differences in the microbiome between people with autism and without.
And then they started to move to other lines of evidence.
First of all, experimental work in mice, but also these small-scale trials in humans where they were testing probiotics or even doing fecal microbiome transplants into children with autism and then claiming that there was some amelioration of symptoms.
And so, you know, surveying the whole field, I think the three of us were, first of all, we were skeptical based on the methods and what we knew about robust methodology.
But we were also, you know, a little dismayed at the hype, frankly, because these claims are not, you know, when they're published, they tend to be published in a quite sort of stark kind of a way.
Like the headline claims are very strong and then they get picked up by the newspapers.
And so it becomes what we took to be, you know, a lot of effectively misinformation.
And, you know, so that's what prompted us to actually dive in here and see, well, I mean, you know, maybe our impression was wrong.
Maybe actually something robust was coming out of the field, but it took it, it takes a deep dive to really assess it and evaluate it.
And that's what we did.
Yeah, Kevin, I think the context here touches on like a few very common sort of public misapprehensions or misconceptions about how things work.
And one of them, of course, is the reification of diagnoses, assuming that they are a hard type of category when any psychiatrist or psychologist knows that they are diagnostic conventions which change as society changes.
What's true of autism is also true of ADHD and any number of other conditions.
And the other aspect, of course, is, I mean, I mean, I first heard about the gut microbiome in the public discourse when we covered Gwyneth Paltrow.
It came up again with Chris.
Michaela Peterson, of course.
And I've forgotten whether Andrew Hoofman is into it.
Has he talked about it, Chris?
Oh, yeah, of course.
Yes.
Of course he has.
Yeah.
So there's obviously this huge public appetite for ideas like this.
I know I think, I mean, personally, it's a bit psychoanalytic of me, but I think we have all have obsessions about what goes into us and what comes out of us, right?
There's certain kind of obsessions that people have.
But, you know, so, but what's kind of most concerning potentially, and we'll get to your results in a second, what you found, is the degree to which the academic and scientific literature, which, you know, as we all know, is voluminous and of widely varying quality and bits and pieces of it that sort of, especially the ones that maybe are a little bit free with their abstract or their titles or their conclusions,
how that can get picked up and functionally serve as misinformation, even if that's not the intention of the people writing the papers.
Yeah.
Yeah.
No, I think that's that's exactly right.
And it's weird, like it's funny to think what makes this topic so appealing.
And I think there's a few things that go into one is that like even for scientists, it's sort of sexy, right?
so novel to think, oh, like, what if I told you, you know, everything you thought about your personality and the way your mind works is actually being affected by the gut, you know, the bacteria in your gut.
It's just a cool kind of sounding hypothesis, right?
I think people find it sort of attractive for that reason.
It's just a bit novel.
Scientists are not immune to trendiness, right?
And then the other aspect for something like autism is that what this topic promises is a simple explanation for a really, really complex condition, right?
And so that's always appealing.
It's simple, it's straightforward.
If you're a journalist, you can write this paper.
You don't have to do any background or, you know, nobody needs any background to understand it.
You can just say autism is caused by the excess of these kinds of bacteria or the imbalance between these different phyla or genera of bacteria in your gut, and you don't need anything else, right?
Whereas to explain the genetics of autism, what we know about that, it's so complex, right?
There's so many risk variants that we know of in different genes that do different things.
They combine together in highly complex ways.
There's effects of genetic background.
There are risk factors for other conditions as well.
You know, even just what I just said, right?
Your eyes start to glaze over.
It's like, shit, that's I can't tell that story.
That's a terrible, that's a terrible story to try to tell, right?
Whereas the autism is caused by this one thing, and you know, it could be the microbiome, it could be Tylenol, it could be fluoride in the water, it could be any number of things that people have run through.
COVID vaccine.
COVID, of course, right?
Absolutely.
I forgot the COVID vaccine.
I forgot vaccines.
Right, right, right.
I forgot vaccines in general.
And of course, there's a link between the original Wakefield fraudulent paper that claims vaccines were at play with Leaky Gut this sort of really vague, nebulous sort of idea, but that was very much at the core of his initial claims.
And so there is a link out there.
I'm not claiming that the people, that the scientists in the field working on it are making this claim, but out in the literature, out in the broader discourse, there's very much an idea that there's some kind of link potentially between vaccines and the microbiome and autism and so on.
So yeah, I mean, it, like you said, it becomes functional misinformation, even if the scientists writing the things don't intend that.
But I'm not going to let them off the hook entirely, Matt, because actually I think there, you know, I think people do have a responsibility to be somewhat cautious and circumspect in the way that they talk about their findings in the title, in the abstract, especially in press releases, where you'll see these individual studies, really usually pretty small-scale studies that people find some positive association or something, and then they get huge press.
Well, that's not a coincidence, right?
That means that the people involved have made a press release and they put it out there and they're promoting things and spinning it in a certain way that, you know, I find, let's say, could be more circumspect and more cautious.
That's a polite way of putting it.
And I will say that this reading this paper reminded me so much of a whole bunch of unrelated literatures that I've looked into where you see similar kinds of issues, right, around small sample sizes and kind of overhype claims.
I mean, mindfulness, meditation, for example, has similar sorts of issues around it.
So basically, anything that gets a lot of hype often comes with these kind of concerns.
But to speak about some of the specifics more, so you cover three lines of evidence: human observational studies, preclinical experiments in mice, and human clinical trials.
And just for my personal preference to start, you can veto this, Kevin, if it doesn't flow coherently.
But the mice studies, I came across these because when Matt and I covered a character, Dr. K, you may or may not be familiar with, but he referenced the fecal transplant.
Oh, really?
Yes, because he wants to link it actually to Ayurvedic notions about diet.
Yes.
these kind of things.
So I went down a rabbit hole into the studies that you're discussing in the paper.
And I remember attempt like one, noting the very small sample size.
I lack the grinding in the relevant animal study methodologies to know how bad the low sample size is.
But the issue that I kept coming up against, and I was like, is this just the norm here?
Is there was no pre-registered studies.
It's absolutely zero.
Like open, it's like open science didn't seem to have occurred in this field.
So anyway, I'm curious about like all three lines of evidence, but maybe we start in the non-human lineage first.
So this is really interesting, right?
So people have tried in trying to understand medical conditions, people often use animal models.
So for example, if you find a mutation in some gene that causes epilepsy in humans, you can make that same mutation in an animal and then you can see, do they develop seizures?
Then you can have a model where you can do some experiments, you can test some drugs, you can work out a mechanism and so on, right?
So that's the rationale for making animal models of conditions like that, right?
Now, for seizures, it's pretty obvious if an animal is having a seizure, right?
Because it looks like they are and you can record from the brain and you can see that that neural activity is happening in that way, right?
The question is for psychiatric stuff, like what do you look for when you make an animal model?
And this becomes really tricky.
And there's sorts all sorts of things that are taken, all sorts of behaviors that are taken to be a mouse analog of depression or psychosis or mania or things to do with autism, right?
Now, for some of those conditions, like for psychosis, for example, there are some lines of evidence that back that up.
And one of them would be like you can give people some drugs that make them psychotic acutely.
And you can give those drugs to an animal and you can see what happens.
And then you can say, here's my model of psychosis in an animal.
It doesn't matter if it looks like psychosis, you know, superficially.
It's that there's some underlying neurobiology that you can presume is the same.
And then you can say, well, actually, I also know I can give these other drugs, these antipsychotics to humans and stop psychosis.
And if you can treat whatever the emergent behaviors in an animal are with those same drugs, then you've got some good confidence that you're in a realm that has some relevance to psychosis in humans.
For autism, there's nothing like that because we don't have any drugs that treat the symptoms of autism itself, the core symptoms, which is delayed language development, differences in social cognition, differences in social interaction, repetitive behaviors, and narrow interests.
Those are the things that those are the core symptoms of autism.
And we don't have any drugs that treat those.
And we don't have any clear analogs in mice, or at least what people claim is an analog in a mouse is just based on what's called face validity, which is a sort of a superficial similarity.
Basically, you can use some of the same words to describe what's going on in your mouse with what's going on in humans, right?
So one of those would be, you know, the most obvious one would be social interactions.
You can see how much interest a mouse has in other mice.
And that's an area of science in mouse behavior.
It's a perfectly valid area of science, but it's a leap to say that indifference to other animals for a mouse is a proxy for whatever is going on.
It's much more complex sort of social cognition in humans, right?
Yeah.
And the burying marbles example too.
Okay, so there's a couple other ones that are taken, right?
One of them is you can measure ultrasonic vocalizations, which is basically how much mice squeak.
And some people will say, well, less squeaking is like a language deficit.
Again, you can see what a leap it is to make that claim.
And in fact, some people say more squeaking is also a proxy for autism.
So it's like, either way, there's some relation.
But the marble burying one is the one that really, yeah, I think is kind of baffling.
So there's this, if you put a bunch of, it sounds funny even describing this, right?
If you put a bunch of marbles in the bedding of a cage that has a mouse in it, they'll tend to kind of dig at them.
And as they dig the bedding, the marbles will tend to submerge.
So it's not that they're trying to bury the marbles.
It's a consequence of how much they like to dig around in the bedding.
And so people have said, well, that digging around in the bedding is like this kind of compulsive, repetitive behavior.
So maybe that's like repetitive behaviors in autism.
Right.
And again, like it's such a leap to make that, but people have sort of taken that to be the case.
Right.
Do they measure how many marbles are like underneath the way the test?
So somebody looks in after 20 minutes or half an hour and counts the marbles.
Yeah.
That's the nature of it.
But at least that should be very easy to pre-register.
It would be.
So, right.
So this is the other issue, right?
It's like, if you're doing these tests, well, let me say the basic idea of the test is that there was all these human observational studies that purported to show a difference or many kinds of differences in the microbiome of people with autism versus without.
Now, just from observational studies, you can't make any causal claims or inferences, right?
It's just an association.
It's just a correlation.
So that prompted people to do some causal interventions in animals where they say, okay, what we're going to do is manipulate the microbiome in some way and then see if we get any effects that we think are related to autism.
And there were sort of two designs.
One is to generate animals that you think reflect the etiology of autism in some other way and then see if manipulating the microbiome affects their symptoms.
And then the other is to take some animals that you think are kind of just a baseline and actually put in the fecal microbiome of people with autism and see if that makes them exhibit these autistic-like symptoms.
Turns out all of those designs have various problems with them.
You referred to, first of all, the lack of pre-registration.
There's many, many sort of researcher degrees of freedom in the kinds of behaviors you can look at.
Usually they weren't corrected for all the tests that people do.
The more things you look for, the more likely it is something by chance is just going to pop up.
The constructs that people used in the first kind of design where they said, we have a mouse that's sort of autistic, let's manipulate the microbiome were also based on epidemiological factors that have no validity.
So things like maternal immune activation, there's a whole literature around that.
But like a lot of epidemiological findings, the supposed association that you get from really small studies just goes away when you look at really, really huge studies in Denmark or Sweden or someplace that has national registers.
It's like when you do the epidemiology properly, these risk factors kind of evaporate.
And we saw the same thing with Tylenol, exactly the same dynamic.
Some small studies claimed an association and then bigger studies, properly done, it evaporated.
One of the other mouse mouse models was maternal high fat diet and that is supposedly the this link between maternal obesity and autism which again comes out of the really small studies.
And then when you do it properly, and especially when you, when you control for potentially confounding factors in your design within, you know if you do a within family study you don't see any effect right.
So when you do the epidemiology studies properly, those kinds of effects tend to disappear.
So anyway, this is a long background, but this is the problem is like it gets really technical to actually get into the details.
Yeah, you actually have to dig in and and see what's going on.
I mean this would be familiar to DTG listeners because we've covered before a kind of pattern where when, especially in the context of meta-analysis, where it can kind of look like there's a lot of evidence there, like there's a lot of smoke so it sort of feels like there's got to be fire yeah, but but when you you look at the details, you see a pattern which is the kinds of studies that are supporting the the proposal tend to be very weak, have lots of flaws,
you know, small n, all kinds of stuff like that.
And then when you, when you look at the studies that are that are stronger, bigger in all of that stuff, they they tend to to not find the result.
But if you kind of look at the all of the studies naively, then it can seem like there is.
Yeah, you know, there is evidence for an effect ivermectin with like ivermectin.
Yeah and Kevin, maybe you were about to cover it, but I, I one of the things that struck me in this section was the notion that, even if you accept like I think to me the clearest analog is okay like, if there's a relatively okay way to measure mice sociality, which is questionable but let's grant it right then okay.
So this is something that is usually disrupted by severe autism.
And then you want to test about these fecal transplants as as a possible way of curing or at least reducing the symptoms.
But that relied on, as you described in the paper, the notion that like a human because like the human gut biome is not the same as rodent gut biome, so you've got that going across and then that that will cure, because that would mean that like essentially, the autism is such a core component of mammalian physiology that it doesn't even matter that mice don't eat the same things as humans generally.
There's so many sorts of underlying conceptual problems.
This is one of the things that's really frustrating.
There's this really loose movement or sliding from human observational studies where basically, you're just finding something is different in the microbiome.
It's not consistent at all as we hope we'll talk about and then it's like okay well, let me test that in the mouse.
But like conceptually, what are you doing?
What do you think is going to happen?
What kind of an effect size do you think you're dealing with?
That manipulating the, the microbiome could cause this Effect in mice.
And so it's a bit baffling, especially as you say, because the gut microbiome in mice is really different from that in humans.
Like it's like something like 85% of the species that are there in humans aren't there in mice and vice versa.
And when you actually implant things, some of the species will engraft and stay there, but many of them won't.
Like you can put human microbiome in a mouse.
It doesn't mean it will have a human microbiome.
Most of those don't live there, right?
So there's a sort of fundamental flaws there.
And then, or questions at least, let's say.
And then the other thing that's funny is like these experiments, many of them start with germ-free mice.
So these are mice that have been raised in completely sterile conditions for generations.
They have no bacteria in their gut whatsoever.
And then the idea is you populate them with bacteria from either children with autism or without.
And then you ask, what are their symptoms, right?
So, you know, that's quite a drastic thing to do to an animal that has never had any bacteria in its gut in its life.
And to populate it like that and then say, you know, what happens behaviorally, it's not really a surprise if something happens behaviorally.
I mean, you might get a shock if someone introduced a whole microbiome to your system that you'd never had before, right?
Well, actually, Kevin, I have other small technical question here.
Yeah.
So if that was their intervention, was their control to just do nothing or was it to inject the mice with healthy, non-autistic humans?
Right.
So in one of these studies, it was the microbiome from healthy, sorry, neurotypical donors versus from other ones.
And, you know, but this particular study, which is cited, I don't know, like over a thousand times, right?
I mean, the influence of this one study is amazing.
Pretty quickly, and so it came out in 2019 almost immediately.
People online started going, wait a minute, what's this now?
Because the data just didn't seem kosher.
There was like something off in the statistics.
And it turned out that the experimenters had made a simple but super important statistical error.
So they used samples from five patients with autism and three neurotypical patients.
And then for each of those, they transplanted the microbiome into a bunch of different mice.
So the number of mice was a lot higher than that.
And they used that number as the baseline for their statistics when looking for significant differences.
What they should have been using is the five and the three, right?
Not the number of mice, because those were, they treated them as independent experiments when in fact they're replicates, right?
This is a hierarchical design, Chris, by the way.
I know that.
Okay.
So people pointed this out like really, really immediately.
And the authors, you know, to their credit, made their data available.
Other people reanalyzed it and the effects go away.
I mean, let me say, I should say what the claim is, right?
The claim is that the mice that are given the microbiome from the patients or people with autism showed some of these symptoms, right?
So marble burying and some social interaction thing and other things.
Actually, it's interesting, just as an aside, to illustrate the flexibility in approach here.
When they did the basic social interaction test, they didn't find any difference.
Then they did another one, right?
So then they said, okay, let's try this other social interaction test.
And then they claim, well, maybe this one's better.
That was the one that showed a positive difference, right?
So you can see there's a kind of a, what people call p-hacking, which is, you know, basically torturing your shifting, shifting the goalposts, shifting endpoints, torturing your data.
Big red flags.
Right.
When you don't, when you don't pre-register even out loud to yourself what you're going to do, there's a temptation to just tweak it a little bit until you see something, right?
Yeah.
I think in that context, it's like for people who don't run experiments just to think that imagine the counterfactual world where the experimenter gets the result in the first measure.
Would they then go, okay, we need to run another measure because maybe that wasn't a good one.
Right.
Like no.
So the only time that it becomes a good measure is when it gives you the result.
And this is also yeah yeah, the keys for field replications and stuff.
They're well conducted when they replicate the result, but when they don't, there's too many things to you.
Exactly right, this is how researcher degrees of freedom work, is that you keep looking until you find the thing that you want to get.
And it's why these experiments that set out to prove a hypothesis, a hypothesis, are much more dangerous than ones that set out, you know, in a disinterested fashion, to test a hypothesis.
And here there's just strong motivation that this is a thing, right and, and so that's, that's kind of what is what has happened.
And anyway, to finish this particular paper, when the data are reanalyzed properly, basically all of the effects go away.
I think there's a very small remaining effect on marble burying, which you know we already talked about, the questionable, and the effect size is very small and it's tiny, right in that case.
So basically, all of the the claims from the mouse literature that we looked at just don't stand up.
I mean, I I think there's nothing solid there that you can say.
Definitely, mouse studies have shown manipulating the microbiome causally affects autistic like symptoms, which is how it's described.
I mean, that's the amazing thing is that in the literature people cite these studies based on claims in the abstract or the titles, and then other people see i've cited the study positively in my introduction or something.
And then matt, you're reading my paper and you go, well, Kevin knows what he's talking about, this must be real.
And then you cite the citation and then it just becomes lore.
Right, it becomes a fact yeah, and so few people go back to the original studies with their own evaluative powers and say that doesn't make much sense at all.
Right yeah, I actually was working on a paper today where i've been, i've been the victim of this in that we wrote a paper a long time ago, had a catchy finding.
I knew it was a bit weak.
It's actually one of the our most highest cited papers and we've tried to correct, we've discovered it's, it's, it wasn't probably wrong, but we've got better refined estimates yeah, and we've published that and they just keep getting ignored because the the sort of um the original one just has too much um impetus behind it.
Yeah, but yeah so yeah, it's a bit.
It's a bad thing.
A lot of this paper reminded me and Matt, you probably note the parallels as well that uh, Dorsa Amir and Chas Faristone, we had them on talking about the visual illusions and the Muller-laire um literature, and there you have a massive literature and they did exactly what you're talking about.
They just went back through it all and find that it's not the Muller Lyre illusion itself but some of the claims made around like cultural variation on it.
Um yeah, but there are massive literature there.
So like, actually going back and looking critically at it, you can often find things that you don't want to find, like in the Stanford Prison experiment.
Oh yeah, exactly right, and that's yeah, that's so.
Those are the kinds of things that become lore.
You know, Stanford Prison experiment is a great one and there's a load of well-known psychological experiments and you know, I always think these days I guess i've become more skeptical or even cynical is that if an experiment has a name, if it's like the Dutch Hunger Winter something, something don't trust it, right?
If it's a one-off, A one-off kind of thing?
It's just far more likely to be wrong.
And what we should do is look at bodies of evidence that consistently builds towards a consensus, right?
The Salian test, that's still okay, right?
I mean, that's a body of literature.
That's not a one-off.
Yeah, that is a body of literature.
Yeah, exactly.
Okay, Kevin, so we've covered the mice.
Yeah.
But there are other lines of evidence that you reviewed with this.
Yeah.
Yeah.
So the lines of evidence that prompted the mouse work are basically human observational studies.
And this is basically epidemiology.
So what you do is you have people with a condition and people without, and you look for some factors, some exposures that are more common in the people with the condition than without.
Those might be causal, right?
So that's how we found out that smoking is associated with lung cancer, for example, because people in people with lung cancer, the rates of smoking are much higher than in people without.
And then you can kind of invert the logic and say, what is the extra, what is the increased risk of having lung cancer if you're a smoker?
But the original data are the other way around.
So the design here would be to say, okay, well, maybe something about the microbiome is different in people with autism versus people without.
Let's look and see what that could be.
Now, the problem is when you sequence the microbiome, this is massively high-dimensional data, right?
You've got tons and tons and tons of variables that you can analyze in lots of different ways.
So if you look at bacteria, you know, they're organized into different bacterial species, but they're related to each other in families and in a genus and in a phylum.
Okay.
So there's like a dozen major phyla of bacteria.
You can analyze those.
You can group them in that way.
Or you can go down a level to the genus.
That's nasty, Matt.
Yeah, absolutely.
Exactly.
And you can go down another level to species and so on.
You can do it any which way, right?
And if you don't, again, if you don't pre-register it or say what you're going to do, then you've got this tremendous kind of flexibility.
You can even come up with measures like, let me measure the ratio between these two types of species, right?
People do that.
Or let me measure the overall diversity.
Maybe it's not one particular species.
Maybe it's the diversity of species that's different.
So that's what the literature basically involves: taking people and then analyzing the microbiome, looking for some differences.
Now, the first studies that were done, and we sort of surveyed kind of maybe a dozen seminal studies that are given as the foundational evidence for this claim that something is different.
You know, they had sample sizes of 10, 20, 30 people, that kind of thing.
And now, many of them reported some significant differences in some aspect of the microbiome between their samples.
But what quickly became apparent was there was no consistency, right?
I mean, you might publish your study, Matt, and then I'd publish mine, and both of them could say, look, there's an association with the microbiome.
And if you don't dig in a little bit, you don't realize that we've actually contradicted each other, right?
I mean, each of us has published a failed replication of the other person's results, effectively.
And so if you look across these studies, what you see, and we have a sort of a figure in the paper that shows I'll have a study that says bacterioids are higher, and you'll have one that says it's lower, and Chris have one that says there's no change.
And then it's different, though.
Yeah.
But each of us has found a difference, right?
And so what has happened in this literature is that you get this kind of apparent replication, which is actually contradictory evidence.
And then people have shifted a little bit from claims about individual species, which just haven't held up, to claims of dysbiosis, which just means there's a change in the pattern and something about the new pattern is pathological.
That's the implication of it.
But your dysbiosis could be different from mine, right?
It just hides the inconsistency, right?
So there was that set of small studies.
And basically what we know from other fields is that you shouldn't be doing studies with 10 or 20 people that are looking at thousands of variables, right?
You're just going to get noise, especially when we know it's a noisy measure.
And we know that because if you measure the microbiome from one person from one day to the next, it's super variable.
Like it's a really noisy measure.
Well, Kevin, just to check, wasn't this like, I think you covered this in the paper, but wasn't this like an issue that cropped up in other fields when people came across technology that allowed them, maybe it's gene-wide association studies or genetic sequencing, but they were dredging the data and showing, okay, this gene might be associated with this, but then it turned out that all of that literature was like,
this is the sort of frustrating thing, I guess, is that we've been through this, right?
We know this is not how to do it from genetics.
And this was like a painful experience in the field of genetics.
People would take a set of genes that they were interested in.
They'd analyze the genetic variants that were present in those genes in people with one condition versus another, whatever it is, schizophrenia, autism, whatever it is.
And they would find some differences.
And then they'd publish those and then someone else would publish and, you know, so on and so on.
But these small studies just generated noise.
And it wasn't until people started to realize that was happening that we fixed the problem by making these huge consortia.
Like we could no longer do this cottage industry.
It's just not okay to have even 100 or 200 people in your samples.
We really need like 200,000.
I mean, it was that kind of scale, right?
And then the whole field had to change the way that they worked from each of us doing individual things to pooling all of our resources and doing things on a huge scale.
And then that really, really robustly showed real findings, but also showed the initial sort of stuff was all noise, right?
And then the other field is in neuroimaging, right?
I mean, if you take a scan of people's brain, there's so many parameters in there that you can compare between groups.
And there's vast literatures claiming to have found a neuroimaging biomarker of depression or autism or whatever it is, right?
In 20 patients versus 20 others.
If you took 20 Geminis and compared them to 20 Libras, you would get differences in their brains.
Yeah.
And so again, what the field has figured out is actually you need sample sizes in the thousands, not in the dozens, right?
And so a couple of papers came out recently for the microbiome stuff, which showed the same thing, to find even the most robust differences that we know about.
So effects on the microbiome due to diet or age or smoking or eating fruits or diabetes, right?
Those are all factors that we know have a big effect on the microbiome.
In order to robustly detect those effects, you need samples of like a thousand, two thousand people, not 10 or 20 people.
So it's very clear that you can't do these studies with these small numbers of people.
And actually, we don't have to spend any time figuring out or thinking about what to make of those data.
You can just, with confidence, ignore them now.
We just don't have to talk about them anymore, right?
Now, what people have done is gone on and done larger studies, right?
So they have done studies with at least in the high hundreds of people.
And those studies are also not consistent with each other.
Each of them finds something, right?
But they're not consistent with each other, except in that what they can look at is the overall amount of variance in the microbiome measures that's attributable to the autistic diagnosis.
So in your sample of people, you've got people with autism, people without, but they also vary in lots of other ways, diet, age, whatever, right?
And you can say how much of the difference in the microbiome is due to the fact that these people are in these two different groups.
And those differences range from 0% to 5%, right?
So in some of the studies, up to 5% of the variance in the microbiome diversity was due to this autism diagnosis, right?
And then, okay, so first of all, like that's a really tiny effect, which is important for thinking about your design of your mouse studies or clinical trials, right?
But secondly, that is still just an association, right?
We don't know what is going on causally.
And it turns out that some of these studies that control for things like diet find that that association is driven by diet, right?
It's driven by the different behaviors of the people.
So it's a small effect in the first instance.
It's not consistent.
And when it's there, it's most likely driven by a confound, not by the causal arrow that people are claiming.
You know, this makes me think, Kevin, that again, something we've come across in people talking about gut biome and also talking about toxic mold.
That's a current damage.
Okay, perfect.
It's taking out Jordan Peterson.
But in that case, there's a whole cottage industry.
And this includes perfectly credentialed people, you know, with medical degrees and so on, where people will talk about that they've got tests that show that, you know, that they were impacted by toxic mold or that the gut biome is out of whack.
And when we were covering Grinnell Paltrow, she also had a doctor on who was talking about how they have, you know, their own bespoke ways that all our other tests don't pick up the differences.
And it felt like there's a very big industry around telling people it is the gut biome.
And there's so many different markers like you indicated because it's a complex system that you can always detect something.
Yes.
Like if you got enough tests.
Yeah.
And the concern is like some of these studies now are moving to machine learning as a way to find the patterns, right?
Now, what machine learning is going to tell you is it's going to find a pattern, right?
It's going to find something that's different between your groups because that's what it's designed to do.
And so it's a sort of a hype, a hyper-powered way to generate spurious findings that are now uninterpretable because you don't know what's going on in the black box.
Like it's not coming to save us.
It's just a terrible, terrible way to analyze your data, right?
If you could take a terrible problem, which is this massive ratio of the dimensionality of the variables to the N, and you go, let's make it much, much worse by fitting it into a much more flexible AI.
Yeah, Matt and Kevin, imagine you had someone, you know, like some listener or whatever, who didn't know much about machine learning and you needed to explain to them basically what that involved.
What kind of thing would you explain why that's a problem?
Not me, you know, just a listener.
Do you want to have a go at that, Matt?
Yeah, I'm happy to.
So, I mean, if you compare a machine learning model fitting some data compared to your normal boring linear statistics, then essentially they're much more flexible.
So if you've got a bunch of data points that are on an XY plot, if you're fitting a straight line to it, it's pretty constrained.
It's only got two degrees of freedom to fit it with.
If you fit it with a really wiggly curve, then you can fit any data that comes along.
You just make the line wigglier.
So that's probably the way to think about it.
Yeah, exactly.
You basically explode your data to a latent space that's massively multi-dimensional.
And then your degrees of freedom explode exponentially.
You don't really have too many degrees of freedom.
But also, I mean, the other thing is that we as researchers then don't know what the thing has done, right?
At least when you do a regression, you can probe it and say which are the bits that are causing it, right?
But the black boxing effect here is really is really an issue.
So this is the opposite of a multiverse analysis kind of approach where you're coming like and saying, okay, when we ran this 1,000 times, this was significant twice.
I don't know if it's the reverse of it, but it's definitely not the same as it.
And, you know, I mean, also there's another sort of issue here.
So there's a couple of papers that have come out claiming that they train their machine learning model on the patterns in the microbiome of their sample with autism and sample without.
And then they claim that they can get a predictor, right?
So their machine learning model has learned what a microbiome signature looks like, and it can predict autism in other people with some greater degree of success than chance.
Not up to 70% or something like that, 80% in some cases.
The problem is, and then they propose that this could be a diagnostic tool, which to me is just such a head scratcher.
It's just like a category error because autism just is the name that we give a cluster of behavioral symptoms.
And the way that we give people a diagnosis is by asking if they demonstrate that cluster of symptoms.
And it's not like you're going to give somebody a microbiome test who doesn't have those symptoms and then say, oh, you know, got news for you.
It turns out you're autistic.
And they're like, well, I don't seem to be autistic.
Well, nevertheless, that microbiome says you are, right?
This is just kind of a weird, or vice versa, right?
It's like someone who has all those symptoms.
And then what are they like going to say, no, you're not autistic.
Your microbiome doesn't say that you're.
I feel you're not cynical enough, Kevin, because I suspect what will happen there is people would get the positive test, would have none of the actual symptomology and then say, well, I guess that means I'm autistic.
So yeah.
So anyway, I mean, like, you know, overall, looking at this human association study literature, we see this pattern where there's a load of small studies, they generate lots of noise, it's really inconsistent.
The sample sizes are like 100 times smaller than they should be.
And effectively, there's nothing there in that literature that actually supports this claim that there's any real association going on, or if there is any tiny association, that it's important, or that it's going in the causal direction that people claim and not confound it.
And in fact, one of the really important things that came out of this, the study when we looked at these things, is that when people do a study with controls within the same family, like sibling controls, then they don't find these differences, right?
Which really suggests that there's confounding with other familial factors that is driving the small differences that people have seen.
So, I mean, the bottom line for us is that, look, you know, Matt, you said it earlier.
You can see that all these studies are coming out.
And even if they're a little inconsistent, it's like, well, look, there's so much smoke.
There must be fire.
But actually, sometimes there's just loads of smoke, right?
And that's what we've seen in these other fields, like the neuroimaging studies of biomarkers of depression or whatever.
It's just smoke.
And the same with these candidate gene association studies, just smoke.
There's nothing there at all.
Right.
So I think there's like one other line of evidence that you considered.
And is it human trials?
Yes.
RCTs, that kind of thing.
Yeah, exactly.
So RCTs.
Not RCTs.
Yeah.
So basically, there's three legs of the stool.
At least that's the way that people talk about it.
So when people are doing the human association study, they'll often say, we know from mouse work that such and such is the case.
And when people are doing the mouse stuff, they'll say, we know from human work that association studies are, you know, associations are valid and so on.
And then the clinical trial people will say, we know from human association studies and mouse work that these things, there's a causal implication.
Let's test that in humans.
And the idea is to test either probiotics, which are, it's a weird term, basically means bacteria that are taken to be good somehow, some beneficial bacteria.
defined in some rather nebulous fashion.
And we'll give those to patients with autism who are enrolled in these small-scale clinical trials.
And then we'll see if they improve, right?
The other one is to actually give a fecal microbiome transplant.
And this has been done in different ways.
One of the studies that's been really highly cited took a fairly drastic approach where they actually gave children like a two-week antibiotic purge that got rid of their own microbiome.
And then they transplanted in the microbiome of healthy people and asked, do their symptoms improve?
And they claimed to find that there were some improvements in symptoms, right?
So that's the headline is fecal microbiomes from neurotypical people can improve the symptoms of autism.
That's the take-home message.
That's how it's been presented.
Now, when you look into it, what you see is, first of all, there were 18 people in this study.
Secondly, there was no control arm.
Everybody got the thing.
It was open label.
Everybody knew what was happening, right?
So very, very prone to placebo effects.
I mean, there's a reason when we do clinical trials that we give some people the treatment and some people a placebo.
And the reason is when people get any kind of treatment, they tend to report their symptoms improve.
And in this case, their parents reported their symptoms improve.
So it's a kind of a manifestation, a wishful thinking sort of effect, but very, very strong.
So what has happened in these studies, if you look at them overall, is that a bunch of studies have reported some positive effects, but they tend to be ones with these open label, single arm, no placebo control, and small numbers of people.
And then when the studies are done properly with randomized control trials where you have two arms, one people getting the treatment, one set getting the control placebo, and the people are randomized between them, then there basically are no effects, right?
So across these different studies, what we're seeing is small samples, again, lots of research degrees of freedom, problems with controls.
And the smaller studies are the ones that show something positive.
And then the bigger studies that are done more rigorously don't show anything.
So again, in looking across all these studies, and I want to emphasize, this isn't just us, you know, the three authors of this paper who are being critical.
It's people in the field, you know, are making, like we quote a bunch of meta-analyses of these studies that basically have negative conclusions.
They say there's just no evidence for any efficacy of probiotics for autism or anything like that.
And nevertheless, you can find, like if you search online, what are the best probiotics for autism?
You're going to get a bunch of sites that are offering you various concoctions that you can give your child to treat their autism.
So this is definitely just out there in the public perception.
I mean, what's interesting is, like I said, these three lines of evidence, people point to them as providing support for each other.
And I would say, first of all, that they're actually not, they're just not commensurate with each other.
They're not doing the same types of things.
It sounds like they are because they're using the same words, but they're actually just premised on very different ideas.
So, you know, I don't think of them as three legs of a stool.
They're just like sticks on the ground.
And so they're really not related to each other.
And then when you pick up any of the sticks, it crumbles in your hand, right?
There's just, there's just nothing left.
So, you know, I think the idea that there's just this very robust connection that has been proven is just not the case.
Yeah, the thing that struck me when like reading through, apart from in general, just my eyes boggling about all the various, every time I come across a review that is like this and the low standards in studies from recent years.
Yeah.
Because like to me, the notion that you would take a clinical trial.
And you give someone the treatment and they know they're getting the treatment and they're likely in many cases to be patients who are actively self-selected into that because of they believe in the treatment right, so they're.
They might be parents who think that it's about the gut biome, exactly.
They're being treated by a doctor who believes the gut biome is the key, and then you've got people doing the analysis who think that it's gut biome and you don't have a control group yeah, like a proper control group.
It's just to me like well that's, that's like a kind of recipe for this and we know it's a recipe for disaster, even with the best intentions of like, everyone involved.
So yeah, that that was just kind of shocking to me.
It's very frustrating that this kind of thing still goes on right, because it yeah, everyone knows that's not a good way to to do things and it's a weird um, sort of dynamic that emerges that people, people think okay well um, I think something might be going on here, so let me do a pilot trial right, as the sort of the way that these things are pitched, but then like, rather than saying okay, I have some preliminary data, now I should do the real trial at scale with the proper conditions, they publish the thing and then they make claims off of it.
Right, I mean, I don't have any problem with doing a small-scale exploratory thing, but just don't don't hype it up uh, you know, as as proof, when you know that everybody knows that this is not the way to do clinical trials.
Well Kevin, I was going to ask you about uh, your conclusion after going through all of this evidence, but I think we already heard it.
We don't have a sturdy school.
We've got three flimsy sticks lying on the ground decomposing slightly.
So a little bit uh um, but you know two, I I think there's.
I mean, I can think of a couple of takeaways, but I want to ask you what the takeaways are.
But for for a member of the public who sort of casually looks at this kind of scientific literature yeah you, look at that citation list it looks incredibly impressive and it it looks like there's really something there even even a casual academic, unless they do a lot of work like you've done, could easily get that oppression.
As well.
As as for Andrew Hoodman or in Jordan Peterson, they have no chance whatsoever.
So, like that's, that's one of my takeaways, but what for you?
Like what what, what do you come away from doing this kind of exercise feeling?
What are the implications for?
Are you okay?
Scientific culture?
Are you okay?
You know drink, you just want to go lie down for a bit.
But what are the implications for scientific culture and also what is the implications for the public understanding of science?
Yeah, I mean, I think there's a few.
There's a few things.
First of all, that that you know, the work that people in some fields have done around reproducibility And robust research practices and so on, clearly has not been socialized to all fields and needs to be.
So we do need to keep banging that drum and talking about good ways to design experiments, good ways to do stats and bad ways, right?
And we need to stop doing the bad ones.
I think what's interesting in this field, you can ask, well, like, how is the public supposed to evaluate these?
Or how are other scientists not in the field supposed to evaluate these when they're peer-reviewed and they're published in really high-impact journals, right?
So those seem to be like often taken to be two markers of quality, where someone who isn't an expert in the field can say, well, look, I don't know what's going on, but I feel like I should trust it.
Some people in the field clearly thought this was good.
The reviewers thought it was fine.
The editor thought it was fine.
This journal thought it was fine.
So I guess one of the keys, well, one of the questions here is like, what's the dynamic within this field in particular that leads to this?
And I think there is a kind of a shared conflict of interest in the sense that, like, if the three of us are working in this field and, you know, I publish a paper and it's sent to you two as reviewers, well, you know, you might quibble with some of the things.
The findings might not be exactly what you've seen, but it's still in your interest for that paper to get published because you can point to it when you put in your next grant proposal or when you put in your next paper to say other people have found something like this in the field, right?
There's a shared interest in just the phenomenon being a thing in the first place.
And, you know, what's interesting in this literature is that all the literature is just continually trying to show that the phenomenon is a thing.
It's just doing that over like it doesn't go anywhere else.
And that's a real red flag is when you see some literature making some claims, and then 15 years later, there hasn't been the normal kind of follow-up where you would say, okay, here this study came out and showed X.
They replicated that in another study.
They dug a little deeper.
They found this mechanism.
We're getting closer to what's going on.
That's the way normal science happens.
And instead, what we see in this field is like, oh, this study shows X.
And then, well, okay, well, this other study shows X prime or Y.
It's sort of like, and then this other study shows something slightly different, but it's continually trying to prove that this thing exists.
And so I think, you know, as consumers of the scientific literature, I think there are some red flags there that we should be wary of.
You know, there's small samples, there's this dynamic of conflicts of interest, just academic ones.
But then, of course, there's also commercial conflicts of interest, right?
So many of the proponents of these claims and some of the authors of the studies that we look at have declared conflicts of interest where they have some commercial interest in the probiotic that they're using or the model that they're using or the biomarker panel that they think they've found.
They have relations with big food companies in some cases, with other sorts of entities, biotech companies, and so on.
So that's a flag that you should take into account, take into consideration when you're looking at these kinds of data.
And I think that dynamic is interesting in the general public because there's always this red flag, like, oh, you know, you're working with big pharma, right?
And there's a huge big pharma industry, and they're really, you know, these terrible, evil people who just want people to have diseases and, you know, so that they can give them drugs.
And I mean, I'm not here to defend big pharma.
There's so many problems with big pharma.
But what's missed in that is like big wellness.
I mean, the wellness industry is like a billion, billion dollar industry.
It's incredible.
So there's a sort of a blind spot that some people have there.
Sorry, Matt, I forget what your question was.
No, those are all really good.
Kevin, you give like a whole bunch of them.
I would also say that like, again, this makes me think of other literatures like the things that are currently around psychedelics, right?
And there you have conflicts of interest with industry as well.
We're going to speak to someone soon who made a report about that.
But one point that just struck me when you were talking about that is like, it actually is a little bit of a hard ask for someone because, you know, when you're talking about a literature that's kind of stalled and is just invested in repeatedly justifying its existence, I think it's hard for someone who isn't like grounded in methodology and statistical analysis and stuff to understand, well, what's the difference between that and when you're talking about that we need replications and robust demonstration of effects.
But I was thinking about very recently, there was a study, maybe you've seen it, the many babies study where they tried to replicate the pro-social effect in very young infants where they prefer like pro-social objects.
And in that case, as a huge study, it's a many lab study.
So it's like 37 or whatever labs involved thousands of babies.
And they're just replicating one methodology.
Took five years.
And the results are null.
Like the results overall are null.
And we know that those kind of things pre-registered, large multi-lab studies without the people that are, you know, or even adversarial collaborations, if you want to deal with the issue about conflict of interest, these are solutions.
So I think one thing I just want to add as an addendum is that even while it is hopeless and there's a lot of things in academia that are like bad incentives and bad studies are still appearing, we do have solutions that we know work or at least help and they could be done.
Absolutely.
And yeah, yeah, no, I mean, so towards the end of the paper, we conclude with this sort of, okay, what should we do now section?
And we sort of put it in two, in two ways.
One is like, if you still think that there's potentially something there that is a hypothesis that's still worth testing, then, you know, do it properly.
Like do it at scale.
Pre-register or at least delineate your hypotheses beforehand so that you don't have this huge garden of forking paths that you could follow in your analyses.
Have a replication sample, do it across different labs and have thousands of people.
That's the right way to do it.
And then you'll have an answer.
Okay.
So if people are still invested in this idea, despite, as I say, there's not a really good rationale for it.
And there's effectively no evidence in support of it whatsoever, in my view.
But if you want to, at least, just don't do it methodologically poorly over and over and over again.
Just face up to the fact, as I said, genetics faced up to this fact that they couldn't just keep doing small-scale studies and had to pool their resources.
The alternative is to say, actually, we have explored this and there doesn't seem to be anything there.
So let's do something else, right?
Let's stop working on this in a sense.
We don't necessarily expect people to do that, but it's interesting, like, to say, what would make you stop?
What kind of evidence would if you keep doing this?
What's the stopping rule?
What would make you stop doing this?
And if they don't have one, then it's not an honest endeavor, right?
You know, it's not testing a hypothesis.
It's really trying to prove it.
And that just becomes bad science and something that I don't think people should engage in.
So I think researchers in the field should think about this.
And again, like all of the problems that we pointed out here are not just ones that we've pointed out from outside the field.
People inside the field have pointed out the issues with animal models, with small-scale trials, with open label, with small sample sizes, all of that kind of stuff.
So it's not just us throwing cynical bombs from the from the sidelines.
I guess the other question then is like, what should funders do?
Because their decisions are often the ones that drive things, as well as like, you know, editors of journals and reviewers and so on.
It's like, you know, what standards should they hold things to?
And there, you know, there's, there's a lot of funding that goes into this.
I just saw there's a new program just been announced by Wellcome.
It's actually called the Wellcome Leap Foundation in the States of $50 million specifically for this.
topic.
And they cite some of the animal studies that we just went through that, you know, are really, really not any kind of strong evidence for any further follow-up.
But clearly, this is still like it's still out there.
It's still a very, very live hypothesis.
And so, yeah, I think people should be cautious in interpreting it.
And I think funders might want to rethink where they're putting their investment.
Yeah.
Yeah, I think I got one final sort of sort of philosophical takeaway for everyone, because, you know, not especially psychologists, we often we have got people who listen to it doing their PhDs or doing postdocs, whatever.
They don't necessarily have million dollar grants to do really strong, powerful studies.
But, you know, I think there is still something you can do, both as a producer and a consumer of perhaps relatively small scale research, which is try to be genuinely dispassionate about the outcomes that you're looking for, right?
It's great to have an interest in knowing whether or not something is true or not and to be passionate about investigating it.
But it's a difficult thing to do.
It's a discipline you have to kind of force yourself.
You have to try not to be invested in the answer and recognize your own motivations, both as a researcher and a consumer.
If you feel that the gut microbiome, that feels right to you, feels real sexy.
It feels sexy.
It feels something that you're jived about, then hold your horses and go with caution because we've seen this in so many psychological fields.
Like so many things, like mindfulness training, meditation, the entire field of positive psychology.
I mean, all of those sexy results that you kind of know from social psychology, they're all stuff that you want to be true.
And it's true of the researchers and it's true of the readers of the research.
So you have to try to be dispassionate.
Yeah, I think that's right.
And, you know, in particular for the microbiome stuff, I should say I've been really critical, obviously, of the way that the research is done.
I don't want to be critical of the motivations because the people who are doing this, they're motivated to want to help people with autism.
And if there's a possibility that the microbiome was really causally involved, then that's great because it gives a treatment possibility, right?
So there's a positive motivation there.
It's just that you're right.
You have to be dispassionate about it and put that to one side and say, okay, I would love to be able to help people with autism.
Maybe this is true.
But if the evidence continues to show it isn't, then at some point you just kind of say, okay, well, that didn't pan out.
Let me try something else.
So, yeah.
It reminds me as well that, you know, in general, I think we're all of us here are on board with it.
But the notion that like, ideally, and for science and progress, you want to be doing severe hypothesis testing, not just saying we'll find a difference from the null.
And so many of these studies seem to be doing the opposite.
Any change from baseline that can be detected is a hit.
And there's another dynamic here, which is which I see all the time, which is that the more sort of extraordinary the hypothesis, the even outlandish the hypothesis, the lower the evidence bar to get this published in really, really high impact journals, right?
So it's exactly the inversion of what, you know, it should be extraordinary claims require extraordinary evidence.
But if you've got something which is like, no, this is a paradigm shift, this is going to blow your mind, totally change things, then much easier to get it into high-knowledge.
You know, we don't have time to talk about it now, Kevin.
But I mean, at some point, I want to convene a roundtable of skeptical people just to figure out what the fuck is going on with the public, with journals and the publication treadmill and so on, because something's not right.
We could get Eric on board.
But, you know, Kevin, we've took up a lot of your time, but we can't let you go without one final question.
And, you know, it's important we've been talking about paradigm shifting ideas and thing.
And Matt's recently become very interested in panpsychism.
He's kind of like an acolyte, you could say.
And I'll summarize his position, but I just, he doesn't believe me that there's an issue.
So I thought I'll ask you because I think you can, you know, steal my response.
So as I understand it, right, the panpsychism, the notion that like consciousness is a fundamental component of the universe, right?
Like not matter is not the unit.
It's consciousness and consciousness exists in everything, including things with no brains or no agentic possibility within them.
Now, from my perspective, my non-philosopher, non-neuroscience-informed perspective, I regard consciousness as a process that emerges from like agentic biological units, as described very nicely in your book, which I read.
But Matt assures me that panpsychism is a deeply serious and very coherent approach.
So I'm just asking, is this true?
Have I been misled that panpsychism is actually, there's a lot to it scientifically?
I see Matt shaking his head there.
Like he doesn't agree with this characterization.
Yeah, that's a characterization of what I said.
But you know, although I have the response.
That's right.
I'm used to this, Kevin.
I'm used to this.
So you go ahead and respond to that.
I'll heap some more abuse on you then, Matt.
Yeah, so panpsychism for me is one of these sort of baffling ideas that doesn't do any work, right?
So it just, it says here we have this mystery.
Some things are conscious or consciousness is a property that we see in the world of some things.
Where could it come from?
It's just this really, really hard problem, right?
As it's, as it's called.
And one way to solve that is to say, well, maybe everything's a bit conscious.
Maybe consciousness is at the root of everything.
And therefore, like you don't solve the problem.
You're not explaining any of the phenomena that you started with.
You just dissolve it.
You just kind of push it under the rug and say you don't need to worry about that.
And it's just a weird sort of premise or conclusion because you could say the same thing for life, right?
You could say, well, we don't understand life.
It's this complex, mysterious thing.
Some things seem to have it, some other things don't, but we don't really understand why or where the transition is.
So maybe everything's a bit alive.
You know, maybe electrons are a bit alive.
And then when you put them together in certain ways and atoms and so on, they get more alive.
It's like, well, no, it's just not doing any work, philosophically speaking.
It's just saying you don't need to worry about the worry about the problem.
So for me, it's not very helpful and not very helpful.
Interesting because it just doesn't work.
I mean, the phenomena that we are looking at is like, first of all, we're having subjective thoughts and experience, right?
But also sometimes we don't.
Like when I'm under anesthesia, I don't.
So there's a contrast case that's interesting.
Now, panpsychism doesn't do anything to explain what's happening there, right?
It doesn't explain what happens when you're unconscious.
It doesn't explain having subconscious psychological processes that you're not consciously aware of and can't really access.
So all of the sort of phenomena of consciousness that make it an interesting thing to study are just sort of pushed under the rug by panpsychism.
They just say, don't worry about that.
Or they offer no explanation for it.
So if you want to think spoons are conscious, fine.
Like, go ahead.
I know you have your dinner to get to.
That was it.
That was a good explanation.
I just want to correct this really rather evil mischaracterization Chris had of me as an offhand comment.
I forget what we were talking about.
What was it?
I was contrasting him with it was something.
Oh, I was saying Alex O'Connor, because Alex O'Connor became a pan or was talking about how panpsychism is.
Yeah, but yeah, we're talking about something.
There was some other sort of thing that was even sillier.
And I said, well, you know, at least panpsychism is kind of coherent and sort of simple.
Like it has a simple, like a simple kind of elegance to it on the face of it.
Kevin fiercely.
Which is, which is exactly what you said, right?
Which is, if you say that there is, you know, even the example you gave, sometimes we're very conscious.
Sometimes humans are less conscious.
And there are other animals, if you admit that they're, you know, conscious to some degree, maybe a bit less than us, who knows?
And you can go all the way down.
And there's, you, you sort of, most people would have to admit there's a continuum.
And the argument behind panpsychism is just, well, if there's a continuum, then the continuum goes all the way down to spoons, right?
Yeah, exactly.
So on that incredibly limited, you know, in that incredibly limited way, I say, well, at least it's kind of elegant or coherent or whatever.
I don't like panpsychism.
I don't know.
I am not, and I have never been a panpsychist.
I don't associate with very coherent.
I just triggered Chris by throwing him a bar.
Sorry, Matt.
I think the label is attached and it's going to stick.
I'm sorry.
He's a panpsychist now.
I'm a panpsychist now.
Kevin, I know, I promised I let you go by.
I know you need to go get your dinner, but I have to ask you one other question while I have you here.
I mean, I can DM you it, but why not do it?
Go ahead.
So there's another debate that Matt and I have been having.
I think you will actually side with Matt here.
Okay.
So I just, just curious.
Wait, it's not a pronunciation debate, is it?
No, no, no.
So I'm definitely not siding with.
No, it is.
It's not because I know I'm right.
In this case, I think you might have more sympathy for him, but I'll see.
I remain unconvinced, even after reading your book.
Okay.
I've read your book.
I've listened to you debate Sapolsky and I've read Chalmers and Daniel Danner.
I'm not an expert, but I've read around the topic.
I just, I lack the bit where I'm kind of finding consciousness and subjectivity, this mysterious amazing.
He lacks the ability to understand that there's even a problem to be solved or even a question.
I do.
I lack this thing because I feel like, well, but just things that have brians and whatnot, and they get more complicated.
And we are agentic beings that like to imagine other futures and stuff.
So like self-consciousness in humans, not a problem.
And then in other animals that have nervous systems and reactions and like in your book, it kind of makes sense to me that this would be on the continuum of thing.
But I agree up to up to a point, right?
So if you think of consciousness as a mode of cognition, right?
It's a way of doing cognition where you have, you know, you're not just sort of, you have like an internal model of the world that you can kind of run simulations over.
That just turns out to be a super good way to do behavioral control in the world.
And you could imagine building robotic systems that have multiple levels of cognition with a sort of a highest level that is running those kinds of simulations, sort of figuring out if I do A, if I do B, what are the outcomes going to be?
What should I do?
Integrating all this sort of data with your knowledge of the world and so on.
So consciousness as this sort of highest level of your control system is actually not mysterious at all.
What's mysterious is why it feels like something, right?
That's what I'm saying.
Subjective says.
But why is that mysterious?
Okay, so that's the bit that I like there because I kind of feel like why, but it good luck, Kevin, because I've been trying to because like to me, okay, right, like the experience of echolocation or bath, that is common example, it likely feels like something.
I have no idea what it like would be like, but it would be an experience.
But like, because Matt and the philosophers like to say, ah, but what if you could have a system where you could produce all the outputs and you have all the things, but you don't have any sensation on the zombie argument, which I don't, which I don't like, because I think if you produced all the things, you would have the subjective feeling.
It's just, but that's not an explanation of why it feels like anything and why it feels the way that it feels, right?
Those are the two aspects of it.
Because imagine, Chris, you built this robot that I was talking about that has these levels of cognition.
It's doing its simulation.
It has a map of the world.
Maybe its map of the world derives from echolocation, maybe it derives from vision or addition, whiskers, whatever, right?
But it's a sort of a relational mapping of where the robot is in the world.
It uses it to navigate around, does all the things that we do with our conscious thought.
The question is like, if you built all those things, would it feel like something to be that robot?
And it's just like, it's really sort of nebulously defined.
What does that mean?
It feels like something.
So when people are talking about consciousness, it's the subjective experience of it that becomes really hard to understand.
And of course, like everyone just takes it for granted.
I think why it doesn't feel like a problem is because we live it, right?
You're just in it all the time.
So the idea that it requires explaining, I think you kind of have to pull back from it.
That's why the zombie argument was made.
I don't like the argument, but the underlying idea is like, let's get a different perspective on it to see the problems.
Okay.
In that case, though, the only, and I promise I'll shut up like this after I say this.
So the like, if you built the, you know, the hypothetical computer where it did all the things and let's just grant the kind of P-zombie thing where it didn't have an internal experience, right?
But it produced all the outputs.
Like to me, it would then just be a case of the potential for, you know, like convergent evolution.
You can build an eye in lots of different ways.
So you might be able to build something that can, in whatever mechanism, like do a version of conscious activity, but it doesn't have the subjective experience.
But a human made of flesh and blood and genes and all that kind of thing, it just produces that kind of sensation from, you know, being near to that material.
So like either way, I kind of feel like while we deal with an N equals one planet with one self-conscious thing, there's nothing but thought experiments as a counter thing where there's something that's conscious that.
So I think we're getting, you know, we're getting to at least a stage with AI and robotics where we can imagine it's not pure science to imagine a scenario where we are going to have to wrestle with this problem.
You know, I think there's also the question of like where the quality of sensations come from.
And, you know, if you're a baby, right?
And the first time you feel something painful, it hurts.
Like it feels like something.
It's not just a signal.
Oh, I should move my hand away from this thing that hurt me.
It's not just a robotic control signal.
It has a feel to it.
It has a raw feel to it.
And the question is like, where does that raw feel come from?
Or like, you know, have you ever seen videos of people giving babies a lemon?
And they taste a lemon and they clearly have this experience of tasting something sour and they have no, they haven't learned it from anywhere, right?
And they make the sour face.
And, you know, you can do it with dogs and it's very funny.
But it suggests there's some raw feels to experience that, you know, it's just really tough to explain where they could come from or why they feel different from each other.
Why does sourness feel that way?
It's just weird.
It's really hard to explain.
I like that answer because I still lack the ability to say, right?
But that speaks to you.
That's an ongoing hypothesis that Chris is a P-zombian.
I was just going to say.
I think we're edging toward that conclusion.
All right.
But I like this because that means that basically Matt Panzaiki study is like he has vindicated that there is some sort of mystery that I just lack the ability to comprehend.
So that's good.
That's balance.
We both got to win.
We've both got to win.
That's good.
Thank you, Kevin.
Yeah, you're welcome.
It's a great paper, Kevin, and we'll point to it.
Is there a preprint?
There's not a preprint, but it's going to be open access in Neuron.
Okay.
Yeah.
Yeah.
Okay.
So we'll point to it whenever it's there in any case.
But it's a great paper.
Maybe we'll cover it, Matt, on the Coding Academia.
That would be fun to get the people to have a look at it.
But yeah, so great work.
And really appreciate you explaining it to us and the audience.
