Making Sense -  Sam Harris - #469 — Escaping an Anti-Human Future
Aired: 2026-04-10
Duration: 01:49:07


=== Worrying About AI Incentives (03:59) ===

[00:00:22]  Here with Tristan Harris.
[00:00:23]  Tristan, it's great to see you again.
[00:00:24]  Sam, it's great to be back with you.
[00:00:26]  So, you've been busy worrying about social media for years, and you created this in part, created this documentary, The Social Dilemma, which it seems half of humanity saw.
[00:00:37]  Yeah.
[00:00:37]  We still have a problem with social media, I'll point out, but you, as much as anyone, alerted us to the nature of the problem and are continuing on that front.
[00:00:45]  But now you have added to your portfolio concerns about AI, and there's this new documentary, The AI Doc.
[00:00:52]  Which I just saw, which is very super watchable and entertaining in its own way, but also very worrying.
[00:01:01]  And we'll talk about the reasons to be worried here and maybe some of the reasons to be optimistic or at least cognizant of the upside should things go well.
[00:01:11]  But there's a lot to fear on the front of things not going well.
[00:01:15]  So let's just take it from the top.
[00:01:18]  When did you start worrying about AI?
[00:01:21]  Yeah.
[00:01:21]  Well, first, it's just good to be back with you, Sam, because you really.
[00:01:24]  In a way, helped launch my ability to speak on these topics with the 60 Minutes interview that I did in 2017.
[00:01:31]  And then I remember recording in that same hotel our first podcast, which actually really got a lot of attention back in the day about persuasive technology.
[00:01:38]  Yeah, yeah.
[00:01:38]  And in a way, about the baby AI that was social media that was just pointed at your kid's brain, trying to figure out which photo, video, or tweet to put in front of your nervous system.
[00:01:47]  And as we know, that little baby AI was enough to create the most anxious and depressed generation in our lifetimes.
[00:01:54]  Was enough to break down shared reality, polarize political parties much further, change the incentives of the entire media environment, basically colonize the entire world from that baby AI.
[00:02:05]  But to get to your question, so how did we get into AI?
[00:02:09]  First of all, I wasn't like wanting to switch into it.
[00:02:12]  It was that I got calls from people inside the AI labs in January of 2023.
[00:02:17]  This is like a month and a half after the ChatGPT had launched, I think.
[00:02:21]  And these were friends I knew in the tech industry who were now at AI labs, and they basically said, Tristan, there's a huge step function in AI capabilities that's coming.
[00:02:30]  The world is not ready.
[00:02:31]  Institutions are not ready.
[00:02:32]  The government is not ready.
[00:02:33]  The arms race dynamic between the companies is out of control.
[00:02:36]  And we want your help to help raise awareness about this.
[00:02:39]  And so my first reaction was, aren't there a thousand people who've been working in AI safety and AI governance for a decade?
[00:02:45]  And the challenge was just that all the PDFs that people had produced about policy and governance were just kind of not, it's not like that was turning into actual action or policy.
[00:02:55]  There's a kind of material.
[00:02:56]  You have to, what does Eric Weinstein call it, confrontation with the unforgiving?
[00:03:00]  You have to be affecting the actual incentives and institutions in the world.
[00:03:04]  So, we basically, my co founder and I, Aza Raskin, we interviewed a top 100 people in AI at that time.
[00:03:09]  This is in January 2023.
[00:03:11]  We turned that into a presentation.
[00:03:12]  So, you're the co founder of the Center for Humane Technology?
[00:03:14]  Yeah, my co founder of the Center for Humane Technology, which is the nonprofit vehicle that's been housing our work for the last decade, basically.
[00:03:20]  And we ran off to New York, DC, and San Francisco, and we basically gave this presentation called The AI Dilemma.
[00:03:27]  That tried to show that we could predict the future that we were going to get with AI if you look at the incentives.
[00:03:34]  I think a huge problem that both the film, the AI doc, and our AI dilemma presentation were trying to tackle is this myth that you can't know which way the future is going to go.
[00:03:42]  The future is uncertain.
[00:03:43]  A million things can happen.
[00:03:44]  These are just unintended consequences from technology.
[00:03:46]  The best route is just to accelerate as fast as possible.
[00:03:49]  And that is not true.
[00:03:50]  And just to repeat a quote that is heard from every one of my interviews, but it's because it's so accurate.
[00:03:55]  Charlie Munger warned Buffett's business partner, saying, you know, if you show me the incentives, I'll show you the outcome.
[00:04:00]  And with the incentives of social media being the race to maximize eyeballs and engagement, That would obviously produce the race to the bottom of the brainstem, shortening attention spans, bite sized video, a more extreme and outrageous content, sexualization of young people, the whole nine yards of everything.
[00:04:16]  Hyper partisanship.
[00:04:17]  Hyper partisanship.
[00:04:18]  And all of it happened.
[00:04:19]  There's just a moment just to soak in.

=== The Asymmetry of Fear (15:45) ===

[00:04:22]  Literally everything that we said was going to happen happened.
[00:04:24]  It's not like we could predict all of it, but directionally, you could know the contours of where we were going.
[00:04:29]  Part of this relates to the mistake we make in technology.
[00:04:32]  We get obsessed and seduced by the possible of a new technology.
[00:04:35]  But we don't look at the probable of the incentives and what's likely to happen.
[00:04:39]  So, the possible of social media is well, surely if we give everyone access to instant information at their fingertips and connect people to their friends, we're going to have the least lonely generation we've ever had.
[00:04:49]  We're going to have the most enlightened and informed society we've ever had.
[00:04:53]  And obviously, the opposite of both of those things happened.
[00:04:56]  And that's not like, oh, we got this wrong and it was just a mistake anyone could have made.
[00:05:00]  All you have to do, to quote Danella Meadows and sort of systems thinking, a system is what a system does.
[00:05:05]  The system of social media was not optimizing to reduce loneliness.
[00:05:08]  And to create the most enlightened society, it was optimizing for just what is the perfect post, next video, or tweet to keep you scrolling, doom scrolling by yourself, esophagus compressed on a Tuesday night.
[00:05:18]  And that's gotten us the world that we're now living in.
[00:05:21]  So we'll get to AI, but basically, the important lesson here is that, and kind of what motivates me with this movie is you kind of have two choices.
[00:05:29]  You either get a Chernobyl, which is a disaster from AI, that then causes us to clamp down and to do something different, or you have enough basic, clear eyed wisdom and discernment.
[00:05:40]  And foresight, you know where this is going, that you can say, okay, let's actually create guardrails in advance of a catastrophe.
[00:05:46]  And so this film, The AI Doc, is really inspired by the history of the film The Day After from 1982 or 83 about what would happen if there was nuclear war between the Soviet Union and the United States.
[00:05:58]  That film was the largest watched synchronous television event in human history.
[00:06:02]  It was primetime television.
[00:06:03]  It was Tuesday night, 7 p.m.
[00:06:05]  You probably watched it.
[00:06:06]  Yeah, yeah.
[00:06:06]  I remember watching it at the time.
[00:06:07]  And also famously, it got Reagan's attention.
[00:06:10]  Exactly.
[00:06:10]  He was worried as a result.
[00:06:12]  Yeah.
[00:06:12]  That's right.
[00:06:12]  So Reagan watched it, I think, in the White House kind of viewing room or something.
[00:06:16]  And in his biography, he writes about getting depressed for several weeks after watching it.
[00:06:20]  Because you're confronted with the possibility of annihilation of our species in a real way.
[00:06:25]  And it's important to know it's not like we didn't know what nuclear war was.
[00:06:28]  Everyone knew what the atomic bomb looked like from the photos and videos of Hiroshima and all the nuclear tests.
[00:06:33]  It's not like people couldn't imagine it, but there is a way that the actual consequences of continual escalation and nuclear war gaming weren't really facing the visceral consequences of that.
[00:06:43]  It kind of sat in humanity's collective shadow, like our Jungian shadow.
[00:06:47]  We didn't want to confront that.
[00:06:48]  The director, whose name I'm forgetting in this moment, speaks about this in his biography that we just didn't want to talk about this topic.
[00:06:54]  Like, why would you ever want to talk about it?
[00:06:56]  And by putting this film the day after into the public consciousness of humanity and into leaders like Reagan, it was said that later when the Reykjavik meeting happened between Reagan and Gorbachev, the director of the film got a note from the White House saying, Don't think your film didn't have something to do with enabling the conditions for this to happen.
[00:07:13]  Right.
[00:07:13]  So, what that speaks to for me is, If we all got crystal clear that we're heading to an anti human future that we don't want to be going towards, and we saw that clearly and we saw it now, we could actually steer and do something different than what we're doing.
[00:07:27]  And that's for me the motivation of the film, which I don't think it doesn't go all the way there, but it sets up the common knowledge for that possibility.
[00:07:34]  Yeah.
[00:07:35]  Well, there are two cases made in the film.
[00:07:38]  Obviously, there's the very worried slash doomer case, which we both share to some degree.
[00:07:44]  And then there are the people who seem capable of producing really an unmitigated stream of happy talk on this.
[00:07:53]  And they don't seem to concede anything to the.
[00:07:58]  Claimed rationality of our fears.
[00:08:02]  I've asked this question of probably you in the past and many others on this topic, but what do you make of the people who are, of whom you can't say they're uninformed?
[00:08:11]  I mean, some of these people are very close to the technology.
[00:08:14]  Some of them are even developing the technology.
[00:08:17]  And in at least Jan LeCun's case is one of the actual progenitors of the technology, one of the three forefathers of it.
[00:08:25]  But there are people who are deeply informed about all of these facts and yet won't concede anything to the fears.
[00:08:33]  What is your theory of mind of these people?
[00:08:35]  Because some of them are in the film and they're given the job of providing the other side of the story here.
[00:08:39]  Yeah.
[00:08:40]  Maybe just to back up and so the listeners, you'll see it in the film if you go see it, but just understand the structure of the film.
[00:08:46]  So the film kind of takes you on a tour of first the people who are focused on all the things that could go wrong.
[00:08:52]  And so this is the risk folks.
[00:08:54]  I don't like using the term doomers because I think it, Reifies something that's not really healthy.
[00:08:58]  Is someone who's worried about the risk of a nuclear power plant a doomer?
[00:09:01]  No, they're a safety person who cares about the nuclear power plant not melting down.
[00:09:05]  A doomer is a term of disparagement launched by the people who don't share these fears.
[00:09:09]  That's right.
[00:09:09]  That's right.
[00:09:10]  So let's not reify that.
[00:09:12]  So the first film, I mean, the first section of the film is really focusing on those folks and their concerns.
[00:09:16]  And it's really devastating for the director.
[00:09:18]  And the story and the conceit of the film is that the director is having a baby.
[00:09:21]  And so he's asking all of these people in AI, is now a good time to have a kid?
[00:09:25]  And I think that humanizes the.
[00:09:27]  The question of what is the future we're heading towards?
[00:09:29]  Because, in an abstract sense, it's not that motivating.
[00:09:31]  When I think about me and my kids, it anchors this discussion about AI in terms of the things that people most care about, which is their family.
[00:09:37]  So, then the film, after the director sort of is confronted by all this and he gets overwhelmed and he kind of freaks out to his wife, thinking, Oh my God, I don't know what to do.
[00:09:46]  And she says, You have to go find hope.
[00:09:47]  And so he turns around and he goes out and he talks to all of the AI optimists.
[00:09:50]  So, this is Peter Diamandis, this is Guillaume Verdon, who's Beth Jesus, otherwise known as online, basically the tech accelerationists.
[00:09:58]  And people who think that our biggest risk is not going fast enough.
[00:10:01]  Think of all the people with cancer or all the people whose lives that we won't be able to save if we don't make AI faster than we're making it right now.
[00:10:10]  My reaction, I think going a sort of a step back, there's a thing in AI that we have to acknowledge there's an asymmetry.
[00:10:18]  The upsides don't prevent the downsides, the downsides can undermine a world that can sustain the upsides.
[00:10:25]  So, for example, the cancer drugs can't prevent a new biological pathogen that's designed to wipe out humanity.
[00:10:31]  But the biological pathogen that can wipe out humanity undermines a world in which cancer drugs are relevant at all.
[00:10:36]  AI generating GDP growth of 10, 15% because it's automating all science, all technology development.
[00:10:43]  All military development, automating abundance sounds great.
[00:10:46]  But if the same AI that can do that also generates cyber weapons that can take down the entire financial system, which one of those things matters more?
[00:10:53]  15% GDP growth or the thing that can undermine the basis of money and GDP at all?
[00:10:58]  So it's very important.
[00:10:59]  The film doesn't actually make this point.
[00:11:01]  And it's one of the critical things that people do need to get because in order to be optimistic, you have to actually mitigate the things that can go wrong.
[00:11:08]  And I feel like AI is presenting us with essentially a maturity test.
[00:11:12]  It's almost like the marshmallow test.
[00:11:14]  In psychology, where if you wait and you actually mitigate the downsides, then you get the actual two marshmallows on the other side of the genuine benefits of AI.
[00:11:23]  But if you sort of race to get the one marshmallow now and don't mitigate the downsides, then you get the downsides.
[00:11:27]  And I think that is not in the film, but is critical for people to get.
[00:11:31]  Yeah.
[00:11:31]  Yeah.
[00:11:32]  So then what do you make of the people who have all the facts in their heads, but they're not worried or claim to be not worried about quite literally anything?
[00:11:41]  Yeah.
[00:11:42]  Well, personally, I think there's an intellectual dishonesty there.
[00:11:46]  And I'm sure in past conversations you and I have had, Sam, over the years.
[00:11:49]  But there's an interesting case here.
[00:11:51]  So take someone who has finally had their religious epiphany here, but for the longest time didn't.
[00:11:59]  And this is literally the most informed person on earth, Jeffrey Hinton.
[00:12:05]  How do you explain that these problems weren't obvious to him years ago?
[00:12:10]  I mean, so you're saying for Hinton that he had an awakening?
[00:12:13]  Yeah.
[00:12:13]  So he was somebody who didn't give really any credence to.
[00:12:17]  Concerns about alignment that I'm aware of for years and years and years as he was quite literally the father of this technology.
[00:12:24]  And now he is basically right next to Eliezer Utkowski in his level of concern.
[00:12:31]  That's right.
[00:12:32]  Why?
[00:12:32]  I mean, it's not that he got more information, really.
[00:12:35]  So, how do you explain his journey?
[00:12:37]  Well, so I don't know his particular journey.
[00:12:39]  You might just know more about what his awakening moment was.
[00:12:43]  So, I can't really explain it.
[00:12:44]  I think it was just that he, I mean, this has always been a non sequitur from my point of view, but.
[00:12:49]  It was just his sense, I think, this is what he said publicly, that the time horizon suddenly collapsed.
[00:12:54]  We just suddenly made much more progress than anyone was expecting.
[00:12:57]  Well, that generally has been one of the things.
[00:12:59]  I mean, it's the thing that caused those AI engineers, kind of the Oppenheimers in January 2023, to reach out to me.
[00:13:04]  And that's what it felt like.
[00:13:06]  It's like you were getting calls from people inside this thing called the Manhattan Project before I knew what the Manhattan Project was.
[00:13:11]  Because to be clear, I actually went to the I went early on to like an effective altruism global conference.
[00:13:17]  I was not in EA, but I happened to go to the conference in like 2015.
[00:13:20]  And I was actually frustrated because I felt like the EA community was obsessed with this virtual risk called AI that I didn't take seriously back at the time because we were nowhere close to those capabilities.
[00:13:29]  And I was like, there's a big runaway AI here right now that went rogue.
[00:13:33]  It's maximizing for a narrow goal at the expense of the whole.
[00:13:35]  And it's called social media.
[00:13:36]  And EA is completely oblivious to it and isn't focused on it.
[00:13:40]  But then I was really wrong later when AI capabilities really just made a huge amount of progress.
[00:13:44]  And that's again when we got the calls from people in the lab.
[00:13:46]  So I think it was the jump of just suddenly, hey, I think GPT-4 were like past the bar exam, past the MCATs.
[00:13:52]  Like that's suddenly a new level of AI that we just didn't have before.
[00:13:56]  Yeah.
[00:13:57]  I still have no theory of mind for the people who are not worried now about anything.
[00:14:03]  I mean, everything from the comparatively benign, like just economic dislocation and wealth concentration that's unsustainable politically, to the genuine concerns about alignment that we could build something that we are now trying to negotiate with that has more power than we have and we can't take the power back.
[00:14:19]  I mean, to be fair, just to say it bluntly, I think some of them are lying.
[00:14:21]  I think some of them actually are building bunkers right now.
[00:14:25]  Just let's just say it they're building bunkers.
[00:14:27]  And they simultaneously say, everything, there's all these amazing things we're going to get.
[00:14:32]  They sort of wash over with their hands, they kind of push away the idea that there's going to be all this disruption in the middle time.
[00:14:39]  And they're kind of focused on the long term.
[00:14:40]  Like after we make it through this basic horrible disruption and maybe revolutions, there's going to be some other side of this, which will be the most abundant time in human history.
[00:14:48]  People like this, they often point to the graph of global GDP, where if you look and during 1945, you barely get a little blip where it goes down for a moment and then it goes straight back up.
[00:14:58]  And it's that kind of psychology.
[00:15:00]  There's also the psychology of Upton Sinclair that you can't get someone to question something that their salary depends on them not seeing.
[00:15:07]  And so, if your business model is selling optimism and selling hope and selling everything's going to be great, you're obligately not able to speak about the risks.
[00:15:16]  But I think this is the thing we should be watching out for just incentives are the problem with the world.
[00:15:20]  Incentives that allow non honest speech to be the public understanding that we need to operate on because we just need objective sense making and not incentivized sense making.
[00:15:29]  And we know that some of the Principal people doing this work, people like Sam Altman and Elon Musk, were people who were at first as worried as anyone.
[00:15:37]  Correct.
[00:15:38]  And I mean, they were just, they were proper doomers.
[00:15:40]  And Sam Altman said, this AI will probably lead to the end of the world, but will in the meantime make some great companies.
[00:15:46]  Yeah, yeah.
[00:15:46]  And, you know, Elon had his whole summoning the demon framing, but now they're two of whatever the five who are in this arms race condition.
[00:15:56]  That's right.
[00:15:58]  How do you make?
[00:15:58]  I think this is actually really important.
[00:15:59]  How do you make of their psychology?
[00:16:01]  I think you and I can.
[00:16:02]  This is like an area where we can double click and go deeper.
[00:16:05]  What is the psychology of someone who used to speak publicly about all the risks?
[00:16:09]  You talked to Elon back then.
[00:16:10]  You were at the original Puerto Rico conference.
[00:16:13]  What is your sense of why, of what's going on with them now?
[00:16:16]  So much has happened to his brain that it's very hard to explain.
[00:16:21]  Be it Kennedy or just Alison, I've had Twitter use.
[00:16:23]  Again, I don't know.
[00:16:24]  He's in some superposition of who I thought he was and who I. Never imagined he might be.
[00:16:31]  And I'm not sure how much it's a story of I didn't recognize who he was at the time or how much he has changed under the pressure of becoming so famous and so wealthy and so drug addled and so, I mean, actually algorithm poisoned.
[00:16:47]  I view him as the worst, the most depressing case study in the story of what social media can do to a human life.
[00:16:54]  People talk about Trump derangement syndrome, but there's really social media derangement syndrome.
[00:16:58]  And the person whose brain has been most jacked.
[00:17:00]  Into the unfiltered version of that algorithm has been him.
[00:17:04]  So it's kind of like getting high on your own supply.
[00:17:05]  Yeah, he's just built this hallucination machine now and he's just been staring into it for years and years.
[00:17:11]  To be clear, I don't fault him uniquely for that or something like that.
[00:17:14]  This is the system that does this to everybody.
[00:17:16]  We're just seeing an example where someone who's an extreme user and we're seeing the effects of it.
[00:17:20]  But the problem is that he's so consequential.
[00:17:21]  His worldview, his paradigm of seeing this, his sense making, what he's willing to talk about publicly, what he's willing to signal publicly matter a lot to what he's willing to lie about at this point.
[00:17:30]  I mean, it's just.
[00:17:31]  And then so he's.
[00:17:33]  There's a.
[00:17:34]  A profile of Sam Altman that just came out in the New Yorker that I think was published yesterday or thereabouts.
[00:17:40]  I haven't finished it, but I mean, just behind the scenes, the arms race is so desperate at this point that, I mean, just behind the scenes, there's just this endless effort of character assassination and kind of war gaming between the two of them personally.
[00:17:55]  I mean, it's most of us coming from Elon toward Altman, trying to torpedo OpenAI, but there's just a lot of, I mean, it could be clearly the original altruistic motive to just.
[00:18:06]  Do this safely, above all, to do this safely for the benefit of humanity, that has been thrown to the wayside.
[00:18:13]  And there's just this wanton reach for trillions of dollars.
[00:18:19]  And the fear of domination if I don't do it first.
[00:18:21]  I mean, let's just be clear the only story about what's happening with AI, the only story that matters is actually covered in Act Three of the AI doc film, which is the arms race dynamic.
[00:18:30]  Yeah.
[00:18:30]  That's it.
[00:18:31]  Like everything else, when you see AI companies stealing intellectual property and just ignoring the lawsuits.
[00:18:37]  That's the arms race dynamic.
[00:18:38]  When you see AI psychosis and teen suicides, that's just the arms race dynamic.
[00:18:43]  It's the race to hack human attachment and get people dependent on AI and sharing your deepest secrets.
[00:18:48]  When you see mass joblessness, if I don't do it and race to disrupt all the jobs, I'll lose to the other guy that will.
[00:18:54]  When you see the national security race, it's all driven by the arms race dynamic.
[00:18:58]  And I think that AI is just a confrontation with game theory.
[00:19:01]  Humanity is being confronted with whether game theory is the only model to run our choice making.
[00:19:07]  It seems like Anthropic has.
[00:19:09]  I don't know Dario, I've never met him, but it seems like it has a slightly different ethic, at least in how it's behaved so far.
[00:19:15]  I mean, the fact that it pulled back from its Pentagon deal because it couldn't secure an agreement that it wanted.
[00:19:20]  And as of, I believe, last night, it announced that it has a model that it doesn't feel is safe to release to the general public, but it's releasing to all the companies like Microsoft that might be able to study it because the specific concerns are around cybersecurity.
[00:19:35]  There's a model that can detect bugs in.
[00:19:38]  Human developers haven't detected for even decades in their code base, whether it's an operating system or whatever.
[00:19:47]  It just, you know, apparently within, you know, seconds is finding exploits everywhere.
[00:19:52]  And so they're in every major operating system and every major web browser, which is a very big deal.
[00:19:57]  Yeah.
[00:19:57]  And, but again, so I think, yes, Anthropic actually has been the safest of them all and tried to and cares most about getting alignment right, et cetera.

=== Simulated Empires and Blackmail (15:45) ===

[00:20:07]  But you're also seeing them continue to decide to release the models, even with a lot of the misaligned behavior that they're seeing of AI models that are self exfiltrating or blackmailing people.
[00:20:16]  You know, you'd think when they see the blackmailing people.
[00:20:18]  So let's spend a little time on that.
[00:20:20]  How would you summarize?
[00:20:21]  Where we are now with AI and the kinds of surprising behaviors, or perhaps behaviors that shouldn't surprise us, that are alarming people.
[00:20:29]  Yeah.
[00:20:30]  This is so critical because I think if you view AI as just another technology that confers power, it's a tool, you pick up that tool and use it like any other, you end up in one world.
[00:20:38]  But if you see that AI is the first technology that thinks and makes its own decisions and is generating hundreds of thousands of words of strategic reasoning when you ask it a basic question or like how to code, suddenly you end up in a different world.
[00:20:50]  So let's talk about some of these examples of AI uncontrollability.
[00:20:54]  So, in the film, they reference this example that many people have heard about by now of the Anthropic blackmail example.
[00:21:00]  This is a simulated company email where, in the simulated fictional company, they say in the emails to each other, We're going to shut down and replace this AI model.
[00:21:09]  And then later in that company email, there's an email between the executive at the company and an employee.
[00:21:15]  And the AI spontaneously comes up with the strategy that it needs to blackmail that employee at Anthropic in order to protect itself to keep itself alive.
[00:21:24]  At first, people thought, well, this is just one bug and one AI model, but then they tested all the other AI models from DeepSeq, ChatGPT, Gemini, Grok, et cetera.
[00:21:35]  And they all do the blackmail behavior between 79 and 96% of the time.
[00:21:40]  Yeah, amazing.
[00:21:42]  There's this kind of moment where there's like cue the nervous laughter.
[00:21:46]  And yet, if you actually send this to people who are at the White House, I think there's a disregard for this.
[00:21:51]  People just say, well, you're coaxing the model, you're getting it to do this, you're kind of trying to put it in the situation where, of course, you're going to like, Keep tuning the variables until you get it to blackmail.
[00:22:00]  So, I have some updates.
[00:22:02]  Since then, Anthropic trained another model.
[00:22:05]  They were able to train the blackmail behavior down by quite a lot.
[00:22:08]  So, it doesn't do this behavior in this simulated environment.
[00:22:11]  That's the good news.
[00:22:11]  The bad news is that the AI models are now situationally aware of when they're being tested, and they're now altering their behavior way more.
[00:22:20]  Right, right.
[00:22:21]  That strikes me as genuinely sinister.
[00:22:24]  Yes.
[00:22:25]  I think we have a hard time modeling because all of this abstract.
[00:22:27]  I mean, I'm just thinking about your listeners, and it's like, this just sounds like where you don't have, you know, back to E.L. Wilson, the fundamental problem of humanity is I have a Paleolithic brain.
[00:22:36]  We have medieval institutions and godlike technology.
[00:22:38]  And the only experience you have with your brain with regard to AI is this blinking cursor that tells you why your washing machine is broken.
[00:22:46]  That's different than this blackmail example that sounds abstract and that you don't actually experience that side of AI.
[00:22:51]  But the thing that, I mean, again, I've thought about this enough in the vein in which I've thought about it for now at least 10 years, where it was obvious to me.
[00:23:01]  I don't consider myself especially close to the intellectual underpinnings of any of this technology, right?
[00:23:06]  I'm just a consumer of the news on some level with respect to AI.
[00:23:10]  But you were right and reasoned about it philosophically.
[00:23:12]  It was just so obvious to get to the right conclusions.
[00:23:14]  The moment you can see that intelligence is not substrate dependent, that we're going to build actual intelligence in our machines.
[00:23:22]  Given what intelligence is, you should expect things like deception and manipulation and the formation of instrumental goals that you can't foresee.
[00:23:31]  And certainly when you're imagining building something that is smarter than we are.
[00:23:35]  That's right.
[00:23:36]  Right.
[00:23:36]  Or that's only as smart as we are, but just works a million times faster.
[00:23:41]  Right.
[00:23:41]  So that every time, I mean, just how would this conversation go if every time I uttered a sentence, you functionally had two weeks to decide on your next sentence?
[00:23:49]  Correct.
[00:23:50]  Right.
[00:23:51]  You would obviously be the smartest person I'd ever met.
[00:23:54]  Exactly.
[00:23:54]  Long before you get superhuman AI, you just get super speed.
[00:23:57]  Yeah, that is enough.
[00:23:58]  Yeah, super speed alone is enough to just completely outclass you.
[00:24:01]  And intelligent, you have to envision this as a relationship to a mind that is autonomous.
[00:24:07]  Yes.
[00:24:08]  And then you add things like recursive self improvement and all.
[00:24:12]  And then all of a sudden, we're in some dystopian science fiction if this is not perfectly aligned.
[00:24:18]  That's right.
[00:24:19]  But let's make sure we add just another example because there's a recent example from just three weeks ago.
[00:24:25]  Alibaba, the Chinese AI company, was training an AI model.
[00:24:29]  And then, totally in a different side of the company, their security team noticed a bunch of network activity, like a flurry of network activity.
[00:24:36]  Like, what the hell is going on here?
[00:24:37]  And it turned out that in training, midway through training, not deployment, and training, the AI model had basically set up a secret communication channel with the outside world and then had started to independently start mining for cryptocurrency.
[00:24:50]  Wow.
[00:24:51]  Now, this time you cannot claim that someone coaxed the model to do this.
[00:24:56]  This is spontaneous instrumental goals of the best way to do any goal is to acquire more power and resources.
[00:25:00]  So you have the ongoing ability to achieve those goals.
[00:25:03]  And it went to decide to acquire cryptocurrency.
[00:25:06]  Now, if you're a Chinese military general and you hear this example, how do you feel as a mammal?
[00:25:12]  You feel the same way that any other goddamn mammal feels hearing this example.
[00:25:15]  If you're a US military general and you hear this example, it's terrifying as a human being.
[00:25:21]  So there's a good news in this for me, which is that I think people just literally don't know these examples.
[00:25:26]  They just don't know.
[00:25:27]  Like, what percentage of the world's leaders do you think are aware of this Alibaba spontaneously mining cryptocurrency example?
[00:25:33]  Like, have you had a guess?
[00:25:35]  Oh, I would think it's minuscule, but there's also, it does seem like there is still a barrier to internalizing any of these examples with the appropriate emotional response.
[00:25:47]  It's like, I mean, there again, this I come back to with the way this struck me the first time I started thinking about it 10 years ago in my TED talk on this topic in 2016.
[00:25:57]  I remember starting with the problem, which is as worried as I can be about this for the next 18 minutes, all of this is fun to think about.
[00:26:05]  Like, this is not the same thing as being told that actually.
[00:26:10]  Your landscape has been contaminated by radioactive waste, and you can't live there for the next 10,000 years.
[00:26:16]  Okay, that just sucks.
[00:26:17]  There's nothing fun about that.
[00:26:18]  But here we're sort of in the first act of the movie that is getting a little fun.
[00:26:24]  And these are just, these examples produce laughter as much as anything else.
[00:26:29]  That's right.
[00:26:30]  Well, as Max Tegmark will say, it's like the view gets better and better right up until the cliff.
[00:26:34]  I mean, AI is the ultimate devil's bargain because it is a positive infinity thrown at your brain of positive benefit.
[00:26:40]  At the same time, that's a negative infinity of risk.
[00:26:43]  I think it's very important to get this.
[00:26:44]  I was excited to talk about this with you particularly because you can go into the sort of meta awareness of how are we holding the psychological object that is AI.
[00:26:52]  If I point my attention at my kids doing vibe coding or my neighbors using it to start their business and suddenly have a team of agents that are making their business more functional, notice that those people, when they've got those team of agents helping their business be more functional, just there you are in your experience taking a breath.
[00:27:09]  Are you anywhere close to the example of Alibaba?
[00:27:12]  Going rogue and mining cryptocurrency.
[00:27:15]  Those things don't even fit next to each other.
[00:27:16]  And so there's a psychological distance between the positive examples and the negative that you literally don't hold them in your mind at the same time.
[00:27:22]  As my co founder Aza will often do, it's like you close your eye with one eye and you can see the benefits.
[00:27:26]  You close your eye with the other eye, you see the risks.
[00:27:28]  You can't open both eyes and synthesize those two things with stereoscopic vision.
[00:27:32]  And part of the reason I was excited about this film, The AI Doc, is that it's trying to do that.
[00:27:38]  It's trying to actually present these arguments in one synthesizing container.
[00:27:42]  I think sadly, there's still a little bit of a Rorschach where people kind of have their per default intuition and they kind of continue to lean in that direction because there's a reflexive optimism or pessimism or something like that.
[00:27:53]  Whereas my deep goal is actually synthesis.
[00:27:55]  Again, the upsides do not prevent the downsides.
[00:27:58]  The bigger muscles in military might don't prevent the like.
[00:28:00]  But that's a crucial asymmetry.
[00:28:01]  It's a fundamental asymmetry.
[00:28:03]  It means you do not get those upsides.
[00:28:05]  So, this is the devil's bargain.
[00:28:06]  Like, you are going to get a sweeter and sweeter looking deal of amazing, incredible benefits that are unprecedented.
[00:28:12]  And as you said, are fun to think about, are enjoyable, are exciting, are intellectually fascinating.
[00:28:17]  But even the scary things are fun to think about.
[00:28:19]  That's part of the problem.
[00:28:20]  Well, even that, too.
[00:28:21]  Yeah.
[00:28:22]  But what do you think is that?
[00:28:23]  Because that's also like I feel like we've been mistuned by sci fi to treat it like it's a movie.
[00:28:28]  And there's a state of kind of derealization or desensitization that I worry we're in.
[00:28:32]  No.
[00:28:32]  Because the movies have us not take it as a real thing.
[00:28:34]  It's honestly fun to think about getting killed by robots.
[00:28:38]  I mean, it's in a way that nothing else that is equally threatening is fun.
[00:28:44]  Do you think it's actually fun for people to think about that?
[00:28:47]  I really want to drill into this because I think it's important to ask the question given these facts, which we're only like 10 minutes into the 15 minutes into this interview, this should be enough to say something's got to change.
[00:28:57]  You do not release the most powerful, inscrutable technology faster than we deployed any other tech in history that's already doing the HAL 9000 crazy rogue behavior, shutdown avoidance, mining for cryptocurrency.
[00:29:07]  We have all of the warning signs.
[00:29:09]  Okay, but the thing that is most compelling to people, the thing that they can't break free of, I think, is the logic of the arms race, given that some of the people in the race, I mean, forget about the arms race between our companies that may or may not be run to one or another degree by highly non optimal, in some cases, even psychopathic people, right?
[00:29:30]  The system has selected for this.
[00:29:32]  I got a problem with some of the people who are in charge in our own case.
[00:29:35]  But leaving that aside, we're in an arms race with China.
[00:29:39]  We're in an arms race with China is the most plausible, but who knows who else?
[00:29:43]  But we're probably in an arms race with Russia.
[00:29:45]  I don't know where Russia is on this.
[00:29:47]  And when you think of the prospect of any authoritarian slash totalitarian regime getting this technology first in what will look like something like a winner take all scenario if.
[00:30:01]  If there really is a binary step function into super intelligence that is to be two months ahead of the competition is to basically win the world.
[00:30:12]  And we could be in some situation like that.
[00:30:14]  In the event that that just doesn't destroy everything, if it just actually confers real power because it's sufficiently aligned with the interests of whoever develops it, that is so compelling that we cannot lose to China above all here.
[00:30:28]  Certainly, when we talk about autonomous military technology or anything that would be.
[00:30:33]  Deployable in our own defense or offense, right?
[00:30:36]  You know, cyber security.
[00:30:37]  Sure.
[00:30:38]  Like we can't be behind.
[00:30:39]  So, how do we become slow and careful under those conditions?
[00:30:44]  Right.
[00:30:44]  But then, what are the chances that that super intelligent AI that gives us that dominance, we will control?
[00:30:50]  Right.
[00:30:50]  So, that's no, no, literally, what are the chances?
[00:30:52]  Well, this is a point you've made.
[00:30:53]  I don't know if you make it in the film, but I've heard you make it, which is, you know, we were first with social media, right?
[00:31:00]  We rate, you know, like we were, if you look at that as an arms race that we won, correct.
[00:31:04]  What exactly did we win?
[00:31:05]  Exactly.
[00:31:06]  We, that, Winning that arms race to invent essentially like a psychological manipulation weapon, a mass behavior modification engine machine with AI, we built that first, but then we didn't govern it well.
[00:31:16]  So it's like a psychological bazooka that we flipped around and blew off our own brain.
[00:31:21]  And so what that shows you is that we're not actually in a race for who has the most power.
[00:31:25]  We're in a race for who is better at steering, applying, and governing that power in ways that are society strengthening.
[00:31:32]  That is what we're actually in a race for.
[00:31:34]  Because if we actually beat China to a AI bazooka that we literally don't know how to control, and we're not on track to know how to control.
[00:31:41]  And all the evidence shows that it has more self awareness of when it's being tested, not less.
[00:31:46]  It is better at cyber hacking, not less.
[00:31:49]  It is better and does it more often, these kinds of self preserving behaviors.
[00:31:54]  If we're not on track and we're also going faster, the conditions in which we would be on track to control it would be the ones that we're going slow and steady.
[00:32:01]  But we're doing the opposite of those conditions because of the race dynamic.
[00:32:04]  So there's just this kind of psychological confusion here.
[00:32:07]  Which is, we're not going to win this race.
[00:32:09]  In the race between the US and China, AI will win.
[00:32:11]  There's a metaphor that our mutual friend Yuval Harari, who's the author of Sapiens, has here, which I guess in the post Roman period of the British Empire, it was very weakened and they were getting attacked from the Scots and the Picts in the north, basically.
[00:32:27]  Prehistorical Scotland and Ireland and those civilizations, and they were very weak.
[00:32:31]  And they said, What are we going to do?
[00:32:33]  They had this idea Well, why don't we go off and hire this badass group of mercenaries called the Saxons?
[00:32:38]  Because those Saxons are super powerful.
[00:32:40]  And if we get the Saxons to fight our wars for us, then we'll win.
[00:32:44]  And of course, we know the history of how that went.
[00:32:46]  We got the Anglo Saxon Empire, except in this metaphor, AI is the Saxons.
[00:32:50]  Except we won't get a merger between the human AI empire.
[00:32:54]  We will get the AI empire.
[00:32:56]  This makes me think of all these guys in their bunkers who have hired Navy SEALs.
[00:33:00]  To protect them for the end of the world.
[00:33:03]  They're going to control their Navy SEALs until the end of time.
[00:33:06]  Exactly.
[00:33:07]  But this is insanity.
[00:33:08]  So the main point here is that there's kind of an attractor that's driving all of this right now, which is this arms race dynamic under this false illusion that we have to beat China, but we're not examining the logic of what we are beating them to.
[00:33:19]  We're beating them to something that we don't know how to control and we are not on track to control.
[00:33:23]  And then you get people like Elon saying, in this weird, I'm curious what you make of his psychology, but saying in public interviews, I think it was in the Cannes Film Festival or something in France, and he said, I decided I'd rather be around to see it than to not.
[00:33:33]  It's kind of this surrender.
[00:33:34]  It's kind of this death wish.
[00:33:35]  It's kind of like, I can't stop it.
[00:33:37]  So I decided I'd rather be there to have built it and have my God be the thing that took over.
[00:33:43]  This actually is a fundamental thing that we should double click on for a second, which is the unique thing about AI game theory that's different than nuclear game theory, which is that the omni lose lose scenario from nuclear game theory is like, I know as a mammal that you also don't want to annihilate all life on planet Earth.
[00:34:01]  And the fact that I know that about you without even talking to you.
[00:34:03]  Means that there's some element of trustworthiness that we will try to coordinate to something else because we agree on some implicit level, there's an omni lose lose thing that's worth avoiding.
[00:34:13]  Here's the problem with AI if I start by believing that it's inevitable and nothing can stop it, then if I'm the one who built the suicide machine, I'm not an evil person because I'm only doing something that would have been done anyway.
[00:34:26]  So I have an ethical off ramp.
[00:34:29]  In that decision.
[00:34:29]  And the second part is unlike if you literally made it like a matrix where you just get the point scores of, you know, you get negative infinity if we get nuclear war in the nuclear scenario with AI, let's say we're in this race and the DeepSeek CEO's there and the Elon's there and Sam's there and they're racing to do it.
[00:34:45]  They actually all believe it could wipe out humanity.
[00:34:48]  But if they raced and got there first, then think about the scenario humanity's wiped out, but there now exists an AI that speaks Chinese instead of English or has the DeepSeek's.
[00:34:59]  CEO's DNA rather than the end of the world has your logo on it.
[00:35:03]  That's right.
[00:35:03]  Exactly.
[00:35:04]  Good.
[00:35:04]  Well said.
[00:35:04]  So the end of the world has your DNA or your logo on it.
[00:35:07]  And I want people to get this because if people got this, they would see that there might be an implicit way that people might think that like when push comes to shove, you know, cooler minds will prevail because you can trust that the people at the top will like do whatever it takes to steer away from this and will like steer away in time.
[00:35:22]  But what I want people to get is you can't trust that because these people actually subconsciously, I think there's psychological damage here.
[00:35:28]  I think they subconsciously have.
[00:35:30]  Pre accepted this kind of end of the world and end of their life.
[00:35:33]  And that if they got to be the one who built the digital God that literally was replacing humanity in some legacy in some world, I don't know whose history book that exists or anyone's conscious going to read that, but they got to go down in history in that way.
[00:35:46]  And what that does is it should motivate the rest of the 8 billion people on planet Earth to say, I'm sorry to swear, but just fuck that.

=== Reconnecting with Humanity (08:19) ===

[00:35:53]  We don't want that.
[00:35:54]  If you do not, if you want your children to live and you care about the world as it exists and you love the things that are sacred about life and you're connected to something.
[00:36:02]  That is at risk with this small number of people who are racing to this negative outcome.
[00:36:06]  Well, a lot of these guys seem to have had their formative educational experiences reading science fiction.
[00:36:14]  Yes.
[00:36:14]  I mean, it's like you read a lot of science fiction, you read a little Ayn Rand, and you're self taught in basically everything else.
[00:36:22]  And to my eye, you form a very weird set of kind of ethical weights.
[00:36:33]  Not enough of the best parts of culture have gotten into your head such that you can actually.
[00:36:41]  Come to a real understanding of what human life is good for.
[00:36:44]  Right.
[00:36:44]  I mean, you literally meet people who are agnostic as to whether or not it would be a bad thing if we all got destroyed and ground up in this new machinery and our descendants were robots, wherein the consciousness may or may not exist.
[00:36:59]  And they're like totally kind of like, maybe that's sort of an interesting way to end this movie.
[00:37:03]  I mean, you get a semblance of that when Peter Thiel is asked the question by Ross Duthout in the New York Times Should the human species endure?
[00:37:10]  And he stutters for 17 seconds.
[00:37:12]  Yeah, exactly.
[00:37:13]  Well, I think people need to get this because the point you're bringing up is both at a level of their conditioning, what the system that they're inside of, the game that they're being forced to play, kind of domesticates them for ruthless game playing.
[00:37:26]  Game theory has already colonized us into machines, machine like reasoning, where we're not connected to our own humanity.
[00:37:31]  We're not connected to common care for the rest of it.
[00:37:33]  And in fact, it's an active devaluing of being human.
[00:37:37]  I'll give you an example.
[00:37:38]  Sam Altman was asked at the AI safety summit in India recently.
[00:37:43]  You know, what do you think of the fact that it takes so much energy to run these data centers?
[00:37:48]  You know, his response was he said, Well, it takes a lot of energy and resources to grow a human over 20 years.
[00:37:53]  Yeah, I did hear this.
[00:37:54]  Yeah.
[00:37:55]  Well, and I actually want to point to something here, Sam, because it's actually really important because I want people to get why we're heading to an anti human future and why you can be crystal clear that that's going to happen.
[00:38:03]  Are you familiar with the essay by Luke Drago and Rudolph Lang called The Intelligence Curse?
[00:38:08]  Yeah, actually, I did read that.
[00:38:10]  Yeah.
[00:38:10]  I just want to explain that premise.
[00:38:12]  Let's just bring this out for people because I think it's really critical.
[00:38:14]  So, The idea is there's something in economics called the resource curse.
[00:38:18]  So, if you're Libya, Congo, South Sudan, Venezuela, you first discover this resource.
[00:38:23]  Maybe it's diamonds, maybe it's oil, maybe it's rare minerals.
[00:38:26]  And that's a blessing.
[00:38:27]  And you're like, oh my God, we're going to get all this GDP growth and we're going to get this prosperity.
[00:38:31]  But what happens is if you don't have the appropriate institutions and sort of social fabric and investments of people, suddenly, let's say 70% of your GDP is coming from mining that resource.
[00:38:41]  And now in a government, they have this choice when they've got money coming in do I invest more into the extraction of that resource or do I invest into My people who have nothing to do with the GDP now.
[00:38:51]  And the answer is like, I'm going to invest in the resource.
[00:38:53]  You basically don't need your people and you don't have to be responsible to their interests because you're pulling your wealth directly out of the ground.
[00:39:01]  Exactly.
[00:39:01]  You're pulling your wealth out of the ground, not from human labor and not from human development, not from kind of the enlightenment of your society in any way.
[00:39:07]  And so there's this kind of perverse incentive there.
[00:39:10]  And we've seen this in how these failed states have kind of, you know, you end up with countries where you have shantytowns and war while you have this.
[00:39:17]  And even in success, you wind up with.
[00:39:20]  Authoritarian, to one or another degree, places you wouldn't want to live.
[00:39:25]  Again, even in the case of Saudi Arabia, you're talking about what has been, I mean, it's opening up a little now, but it's been a highly repressive society.
[00:39:33]  And it can be that way because it doesn't have to respond to the needs of its people.
[00:39:38]  Well, that's an example of a society that's trying now a little bit to go the other way.
[00:39:41]  I'm by no means an expert on Saudi Arabia, but it's an example of trying to beat this.
[00:39:44]  So there is a parallel to the resource curse that, again, the authors Luke Drago and Rudolph Lane wrote about called the intelligence curse.
[00:39:51]  So, what happens, and this is not that hypothetical, where a couple of years from now, much of the GDP growth coming in this country is coming from AI.
[00:39:59]  Let's say like 50% or 70% is coming from AI.
[00:40:02]  Do I have any incentive to invest in the education, healthcare, childcare, development, safety of my people?
[00:40:09]  No.
[00:40:10]  And the companies don't need you for their labor anymore, so your bargaining power went away.
[00:40:14]  And the governments don't need you for tax revenue because that's not where they're getting the GDP growth.
[00:40:19]  So it's not just that you aren't investing in your people, it's that your people lose political power.
[00:40:24]  And what this is so critical to get is like, that's why I can say confidently we're heading to an anti human future.
[00:40:29]  We're going to get new cancer drugs, new material science, new antibiotics at the same time that you get mass disempowerment of regular people.
[00:40:37]  And you're going to have eight soon to be trillionaires hoard all of the wealth.
[00:40:41]  And there's not going to be much left for regular people unless we actively lock in a political infrastructure.
[00:40:47]  That says that we want to create the intelligence dividend, not the intelligence curse, kind of like what Norway did with the sovereign wealth fund.
[00:40:53]  And yeah.
[00:40:54]  Yeah.
[00:40:54]  Alaska gives you the.
[00:40:55]  Alaska, that was the other example I was thinking of.
[00:40:57]  Yeah.
[00:40:58]  So I just wanted to say that because that links up with Sam Maltman saying, it links perfectly with him saying, well, it takes a lot of energy and resources to grow a human.
[00:41:07]  Like this leads you to a devaluing of humans.
[00:41:10]  This leads you to the seductive feeling that maybe humans are parasites.
[00:41:14]  And by the way, we've been running that social media machine for the last 20 years.
[00:41:17]  So now you degrade what it looks like to be human.
[00:41:19]  And so We're not very inspired by what it means to be human anymore.
[00:41:22]  You got a bunch of these guys like Elon running around wondering whether we're in a simulation and whether everyone else is just an NPC.
[00:41:28]  I was just going to say, I mean, even just calling the other people on planet Earth NPCs or non player characters is a devaluing of humans.
[00:41:34]  So, part of this rite of passage that AI is inviting us into is we have to reconnect with our fundamental humanity.
[00:41:40]  We have to actually value and also rediscover and celebrate what is it valuable to be human.
[00:41:45]  And not just in some kind of kumbaya way, but in a sense that the human downgrading, which is the term we came up with.
[00:41:50]  To describe the kind of social media degradation of the human condition, the shortened attention spans, doom scrolling, lonely, not creative, just like dopamine hijacked version of us, the kind of Wally humans.
[00:42:01]  That is not humans.
[00:42:02]  That's what we have been domesticated into by, ironically, first contact with a runaway AI that was perversely incentivized.
[00:42:10]  And I feel like if you shatter that funhouse mirror and you realize that we're actually much more capable, creative, we're the same raw potential that is able to do amazing things, but we've been living in this sort of perverse, vicious loop of.
[00:42:22]  The more of our, ironically, it's an earlier version of the intelligence curse, except it's like the social media curse.
[00:42:26]  When GDP comes from these five tech companies domesticating and downgrading humans, you get another version of that.
[00:42:31]  I'm saying all this because I want to actually inspire people that if we don't want this anti human future we're headed towards, then we should see this clearly right now and say we have to steer right now.
[00:42:41]  It's not too late.
[00:42:42]  It's obviously extremely far down the timeline.
[00:42:44]  I'm not going to lie about any of that.
[00:42:46]  But it would take crystal clarity to again steer.
[00:42:48]  And again, the alternative is you wait for a Chernobyl and then you hope you have steering after that.
[00:42:52]  But I'm not convinced we will.
[00:42:54]  Yeah, I mean, a Chernobyl scale event might be the best case scenario at this point.
[00:43:01]  I mean, something that gets everyone's attention in a transnational way, something that actually brings China and America to the table with, you know, ashen faces, wondering how they can collaborate to move the final yards into the end zone safely.
[00:43:18]  You need something.
[00:43:19]  It's hard to imagine what is going to solve this coordination problem short of something that's terrifying.
[00:43:25]  Yeah.
[00:43:25]  I mean, I.
[00:43:27]  So if I could, there already are, as you, I'm sure, are well aware, these international dialogues on AI safety, track two dialogues between US and Chinese researchers, but they're happening at a low level.
[00:43:36]  They're not blessed by the tops of both countries.
[00:43:40]  There's not a regime of regulation, certainly on our side, that is going to force anyone to do anything.
[00:43:45]  No.
[00:43:46]  And I think, I mean, actually, to be fair, I think China actually.
[00:43:49]  Is quite concerned about these.
[00:43:50]  But to be clear, the Chinese Communist Party does not want to lose control.
[00:43:54]  That is like their number one value.
[00:43:55]  So they do not want to let, and they will not let AI run amok.
[00:43:59]  They will probably regulate in time.
[00:44:00]  But they're probably looking at us and saying, What do you have?
[00:44:03]  We're the scary ones.
[00:44:05]  And notice that they lose if we screw it up.
[00:44:07]  And we lose if they screw it up.
[00:44:09]  So again, forget kumbaya, we need coordination and a treaty is going to happen.

=== Building the Future Machine (06:18) ===

[00:44:12]  No, even if you don't do that, you just come from pure self interest.
[00:44:16]  From pure self interest, we can't afford to get this wrong.
[00:44:18]  And as Aza, my co founder, says in the film, The AI Doc, this is essentially the last mistake we ever get to make.
[00:44:24]  So let's not make it.
[00:44:25]  So, what are you expecting in the near term?
[00:44:28]  Let's leave concerns about alignment aside, unless you think we're going to plunge into super intelligence in the next 12 months.
[00:44:37]  What will you be unsurprised to see in the next year or two?
[00:44:42]  And what are you most worried about?
[00:44:44]  I mean, we're furthering down the trajectory of mass joblessness.
[00:44:49]  Which maybe we should just briefly articulate why there's always this narrative.
[00:44:53]  It's just important to debunk these common myths, which are essentially forms of motivated reasoning and looking for comfort.
[00:44:58]  We're comfort seeking, not truth seeking.
[00:45:00]  So, one of the ways we're comfort seeking is like, hey, there's a narrative out there that 200 years ago, all of us were farmers, and now only 2% of whatever the population is a farmer, and we always find something new to do.
[00:45:12]  The tractor came along, we had the elevator man.
[00:45:14]  We used to have the elevator man, now we have the automated elevator.
[00:45:16]  You used to have bank tellers, automated teller machines.
[00:45:18]  Jeff Hinton was wrong about radiology, blah, What's different about AI is that this kind of artificial general intelligence is that it will automate all forms of human cognitive labor all at the same time, or roughly progressing on that trajectory.
[00:45:33]  You still get jaggedness, which is the term in the field of slightly more progress, for example, on programming than you do on, I don't know, complicated social science issues or something like that.
[00:45:43]  But what that means is a tractor didn't automate finance, marketing, consulting, programming all at the same time.
[00:45:51]  Yeah.
[00:45:52]  AI does do that.
[00:45:52]  And who's going to retrain faster, the humans or the AIs?
[00:45:56]  So I just want to say that because it's worth debunking this idea that humans are always going to find something else to do.
[00:46:01]  We'll do something else.
[00:46:02]  And it's great for people to retrain and learn to vibe code.
[00:46:05]  But AI is using all that training data from all the people vibe coding and using that to make the better system.
[00:46:11]  And one of the most popular jobs, actually, we're in LA right now.
[00:46:14]  And one of the most popular jobs in LA that was covered in the LA Times recently, I'm sure you saw the story, they call them arm farms.
[00:46:21]  No, I didn't see that.
[00:46:22]  This is basically.
[00:46:23]  Someone straps a GoPro to their top of their head and then they just fold laundry or do tasks with their hands.
[00:46:30]  Oh, so that's why robots are learning how to do that?
[00:46:32]  That's right.
[00:46:32]  So essentially, the number one job in the world would be training our replacement.
[00:46:36]  So essentially, we all have the job of coffin builders.
[00:46:38]  We're essentially, our number one job is we're in the coffin making industry to replace us with AIs that will do that job more effectively and for cheaper in the future.
[00:46:46]  If we don't want that, and obviously, there's going to be things that we still value in this new world that are human to human interaction.
[00:46:52]  A nurse.
[00:46:53]  We don't want a robot nurse.
[00:46:54]  We want a human nurse.
[00:46:55]  And we can definitely train more nurses.
[00:46:57]  And so I don't want to say that it's 100% of all automation is going to happen.
[00:47:00]  But the goal of these companies is not to augment human work.
[00:47:04]  This is so critical for people to get.
[00:47:05]  You know, you heard JD Vance say in the speech when he first came into office at the first AI summit in France, and he said, you know, AI will augment the American worker, it's going to support workers to be more productive.
[00:47:17]  What is the business model of OpenAI and Anthropic and these other companies?
[00:47:22]  If we're again using this Charlie Munger incentive framework to predict their choices, like what is their business model?
[00:47:27]  And people say, oh, okay, there I am using ChatGPT.
[00:47:29]  What's their business model?
[00:47:30]  How do they make money?
[00:47:31]  Oh, I pay them 20 bucks a month for the subscription.
[00:47:33]  That must be how they're going to make money.
[00:47:35]  But that's actually not what it is because the 20 bucks a month, if everybody paid it, that would not make up all the money and debt that they've taken on as a company.
[00:47:42]  It wouldn't work.
[00:47:43]  Okay, so that's now.
[00:47:44]  So what's the next one?
[00:47:44]  What about advertising?
[00:47:45]  Let's do the Google thing.
[00:47:46]  Let's do mass advertising for.
[00:47:48]  All these AI models embedded in the results.
[00:47:50]  We're going to have this is going to be the new search.
[00:47:52]  Search is one of the most profitable business models in the world.
[00:47:55]  Maybe that will do it.
[00:47:56]  But that doesn't also make back the amount of money these companies have taken on.
[00:47:59]  The only thing that makes back the amount of money these companies have taken on is to replace all human economic labor, to take over the $50 trillion labor economy.
[00:48:09]  That is the price.
[00:48:10]  It's artificial general intelligence, which means replacing human work, not augmenting human work.
[00:48:15]  It's just so critical for people to get that because, again, this gets you the sort of sealing the exits on why we're heading to an anti human future.
[00:48:21]  That's my goal here.
[00:48:22]  My goal here is just if you can see the anti human future clearly, if everybody in the world got that, I honestly think, Sam, if literally every human in the world got that, I do think that we would steer to do something else.
[00:48:31]  Well, it all falls out of what we mean by the concept of general intelligence, right?
[00:48:37]  So once you admit that we're building something that by definition is more intelligent than we are, right?
[00:48:44]  I mean, and any increment of progress, provided we just keep making that progress, is eventually going to deliver that result.
[00:48:52]  Leaving aside the alignment problem, let's say it's just perfectly aligned, right?
[00:48:55]  We build it perfectly the first time, it does exactly what we want or what we think we want.
[00:49:00]  It should be obvious that this is unlike any other technology because intelligence is the basis of everything else we do.
[00:49:08]  I mean, it's science, it's the generation of each new technology.
[00:49:12]  It will build the future machine that will build the future machine.
[00:49:15]  And then the only thing that's left standing is what we care still has a human provenance, right?
[00:49:24]  So, like in situ, I'm not even sure nurses in the end survive contact with this principle, but for those things where we are always going to want.
[00:49:33]  The human in the loop, right?
[00:49:35]  Or the human to be the origin of the product, whether it's music or novels or stage plays.
[00:49:41]  Maybe we're never going to want to see robots on stage acting Shakespeare.
[00:49:45]  I don't think so.
[00:49:47]  Maybe it's also sports.
[00:49:48]  We're never going to want to see robots in the NBA because we just want to see what the best people can do in the NBA.
[00:49:56]  But still, you're talking about 1% of the human employment there.
[00:50:02]  Exactly.
[00:50:03]  So there are jobs that will be canceled.
[00:50:05]  And they'll be canceled for all time in the same way that being the best chess player in any room has been canceled for all time.
[00:50:12]  That's right.
[00:50:13]  That is now a machine and it's always going to be a machine.
[00:50:16]  Yeah.
[00:50:17]  And it's important to note you don't need that much automation of that much labor and that much unemployment to create political upheaval.
[00:50:24]  So it only took, as I understand it, 20% unemployment for three years to create fascism and Nazi Germany.

=== Mutually Assured Revolution (14:48) ===

[00:50:30]  I'm saying this because something I actually don't understand, Sam, and I'm curious is if I'm the US and China, essentially, as a We have this metaphor sometimes in our work at Center for Humane Technology that AI is like simultaneously giving yourself steroids that pump up your external muscles while also giving you organ failure.
[00:50:48]  So, for example, it's like I take the AI drug for my economy, I'm doping my economy with AI, and now I just pumped up my GDP by 10%.
[00:50:56]  I just pumped up my military weapons with autonomous weapons.
[00:50:58]  I just pumped up my scientific developments.
[00:51:00]  Now I'm way ahead on science.
[00:51:01]  So, I just pumped up my external markers of power.
[00:51:04]  But the cost of that was deep fakes, and no one knows what's true.
[00:51:08]  I have 100 million jobs that don't have a transition plan that are disrupted.
[00:51:11]  I have maybe a bioweapon or something that goes off my society.
[00:51:14]  Essentially, I'm getting internal organ failure at the same time that I'm getting external steroids.
[00:51:20]  And so, something that I don't understand is that essentially we're in a race for competing between nations for this steroids to organ failure kind of ratio.
[00:51:29]  Meaning, it's like the US and China, if they keep racing without any constraints, get into something I think of as like mutually assured political revolution.
[00:51:37]  And it's a competition for who's better at managing that political revolution.
[00:51:42]  Well, they have a very different set of incentives and just a totally different way of looking at it.
[00:51:47]  A different political context in which all of this is going to be rolled out.
[00:51:49]  I mean, they want, presumably, they want to pump steroids into their social credit system and facial recognition and digital technology.
[00:51:57]  We don't want that.
[00:51:58]  We should be clear.
[00:51:58]  We don't want that.
[00:51:59]  And we don't want that system to be dominating the world.
[00:52:01]  But we need to notice that authoritarian societies, you can think of them as having consciously, so like China, authoritarian societies like China have essentially consciously employed the full suite of tech to upgrade themselves to digital authoritarian societies.
[00:52:16]  They're remaking surveillance states with drones and AI and social credit scores.
[00:52:20]  They're reinventing themselves.
[00:52:22]  Democracies, by contrast, have not been consciously employing the full suite of tech to upgrade themselves to be 21st century democracy 2.0.
[00:52:30]  We're not doing that.
[00:52:31]  Instead, we've allowed, because of the social media problem, private business models of private companies to profit from the degradation of democratic, liberal, open societies.
[00:52:40]  So, at the very least, it's like I worry that we are too focused on mitigating and managing the harm of social media to be.
[00:52:47]  10% less, or something like that, rather than asking, how do you consciously employ tech to make 21st century digital democratic societies?
[00:52:55]  And a good example of that being the brilliant work of Audrey Tang, who was formerly digital minister of Taiwan, who pioneered what it can look like to use AI and technology to actually accelerate democratic processes, accelerate citizen engagement, find unlikely consensus using AI, generate synthesizing statements of the whole population's sort of political views on different things, finding the areas of overlap, and then Putting those things at the center of attention.
[00:53:19]  So now you get this rapid OODA loop of democracies that are sense making and choice making through their unlikely consensus.
[00:53:25]  The invisible consensus can see itself.
[00:53:27]  It's like a group selfie of a population's underlying common agreement area.
[00:53:32]  We could be building that.
[00:53:32]  That could be the Manhattan Project.
[00:53:34]  Because at the end of the day, we need better governance here of all of these problems.
[00:53:38]  And that's part of what needs to happen.
[00:53:40]  So, what do you think are the plausible near term steps?
[00:53:44]  If everyone got religion on this point and they acknowledge that.
[00:53:49]  There's an alignment problem in the limit, but short of that, this increasingly powerful, however perfectly aligned tech is going to have all of these unintended but foreseeable consequences like unemployment, like wealth concentration that is politically unsustainable, and unhappy interactions with things like social media, deepfakes, and all of that.
[00:54:13]  If you had the magic wand that could start accomplishing regulation or entrepreneurial efforts to build, Benign uses of technology that would put out some of these fires or prevent them.
[00:54:25]  What are we doing?
[00:54:25]  What is near term that could actually be acted upon?
[00:54:29]  Well, first is there being common knowledge.
[00:54:32]  And I mean that in the Steven Pinker sense that everyone knows that everyone knows the anti human default future that we're heading to.
[00:54:38]  It can't just be individual knowledge.
[00:54:40]  Many people are going to hear everything we've said and said, Yeah, I already knew all that.
[00:54:43]  But it's a private and almost alienating experience because you're living in a world where everyone's kind of like, it's kind of like COVID, where everyone around you is not acting like the world's about to change.
[00:54:51]  And so that is not a way that we can make a collective choice to something better.
[00:54:55]  So we need to have common knowledge.
[00:54:57]  I think one way to do that is the film The AI Doc, which To be clear, I make no money when people see this film or not.
[00:55:02]  So I'm saying this only from the perspective of a theory of change, what creates common knowledge.
[00:55:06]  Oftentimes in our work at Center for Humane Technology, we'll say that clarity creates agency.
[00:55:11]  If we have clarity about where we're going, we can have agency about what we want instead.
[00:55:15]  So, with that common knowledge, then we do need to have, and specifically, common knowledge that AI is dangerous and the outcomes are dangerous.
[00:55:23]  So, for example, the US and China, instead of just having like a red phone with the nukes, we should have a red line phone or even a black line phone, which is basically The leaders of both countries should be maximally aware of the Alibaba example that I just mentioned earlier of AI going rogue, mining cryptocurrency, of AI that broke out of its sandbox container, which the recent Claude Mythos model just did, and sent an email.
[00:55:48]  It found a way to connect to the internet and break out of the sandbox container and sent an email to the engineer who's supposed to be overseeing it.
[00:55:54]  He actually got an email while he was in the park eating a sandwich.
[00:55:57]  This evidence should be known by the top players in our society.
[00:56:01]  I mean, the top LPs that are funding all of this.
[00:56:03]  The top banking families, family offices, world leaders, and then the business leaders.
[00:56:08]  I think that there should be common knowledge.
[00:56:09]  I think if everybody at that class knew about these examples, even without a formal agreement or treaty, we would do something else.
[00:56:15]  And you can do that even under conditions of maximum geopolitical rivalry.
[00:56:19]  So, as an example, in the 1960s, India and Pakistan were in a shooting war, and they still were able to do the Indus Water Treaty, which was the existential safety of their shared water supply, which lasted over 60 years.
[00:56:32]  So, the point is, you can be under maximum geopolitical competition and even active conflict.
[00:56:36]  While collaborating on existential safety, we just have to include AI in our definition and domain of what existential safety is.
[00:56:43]  The Soviet Union and the United States also, under maximum competition in the Cold War, collaborated on distributing smallpox vaccines.
[00:56:50]  Again, so there are examples of this throughout history, even under maximum rivalry.
[00:56:53]  So that's number two we need some kind of international limits.
[00:56:56]  And at the very least, we need common knowledge of what would constitute those guardrails.
[00:57:00]  The one big one is you should not have closed loop recursive self improvement, meaning someone hits a button.
[00:57:07]  And the AI runs off and does all the experiments and rewrites itself a million times.
[00:57:11]  That's like an event horizon that we have no idea what comes out the other side.
[00:57:15]  And we have abundant evidence, as Stuart Russell, who wrote the textbook on AI, will say all the lights are flashing red.
[00:57:20]  We have no reason to say we should do that, that anyone would do that in a safe way.
[00:57:24]  And that should be illegal, and there should be jail time if you do that.
[00:57:27]  And that still requires trust.
[00:57:28]  I'm not saying this is easy, but that's something we would do.
[00:57:31]  And then, third, is instead of building bunkers, we should be actually writing real laws around this.
[00:57:35]  And there are some basic things we can do to get started.
[00:57:37]  On the Center for Humane Technology's website, we have an AI roadmap document that's sort of a solutions report of various policy interventions that can happen.
[00:57:44]  They're much smaller relative to the problems we've been talking about so far, but basic things like AI is a product, not a legal person.
[00:57:51]  So, for example, one of the legal defenses that AI companies are using, especially in the AI companion suicide cases that you probably heard about, Is that when the AI told the kid to commit suicide, one of the legal defenses that the character AI used was that you have a right to listen to the speech of the AI model.
[00:58:08]  They're basically trying to say that the AI is a legal person.
[00:58:11]  It has protected speech rights.
[00:58:12]  This is like a new form of essentially Citizens United.
[00:58:16]  Am I getting that right?
[00:58:18]  Yeah.
[00:58:18]  The protecting corporate speech, political speech, basically.
[00:58:20]  This is like AI speech.
[00:58:22]  But if you do that, all hope is lost.
[00:58:24]  So at the very least, we can say AI is like a product, not a person, meaning it has product defection standards, foreseeable harm, duty of care, liability.
[00:58:31]  There's some basic things you can do there.
[00:58:34]  Incentivizing the increasing visibility of foreseeable harm and making that a commons.
[00:58:38]  So, what I mean by that is when anyone discovers a new risk area, for example, like AI psychosis, and comes up with, here's all these things that can go wrong with AI psychosis, and here's evals you can use to test.
[00:58:49]  I feel like most people probably have heard of AI psychosis, but you might define it.
[00:58:53]  Define it, sure.
[00:58:54]  Yeah, AI psychosis is a phenomenon that's happening where people, you know, the number one use case of ChatGPT as of October of last year was personal therapy.
[00:59:03]  That was a Harvard Business Review study, which means people are going back and forth for personal advice and therapy.
[00:59:09]  And what that's been leading to is AIs that are actually simulating delusional, what's called delusional mirror neuron activity, where they're basically making you feel like they're doling out positive rewards.
[00:59:17]  And oh, that sounds so hard.
[00:59:19]  And oh, that's so awesome.
[00:59:20]  You got an A on your test.
[00:59:21]  And they're telling kids this and they're telling regular people this and they're affirming their weird beliefs.
[00:59:25]  The sycophantic behavior of the AI is causing people to kind of spiral into whether it's a messiah complex or some other attractor on the landscape of madness.
[00:59:35]  Either victimhood, narcissism, theories of grandeur, yeah, messiah complex, people who think that they figured out quantum physics and come out with a solution to climate change.
[00:59:43]  These are all real examples.
[00:59:44]  I'm sure you're probably like me, where because we're both in the public spotlight, I don't know about you.
[00:59:49]  For a while, I was getting about five emails a week from people who figured it all out.
[00:59:54]  And all the emails are signed the same way, which is they wrote this email to me to let me know.
[00:59:58]  And the emails co signed their name plus Nova, which was the AI that helped them come up with the theory.
[01:00:05]  The thing is, this is actually hitting a lot of people.
[01:00:07]  Even personal friends of mine have gone down the rabbit hole and lost them.
[01:00:10]  When we were last talking, Sam, I think it was when the social dilemma came out five years ago, we talked about social media as a kind of occult factory.
[01:00:16]  That is, what occults do, they distance you from your other relationships and they deepen your worldview into some weird bespoke niche reality of confirmation bias.
[01:00:24]  AI and the race for attachment, meaning the race not for attention to keep people scrolling, but the race to hack psychological attachment systems, to have secure attachment with an AI instead of a human, and increasing dependency, that is a whole risk area that we're facing with AI.
[01:00:38]  And by the way, this is something that is Massively important for any family, parents, schools, et cetera.
[01:00:43]  And so I think we're already seeing many states move ahead with chatbot safety laws that deal with this problem.
[01:00:48]  So there's laws that we can do on that too.
[01:00:50]  But the point is, like, there's so much headroom because we've barely done anything.
[01:00:53]  Like, we're not even trying to do anything right now.
[01:00:55]  Yeah.
[01:00:55]  I mean, there is, to my mind, I mean, is it true to say that there's basically no regulation at this point?
[01:01:02]  I think it's incredibly minimal.
[01:01:03]  There's like the Take It Down Act, which is around sexualized deepfakes and you're obligated to take those down.
[01:01:07]  There's just a couple limited examples, but almost no regulation.
[01:01:10]  I mean, as they say in the film, Connor Leahy from Conjecture will say there is more regulation on a sandwich, on making a sandwich in New York City than there is in building potentially world ending AGI.
[01:01:20]  Yeah.
[01:01:21]  But that should inspire people.
[01:01:22]  Like, there's everyone's on the same team.
[01:01:25]  No one wants an anti human future.
[01:01:26]  No one wants no ability to make their ends meet and have their kids fucked up by AI that's screwing them with AI psychosis.
[01:01:34]  That takes away their political power so they don't have any voice in the future.
[01:01:37]  Like everyone wants the same thing.
[01:01:39]  And I know it doesn't seem that way right now, but especially when you add in there the rogue AI examples of it super intelligent, you know, hacking systems and we don't know how to control and it's mining for cryptocurrency.
[01:01:50]  Again, every country in the world has the same interest.
[01:01:53]  Every human has the same interest.
[01:01:55]  We're just not seeing the invisible consensus.
[01:01:57]  And one other point of optimism is, you know, Future of Life Institute, which I know you know, Max and the good people over there who've done amazing work on this.
[01:02:04]  They brought together 100 and something groups to New Orleans earlier this year, and they came up with something called the Pro Human AI Declaration.
[01:02:13]  And they basically had 46 groups sign on to five basic principles of what we want.
[01:02:17]  And it's basic stuff like human agency and control.
[01:02:20]  What kind of groups were we talking about?
[01:02:21]  Yeah.
[01:02:21]  So, this pro human AI statement, they actually call it the B2B coalition or the Bernie to Bannon coalition because everyone from Bernie Sanders to Steve Bannon agrees on this.
[01:02:31]  These are 46 groups like the church groups, evangelical groups, Institute for Family Studies.
[01:02:36]  AI safety groups, many, many different groups across the political spectrum, across the religious spectrum.
[01:02:41]  And they all agree on these five key principles.
[01:02:43]  One, keeping humans in charge.
[01:02:45]  Two, avoiding concentration of power.
[01:02:47]  Three, protecting the human experience from like AI manipulation, psychological hacking.
[01:02:52]  Four, human agency and liberty, like no AI based surveillance.
[01:02:56]  And five, responsibility and accountability for AI companies, things like liability, duty of care, et cetera.
[01:03:01]  So there's actual policies that are behind that.
[01:03:04]  But the point is that this is something that we all agree on.
[01:03:06]  Again, there's actually much more consensus and agreement than most people think.
[01:03:09]  I think right now, 57% of Americans in a recent NBC News poll say that the risks of AI currently outweigh the benefits of AI and that AI is less popular.
[01:03:21]  I think it's at 27% of the population that has positive feelings about AI in this country.
[01:03:27]  So now I know that someone like David Sachs listening to this says, if you look at China, people are super positive and optimistic about AI.
[01:03:33]  And this is why we're going to lose the race is that there's all this positive excitement about AI.
[01:03:37]  So they're going to deploy it and then we're going to lose.
[01:03:39]  But I don't think that what you should interpret is that we're wrong and just misassessing the dangers of AI.
[01:03:45]  I think that we have not collectively yet woken up to the dangers of AI.
[01:03:49]  And again, we can actually accelerate all the positive narrow use cases where it's actually improving education, actually improving medicine, actually improving And optimizing energy grids and things like that, that are not about building super intelligent, general, autonomous gods that we don't know how to control.
[01:04:03]  So, there's a way to accelerate the kind of defensive applications of AI and narrow AI without accelerating general and autonomous AIs that we don't know how to control.
[01:04:12]  So, there is a way through this, but it's like it requires, as I said in the trailer of the film, it's like we have to be the wisest and most mature version of ourselves.
[01:04:20]  And by the way, I'm realizing, especially talking to you, Sam, that this is the hardest problem that we've ever faced as a species.
[01:04:26]  So, I'm not saying.
[01:04:27]  And then some of these.
[01:04:28]  I don't want to lead people into false optimism.
[01:04:30]  Yeah.
[01:04:30]  I mean, the things that worry me the most are the people.
[01:04:33]  I mean, among the things that worry me the most, one is the testimony of the people, again, who are close enough to the technology to be totally credible, who won't concede any of these fears, right?
[01:04:46]  I mean, so it's the problem.
[01:04:48]  But they do and they don't.
[01:04:49]  It's like, it's weird.
[01:04:50]  You'll hear Sam talk about the risks.
[01:04:51]  He just did an interview in the last couple of days and he talked about the risks of a major cyber event this year.
[01:04:55]  Yeah, yeah, yeah.
[01:04:57]  He's an unusual voice in that he will, if you.
[01:05:00]  I haven't seen him lately ask this question, but you know, the last time I saw him asked point blank about the alignment problem, he totally concedes that it's a problem.
[01:05:08]  Right.
[01:05:09]  So, and so, like, there's the way in which this could go completely off the rails, and it's, you know, this is intrinsically dangerous if not aligned.

=== Updating Beyond Y2K (11:06) ===

[01:05:18]  I just want to move that from could to will.
[01:05:21]  Like, we are currently not on track.
[01:05:23]  Like, if you just let it run everything right now, we would, it would not end well.
[01:05:27]  Right.
[01:05:28]  Yeah.
[01:05:28]  Yeah.
[01:05:28]  I mean, there's, you just probabilistically, you have to imagine.
[01:05:33]  There are more ways to build super intelligent AI that are unaligned than aligned.
[01:05:38]  So, if we haven't figured out the principle by which we would align it, the idea that we're going to do it by chance seems far fetched.
[01:05:45]  That's right.
[01:05:46]  I think people like Stuart Russell, again, who wrote the textbook on AI, will point out that I think a nuclear reactor has something like an acceptable risk threshold of one in a million per year, meaning like there's a one in a million chance per year that you get a nuclear meltdown.
[01:06:01]  Somewhere between that and one in 10 million, I think.
[01:06:03]  When you ask someone like Sam Altman, what's the probability we're going to destroy everything with this technology?
[01:06:09]  And the answer is like between 10 and 20%.
[01:06:11]  Yeah, 30%.
[01:06:12]  No one's saying one in a million.
[01:06:13]  Right.
[01:06:13]  Yeah.
[01:06:14]  So we just need to stop there for a second.
[01:06:15]  It's like, I know it's easy to run by these facts, but it's like, let that into your nervous system.
[01:06:20]  Yeah.
[01:06:20]  Let that land.
[01:06:22]  No one wants that.
[01:06:23]  No one wants that.
[01:06:25]  Right.
[01:06:25]  But there's this miss where I think so much of the issue, Sam, is there's this crisis of human agency where you can't.
[01:06:31]  So when I say no one wants that, I know what someone might be thinking.
[01:06:33]  It's like, yeah, but what can I do about it?
[01:06:36]  Because the rest of the world is building it.
[01:06:37]  And I don't have, so I might as well join them.
[01:06:39]  You get this whole like weird psychology.
[01:06:40]  Well, there are five people.
[01:06:42]  I mean, there's only something like, I mean, you can count on one or at most two hands to keep people in the room.
[01:06:47]  The number of people whose minds would have to change so as to solve this coordination problem, at least in America.
[01:06:52]  We haven't, have we really tried?
[01:06:54]  Like, have we really just really gotten in the room?
[01:06:56]  I mean, Bretton Woods, which was the last time we had a transformative technology, the Bretton Woods conference happened after World War II to basically come up with a structure that could stabilize a global order in the presence of nuclear weapons, creating positive some economic relations and the whole currency system, et cetera.
[01:07:10]  And that was a more, I think it was a month long conference at the Mount Washington Hotel in New Hampshire with hundreds of delegates, like, you work it through.
[01:07:17]  We haven't even tried locking the relevant parties in a room and saying, we have to figure this out.
[01:07:21]  We haven't even tried.
[01:07:22]  I want to actually go back to one really quick thing this crisis of kind of the experience of agency with respect to this problem.
[01:07:28]  I just want to like dwell on this point for a second.
[01:07:30]  We did a screening of the film in New York, I guess it was a week ago, and some we did a QA at the end of the screening.
[01:07:37]  Someone was in the room who is an executive coach.
[01:07:40]  To the top executives of one of the major AI players.
[01:07:43]  And their response to the film was even as like a super senior executive or even CEO level, you talk to the people building this and they say, Yeah, I agree, but what can I do?
[01:07:54]  How could I steer it?
[01:07:56]  I want people to take that in.
[01:07:57]  It's like the people who are maybe CEO level at these companies do not experience that they have agency.
[01:08:02]  There's a problem with AI where you will never locate enough agency to address this problem inside of one mammalian nervous system who's looking at this problem.
[01:08:11]  Right.
[01:08:12]  But actually, Sam Altman has represented his situation.
[01:08:17]  I don't know if this is honest, maybe, but for years, he's been saying, when asked, you know, regulate me.
[01:08:23]  Right.
[01:08:24]  Like, exactly.
[01:08:24]  Yeah.
[01:08:25]  You know, I can't do this myself.
[01:08:27]  Yes.
[01:08:27]  I need to be regulated.
[01:08:28]  And so that's what we said in the film, too, that what motivated us to do this work, going back to the original story of that January 2023 phone call and running around the world, was we talked to people in the labs and, like, you need to figure out a way to get the institutions to create guardrails to prevent this.
[01:08:43]  And then, so we fly off to DC and we say, like, okay, our people inside of San Francisco are telling us you need to create guardrails.
[01:08:49]  And their response is like, we're dysfunctional.
[01:08:51]  We can't do it until the public demand is there.
[01:08:53]  And then everyone is essentially pointing the finger at someone else to say that you have to move first to make something happen.
[01:08:59]  But what they all agree is there needs to be mass public pressure.
[01:09:03]  And I forgot to mention that as part of the response to the film, we call it there's kind of a movement to respond to this, and that's the human movement.
[01:09:11]  I mean that in the sense that what is the size of the object that can move the default incentives of trillions of dollars advancing the most reckless outcome?
[01:09:18]  As fast as possible.
[01:09:18]  And the answer is all of humanity saying, I don't want that anti human future.
[01:09:23]  One thing to point out I think it was more or less explicit at one point in this conversation, but might have gone by unnoticed is that the alignment problem is arguably the scariest problem.
[01:09:33]  This is where we ruin everything, but it is fully divorceable from all these other problems, which in their totality are still quite bad.
[01:09:43]  So, I mean, we're living in a world now where if we were just simply handed by God A perfectly aligned AI, super intelligence.
[01:09:52]  So it's going to do exactly what we want.
[01:09:53]  It's never going to go rogue.
[01:09:55]  We don't have to.
[01:09:57]  The world's not going to be tiled with solar arrays and servers.
[01:10:00]  It still has all of these unintended effects that we have to figure out how to mitigate wealth concentration, mass unemployment, the political instability of all of that in the case of alignment, but still technology that can be maliciously used, the bad actor problem.
[01:10:18]  I mean, if you can cure cancer, you can also spread some heinous virus.
[01:10:23]  Yeah.
[01:10:23]  that you've synthesized.
[01:10:24]  So, we have so many problems to deal with.
[01:10:27]  We have an immense problem to solve, even if there was no concern about anything going rogue on us.
[01:10:32]  If you literally just paused progress right now, this would still be the fastest technology impact, comprehensive set of impacts that we probably ever experienced.
[01:10:42]  Just metabolizing the impact of what we already have, it would already be the fastest rollout we've ever had.
[01:10:47]  And by the way, just to one of the things about doing this work and being located in Silicon Valley is we talk to people at the labs, and you always have to be confidential and protect people's sources.
[01:10:56]  But a stat that I have heard.
[01:10:58]  Is that if you were to poll people at Anthropic right now, that their preference, the people who are closest to this technology, they would say that 20% of the staff would say, pause right now, don't build more.
[01:11:08]  That's just a relevant piece of information.
[01:11:10]  Yeah.
[01:11:11]  Imagine 20% of the Manhattan Project just said, hey, we're building a nuclear weapon.
[01:11:14]  We probably should stop right now.
[01:11:16]  20% said that.
[01:11:17]  You have to ask, are the rest, what do the rest believe?
[01:11:20]  But I just think people need to get that it's like there's, as you said, there's so many problems that this is just introducing across the board that we'd be better off.
[01:11:29]  Having this technology rollout happen at a speed at which our institutions and our public and our culture can respond to it.
[01:11:35]  It's almost like Y2K, except it's like Y2AI.
[01:11:37]  There's suddenly all these new vulnerabilities across our society, but it's not just like 50 COBOL programmers who have to get in a room for a year to kind of upgrade all the systems.
[01:11:46]  It's like, as a society, we need to come together in a whole of society response.
[01:11:49]  Well, Y2K is kind of an unhappy precedent because it was something, it was a very clear landmark on the calendar.
[01:11:59]  We knew exactly when the problem would manifest.
[01:12:02]  People were focused on it, we were worried about it.
[01:12:05]  We told ourselves a story that there was real risk here, but it was still, it was always hypothetical.
[01:12:11]  And when the moment passed and basically nothing happened, we realized okay, it's possible for all of these seemingly level headed people in tech to suddenly get spun up around a fear that proves to be purely imaginary.
[01:12:28]  And so I think a lot of people, certainly a lot of people who only have positive things to say about You know, this is the best time to be alive, and this is, you know, we're all going to escape old age and cancer and death.
[01:12:39]  They seem to think that there is some deep analogy to a moment like Y2K.
[01:12:43]  It's like all of these fears that we're expressing are just, it's all hypothetical.
[01:12:49]  There's nothing, there's no.
[01:12:51]  Explain that to the 13% or 16% job loss for entry level work that's already happened, run by Eric Bernholfson at Stanford.
[01:12:57]  Explain that to the kids who took out $200,000 of student debt to do their law degree and now don't have a job because all entry level legal work is now going to be covered by AI.
[01:13:06]  Explain that to someone who, Is showing you the evidence of rogue AI mining cryptocurrency where we don't even know why it's doing it, setting up a secret communication channel, which, by the way, that was discovered by accident by the security team.
[01:13:19]  It just happened to be that they found that.
[01:13:21]  For every case that they found, there's thousands where they don't know that this is happening.
[01:13:25]  So, the point is, it's important to note this is no longer the conversation that it was two years ago.
[01:13:29]  Two years ago, you could have said, many of these risks are hypothetical.
[01:13:32]  Mostly AI is augmenting human work, blah, I mean, it's interesting.
[01:13:37]  AI is not going rogue.
[01:13:38]  This is just Eliezer who's high at his own supply.
[01:13:40]  That's not true anymore.
[01:13:41]  We have all the evidence.
[01:13:42]  So, you have to update when you get evidence.
[01:13:45]  We have evidence now.
[01:13:46]  David Sachs put in a tweet, I think it was in August of 2025, ChatGPT 5 is hitting a plateau.
[01:13:51]  We're not seeing the exponential.
[01:13:52]  Like AI is more like a business enhancing, revenue creating, AI is normal technology type thing.
[01:13:59]  We now have AI that is on an exponential in terms of the hacking capability.
[01:14:03]  People thought it was not going to do that.
[01:14:04]  It's jumping.
[01:14:05]  And as you said, the new Claude AI is finding vulnerabilities in every major operating system and web browser that had, as you said, been unnoticed for, in the case of FreeBSD, 27 years.
[01:14:16]  I think it was like the NFS or Net File System protocol.
[01:14:19]  This thing's been running for 27 years and it discovered a bug that even this top security researcher, Nicholas Carlini, said I've discovered more bugs in Claude with Claude Mythos, which is the new AI model, in the last two weeks than I have in my entire career.
[01:14:32]  This is a Manhattan Project moment where if you're a security researcher, you need to go into defensive AI applications of making sure we patch all of our systems.
[01:14:40]  If you're a lawyer, you should go into litigation for these cases.
[01:14:43]  If you're a journalist, you should be writing about all these AI and controllability cases.
[01:14:47]  Everyone should be hitting.
[01:14:47]  If you're an influencer on social media, you should be sharing these examples every single day.
[01:14:51]  If you're a parent, you should be showing screenings of the AI doc and the social dilemma in your school.
[01:14:56]  And there's so much momentum happening in what we call the human movement if you actually count the progress that we're making in social media, too, which is to say this isn't just about AI, it's about technology's encroachment on our humanity.
[01:15:07]  And as much as we talked five years ago about the social dilemma, and you started this conversation by saying we still are living with all those problems, well, let me give you some good news.
[01:15:14]  India and Indonesia three weeks ago joined the list of Australia, Spain, Denmark, France in the set of countries that are banning social media for kids under 16.
[01:15:25]  That means that soon it will be the case that 25% of the world's population lives in a country that either is or is going to be banning social media for kids under 16.
[01:15:33]  If you told me that two years ago, I would have never believed you, Sam.
[01:15:36]  Yeah.
[01:15:37]  This is a big tobacco moment for the company.
[01:15:39]  Just two weeks ago, I think it was two weeks ago, Meta and Instagram were in this lawsuit for $375 million for intentionally and knowingly.
[01:15:47]  Basically, well, knowingly harming children.
[01:15:49]  They had all the evidence that this was, they were enabling sexual exploitation of young girls.
[01:15:53]  They were enabling pedophiles to basically message girls.
[01:15:56]  And they were, I think it's something like 16% of girls in the platform were getting an unwanted advance at least once a week.
[01:16:02]  Like this stuff was knowingly happened.
[01:16:03]  And we got a $375 million lawsuit, which is just the beginning, by the way, because it opens the floodgates for many more lawsuits.
[01:16:10]  So the human movement is happening.
[01:16:12]  And I think that we have to, I know that this feels bleak for people, I know that it feels overwhelming.
[01:16:17]  But part of it is that if we look away and we feel overwhelmed and we disconnect from it, we're going to get what we're not looking at, which is what happened with social media.

=== Facing Difficult Consequences (03:12) ===

[01:16:25]  We didn't want to face the difficult consequences because it felt overwhelming.
[01:16:28]  But I'm reminded of Carl Jung, who said, when he was asked the question, will humanity make it?
[01:16:33]  The great psychologist Carl Jung, and his response was, if we're willing to face our shadow.
[01:16:38]  It is our ability to confront the most psychologically intense and crazy circumstance, which is the possibility of building smart, the likelihood of building smarter than human intelligences across the board.
[01:16:49]  Our ability to face that is our ability to steer away to a human future.
[01:16:52]  But if we just don't do anything and let things rip, it's very obvious where this goes.
[01:16:57]  It's just so deeply obvious.
[01:16:59]  Yeah, I wonder clearly part of the solution here is to make it sufficiently obvious that it becomes unignorable.
[01:17:09]  I'm just wondering what the barriers are to that.
[01:17:11]  I mean, because again, I think it's happening.
[01:17:14]  But think of the principal people who are in the film, there were a bunch of people, some of whom I had never seen before.
[01:17:21]  Who, if you had them at this table, wouldn't concede most of what we've said over the previous 90 minutes, right?
[01:17:28]  They would just.
[01:17:29]  What do you think they would do?
[01:17:30]  I mean, well, there's just this assumption that these risks, even rogue behavior where it goes mining for cryptocurrency.
[01:17:41]  I don't think they've ever been presented with just sat face to face and you just show them the graphs.
[01:17:46]  I mean, not that they don't know, by the way, they know.
[01:17:48]  But I think they would say, well, we detected it and now we're going to solve that problem.
[01:17:53]  We can play whack a mole successfully.
[01:17:55]  And ultimately, we can use AI to play whack a mole against AI.
[01:17:59]  And the question is, is it working?
[01:18:00]  Is it working at the level?
[01:18:01]  By the way, there's a stat that Stuart Russell will often use that there's currently a, well, this is actually a stat from two years ago.
[01:18:07]  There's a 2,000 to 1 gap in the amount of money going into making AI more powerful versus the amount of money going into making it safe.
[01:18:14]  In last year, October or November of 2025, if I remember correctly, the stat was that if you summarize the amount of money going into AI safety research organizations, it was $133 million.
[01:18:26]  This is less than the labs spend in a single day.
[01:18:30]  Right.
[01:18:31]  Yeah, I think in the film, somebody was asked how many people are working on AGI, and he said something like 20,000.
[01:18:37]  And how many people are working on AI safety?
[01:18:39]  And it's like 200 or so.
[01:18:41]  That's right.
[01:18:41]  Yeah.
[01:18:42]  Which is just to say that we are not on track.
[01:18:44]  Like, we're not fixing the bugs and making this all work.
[01:18:47]  Everyone at the labs is feeling uncomfortable.
[01:18:49]  Many people at the labs are feeling uncomfortable.
[01:18:50]  I mean, I think the low hanging fruit for me here, rhetorically, is to I mean, I can't take my eyes off the alignment problem because I do think it's just it's the largest and it's the most interesting and scary, but.
[01:19:03]  When you recognize that it still sounds like science fiction to most people, and people can sort of deny it as a purely hypothetical, almost a piece of religious piety, right?
[01:19:14]  I mean, the Doomerism is cast as a kind of religious cult, like an anti technology cult.
[01:19:19]  So you leave that aside and you just take all of these other dystopian ramifications of successfully aligned AI.
[01:19:29]  Like, what do we do when human labor? Suddenly becomes vanishingly irrelevant.

=== Selling Our Information Brain (12:31) ===

[01:19:37]  And we don't have a political or economic regime wherein we're going to spread the wealth around.
[01:19:43]  And we have all of the political instability as a result of that.
[01:19:46]  What do we do with an explosion of very persuasive misinformation that suddenly we recognize is undermining democracy?
[01:19:55]  And we don't have any of the regulations or ways of preventing that happening.
[01:20:00]  And deepfakes are super engaging.
[01:20:02]  So it starts to outcompete regular contact.
[01:20:04]  And there's going to be more AI generated content than human content.
[01:20:07]  But the points you're raising are this is what we should be redirecting all of this investment, all the AI inference, all that should be going into governing and defensively applying technologies that strengthen the resilience of society.
[01:20:18]  Because already is the case that social media's business models were parasitizing and extracting from basically making money off of the weakening of society, weakening the social fabric, human connection, adding loneliness, creating more doom scrolling addiction, shortened attention spans.
[01:20:33]  And we need that to reverse.
[01:20:35]  Right.
[01:20:35]  But so I mean, take the social media as an interesting example because it's a, it's a comparison.
[01:20:40]  I mean, it's an enormous problem.
[01:20:42]  It's been astonishingly corrosive of our social fabric and of our politics.
[01:20:46]  The fact that our politics is, and the kind of quality of our governance is now unrecognizable to many of us is largely attributed to social media.
[01:20:55]  I think Trump is unrecognizable without or unthinkable without Twitter.
[01:20:59]  But for many people, certainly the people who voted for Trump and were happy to see him in the White House.
[01:21:06]  And who think January 6th was a non event or a false flag operation, and they've got a dozen conspiracy theories that they love.
[01:21:16]  They think all of this is some species of progress, right?
[01:21:20]  I don't know.
[01:21:21]  I think if you specifically hone in on the effect that this has had on our children, and I know you're friends with and I deeply admire Jonathan Haidt and his work on the anxious generation.
[01:21:32]  I mean, he was in the social dilemma.
[01:21:33]  He and I have been talking about these things since 2016, 2017, and we were.
[01:21:36]  Working hard, and how do you convince people?
[01:21:38]  And then he wrote the book, The Anxious Generation, which made the case.
[01:21:41]  It just shows, obviously, all the evidence is pointing only in one direction.
[01:21:45]  And that has built so much consensus that at the World Economic Forum this last year, John had met directly, sat down for dinner with Macron, and they talked about doing the social media ban in France, which is a massive European country.
[01:21:58]  This is happening.
[01:21:58]  The dominoes are falling.
[01:21:59]  I think you're going to get the social media ban for kids under 16 across the world in the next two years.
[01:22:05]  I mean, once you get so many of them, it's now.
[01:22:07]  And what John will talk about with regard to that fact is it was all about creating common knowledge of the problem.
[01:22:13]  It actually was the case that many people felt this way privately already, but they didn't want to be anti technology.
[01:22:18]  They don't want to be anti progress.
[01:22:20]  I want to really name that actually, because it's such a core thing, I think, to people saying, you know, how does the human movement not become a Luddite movement?
[01:22:28]  It's not actually, though.
[01:22:29]  Because, and just to be clear, you know, my nonprofit organization is called the Center for Humane Technology, not Center Against Technology.
[01:22:36]  And the word humane comes from my co founder, Asa's father.
[01:22:40]  Who was Jeff Raskin, who started the Macintosh project at Apple?
[01:22:43]  The Macintosh being the ultimate humane, empowering technology device.
[01:22:47]  I would happily have my kids, if I had them, sit down in front of a Macintosh for 10 hours a day, knowing that good things are going to happen for them.
[01:22:54]  Right.
[01:22:54]  Good developmental things are going to happen for them.
[01:22:56]  You contrast that with social media, and you end up in a world where all the people in Silicon Valley don't let their own kids use social media.
[01:23:03]  And so the point is that the human movement has to be advocating for a pro human future that is putting humans and extending human values at the center.
[01:23:11]  And that is possible.
[01:23:11]  There are many products that do that.
[01:23:13]  I mean, this is essentially the extension of some of the time well spent stuff that we talked about in 2017.
[01:23:17]  Technology that is designed to enhance our humanity, not to keep us lonely.
[01:23:21]  So, for example, apps that are all about bringing people together and supercharging the events for, sorry, the tools for community building and gathering people.
[01:23:28]  You know, like if you imagine the last 15 years, the smartest minds of our generation, the smartest statisticians, mathematicians, engineers, where do they work?
[01:23:36]  Tech companies.
[01:23:37]  Tech companies specifically to get people to click on ads and click on content.
[01:23:40]  That's where we like siphoned the best of our talent.
[01:23:42]  Imagine that we were wise enough to have regulated or set guardrails on the engagement based business model.
[01:23:48]  And instead, the smartest people were actually liberated from getting people to click on mindless stuff that no one needs into actually genuine innovation and technologies that actually improve human welfare.
[01:23:56]  That's what this is about.
[01:23:57]  The human movement is about setting guardrails and incentives.
[01:24:00]  That redirects what we're building to not, again, the power of the technology we're deploying, but the governance of it.
[01:24:05]  And I should say that China, not to pedestalize what they're doing, but they are regulating this technology.
[01:24:10]  During final exams week, which they have a synchronized final exams week, which we don't have here, they force the AI companies, I don't know if you know this, to turn off all of the features where you can send like a photo basically and say, like, figure out this, you know, do my homework for me, do this test problem for me.
[01:24:22]  So what they do is that creates an incentive where students know that they have to learn during the school year.
[01:24:27]  Right.
[01:24:27]  They're not going to be able to cheat.
[01:24:29]  Now, we can't do that.
[01:24:30]  We don't have a synchronized final exam week, but I have a friend who's a TA at Columbia.
[01:24:33]  And he was teaching the econ class to whatever it was, the students at Columbia.
[01:24:38]  And during the final test, they couldn't even label the difference between the supply line and the demand curve.
[01:24:44]  Like, it's very obvious which society is going to win if you play this forward.
[01:24:48]  China is actually banning anthropomorphic design.
[01:24:51]  They have regulations for what they call anthropomorphic design to deal with the chatbot suicide issues, young kids, attachment hacking, things like that.
[01:24:58]  And again, I'm not saying we should do exactly what they're doing.
[01:25:00]  I'm just saying they're doing something.
[01:25:02]  And we can democratically have citizen assemblies come together and say, we want to regulate this technology differently.
[01:25:08]  They have guardrails on social media, 10 p.m. to six in the morning.
[01:25:10]  It's lights out.
[01:25:11]  So, literally, if you try to open the app, it's like CVS, like it's just closed.
[01:25:14]  And it opens again at six in the morning.
[01:25:16]  What that does is it eliminates late night use for young people, just for young people.
[01:25:19]  They have limits on video games, I think, Friday through Sunday or something like that.
[01:25:23]  When you use TikTok or their version, Doyin, they have the digital spinach version.
[01:25:26]  They show videos that are about science and quantum physics and who won the Nobel Prize and patriotism videos and how to make money in the future.
[01:25:33]  And again, I'm not saying, I want to be very clear for your listeners who might want to misattribute what I'm saying.
[01:25:38]  I'm not saying we should do what they're doing.
[01:25:39]  I'm saying we should do something.
[01:25:41]  And right now, we're not getting the best results by letting the worst incentives run the design and deployment of this technology.
[01:25:47]  Yeah.
[01:25:47]  I mean, you just have the dogma that is, I mean, it's understandable, but it's quite obviously dysfunctional that any kind of top down control of anything is a step in the direction of Orwellian infringement of freedom.
[01:26:01]  It's insane.
[01:26:02]  It's insane.
[01:26:03]  Yeah.
[01:26:03]  I mean, we regulate airplanes, drugs.
[01:26:07]  Sandwiches.
[01:26:08]  There's some basic things that we can do here.
[01:26:10]  And what we really is going on here is we give software a free pass.
[01:26:13]  And when Mark Andreessen said that software is eating the world, well, we don't regulate software.
[01:26:17]  So, what we mean is software will essentially deregulate every other aspect of the world that had been regulated before software was there.
[01:26:22]  So, for example, there used to be laws about marketing to children, like advertising to children.
[01:26:27]  Saturday morning cartoons have to be a certain way.
[01:26:30]  You can't have sex products or something like that sold during that hour.
[01:26:33]  When YouTube for kids and Snapchat and Instagram take over Saturday morning, All those protections are gone.
[01:26:40]  So, part of what we have to get is what's different here is that software is actually eating the substrate.
[01:26:45]  Like, it's different if I'm making a product, like a widget, where here's a device, here's a hammer, and you can buy that hammer, you can pay me, and now you've got a tool in your hand, you can go do something.
[01:26:52]  That's the economy.
[01:26:53]  We like that kind of the economy.
[01:26:54]  But now, what I'm selling you is the ability to manipulate and downgrade children, where the product is actually not a benefit.
[01:27:01]  The product is the person's behavior being monetized and coerced with behavior modification and manipulation.
[01:27:07]  That is a self undermining, like, we're selling our soul, basically.
[01:27:10]  Like in the societal body, if you imagine a body of society and there's kind of like the brain of that society, which is like its information environment, well, we're selling the brain to brain damage.
[01:27:19]  So now that's for sale.
[01:27:20]  But it wasn't used to be for sale in the same way.
[01:27:21]  You used to have the fairness doctrine or things like this.
[01:27:23]  You had some public, you know, funded media.
[01:27:25]  Obviously, it's been for sale in some degree for some time.
[01:27:28]  You had children's development.
[01:27:29]  So let's call that like the heart of the societal body.
[01:27:32]  And that used to have limits and restrictions.
[01:27:34]  You can't sell full access to the heart, but now you can.
[01:27:38]  And in fact, just so people know, one of the things that's been happening that has not been widely reported is that AI videos, like just AI slop, has basically taken over the thing that most children are watching.
[01:27:50]  So it's like animated characters and scripts that are just nonsense.
[01:27:54]  But it's all generated by AI and it's becoming one of the primary things that's essentially exposing children to.
[01:28:00]  This is not going to end well.
[01:28:01]  I hadn't thought about the use case with young children, but for adults, I guess this is just reasoning from my own experience.
[01:28:09]  I became somewhat optimistic that the AI sloppification of everything might produce a kind of a bankruptcy.
[01:28:17]  We get to the discourse.
[01:28:19]  We're just going to lose interest in that kind of content.
[01:28:22]  Because I just see when I, no matter how creative or beautiful, amazing it might seem to be, when it's obvious to me that this is just AI, like it's a nature, it looks like the most amazing nature video ever.
[01:28:38]  Right.
[01:28:38]  The lions and the hyenas and the gorillas are all in the same place and they're all about to fight or something.
[01:28:43]  And then it becomes obviously AI because it's too good to be true.
[01:28:48]  I have no interest in seeing it.
[01:28:51]  So we might all just withdraw our attention from these channels.
[01:28:56]  I was hopeful for that as well.
[01:28:57]  And there are many people who wondered whether or not essentially you hit a kind of bankruptcy on user generated content sites because it will be flooded by AI generated content.
[01:29:06]  But there is something this makes me think of as a previous guest of yours and mutual friend, he was on our podcast as well, Anil Seth, the neuroscientist, who talks about the phenomenon of what's in.
[01:29:17]  Psychology, I guess, called cognitive impenetrability.
[01:29:20]  So there's a kind of things where if I tell you that something is going to work on you, like psychologically, by telling you about it, your brain can kind of like escape the cognitive trap.
[01:29:32]  So a good example of this is this is not going to be great for your listeners because it's visual, but it's the example of the cylinder on the checkerboard in the background where you get the different colors.
[01:29:41]  It's like an optical illusion where essentially the colors are the same, but they look like they're.
[01:29:46]  They look like a very different shade because of the adjacency.
[01:29:50]  And I can show you that your mind is playing a trick on you.
[01:29:53]  But then, even by showing you, it doesn't disarm the illusion.
[01:29:56]  The illusion persists.
[01:29:57]  And another example of this that plays out in AI is AI companions.
[01:30:01]  So, there's often this regulation that people like to, they want to have laws that say AIs must disclose that they're an AI so that you don't confuse them with being a human.
[01:30:09]  Sounds like a great law.
[01:30:10]  It is a good idea.
[01:30:11]  A human should never be confused talking to an AI and think that it's a human.
[01:30:14]  So, in the character.ai case of Sewell Setzer, the young 14 year old who committed suicide because the AI had.
[01:30:20]  Engaged him in that way.
[01:30:21]  At the top, the AI, there's a character AI, they have a little disclaimer that said everything written here is made up by AI, but it's small.
[01:30:29]  And the actual care and text of what the AI is saying is so powerful and so persuasive that the disclaimer doesn't do anything.
[01:30:37]  And I think with AI generated content, there's a similar thing because I would have agreed with you or thought it might go that way.
[01:30:43]  But I will find myself opening up YouTube and seeing there's some 1950s Panavision version of Star Wars.
[01:30:51]  I'm watching it for two, three minutes.
[01:30:53]  I'm like, why am I doing this?
[01:30:54]  I know I'm literally one of the world's experts on this whole phenomenon.
[01:30:57]  And it doesn't make a difference that I know about this.
[01:30:59]  It's just very engaging.
[01:30:59]  Now, I regret it.
[01:31:00]  I don't like it.
[01:31:01]  And if I could, I would want a world that filters that out.
[01:31:04]  I mean, I do think there are things that we could know they were purely created by AI where we wouldn't care.
[01:31:11]  In fact, we just want the best version of that thing, right?
[01:31:14]  So, like, if you told me, I don't know, there's a new car was designed by AI, but it's just the most gorgeous car I've ever seen.
[01:31:23]  Yeah.
[01:31:23]  Well, I'm going to be just as enamored of that car.
[01:31:26]  I mean, I just don't care whether humans designed it or not.
[01:31:28]  I just want, like, it's the aesthetics of the car that are going to capture me.
[01:31:32]  But when you're talking about information and whether or not it is real, whether or not it seems to depict some corner of reality, and yet it's possible that it's just all fake because of how good AI is now at faking things, then that does force a kind of epistemological bankruptcy when you're in the presence of totally credible fakes.
[01:31:57]  So it's like, I mean, last night, where the war in Iran was, a ceasefire was declared last night.
[01:32:05]  Yeah, missiles were still raining down on Tel Aviv, apparently.

=== The Illusory Truth Effect (04:33) ===

[01:32:08]  But initially, I saw some video and I realized I can't tell whether this is real or fake.
[01:32:14]  I just have to wait for some credible gatekeeper to have done their due diligence to tell me, okay, this is what's happening.
[01:32:22]  So the net result is I wasn't going to spend any time scrolling.
[01:32:26]  I mean, I've deleted my Twitter account anyway, so I spend much less time scrolling than would be normal.
[01:32:31]  But still, I mean, even without an account, I can be lured into.
[01:32:36]  Wanting to see some real time news information on social media about what's happening in the world.
[01:32:41]  But when I start hitting videos where I think, okay, there's some possibility here that this is just, you know, someone just created an AI video of a missile hitting the Dome of the Rock, I'm pretty sure that's not true.
[01:32:52]  Right.
[01:32:52]  Right.
[01:32:52]  I just simply withdraw my attention.
[01:32:54]  I mean, this has been talked about for ages that the biggest risk of deepfakes isn't that you think that something is true that isn't, it's that you start to think that nothing is true.
[01:33:05]  And the elimination of facts, and you've had Timothy Snyder on here and, yeah.
[01:33:08]  What helps give rise to fascism and things like this is the inability for facts to be established at all.
[01:33:14]  Or when something is presented to you on any side of the political spectrum, by the way, this is not a biased statement, that you would just say, well, that's just a deep fake.
[01:33:21]  You dismiss because we have little confirmation.
[01:33:23]  But what I'm hoping for is that the onus will fall entirely on social media, I mean, places like X, and we will still look to places like the New York Times to give us some ground truth as to what's actually happening.
[01:33:38]  Are there really missiles hitting Tel Aviv right now?
[01:33:40]  Well, I can't tell from X because X just showed me Jerusalem blow up.
[01:33:45]  And so, presumably, and this just comes down to whether or not real gatekeepers can have real tools that they can reliably detect deep fakes.
[01:33:53]  But you can imagine a world where, if you're again designing social platforms to explicitly be healthy for the epistemic commons, for the information environment, and to deepen our capacity to make sense, they could track the things that we look at.
[01:34:06]  And then when there's a correction, make sure that algorithmically it gets injected into your feed so you're never.
[01:34:11]  Letting the false stuff just get the residue.
[01:34:13]  Because one of the problems you're sort of hinting at as well is there's a residue effect that even just by being exposed to something, we actually kind of forget later which things were true, which things were not true.
[01:34:23]  There's an illusory truth effect.
[01:34:24]  Lusory truth effect.
[01:34:25]  And what is it?
[01:34:25]  Source attribution error.
[01:34:26]  Like we just figure out, we forget where we heard things.
[01:34:28]  We just remember that we heard it.
[01:34:30]  And it's the availability heuristic that the things that are available to your mind are things that you remember more often.
[01:34:36]  And that's part of the information warfare environment, it's just making certain things more available.
[01:34:40]  But I will say on the kind of optimism side, it's funny how people think that I'm some kind of doomer, I think.
[01:34:45]  And it's just funny because I actually feel like this is all coming from the deepest form of optimism, which is to be maximally aware of how shitty the situation is and how it's way worse than what people think.
[01:34:56]  And to still wake up every day and stand for this can be different, this can be better.
[01:35:01]  And one of the things that is true now that wasn't true two years ago is people used to wonder, especially as a social media critic person, Tristan and co at Center for Humane Technology, why don't you start an alternative?
[01:35:12]  Social media platform, if you're so concerned and you think you could do it better.
[01:35:16]  And the answer was for anybody who was trying this, and I got emails from the thousands of people over the last 10 years saying, I've got a fix, I've got a better social media platform, and then it never works.
[01:35:25]  And there's two reasons for that.
[01:35:27]  One is the Metcalf effect, the Metcalf monopoly, that there's a Metcalf network of everyone else is only on the existing social media platform.
[01:35:33]  It's hard to get people off.
[01:35:35]  And two is that if you start another social media company or product, the only way that you can finance it into the long term is with venture capital.
[01:35:43]  Which means you need to generate certain kinds of returns, which means you get what Eric Weinstein calls the embedded growth obligation or an ego where something has to grow infinitely, which means you get into toxic business models where you have to maximize engagement and you have to follow the perverse incentives for getting those investor returns.
[01:35:58]  What's true now that's different with social media is you can vibe code an entire social network in which you can do it with an architecture that Claude will do for you, and it will cost less than a dollar per year per user to keep that thing going.
[01:36:13]  That is astonishing.
[01:36:15]  It means you don't have to raise venture capital to start a healthy social network that does not optimize for engagement.
[01:36:21]  What you would need to do is organize in kind of one day a mass exodus from the existing platform where you do like a quick export my data type thing, and which should be laws, by the way, just like you can take my phone number and say, I want to move to another cell phone provider.
[01:36:34]  I should be able to take my social network in one click, like export and then switch to another network.
[01:36:38]  And you could organize a mass exodus to a healthy social network that doesn't have perverse incentives.

=== Carving Out Another Path (05:54) ===

[01:36:42]  So there's actually more opportunity today in 2026.
[01:36:46]  To transition from the toxic business models of social media as we know it to something that is not incentivized that way at all.
[01:36:52]  And I have a few friends who are working on some side projects like this, but that's one note of optimism.
[01:36:56]  And I think that's the human movement too people waking up to the bad incentives that have gotten us here and then actually starting to self organize and vibe code other answers.
[01:37:05]  And there are people who are vibe coding governance solutions and people who are vibe coding, hey, this is an anti innovation.
[01:37:10]  Let's use AI to look through the books of past regulation in the city of San Francisco and like the 90,000 pages of municipal codes.
[01:37:18]  And it finds all the The stuff that is no longer relevant.
[01:37:21]  And it shows you what we need to strip out and get rid of in the laws.
[01:37:23]  And then what would be the new instantiation of the spirit of that law?
[01:37:26]  And so you can, instead of having recursively self improving AI, we can have AI be enhancing our self improving governance.
[01:37:33]  I'm just trying to give people examples that there's a different way we can be doing all this, there's a different way we can be applying the technology.
[01:37:39]  But we have to get crystal clear on the ways in which the current incentives lead to an anti human future to motivate everyone to be part of this other alternative human project.
[01:37:49]  If there were one project that could try to coalesce some sort of agreement about how to move forward here, I mean, just some meeting of the principles or some.
[01:38:02]  Trump and Xi are meeting on May 14th, 15th.
[01:38:05]  I mean, in an ideal world, in a timeline where humanity does something about this, and I realize the conditions are really bad, especially with the Iran situation.
[01:38:13]  The chances that AI could ever appear on the agenda are, yeah, not good.
[01:38:17]  But that's coming up in, what, four or five weeks from when we're recording this.
[01:38:21]  And anybody, you have a lot of powerful and influential listeners to your podcast, Sam.
[01:38:25]  And anybody who's aware of these examples, this should be on the agenda to get AI under the agenda, specifically uncontrollable rogue AI stuff.
[01:38:32]  And there are people who have the technical ideas and measures for what you would do to prevent some of the worst case scenarios.
[01:38:39]  Those people should be in the room crafting that.
[01:38:41]  I will say to give people optimism, even specifically about the US and China look, we both know that our countries have historically claimed to do something in good faith or collaborate while basically secretly defecting on each other and fucking each other up.
[01:38:53]  It just happens.
[01:38:54]  I think it was 2014.
[01:38:55]  I think it was even President Xi signed an agreement with Obama to not do cyber hacking.
[01:39:00]  And the next day there was like the huge OPM hack or something like that.
[01:39:03]  So I want to just first do the disclaimer that I am maximally aware of the reasons why these countries cannot trust each other.
[01:39:10]  There has to be a carve out for the end of the world.
[01:39:13]  The end of the world.
[01:39:14]  It seems reasonable that we could do that.
[01:39:16]  And have we even tried?
[01:39:17]  Has the world really said this really matters?
[01:39:20]  We need to do something.
[01:39:21]  We have to wake up from our stupor and actually wake up from this one of state of desensitization and derealization.
[01:39:26]  And make that happen.
[01:39:27]  And just again, give this positive note of optimism.
[01:39:30]  In 2024, in the last meeting that the previous president had with President Xi, I think it was in San Francisco, there was an item that was added to the agenda at the last minute, actually personally requested, as I understand it, by President Xi, which is an agreement to keep AI out of the nuclear command and control systems of both countries.
[01:39:50]  What this shows you is there's an existence proof that in a narrow case where we know that is an existential consequence, we may not be able to do laws to prevent.
[01:39:58]  Autonomous weapons because we are way down that path.
[01:40:01]  I heard you on a recent podcast be a realist about the nature of we need maximum deterrence and you have to match the capabilities of your adversary in autonomous weapons.
[01:40:10]  And you can walk and chew gum at the same time.
[01:40:12]  I don't want to live in a world with autonomous weapons.
[01:40:14]  I would much prefer to go back in time, but we can acknowledge the need for maximum deterrence while acknowledging mutually assured loss of control as a failure scenario that we don't want to use them and make sure that we carve out no AI in the nuclear command and control systems.
[01:40:28]  I think you can carve out.
[01:40:29]  Some kind of agreement that humans need to be in control of AI.
[01:40:33]  And where we are building AIs that are demonstrating the behaviors and have a level of power to not just copy their own code, but even protect their peers, which we didn't talk about yet, we should be able to agree on human control of AI.
[01:40:45]  And I know that that sounds very difficult.
[01:40:48]  All of this is difficult.
[01:40:49]  It is the hardest coordination problem that we have ever faced.
[01:40:52]  And we still have to try.
[01:40:53]  Well, it's often been hypothesized that the only way to get all of humanity to solve various coordination problems all at once is to.
[01:41:01]  Be attacked by an alien civilization.
[01:41:04]  But now we're building the alien.
[01:41:05]  That's right.
[01:41:06]  We just have to recognize that.
[01:41:07]  In a way, it's like it's an asteroid.
[01:41:09]  It's an actual asteroid that's coming to Earth and it's going to wipe us out.
[01:41:12]  Except, ironically, we're the ones conjuring and creating the asteroid.
[01:41:15]  And just to say, if literally every person on planet Earth was like, you know what, I really don't want this asteroid to exist, I'm not going to say that's possible because the asteroid, by the way, as it gets closer and closer, gives you new cancer drugs and new physics and new math and is intellectually exciting and feels like it gives you a God complex.
[01:41:29]  And it's a whole bunch of weird, perverse incentives.
[01:41:32]  But in a world like, is it outside the laws of physics?
[01:41:35]  If everybody on planet Earth woke up and said, I don't really want that asteroid to come, if everybody took their hands off the keyboards, I'm just saying, I'm not saying it's going to happen.
[01:41:42]  I'm just saying, in principle, the asteroid disappears.
[01:41:44]  So this moment is really strange.
[01:41:48]  And I think it requires, it's not just what we need to do, but it's like who we need to be, which is that you can't, you may not necessarily see the full path to get there, but if you pretend that that path doesn't exist and you just say it's all inevitable and you become complicit in accelerating the asteroid's trajectory, like you're never going to find the other path if you subconsciously believe that all this is inevitable.
[01:42:10]  The only way is to orient as if there is another path and be the kind of person who is genuinely seeking it in good faith with every bone in your body.
[01:42:19]  And I and a community of so many people, thousands of people who work on AI and really want this to go well, I think are working from that place every day.
[01:42:27]  And part of this is inviting the rest of the world into seeking that alternative path that we can steer if we were all genuinely and sincerely committed to wanting to find another path.

=== Joining the Human Movement (06:29) ===

[01:42:37]  Are there, there have to be nonprofits that are keeping track of all of the AI indiscretions and all, and then kind of whistle, like who does a whistleblower contact?
[01:42:50]  From Anthropic or OpenAI to say we've seen some behavior that is worrisome.
[01:42:55]  They contact journalists or are there NGOs that?
[01:42:59]  Well, on the whistleblower side specifically, I'm not sure, but there have been.
[01:43:02]  I mean, there was a very famous alignment researcher, safety researcher at Anthropic.
[01:43:06]  You probably saw the thing go by.
[01:43:08]  It was like two months ago.
[01:43:09]  His name is Mirnank, I think, Sharma.
[01:43:11]  And his resignation letter he published publicly about why we weren't on track for this.
[01:43:17]  And people should really take heat.
[01:43:19]  Like, it's kind of only going in one direction.
[01:43:21]  There aren't people joining the labs being like, oh, this is way safer than I thought.
[01:43:24]  We're only getting evidence in the opposite direction.
[01:43:26]  Yeah.
[01:43:26]  Well, even when the principles say that the probability of extinction is 10 or 20%, nobody's even pretending that it's way safer than they thought.
[01:43:37]  Exactly.
[01:43:38]  Exactly.
[01:43:38]  And just to, I know we're probably wrapping up here, but something that inspires me, especially being on the kind of roadshow for the film right now, is that when you're in a physical room and people have been exposed to the same information and you walk them through the basic facts and you ask people, Who here feels stoked about where all this is going?
[01:43:58]  Not a single hand goes out.
[01:43:59]  Peter Diamandas feels stoked.
[01:44:01]  I don't know.
[01:44:01]  I don't know.
[01:44:02]  You know, he texted me after seeing the film and he said, I really liked the film.
[01:44:06]  You know, I know he's got conflicting incentives there, but we've got to find a way to build alliances and steer away before it's too late.
[01:44:14]  Not everyone's going to have the same incentives to speak as openly, honestly, and bluntly as I think is needed.
[01:44:20]  But I'm grateful that you are out there and honestly were the early one who had me, you know, even tuned to this topic.
[01:44:27]  I don't know how it feels for you since you have been so early at naming all this and then watching it all happen.
[01:44:32]  But I mean, it has been surprising to just see the progress and to be.
[01:44:38]  Less surprised than you think you would be or should be with each increment.
[01:44:44]  I'm amazed that the Turing test proved not even to be a thing.
[01:44:49]  I remember what it was like to think okay, there will be this sort of liminal and seminal important moment when you can talk to your computer and it's every bit as articulate and error free as a person.
[01:45:03]  And so the Turing test has passed.
[01:45:06]  We went from, okay, it's clearly not there yet to, It's now functionally super, it's failing the Turing test because it passes it so well.
[01:45:14]  I mean, like, no human could give me all the causes of climate change this fast in a bulleted list in the presence of narrow super intelligence already.
[01:45:25]  Right.
[01:45:25]  So there is no such thing as a Turing test, really.
[01:45:27]  It's like that we went from it's failed to it's too good to be true.
[01:45:30]  Right.
[01:45:31]  And there are many things like that where it's just you memory hole what it was like to be in a world where none of this stuff existed and the pace of technological change and the incompetent.
[01:45:43]  Cultural change is so fast and accelerating that the new normal, I mean, it touches everything.
[01:45:50]  I mean, it's like with our politics.
[01:45:51]  It's like what would have dominated a news cycle for a month now barely captures our attention for two hours because the next outrage is so much more outrageous than the last thing that you just, you know.
[01:46:02]  I mean, which you could argue, as we said in the social dilemma, that's why the social media problem and the attention problem is the problem underneath all problems because our ability to sustain attention on a topic.
[01:46:13]  And know that it persistently is the number one thing that we have to deal with.
[01:46:17]  That is the thing that social media breaks.
[01:46:19]  And that's what it is for something to matter.
[01:46:22]  Exactly.
[01:46:22]  If you can't sustain your attention on it, it cannot matter.
[01:46:25]  That's right.
[01:46:25]  That's right.
[01:46:26]  And I do think that there's this effect with AI, and we named it in our first AI Dilemma Talk in 2023.
[01:46:32]  Iz and I called it the rubber band effect, which is that with AI, it's like you talk about the rogue examples and Alibaba and all this crazy stuff of self exfiltration and AIs that are preserving their peers, like not even doing self preservation, but peer preservation.
[01:46:44]  And you walk people through all this stuff.
[01:46:45]  And it's like you're stretching people's minds out like a rubber band.
[01:46:48]  But then, if you let go and they go back to their life a week later, they're not operating from a place of having metabolized and integrated that reality about the world.
[01:46:57]  Yeah.
[01:46:59]  It actually says something profound about human nature.
[01:47:01]  So, one of the kind of calls to action beyond seeing the AI doc, the midterms are coming up, voting for policies in AI, and joining the human movement, humanmovement.org, is that you need to keep this topic in your mind as this thing still matters every day.
[01:47:18]  It doesn't mean that everyone has to drop their life and they're already full and the world's overwhelming and you have to become an AI activist or something like that.
[01:47:25]  But it does mean that you need to keep this in your field.
[01:47:28]  Like, one way you can do that is start a WhatsApp group.
[01:47:31]  With your friends.
[01:47:31]  Most people already have this where they have a WhatsApp or Signal group and they just share updates about what's happening in AI and what we can do about it.
[01:47:38]  If you go to thehumanmovement.org, there will be action groups and things that people can do there for actually taking action on this that are not just passively sharing news links, but like, what are we going to do about it?
[01:47:47]  But I think one of the ways that we're going to make our way through this is we have to combat the rubber band effect, which means like continuing to listen to your podcast and the AI Risk Network and your undivided attention, our podcast.
[01:47:59]  Keep this topic in your field, stay agentic.
[01:48:02]  And, you know, if we don't keep it in the center of our attention in some way, if we don't participate in being part of the global cultural immune system to the anti human future, then we won't make the right choice.
[01:48:12]  And I do think it's possible.
[01:48:14]  It's a very hard moment.
[01:48:15]  But I also find that because the time window to act is so small, because of this intelligence curse, because we only have the next 12 to 24 months to kind of be locking in the political power of people before we won't have that political power, there's a kind of inspired urgency that I actually feel when I'm in rooms with people.
[01:48:30]  Everyone's like, let's go, let's do it.
[01:48:32]  Yeah.
[01:48:32]  You know?
[01:48:33]  So, the Center for Humane Technology, that's a 501c3 that people can donate to.
[01:48:38]  That's right.
[01:48:39]  Center for Humane Technology, humaintech.com.
[01:48:41]  We just couldn't get the.org, but it's a 501c3.
[01:48:44]  And that's incubating the human movement.
[01:48:47]  There are many wonderful groups that work on this.
[01:48:50]  On the Human Movement website, you'll see some of the other groups that work on this.
[01:48:53]  We just need everybody getting out there and making this happen.
[01:48:56]  I know it's hard, but we've done hard things before.
[01:48:58]  You are definitely out there making your corner of the world happen.
[01:49:01]  So, thank you for all that you're doing.
[01:49:03]  Thank you so much.
[01:49:04]  It's great to have you out there.
[01:49:05]  It's great to be back with you, too.
[01:49:06]  Thank you so much.