The Dawn of AGI: Bret Speaks with Alexandros Marinos on the Darkhorse Podcast
Bret Speaks Alexandros Marinos on the dawning of the AI age. Where are we, where are we headed, and how should we react? Find Alexandros on Twitter: @alexandrosM (https://twitter.com/alexandrosM)Find Alexandros at his Substack: Do Your Own Research (https://doyourownresearch.substack.com/) ***** Find Bret Weinstein on Twitter: @BretWeinstein, and on Patreon (https://www.patreon.com/bretweinstein). Please subscribe to this channel for more long form content like this,...
I think you're really going to enjoy what you hear.
There's a lot of provocative stuff in here and at least one bombshell.
I do have to tell you, though, that when Alex and I recorded it, we started a discussion at the beginning that, while important, is not the central focus of our talk on Dark Horse.
So in the editing process, I'm having Zach move that first half hour to the end of the podcast.
It's still there.
I hope you listen to it and enjoy it.
But I want you to get to the AI stuff first and foremost.
So that's where it's going to start, and Zach is going to do his best job editing it so that it makes sense.
But anyway, shall we switch gears and talk about AI, LLMs, AGI, pauses, Doomers, all of those topics?
Doomers, Foomers, the whole shebang.
Yeah, you're going to have to help me with Foomers.
I never quite remember what that is.
But let's say a couple of things up top in this segment.
A, I want you to be extra careful to try to talk In English, right?
This is a topic that is so rich with technical detail and with, you know, with an entire culture's personal language of thought problems that it's very easy to have an inside baseball discussion that will lose average folks.
And so I think one of the things that we should endeavor to do here is be very careful to present this so that somebody who has no background but is interested can follow what is being argued about, by whom, what are the different camps to the extent there are camps, what are the assumptions that they share, where do they differ, that kind of thing.
So do you want to You have taken a number of people to task.
You've called them, I don't think you invented the term, but you've called them doomers.
I've taken you to task for calling them doomers because I think there are a lot of us who believe that the dawning of the AGI era contains a great many hazards that could lead to something extremely dire up to and including the possibility of human extinction.
And so if you call folks who are saying that human extinction is a near certainty, doomers, then the problem is it Because Doomer makes it sound like, oh, people who see a very dire future, it leaves those of us who believe we are in very serious trouble and better think very carefully about our next move.
It leaves us sort of artificially dismissed along with those who say the risk is a near certainty.
Right.
So let's talk about, let's talk about doomers.
Good.
Let's talk about the term.
Right.
So as we've, I mean, we've spoken about this privately, but it's probably worth articulating at least the, the two sides of that conversation.
So I, Look, no term is perfect, right?
Language is compression and lossy at that, right?
So, there's always going to be something you don't like about a terminology.
Me first.
You know, me before most.
You know, is wokeness the right way to do it?
Well, you know, there is a history of that term, the black community.
Are those people worth packing up with, you know, what we call, you know, No, you know, fundamentally it's unfair to use any, you know, reductive term to group a set of people and they might not all agree, et cetera, et cetera, et cetera.
However, you know, for the purpose of communication, you need a label, right?
So, I mean, what I told you in private and what I've said a few times on Twitter when people ask, it's like, okay, well, you know, give me something better.
Like I was playing with the idea of like AI safetyists.
I think it's probably marginally better, but also I don't control the conversation, right?
I need symbols that people will dereference in a way that is useful.
If I say the AI safetyists, what will people think then?
So I ran a poll on Twitter actually asking, if I call somebody a doomer, if somebody calls somebody a doomer on the from the AI question, what do you think they are saying the other person assigns as a likelihood of AI doom?
And I was very happy to see that about three quarters of people said above 50%.
So it was either 50% plus or 95% plus.
It was about split between these two camps. - Our first sponsor this week is Vivo Barefoot, shoes made for feet.
Everyone should try these shoes.
Most shoes are made for someone's idea of feet, but not vívos.
Vivos are made by people with feet who know how to use them, and word is spreading.
People often approach us because of the vívos we are wearing, saying they've heard they're good.
And they are.
These shoes are every bit as good as you've heard.
Here at Dark Horse, we love them.
They are beyond comfortable.
The tactile feedback from the surfaces you're walking on is amazing, and they cause no pain at all because there are no pressure points forcing your feet into odd positions.
They are fantastic.
Our feet are the product of millions of years of evolution.
Humans evolved to walk, move, and run barefoot.
Modern shoes that are overly cushioned and strangely shaped have negatively impacted foot function and are contributing to a health crisis.
People move less than they might, in part because their shoes make their feet hurt.
Enter Vivo Barefoot.
Vivo barefoot shoes are designed wide to provide natural stability, thin to enable you to feel more, and flexible to help you build your natural strength from the ground up.
Foot strength increases by 60% in a matter of months just by walking around in them.
Vivo Barefoot has a great range of footwear for kids and adults and for every activity from hiking to training and everyday wear.
They're also a certified B Corp that is pioneering regenerative business principles and their footwear is produced using sustainably sourced natural and recycled materials with the aim to protect the natural world so you can run wild upon it.
Go to vivobarefoot.com and use Dark Horse 15 to get an exclusive 15% off.
Additionally, all new customers get a 100-day free trial so you can see if you love them as much as we do.
That's v-i-v-o-b-a-r-e-f-o-o-t dot com and use the code DarkHorse15 at checkout.
Our second sponsor for this episode is Maddie's all-time favorite, Sundays.
Sundays makes dry dog food.
When they approached us about being a sponsor, we were dubious.
Maddie, our dog, is a Labrador.
Labs will eat basically anything.
What possible difference was she going to show in interest between her usual kibble, a widely available high-end brand, and Sundays?
We were wrong.
Maddie clearly loves the food Sundays makes.
Seriously loves it.
In fact, she's become discerning in what she wants to eat now.
She wants her Sundays.
Why?
Sundays is far better than the standard burnt kibble that comprises most dry dog food.
Sunday's is the first and only human-grade, air-dried dog food.
I don't think Maddie knows that, but it does make good stuff.
It combines the nutrition and taste of all natural human-grade foods with the ease of a zero-prep, ready-to-eat formula.
Sunday's is an amazing way to feed your dog, And in a pinch, a person too.
I have tried it.
It was all right.
Sundae's is easy for humans as well.
No fridge, no prep, no cleanup, no gross wet dog food smells.
Sundae's is gently air dried and ready to eat.
No artificial binders, synthetic additives, or general garbage.
Seriously, look at the label.
All of Sundae's ingredients are easy to pronounce except quinoa.
It's healthy for dogs to eat.
In a blind taste test... the taste test was blind, not the dogs... Sundays outperformed leading competitors 40 to 0.
That sounds like a made-up number, I know, but here's the thing.
When I have a bowl of Maddie's previous food ready for her, she's certainly enthusiastic.
Again, she's a lab.
But when I have a bowl of Sundays ready for her, it's a whole different level of enthusiasm.
Do you want to make your dog happy with her diet and keep her healthy?
Try Sundays.
We've got a special deal for our listeners.
Receive 35% off your first order.
Go to SundaysForDogs.com slash Dark Horse.
Use Dark Horse at checkout.
That's S-U-N-D-A-Y-S F-O-R-D-O-G-S dot com forward slash Dark Horse.
Switch to Sundays and feel good about what you are feeding your dog.
Our final sponsor for this episode is House of Macadamias.
Trees aren't edible, but tree nuts are delicious and nutritious.
They are generally high in fat and low in carbohydrates.
Unlike what various food pyramids and government agencies may have led you to believe, high-fat, low-carb foods are increasingly understood to be both satiating and good for you.
But each species of nut is different, requiring a special mindset and appropriate footwear.
You heard me.
For many of us, macadamias are the best nut there is.
Macadamia nuts take a very long time to grow, however, and because they are both rare and highly sought after, they have the dubious distinction of being the world's most expensive nut.
Between the taste and the health benefits, though, they're worth it.
They have even fewer carbohydrates than most other nuts, for instance, half of what cashews or pistachios have, and two-thirds of what almonds have, which makes them the perfect snack for breaking a daily fast and controlling blood glucose.
They're also uniquely rich in omega-7s, including, especially, palmettoleic acid, an unsaturated fat that has been linked to natural collagen production, fat loss, and heart health.
And House of Macadamias is intent on making this amazing food accessible to everyone.
They have partnered with more than 90 farmers in Africa and now make one-of-a-kind vegan keto and paleo snacks.
These include their dark chocolate dipped macadamias and a delicious assortment of bars made with 45% macadamia nuts in flavors including salted caramel and chocolate coconut.
But our favorite product of theirs is the simple salted macadamias made with Namibian sea salt.
They are amazing.
We love them and we think you will too.
House of Macadamias also makes a delicious macadamia nut oil, which is 100% cold-pressed, rich in monounsaturated fatty acids, and has a higher smoke point than olive oil, so it is well-suited to high-heat cooking and baking.
And check out their new macadamia nut butters, which are absolutely delicious.
Our House of Academias highly recommends House of Macadamias for all your macadamic needs.
For a limited time, House of Macadamias is offering listeners a free box of their bestseller Namibian Sea Salted Macadamia Nuts worth $35 with your purchase at houseofmacadamias.com slash darkhorse plus 20% off your whole order with the code darkhorse.
Once again, that's houseofmacadamias.com H-O-U-S-E-O-F-M-A-C-A-D-A-M-I-A-S dot com and use the code Dark Horse for 20% off every order.
You won't be sorry.
So I think most people understand that when you call somebody a doomer, they think it's more likely than not that we are doomed, right?
And just for clarity's sake, I also consider that there is a serious chance of doom.
Some chance, anyway, that things go really bad.
And in fact, this is why I'm animated, because sadly, kind of the And really, that's where AI safety is actually a better term.
In pursuit of safety, we are making things less safe.
Like with so many other things, this sort of creation of panic in the public, telling them about how, you know, some AI is going to order viruses from a lab and they're going to do these completely, you know, very complicated things or whatever.
Could it happen?
Sure.
But a thought experiment is one thing.
In reality, it's a very different thing.
A thought experiment is not an experiment, for instance, because, you know, you actually have to get data back from not you to consider something an experiment.
But so, unfortunately, I think that the net effect that this group of people who have been preparing for this moment for decades, right, they should have had something.
And what they had was You know, ill-advised essays that called for extreme measures, some of them more extreme than others, some of them more sort of policy-worded than others, but all of them, by my estimation, at best, lead to a world where 1984 is a utopia we aspire to.
Let me put it that way.
Yeah, 1984 is the best case scenario in the world that's being proposed.
So look, there are a number of things to dissect here.
You and I are not in exactly the same place with respect to what we think is likely to come out of AI.
You and I are in the same place, as far as I know, in believing that the possibility of something absolutely terrible coming out of this is not small.
Right?
It's unacceptably large.
Right?
Now, I've laid out, as you know, a taxonomy of five different types of catastrophe that could come from AGI, three of which I believe to be literally guaranteed.
They are going to happen and are happening, in fact, even pre-AGI, if that's where we are.
So, I might be farther down the road of thinking we are headed for a disaster, and the disaster is certain.
But the people you're calling doomers actually believe human extinction is extremely likely, which is why they are calling for absolutely draconian measures, which you and I agree Those measures are more likely to cause catastrophe than to prevent it.
The measures themselves are ill-considered, and I would—there's one thing I 100% want on the table here.
You've got a group of people who have just come through a most shocking piece of history in which their toolkit should have set them up to beat almost everyone, and instead, they completely fucked it up.
It was a good toolkit.
I used it.
It worked.
It was a good toolkit, and if they had actually applied it to COVID, They would have come out somewhere very different.
They would have been much less welcome at certain cocktail parties, but they would have been ahead of the curve.
Instead, they botched it, and worse, they never admitted it.
Having just demonstrated that whatever their movement is composed of is incapable of dealing with something as tractable as COVID, Those same people are now demanding that the world do absolutely draconian things, you know, including airstrikes on server farms.
Effectively, they want the ability to declare war on people who do things that they say are Going to cause humanity to go extinct, right?
They have not demonstrated the mental discipline to do this.
Their model, to the extent that they have revealed it, is full of cheats and errors and inappropriate uses of principles.
And so it is therefore very frightening, to me at least, to see them They should be very uncertain of where we are and what to do about it, right?
And they should be working their way forward from an uncertainty.
Rather, what they have done is told everybody else you should be uncertain, and they have doubled down on certainty.
And what they are asking for, those of us who understand game theory, complex systems, etc., are looking at and thinking, holy hell, do you realize what you are inviting, right?
You are inviting a disaster that might not otherwise happen.
Let's take a very simple approach to risk, and I'm sure we're going to go many ways with this, but here's my very basic framework.
Risk comes because a naturally or otherwise occurring single point of failure or single point of control goes wrong.
Let's say we all depend on the Earth.
The Earth is our single point of failure.
If an asteroid big enough hits the Earth, we're gone, because the Earth is a single point of failure.
Sure.
What is a good way to deal with that?
Well, you know, you might or might not agree with Elon.
I know you've got your doubts, but making life multi-planetary is kind of that.
Well, then we've got the solar system, et cetera, et cetera.
Sure.
But, you know, for that limited question of like, what makes the Earth robust?
You know, what makes us, you know, the Earth a single point of failure?
You can sort of reason about that, right?
Yeah, yeah.
Hold on.
Hold on.
Point of order.
It's very important to me, especially since Elon might see this, that he understand I am not arguing it is not a very good idea to make humans multi-planetary.
My concern is that that is not a near enough term solution, that the biology on Mars or anywhere else is a tricky enough problem that this does not deal with our immediate crisis, right?
You know, in other words, it's like we've got a house fire and the proposal is, well, what we need is a house built out of different materials.
And it's like, Okay, but no, we've got a house fire, there are people in that building, it's burning, and it's like, yeah, but if that building was made out of concrete, you know, and it's like, I don't want to talk about building materials now, I want to talk about the house fire, right?
Right.
So anyway, my objection is not to being multi-planetary, I think it's a good idea.
I know it's nuanced, and yeah, I didn't mean to imply that you are as well.
Yeah, yeah, no, it wasn't you, I just want it clear.
Yeah.
Okay, so then, you know, that's basically the same pattern as you say, if you say like, well, you know, if you put, I don't know, a single, you know, incident of health of some nation in charge of, you know, by extension, the whole world's response to COVID.
Even if they were staffed by the best people, any mistake they make is going to go very far.
It's the same problem with monocultures, right?
One virus gets in, it gets in everywhere.
All of these are the same thing.
Existential risks exist because, as a civilization, we have not weeded out all of the single points of failure, some of which are single points of control that we have created.
Right, so governments we have created, right, to solve certain problems, but they themselves are juicy targets.
Let's put it that way.
So in that scope, right, my thought And sort of where I'm aghast at my fellow rationalists, it's like, if you are discussing how to reduce existential risk, you couldn't possibly be suggesting that we add a single point of failure to the mix, right?
Especially when you're talking about an intelligent agent that can do what all the other intelligent agents are doing and try to capture that thing, right?
Let's say we made a world AI organization, a WIO.
You know, in the spirit of the WTO and WHO, right?
Do you think, maybe, that there is enough literature on regulatory capture that an AI that just woke up would be like, oh, I know what to do.
You know, let's get an island somewhere on some plains.
I mean, I don't know.
It's kind of like, you don't need to go to, you know, advanced biology to figure that one out.
So, let's make that perfectly clear, right?
First of all, Yudkowsky himself is famous for pointing out the high likelihood that an AGI could talk its way out of a safety protocol, a box that it was designed to contain it, right?
So, you're talking about a single point of failure that an AGI would then have as a target, and were it to capture that thing, it would have de facto control over the board, right?
Over any competition that might arise, because the function of that International Atomic Energy Commission, whatever, agency, That's how Altman proposed, for instance, is to keep AIs below a certain capability.
So all you have to do is capture it and make it effective.
So long as it doesn't catch you.
I mean, it doesn't... It's exactly what, in a business context, this would be obvious, right?
You want to hobble all of the competitors in their crib, right?
That's the game.
And the idea that you're going to build the mechanism whereby an AGI could do that to its would-be AGI competitors is foolhardy.
Now, I'm troubled by the fact that we find ourselves staring down the barrel of the AGI era, and that we are now playing catch-up.
The horses are out of the barn, what are we supposed to do about it?
And the answer is, I understand this isn't the time to figure out what dumb fuck let the horses out of the barn, But wow, was that a big error, right?
Now, I'm not sure what to do about it.
I don't know what the non-authoritarian response is that might have contained the developing technology that will produce AGI, but nonetheless, We are left playing a very difficult game.
This thing now available in public and the keys to how to produce it more or less Distributed about, I know there are details which are not, but anyway, that's a difficult puzzle to solve.
On the other hand, watching again, the folks who fucked up COVID so badly, arguing for the absolutely most draconian reorganization imaginable, basically handing over power to people who think that they've gamed this through thoroughly, have done some obviously broken math,
to produce a level of certainty that would if they're effectively like creating the inverse of Pascal's wager in order to require us to engage in a behavior that they think is the right one but it's like you'd have to be mad not to do what they think and the answer is no actually I just disagree with your calculus I'm not mad.
Here's the thing, right?
Yes, I think here's something very interesting I learned over the last couple of days.
Most people in that community do not actually agree with Youkowski on his estimate.
So his estimate as articulated, I believe in the Bankless podcast, though it might have been in one of the other ones he appeared in the same period.
He said he's 99.5% and maybe a little bit more sure that we're doomed.
Right?
And doom is a word that's used sort of as a term of art, as in we either go extinct, or we get kept in a zoo that we can't get out of, or like, you know, all sorts of bad, you know, things that I don't want to get further down that tree.
It's not, yeah, the sort of thing that people want to talk about.
But, you know, we're doomed.
Like, it's a bad outcome.
Like, we're, you know, the possibility of, you know, occupying universe, let's call it, is permanently gone.
And probably a lot worse than that.
So the problem with that is that if you believe that we are 99.5% doomed, if I offer you a 99% doomed scenario, you would take it.
It is an upgrade.
Yep.
Right.
And that is really bad.
So I want to call out to the rationalists who think it's, you know, it's 30% likely 20% likely, you know, I even quibble with the idea of putting percentages on these things, but whatever, like, let's just go with it.
Um, Even if you think that, please understand, you think Eliezer Yudkowsky is mostly wrong.
And if he would take a 30% risk when you think the current risk is 20, he is advocating for making your situation worse.
This is not a situation where it's like, well, he's making people aware of AI risk.
That's not that kind of timeline.
Yes, people are going to be aware of their risk, trust me.
The point is to have a realistic understanding of that risk so we don't overreact and make things worse, as we have in similar situations.
In fact, let's get to that about the rationalists and COVID, because I think if they hear you say they messed up COVID, I know what they'll say, and I think I want to bring that up, handle it, and bring us back here.
which is to say, rationalists think they did excellent during COVID because they picked up the signal early, which is true, right?
By February and March, rationalists were all out buying masks, et cetera, et cetera, et cetera, much earlier than most people taking it seriously.
Great.
However, from that sort of triumph, the next thing they did is they gave themselves to modeling exercises, basically, right?
A lot of rationalist or rational adjacent money went to the Imperial College for these sort of simulations they were doing, and these models, etc, etc.
And a lot of that was sort of built on that.
And you know what, a pandemic is actually, you know, after we deal with the original Horrible mistakes.
It's pretty good for a thing you model.
Like, honestly, if you're going to model something, you know, how a virus spreads in a population is a pretty well understood problem.
And we still did it pretty poorly.
But that actually fed into this mindset that you can just go off in your sort of crib on a notebook or on a computer and like put some numbers down and know what's going to happen.
And first of all, that was wrong with COVID for reasons we can get into.
But secondly, Right now, they have the mindset that they've basically discovered psychohistory.
They kind of know the broad patterns of history and how it's all going to end and what we should all do.
We should just shut up and listen.
And it is built on layers and layers and layers of bad assumptions.
Reality is complicated.
Complex systems are complex.
Just because you know what to eat to not starve does not mean you can model A novel technology with superior intelligence five times down the road, or what regulatory authorities are going to do if you bring them into a conversation they fundamentally do not understand.
You get, at best, AI czar Kamala, and then it gets worse.
You get that.
I mean, yeah.
For one thing, there's no way that given the level of capture, you don't get somebody who thinks, oh, with this level of control, I can make a mint, right?
It's not even, I mean, it's not even worth it.
I don't know.
The ring of power shows up.
What do we do?
Oh, I don't, I don't know.
Get Sauron involved.
Right.
So I want to come back to a couple of things.
One, you said that Yudkowsky is 99% sure we're doomed, which is very different than there is a 99% chance that we are doomed.
It's subtle.
But let's say that you are being sucked over a waterfall, right?
You're in a canoe and you're being sucked over a waterfall and you're paddling as hard as you can.
There's a point at which there's no amount of force you could generate that's going to prevent you from going over the falls, right?
So that's 100% you're going over the falls.
And you might be 99% sure that you're past that point.
Sounds like that's where Yudkowsky has us.
That's very different than, you know, 99% of the trajectories we might take from here fail to save us, but there is one and we're looking for it, right?
You have to calculate those two things differently.
But the other thing I'm getting from this, two of them, is you have to watch out for somebody who has 99% chance of doom.
Either of the versions I've just spelled out.
Either there's a 99% chance we are past the point of no return, or there's a 99% chance we will not succeed in escaping.
Either of those is essentially... I mean, I know this well because I'm a guy who does very well in emergency circumstances, right?
At the point at which you're 99% doomed, there are things that become very rational for you to attempt that you would never consider if you were only 90% doomed, right?
That's exactly it.
And so, why do I not want to hand over control, analytical or otherwise, to anybody who's so certain in circumstances in which you really couldn't be so certain?
Right.
The uncertainties here are many.
And, you know, just the simple idea that he's potentially doing the math wrong, which really looks to be the case, the more you scrutinize the analysis, the more it looks like it is, you know, I took him to task on on Twitter and the podcast for treating a complex system as a complicated system.
And if you do that, you get very funny things out of your analysis for things that will lead that will mislead you dramatically.
And I'm reminded a little bit about, you know, about Mandelbrot and the It's not really the discovery of fractal math, but the innovation of modern fractal geometry.
Probably not to the degree of detail you do, so go for it.
I'll say it very simply.
There are some paradoxes that emerge if you look at things like, how long is the coastline of this landmass?
It turns out that the finer a measuring device you use, the longer the coastline is.
An infinitely fine measuring device says that the coastline is infinitely long, irrespective of the size of the landmass, and that can't be right.
They can't all have an infinitely long coastline.
You need something that deals with the fact that the finer you measure, the more convolutions there are, but that's not really adding to a linear calculation.
There's something about what is being calculated here where the number of tiny fractional possibilities that are being added up and coming to effectively one is not how you would do this calculation, right?
There are whole branches of the tree that are being excluded here because the one branch that's being calculated is being allowed to take up the entire possibility space and all I would say is It doesn't have to be the exact description of how you're getting a weird number out of this, but that number that he's calculating in light of what we all know ought to tell you there's too much certainty here for the degree to which none of us know where we are, right?
Yes.
I mean, here's how perhaps he gets there, and I'll put a little bit of a question mark, but that's how I understand it anyway.
So that's where the Foomers come in.
So Foom is this idea, much like a fire that sort of just, you know, captures something and then kind of goes, boom.
That not only AI, you know, self-improvement is possible, so you get an AGI that can now program itself better than Humans could, right?
Hey, it's better at us at least that, right?
So therefore it can arbitrarily keep upgrading itself and therefore will become, you know, all-powerful.
But that will happen very quickly, right?
So it could be in the scope of seconds or minutes or hours or even days or even a few weeks, but not years.
And definitely not decades.
So that's the FOOM position.
And from that position, you can sort of see, especially if you're convinced that a superintelligence will, that is a singleton, as they call it, is uncontested.
Let's say it doesn't have something else that will stop it, will just do away with us because, hey, why not, or whatever.
And again, we can't really be that sure about that, but let's just say that that is sensible.
Right?
But you can sort of see how that all holds together.
The problem is that the way we got into this, in reality, is these large language models, which, you know, do not appear to be agentic at all.
They don't want anything, right?
If they want anything at all, it is to compute the next token, as they call it, find the next word in a text.
I'm using the word with some liberties here, but for most people that's roughly accurate.
But, you know, that's... And you can sort of see how that can be used, because, well, hey, if you can do that, you can write code, et cetera, et cetera.
But the important part is that, you know, when we've tried to turn them into agents, right?
So, we did this whole, like, auto GPT, or somebody else made chaos GPT, or whatever.
Baby AGI.
There's a few projects on GitHub.
Well, they, you know, besides the original, like, Hey, that's amazing or whatever, like they don't work that well.
Right.
Like once you chain enough, like uncertain tasks on the back of the other, the uncertainty is dominant and the thing like crashes and burns and it doesn't crash and burn in a way that it like, I don't know, it takes over the world and kills us all.
It just crashes and burns.
And like the way programs crash, like it just doesn't do much.
Maybe it spends 20 bucks off your credit card.
Will that be sorted out?
It totally could be.
Like, I'm not here saying that this is not possible.
What I'm saying is, we still got room to go, and humans are still in charge, mostly, and there is nuance in the path.
The path matters, right?
If it's a foom, Well, then the path does not matter, because here we are sitting one day, and the next day, our new overlords announce their presence.
But the way it is, the sequence of operations matters.
And if you hear a lot of Youkowsky's arguments, there are, well, if a superintelligence showed up, Yeah, sure, but it's not going to just appear, right?
We have some control authority here to do stuff like figure out how it works and take various adjustment courses.
This is going to open up a whole other path.
My question is about symbiosis, right?
Are the two even distinct?
Is it even worth talking about AGI versus human?
Or should we be talking about human plus AGI?
There's so much to talk about.
That's my third of five killer scenarios, or deadly scenarios, is malevolent use of AI.
So that is human plus AGI.
Sorry, I didn't mean to interrupt you.
No, but well, but that's and that brings us back to the single point of failure thing, right?
If we were sane, we would be dismantling all of these sort of low-hanging fruit, as they call it, like, you know, easy ways to world dominance.
Not because we are anarchists or, you know, into radical equity, but because, well, those things are the levers you do not want, you know, the checkouts guns that you do not want on the stage.
an all-powerful criminal breaks into your house.
You don't want the gun just sitting there.
Hold it or throw it away.
Do not just leave it next to the fireplace from where the thief is going to go.
Just don't do that.
So yeah, that's sort of where I get into the whole sequence operation matters.
Feedback matters.
I think that's my difference, actually, with most of the rationalists.
Having run a company for 10 years, I understand the dynamic decision-maker making, and that planning tends to go awry.
You don't want to over-plan.
You don't want to over-specify.
You want to be on your feet.
In my mind, Youkowski should be furiously experimenting right now with the latest technology to figure out what we can find out, not sort of making overconfident predictions.
But again, I'm not going to tell them what to do, right?
Anybody can do whatever they want.
The degree to which I'm interested in all of that is a degree to which there is panic spread, and that panic will not be used by Youkowski, but it will be used by others who see the ring of power and will move the appropriate levers to shut the rest of us out.
And that's really where the rubber meets the road.
It's not about what Yudkowsky should or should not be doing.
Everyone is free to, you know, and he knows more than most.
His perspective belongs in the discussion, right?
Maybe he's right.
On the other hand, It's a question about listening to him and acting on this suspect analysis, doing absolutely draconian things that you would never do if you weren't effectively convinced that there was nothing to lose, which is where he seems to have taken us.
I want to ask you, maybe this has come up in conversation somewhere, I have not seen it yet.
But I know that there is a chapter in the history of rationalism that I think actually belongs in the discussion of how we should take this current claim.
And that is the chapter of Rocco's Basilisk.
Can we talk about that?
To the degree that I can articulate it, sure.
Yeah, you want to lay it out?
Yeah, again, I'm not current on it, but I'll do my best to reconstruction.
So Rocko's Basilisk is this idea that So, okay, before we get to that, I will try very hard to not go sort of all technical, but you'll help me.
First of all, Rocco's a guy, a rationalist, right?
Rocco's a guy.
Rocco's a guy.
And he posted something... Let's have the human interest story here.
Rocco's a guy whose basilisk was the reason that Elon met Grimes.
So, they both searched for the same joke, which was... I did not realize the basilisk had introduced those two, but okay.
Interesting.
It does strange things, that basilisk.
Yeah.
So, whatever.
Okay.
So, let's get a little bit technical.
So, the first idea is a causal trade, right?
And this is kind of the hard one, right?
So that two agents can cooperate over a gap, right?
It could be a time gap or space gap.
This is something that Rocco posted on Reddit or something?
LessWrong back in the day.
I'm talking like 10 years ago.
LessWrong back in the day.
So, the idea of a causal trade I don't think comes from Rocco.
The Basilisk does.
A causal trade comes from somewhere in game theory.
But the point is, if two agents understand each other sufficiently, they can do what the other wants on the condition that the other agent will also do what they want without any conversation exchanging hands, right?
Like exchanging sort of...
You know, going across each other, right?
So, and this is kind of often thought in terms of like the speed of light.
So if you've got, you know, like a very large, very large distances, you don't even want to send any message at all.
So you kind of like, well, you know, what would, you know, what would they do if they were cooperating with me?
I should do that, et cetera, et cetera.
And if they, if they're not doing, this is the important part, if they're not doing the thing that I want them to be doing, I would want them to be doing based on my fair calculation, whatever.
When I get there, I'll punish them.
Right.
So you kind of, you've got two people, like you've got prisoner's dilemma, but there is no communication.
And at some point the walls are lifted.
Right.
So you say, well, I don't want to defect because if I do defect, yes, I get my benefit, but then the other person is going to know.
Um, and I should kind of do, I should affect my behavior.
Okay.
So we, we take all of that.
And again, I appreciate this can get complicated.
Um, but we take that idea, uh, and we say, well, maybe the AI is already kind of posing that dilemma to us.
Right?
So maybe it's already saying that, um, if you help me come into existence, I will reward you because, Hey, that's cooperation, right?
With it.
Um, but if you don't, and I'm just going to.
That's the Basilisk, right?
That's the scary thought of, if it is going to happen, then it is at the benefit of people today to make it come into existence, because us not doing that or not bringing it to existence earlier will be seen as a defection.
You can even articulate it like, we let more people die than there should have been, or whatever.
So, the idea is you could have the emergence of cooperation without communication over a distance based on the fact that when there is no distance, the person on the other side of the communication will behave in a way that you can predict.
And that can be done in time as well.
The AGI that does not exist, how will it feel about what I am doing relative to bringing it into existence at the point that it does arrive?
Right?
And so I better start working now before there is AGI to make the AGI happy when it arrives with my behavior so that it doesn't hurt me.
Right?
Something like that.
Okay.
It's a good way to put it.
Okay.
I didn't mean to say like there's something.
No, no.
Good enough.
All I, you know, for the relevance to the current story is not the details of Rocco's Basilisk.
What happened after Rocco posted this on LessWrong?
What year would that have been, roughly?
It would have been early 2010s.
I don't know if it was 2010 or 2011 or 2012, that period.
or 2011 or 2012 that period so what happened Yukowski lost his mind He went at him really hard in ways that I'd never seen.
Like, I mean, it was like all caps, you know, you idiot, or you kind of let the cat out of the bag basically, right?
By spreading this knowledge, you have now made it a thing.
Even if it wasn't, even if that deal was not extended, well now people know about it, so they can't even claim they didn't know.
So therefore?
So therefore that that might You making it more likely to happen?
Near certain, as I understood the, I mean, I only learned about this long after it had happened, but that the idea that Rocco even posting this hypothetical basically condemned people to a future of some kind of hell
If they did not start doing their best to figure out what the AGI would want and doing its bidding in a present before the AGI arrived.
Is that right?
I don't know the certainty that he expressed, but he definitely was sure enough to cause a big scene, let's say, right?
And to sort of turn a lot of people off.
So he must have thought it was Wasn't, you know, wasn't the post purged?
And people forbidden to go looking for it or something?
I think it was.
I think it was.
Yeah.
I don't know if Rocco was on board with that or whatever.
I mean, Rocco is on good terms with Yukovsky now, so I don't know how all of that stuff played out.
But yeah, it was, it was sort of, you know, banned and purged for a while or whatever.
I mean, probably still is.
I don't know.
It's in a hundred places on the internet, right?
Right, right.
Look, it doesn't matter because I'm not really interested in the Basilisk problem per se.
What I'm interested in is that it is a early sign of a kind of inappropriate thinking and inappropriate interaction surrounding a problem that is quite parallel to the one that we are currently facing.
In other words, I believe that in evaluating Yudkowsky's View on The certainty of our doom and therefore what the rational behavior is that this is actually an indicator of how this person Behaves and what they perceive to be an emergency Relative to something that they can calculate that others don't necessarily see in the same terms, right?
I think the point is this is our test case and it tells us an awful lot, right?
It tells us I think That this is a person who is so wrapped up in their own thought process, so lacking in what I have heard you and others call epistemic humility, that they panic, they reach a conclusion about which they are very certain, and they publicly panic over the fact that something dire has occurred that
Basically, anyone who doesn't see it is a fool.
Right?
Yeah.
I mean, if I'm still manning his position, I would say he would say that, like, he wanted to create a social, you know, negative around doing things of that nature.
Now, again, that I don't think that that was exactly how it played out.
It was a lot more, you know, foaming at the mouth and a lot less, you know, the next person who does this, uh, kind of thing.
But again, it's, it's, it all leads down an authoritarian path, right?
Because what do you want to do?
You want to control information.
And, and, and the other thing I want to say, I'm trying to think how to say it because I, I, unfortunately I don't have a lot of the details, but I think it's important is that on the basis of such ideas, people in those circles have had, um, pretty dramatic psychological events.
Right.
So this is not events.
I don't know if it's psychotic breaks or stuff like that, but like it, these are people who, as they say for themselves, they take ideas seriously.
Right.
And Hey, I, I, you know, I admire that.
But there is a reason why most people don't because most ideas are worth taking seriously.
You know, there's a, there's a sort of whatever-ness baked in inside of us.
And if you're able to disable enough of those safeguards and like really believe that something's going to happen.
Um, things can, can, I mean, that can be really powerful or well, really powerful, but in not in kind of the way you were intending.
Yeah.
Okay.
Um, so the question then, I guess, you know, I, I don't, I don't mean to be, I don't mean to be so focused on Yudkowsky.
In fact, I wouldn't be focused on him at all if he weren't In my opinion, pulling a fast one in which he's saying, look, here's a scenario.
Either you get it, in which case you'll agree with me about what needs to happen, or you don't get it, in which case you're not smart enough to get it.
So basically it's a cheat that results in everybody who gets it knows that what we have to do next and the people who don't get it are the obstacles, you know.
Forgive me for being traumatized over being told a thousand times that we had a pandemic of the unvaccinated, right?
It's like that kind of trick where you create this idea that anybody who doesn't see the world as you do is the obstacle to all of our well-being.
And it's kind of your predicament, really.
What?
It's kind of your predicament.
Yeah, exactly.
But nonetheless, I don't like This is not how I would expect somebody who actually did spot an emergency that other people didn't see.
This is not how I would expect them to behave, right?
I would say especially if you had 20 years to prepare for that moment.
Yeah, that's a good point.
I would expect something a little bit more polished, at the very least.
Like something more thought out than... Hey, not even polished.
I would be suspect of polish, but I would say something more persuasive and less manipulative.
Anyway, by the way, for people who might be watching this and saying, you know, what are the alternatives?
I do want to point people to Paul Cristiano, who, you know, is definitely not unconcerned.
I think his probability that something goes wrong is in the, I don't know, 20 to 40 range, let's call it, maybe even a little bit above some days.
But I think sub 50 is important.
I think 50% is a very important threshold.
Yeah, it's a very important tipping point.
And he has made actual contributions to the modern state of AI.
And he has written... Yukowsky has cited him as somebody who gets it and has had long debates with Yukowsky or something along those lines.
And Paul Christiano has written, you know, very eloquent.
He's not like me.
He's very, you know, measured in his tone and stuff.
But he has said, Look, I feel Youkowski hand waves a lot in some of his arguments.
I feel like, you know, he's overly certain.
I feel like he doesn't have full understanding of the cutting edge of the field, and he downplays certain things that he shouldn't, etc, etc.
Which, again, is not to knock anybody, except if, you know, you're shouting fire in a crowd.
Which, by the way, is not a free speech violation these days.
Mind you.
Right.
But, just to say.
But, you know, again, I support basically, look, I support Eleazar's right to say all of that stuff, but I think the correct response is to What we're doing, which is to speak the opposing view, that is, no, we're not that doomed unless we make it.
Unless we actually take actions that will concentrate all of the risk in one place and just offer a juicy soft target to an AI that maybe is not that capable.
Oh, I have a terrible idea.
We can put it on an island.
No, that's a truly terrible idea.
I have a terrible interpretation of the present.
I would like to get you, I mean, as long as we're over here in this territory.
Go for it.
Eliezer Yudkowsky believes in Rako's Basilisk.
You got it?
You got it?
Yeah, I mean, hey, I can't rule it out.
So let me spell that out for others.
I'm glad you got it immediately.
You tell me if you in fact have what I'm about to say.
If Eliezer Yudkowsky believes in Rako's Basilisk, and he also believes the AGI era is about to dawn, and therefore the AGI is coming and is going to look at Eliezer Yudkowsky and say, well, what have you done for me?
Eliezer Yudkowsky had better get the world to create a single point of failure that the AGI can capture, right?
That is the way Eliezer is going to survive the AI apocalypse, is by being the AI's right-hand man, and he is now doing its bidding to make the world more vulnerable.
You know, I've heard... It explains a lot.
It could.
He's totally capable of conceiving that thought.
In fact, how could he not have?
He probably is.
So, I don't know.
I mean, it makes sense.
Look, I've got to tell you, when I offered that... You know, I feel like there's an Occam's razor violation here.
On the other hand, I can't tell you what is a simpler explanation.
So... No, no.
Look, let me be perfectly clear.
I don't think that's what he's doing.
But the point is, when somebody has had the reaction to Rocco's Basilisk that Yudkowsky has, and then the AGI era is upon us and he's arguing for things that seem to make matters worse, And doing everything in his power to persuade us that actually we must take these actions because the certainties are so high.
That is at least consistent with the idea that he is in fact covering his own ass in light of the fact that he thinks the AI is about to be in charge and, you know, it's going to view us differently based on what we've done in this moment.
I think it's a meme worth existing.
Well, it is.
I think it is also just worth pointing out.
Hey, Eliezer, you've put these things on the table.
If we put them together, they add up with the possibility that, let's say that you're not doing this, right?
Just recognize that some things, some arguments you have made very forcefully when put together, create a third argument about what you yourself are doing in the present, which requires us to ignore you.
Well, well, for sure.
But here's what makes that argument, you know, as plausible as it is, which is that, you know, let's just accept for the purposes of this argument that probability of doom is where it is or whatever.
The question is, how do you know that your proposed remedy is going to make things better?
Right.
Because, you know, for somebody who demands, you know, very rigorous proof, formal proof, if possible, you know, very detailed argumentation for, you know, why AI risk is not that high or is lower or is very low or can't happen or whatever.
Right.
And some of those critics are just not very informed critics.
Right.
And a lot of the arguments he gets are not very good arguments.
I get it.
OK.
Now, let's go on the other side of the ledger, right?
He is advocating for, I mean, I don't even know if he would agree that he is advocating for it, but that's what he put out.
If you read this essay at Time Magazine, what did he suggest?
He suggested a multi-country treaty.
let's say, I think the example countries you mentioned is the US and China, right?
That would agree that they would perform a preemptive strike on Russia, on a non-signatory country.
This is very important, right?
So the treaty part is almost like a coalition of the will.
It doesn't really, the treaty part doesn't really matter in this case.
It is a bunch, we could say the 50 states of the United States got together in time to treat.
Sure.
The point is, there's a bunch of the world, right?
That gets together and says, whoever else, we're not going to develop AI.
but if you catch anybody else, we're coming for you, right?
That is the plan that was put forth in the Time Magazine essay.
And the question is, so basically that requires, right, this sort of international atomic energy agency.
I don't know the name.
The E-I-E-I-E-A-I, I think.
Anyway.
That sort of international monitoring body, right?
To go everywhere and check everybody's, you know, data centers and GPU production.
And people have said like all GPU production should be tracked.
People have suggested that we stop, you know, GPU production for 10 years, all sorts of crazy things.
But the point is, if you build that centralized control authority, then you have that problem.
And there's no way out of the problem.
And that's why we call it the AI lockdown.
It's the same idea.
Stop the world, basically, is the vibe.
Yeah, but much like the lockdowns, where is the case?
It's like, we'll do that and it'll stop.
But why do you believe what you believe?
This is a key rationalist question.
Why do you think it will do that?
Because I've got a lot of evidence for you, man, of so many things that went the opposite.
Not that didn't work.
How could it possibly go right?
Especially in a world with AGI.
Without AGI, it still adds an extra single point of failure.
With AGI, it's like throwing gasoline on the fire.
And I must tell you, even hearing you describe what I had already read in his Time Magazine article If you then take the yes, tongue-in-cheek interpretation that he might be doing the AGI's evil bidding for his own reasons and that that's why he's advocating these things, which again, it's not what I believe he's actually doing.
I take him at his word, but I cannot put aside that if I take everything I know about his thought process, that it would make perfect sense for him to do this.
But if you do that, then what you have just described is Oh, maybe the AGI is going to want an Air Force.
How are you going to get the AGI an Air Force?
It's not easy, because Air Forces are expensive, hard to come by.
People who have them are not very likely to cough them up.
Well, you need a rationale, and this would be it, right?
Air strikes on server farms are the rationale for the AGI to get airplanes.
So anyway, again, I am not...
I'm not arguing that this is what he's doing.
It could be.
But what I'm arguing is that the fact that this is a rational interpretation of his behavior, of the reasons that he would make arguments that to some of us look preposterous, right?
This would be an explanation for that.
And the fact that it would be means either it's what he's doing, in which case we have an obligation to ignore it, or it's not what he's doing.
And there's some large incorrect calculation, at least one of them buried somewhere in this matrix of things that he has claimed are true.
By the way, before we, I, you know, I don't really want to focus.
I don't, I don't even know how much time I spent on, on specifically Yachowsky, but, um, Just to sort of deal with a possible objection, Some people will say, well, you know, isn't that how the nuclear non-proliferation treaty kind of works anyway, right?
That's one that comes up often.
So let me put a few facts on the table.
First of all, There's 50% more nuclear armed powers today than when the non-proliferation treaty was signed in 1970.
Secondly, the signing of the treaty is based on the fact that some countries have nukes.
Right, so let's transfer that analogy over.
You would need some AGIs or superintelligence or whatever to exist to make a credible threat and make other people sign that treaty.
So, you know, it doesn't really work.
And honestly, all the nukes we have now have been built after that time because it didn't actually stop the US and the USSR from building more nukes.
The peak was much later.
So, you know, that's not the analogy you're looking for.
It's also hard to make nukes, right?
So the idea that some kind of control coupled with the difficulty in producing them would have some sort of a positive benefit in a world that does not mirror the AI world of today.
It's a terrible analogy.
Let's put it that way.
We haven't destroyed the smallpox we have.
Yeah.
People like to keep powerful things.
Vladimir Putin is on the record in 2017 saying whoever controls AI will control the world.
I don't know what world they're... Yes, that's the sort of thing it will require.
It will require world war.
Right?
If you could get a few powers to be sane.
But people are not, the US and the USSR, well, Russia today, not the YouTube channel, have not destroyed their samples of smallpox, right?
You tell me what the beneficial use of that is.
I can tell you many beneficial uses of AI, right?
But people are just not in the power of giving up, not in the habit of giving up things that yield power.
you know, the non-workiness of that idea.
And you know what will happen, actually?
Let's put it on a table too.
So there was this AI pause letter, right, that was circulated, and they said, you know, we want for at least six months, I want to highlight that letter said at least six months, not six months, at least six months, pause, right?
Again, as if there's like some sort of switch that can be pressed somewhere, and that will apply to the Chinese labs and the secret NSA facilities and, you know, the basement labs under the Kremlin and whatever.
But whatever.
What happens is that the research goes underwater, right?
It goes to places where we can't see it.
It goes to places where people can't make arguments about why it's unsafe.
And just to give you an example, that very letter cited certain things that were successful examples of pausing research.
One of the things it cited was gain-of-function research.
You know, I wish I was making this up, but literally their example of a pause technology was gain-of-function research.
For anybody who's watching this who has not followed along, even if we accept that gain-of-function research, that particular gain-of-function research did not give rise to SARS-CoV-2, what we do know is that that research was outsourced by the National Institutes of Health in the United States to China, and that that very pause was a factor in that outsourcing because they just couldn't do it here.
So they gave it to the Wuhan Institute of Virology, which, instead of doing that research on their BSL-4 facility, did that research in their BSL-2 facility.
What is a BSL-2 facility, you might ask?
Well, let's say it's got the biosecurity level of your dentist's office.
Right.
Something like that.
Right.
Like, I mean, and they said, well, you know, it wasn't, I don't know, there was some, some complicated argument, but that knowledge made several top virologists who were previously supporting WIV to say, okay, no, that is, wait, you didn't tell me that.
Right.
I thought you were doing BSL 4.
Point being, Here's an example where this pause led to a drop in safety, hidden from everybody, even if you don't believe that SARS-CoV-2 came out of that research, right?
Just that they were doing research that was, like, goddamn similar to that particular output.
Everybody agrees that that was a drop in safety, that that should not have happened.
But why did it happen?
Well, because we said, well, the U.S.
is not going to do that work unilaterally.
We're just going to step back.
Right.
So, you know, I think people should really study these examples, should really, truly try to understand what that means, because, you know, there's this sort of almost magical belief in regulation and treaties that they'll do something.
And it's just not not the experience.
Yeah.
Now, anybody who believes that, I mean, look, I'm a dyed-in-the-wool liberal.
I believe good governance is the key to a truly free society and that wise regulation is the mechanism that produces a society in which negative externalities are reduced to a tolerable level, etc., etc., etc.
But there's no way on earth that I want any of these regulators further empowered in this era because I've seen the that power under capture is a huge fraction of the problem.
And there's just no indication that the folks proposing this in the AI context understand the degree of capture or have any answer to how it is that it would be excluded from this set of regulations.
And I just don't think they have the goods.
I mean, this is what I've been pressing everybody on.
And what I'm hearing back is just not.
All right.
So let us move on to what I think should be our next topic.
quick.
You and I have had a friendly argument going about the value of the dawning AI versus the horror of the dawning AI.
And I don't want to caricature either one of us here.
Both of us see both things.
There are things that this AI liberates us from that are truly, or this proto AI liberates us from, that are truly remarkable and very positive.
Some things that have been wicked problems dogging humanity since the dawning of the technological age, let us say.
And then there are other things in which this opens fresh horrors that we may soon face, And the question is, A, is it just going to be a chaos and we're going to find out what the balance is?
Is the balance positive or negative over time?
Can we deduce that the balance might be reliably more positive than negative based on some kind of related rates calculation or the reverse?
Can we say that actually, yes, this is going to be more harmful than it is beneficial, even though the benefits are real, as we've seen with many technologies, right?
We've seen that.
With smartphones, we've seen it with the internet, including and especially social media, right?
I don't think any rational person should argue that a smartphone has not provided tremendous benefits to humans.
It really has.
But the net effect may well be deeply negative.
So, are we looking at another one of those?
Anyway, you and I have been arguing back and forth about that, and I believe we've gotten to some interesting places.
You want to lay out your perspective on the positives and negatives of the AI that appears to be emerging?
Yeah, so let's putting aside sort of the medium long term risks of, you know, just world ending scenarios, right?
Like, let's leave those aside for the moment.
There are other things that this technology can do that are Not very desirable, at least from where I stand.
I'm sure some people will find it desirable.
In particular, here's an experiment I ran, right?
This is not something I read somewhere.
I just did it myself.
I just ran into this.
I took a podcast, right?
Not a podcast from you, a podcast from ZDogg and Poloffit.
Right.
Like the, you know, the kind of the establishment contrarian types, let's call it.
Right.
And I asked, so I put the whole sort of transcript into chat GPT and I asked, tell me where the speakers diverge from the mainstream consensus.
Okay.
This was the question I asked.
And it's, here you go.
Right.
They talked about COVID vaccines and fertility.
Here you go.
Right.
And that is shocking.
Right.
The degree of censorship that this technology enables is a wet dream.
To the people who are trying to do this with probably the precursors of this technology during COVID.
It can, you know, like sometimes they say, well, you know, censorship is fine because, you know, it creates art because we speak in subtext and, you know, read between the lines.
This thing can read between the lines.
Right.
This thing can explain jokes.
Give it a joke.
Ask it to explain it to you.
It will do that.
So it's, you know, it's, it's, it's, it's really, you know, I, I never was fully convinced by 1984 because you could always, you know, the idea of newspeak, right.
That a bad thought was not possible in newspeak because I said, Hey, I can always say big brother, double plus on good.
Right.
Okay, so the idea of revolutionary thought not being thinkable, in you speak, didn't make full sense to me.
But here, you can wipe out the concept, right?
If you can get in the wires, right, between people, you can eliminate classes of conceptual conversation, even if you have the thought yourself for the first time.
You'll think you're crazy because nobody else is going to be saying that stuff, even if they are also thinking it, right?
So, okay.
After this vivid example of why exactly I'm truly worried, even if existential risks are out of the picture, about this technology being captured by the very people who are doing this very thing, this very moment.
You know, giving them, you know, a 10x in capability boost, or more probably, why that might be a bad idea.
It can do good things as well, right?
The sorts of things I've been able to do, for example... So wait, wait, wait.
The example where it reads between the lines is clearly a good thing.
It is a thing, right?
Which could be good or bad, depending on how it's used.
Well, the thing that worries me about it is that there's an arms race that will unfold, and the technology, to the extent that it can point to the places where Offit and ZDogg are off-narrative, also allows one to use it as a refinement technique for making manipulation so subtle that it can't be detected, right?
That's exactly the class of attacks it enables.
Yeah, and so to me, I'm concerned that there are some basic features of the universe that say that those who wish to use this for ill will outmatch those who wish to use it for good, although both of them will find it useful.
The fact is those who wish to articulate the truth have a limited canvas on which to paint.
And those who wish to obscure the truth effectively have the rest of the design space.
And I'm concerned that therefore what we you know, this is a loose analogy, but What we can expect to come out of the AI amplifier of both of their power is a greater amplification for the liars and propagandists than for the truth tellers.
And I think it is if you're a truth teller and you Check out the first round of the arms race.
You may be very happy with what comes out of it, but I think the point is it's a short ride before you get to the point where, as I argued on an episode of Dark Horse with Heather, we're going to come to a place where the only rational thing to do is not believe anything.
My point is that is better than believing stuff, but absolutely unsustainable.
Not believing stuff is not a way you can live.
So, okay.
I think if you see every fact as being a vote in a ballot, then you're right, right?
The number of possible facts is infinitely larger.
I hear Patterson screaming in my ear.
Big number larger than sort of the true facts, right?
So obviously you can swamp them out.
So the question then becomes, what do we do?
Thankfully, I've been thinking about this problem for a good decade, or I don't know, 15 years by now.
So I've got some, what looks like good news, it's conditional good news, right?
It needs certain things to happen in certain ways for that good news to take effect.
But I don't remember who said it, and he very well might have been Eliezer Yakovsky.
But there's this idea that all true facts...
All true facts must align.
At least that's how I say it.
The idea exists in other places.
I don't know if you're quoting me or not.
I have said all true narratives must reconcile.
The idea definitely exists in the rationalist corpus somewhere, and I'm pretty sure it's echoing other ideas, because hey, we're all climbing the same mountain, we're gonna get the same peaks.
But the point of that is, if I can have my personal system sweep up facts from all sorts of sources, and I'm looking for essentially the largest set of facts that are mutually compatible, Then that, you know, broadest network is going to be a network that must be biased towards truth because those things match each other, right?
A lie is by definition intention with something that happened, right?
The things that happened are always in agreement with all the other things that happened because they'd happened.
I don't know how to put this this way.
But that kind of speaks to the need for this technology to be distributed, to be available to all.
I understand the critique of positive rights, but this is one of them that is going to come out.
If it's not a human right, it's almost up there, where it's just going to be essential for each one of us to be I think you're missing something.
bring the first amendment if you want to bring the second whatever however you want to play it we we need to be uh equipped with artificial intelligence that that we control that we tell it how to act that we can uh audit to um filter the outside world um i think you're missing something okay let's play your scenario through
everybody gets an a an agi that works on their behalf Well, I wouldn't... It doesn't need to be necessarily agentic, right?
It doesn't need to have a will.
It's just... Think of it as a firewall.
No, I'm thinking that the arms race leads you to need it to have a will.
That effectively, if you imagine a world in which everybody has an automaton that simply does its bidding the way your computer does, or did until the bad people started algorithming us to death.
But that then results in people who have one that actually mirrors their own values such that it pursues their interests while not being directed will outcompete those that have one that will simply do what it's being asked.
Here's a question, right?
And this is a tension inside the human mind.
If asked, do you want the truth or do you want a narrative that will make you feel better?
Can most people bring themselves to consciously say, I want a narrative?
Tell me a story.
Some people will.
I've met some of them.
I don't know what most people will say, but I know that most people will choose.
Their revealed preference is for B.
Perhaps, perhaps.
Perhaps it's a systemic distortion of a different nature that that is what happens.
But the point is that there's another element here, right?
It's that having more truth, there's a reason to want more truth, right?
And that's that it makes you more capable.
All else being equal, it does.
Well, yeah, so, you know, having, and I mean, you can see it in the real world, right?
Who is the most competent political operative?
Is it one that doesn't know what the other side is thinking?
Or is it one that actually is pretty close to the truth, but, you know, is on your side?
Right?
There are forces, basically, that are pulling things towards the center that aren't, you know, self delusion is useful to a point.
It's not useful, you know, without measure.
And anyway, I think these these elements all come together to Create at least possibilities, right?
The whole point is, I think people should be more uncertain.
I'm not here to say what's gonna happen.
I'm here to say, these are potentials that are opening up in front of us, and that's the kinds of things that are possible.
There are some really good things.
For instance, we talked about this briefly, Iran.
It was a joke, of course.
A thing where, you know, I put in some proposal by somebody and just wrote underneath what could possibly go wrong.
Right.
And sure enough, it gave me a list of pretty good ideas about what could go wrong, some of which I hadn't come up with.
Right.
And I'm not saying that that is a verdict.
I'm saying that that is a good checklist to go through before doing something.
Right.
At least.
It was a very powerful demonstration of a capacity, of a useful capacity, a way in which the current LLMs can be consulted about problems and the various risks delineated.
It was pretty cool stuff actually.
I mean, your example was something like, uh, I'm going to have some cereal with milk.
What could go wrong?
And it gave you a pretty complete analysis of various ways, you know, from food allergies to choking, that your plan might be bad.
And then you had it analyzed whether the net risk from those things made it a good or a bad plan, and it gave you the go-ahead.
Yeah.
And again, I'm not advocating for anybody to actually use that as a, follow that blindly, right?
Use your, use your brains.
No, no.
When it comes to cereal, I think that they could probably trust it, you know, especially since the LLM correctly deduced that were they to have a serious food allergy, that they would probably know it already.
And therefore it was safe to greenlight the plan because the person themselves could be trusted not to eat the cereal if it wasn't good for them.
Yeah, right.
And so then I asked some ideas about putting sulfur in the atmosphere, SO2 in the atmosphere, to fight global warming.
Or as I say, do the thing that took out the dinosaurs, but do a little of it.
Take a homeopathic approach to the dinosaur extinction question.
And it did not greenlight that.
It did not greenlight it.
And again, that's just two random examples, right?
It happened to be correct on these ones.
It might be right or wrong, and there's all sorts of biases in these systems.
I'm not even going to the thing, you know, it's been trained to do certain things and certain things not, et cetera, et cetera, right?
I'm just saying, it's an interesting, it's a good second opinion.
Even though it suffers from the same things we suffer, it is unable to see the future.
It is at least able to do a more complete job of gathering what could possibly go wrong.
What else have I made it do?
I've used it to create bridges across scientific fields, for instance.
There were certain principles that I knew from Historical analysis and law.
I think the particular example I used was the principle of embarrassment in historical and biblical analysis and admission against own interest, I think, in law, right?
Those two things are basically the same thing.
It's the idea that if you are admitting something that doesn't help you, that that thing is probably, you wouldn't do it, you're not doing it because it's helping you by definition.
So it's more likely to be true.
If you read the Gospels and they are admitting something about themselves that doesn't make them look good in the local context, that is more likely to be true than some other thing that they're writing that makes them look good.
That's the idea, right?
And this is the same in law, that if a witness says something that makes them look bad, that's more likely to be true, etc.
Okay.
But the idea that...
The same principle existed in two places.
I was then able to use it and say, like, give me more correspondence.
It gave me a bunch of them across law and history, right?
And then I said, well, add another discipline.
And it was like, well, here's journalism, right?
And it, you know, it added that and basically created this kind of Rosetta Stone across different disciplines.
And as we said, it's really good at translation.
Oh, no, we haven't said it here, but As we know, it's really good at translation.
So then you're like, well, it could be used for translation across fields.
And long story short, this is something I've done in my life a lot, right?
I've gone into one field, I've learned some ideas, then I've gone to some other field, and I've transferred some good ideas that haven't made it yet.
So the next thing you do is you say, well, what epistemological ideas do exist in history that do not exist in, I don't know, entrepreneurship, right?
What happens if we apply that?
Boom, you know?
Innovation of some variety.
Yep.
You get a huge burst of wealth creation, actually, from leveraging what has been discovered in one place in places that are analogous.
Yeah, absolutely.
I had much of the same experience doing my dissertation work, where I discovered there were things that engineers, for example, knew that were perfectly relevant to biology that biologists didn't even believe, let alone, you know, they couldn't accept it.
Yeah, active disbelief.
So yes, that is a very powerful thing that the LLMs are apparently already capable of.
I will say, interestingly, among the experiments that you have Revealed publicly.
I saw one, I think just today, you put out yet another children's book, an alphabet book, where the LLM, so the LLM created text and presumably you fed some graphical AI a prompt and it created the pictures for an alphabet book in which everything is relevant to volcanoes.
Right.
So my son is super into volcanoes, right?
And how would I find an alphabetic book that's about volcanoes?
Like that's not the sort of thing that exists.
So we were just hanging out here this morning and I was like, Hey, do you want to do that?
And you know, he, He made choices, right?
He was like, there was multiple options.
Like, I like that one or whatever we could play along.
But yeah, so that's like, I think like making it a little bit, I mean, it was, it looked good to me and a lot of people said it looked great to them.
Actually, can I give you one bit of critique?
It's not a critique of you.
It's a critique of, of the, uh, the AI that you used.
Um, I'm somebody who A has, Written a bit of poetry, not much, but enough to really know what's difficult about it.
And getting the number of syllables and the way you would naturally read it to line up so that somebody who wasn't in on the creation of the thing reads it correctly is very difficult.
And your AI failed at it, right?
It came up with lines that had the correct rhyme, Right, right.
But they did not have the they did not have a parsable meter that would have, you know, an adult reading it to a child could have fudged enough to make it make sense.
But it did not just, you know, Let me tell you a little bit about that.
This is actually very important to understand about what the language models are doing.
For instance, I can make it write software for me.
Now, here's something counterintuitive you can do once you make it write software.
You can say, improve this.
And it will improve that.
And you're like, why does this work?
Like if you could do something better, why didn't you do it in the first place?
Yeah.
Right.
But the thing is that what it tries to do is not to do the best at X. It is literally just trying to satisfy your request on some, you know, regression to the mean kind of way.
Right.
Which means that it's up to you to constrain it enough to get something valuable out of it.
What happened this morning is I had an hour, right?
So I just like, I used the first thing it made for me.
I didn't, you know, whatever.
I've done situations where I've tried to make haikus, right, and I actually put it through a process like, make me multiple, and then I used it to judge the multiples against each other and filter them out and create variations.
Like, I even ran a somewhat evolutionary process of like, take the best ones and combine them, you know, take the best features from the best ones and combine them.
You can go a lot deeper and get a lot better.
One actually funny thing is that somebody has found out that the human text tends to have these two features called perplexity and burstingness.
Long story short, we don't write sentences of the same length.
We have small ones and long ones and we don't like to repeat ourselves.
The LLMs don't care about that.
So that's how AI text detectors figure out at the moment what is what.
However, you can tell it to do that and it will.
So it gets weird.
But long story short, yes, don't take that to be an example of the best that can be done with these things.
Take it as something that can be done when you're not double checking what you're doing.
Just kind of like running through it.
Yeah.
I mean, the whole thing was extremely impressive.
This was the one thing about it that I thought, "Ah, it's not quite there." And I'm not saying that it couldn't be there if you worked on it a little more and refined the prompt and constrained it further or simply somehow convinced it.
I am a person who is discerning about language.
Don't disappoint me.
Make it sound good, you know?
Um, so I don't know, I don't know how possible that is because I have not been, I've been deliberately avoiding messing with this.
Some of that are, some of that are, some of that stuff is actually, actually works.
Some, some of it is, um, you can tell it, uh, I don't, I haven't seen research on this, so I have a small question, but it does definitely seems to me to work.
Um, if you tell it you are a world-class poet and then do the thing, it actually make it better.
Do you think it would understand if I said, uh, I'm a poet and I'm not even aware of it?
Uh, what?
Sorry, what?
One more time?
Bad joke.
You're supposed to say, I'm a poet and I don't know it.
But if you say I'm not even aware of it, then it doesn't doesn't work.
And I don't know.
I find it funny.
But yeah.
All right.
Cool.
So there are a lot of good things that I see I can do today.
Right.
The question is, upon reflection, how does it all shake out?
And I'm not here to tell anybody that it's going to it's all going to be OK or that it's all going to not be OK.
I think my main message is to people to be less to be less certain.
And to not forget what we already know.
Okay, not forget what we already know is really important.
I would say one of the things, the more modern we become, as humans, the less often panic is useful.
Panic evolved for a reason.
There's no question about it.
There are circumstances in which it works.
But for a modern person, the number of places where panic gets you into more trouble rather than less trouble is huge, right?
The fraction.
And so to me, this is a very serious moment where...
We are in trouble.
As I say, I think there are five ways this could go disastrously wrong, and three of them are guaranteed.
So, I've just said three disasters are coming, I think, no matter what we do.
And I certainly understand that I could be wrong about that, but I'm happy to enumerate what the three disasters that are guaranteed, in my opinion, are.
They're not necessarily existential.
They could be amplified by something into existential crisis, but they're very serious challenges for humanity to deal with.
with.
But the thinking through this methodically while we are here, Taking the time to think it through carefully to figure out who has a model that is predictive and proceeding from there is clearly the right thing to do no matter what.
And avoiding panic is, I think, top priority.
The chances that we are going to panic ourselves into a catastrophe that we could otherwise avoid is too high.
It's unacceptably high.
One thing I was advising everybody during COVID, I wasn't saying it as loudly perhaps as I should have because I hadn't run it through, but now after COVID I can tell you it definitely would have been the right idea, is if you don't have principles in an emergency, you don't have principles.
Right.
What are principles, right?
Principles are things you have pre-decided to do when things are really, you know, up in the air, so you don't have to think, right?
You have committed yourself to certain, you know, arguably when When things are not emergencies, you might be able to say like, hey, I've thought it through.
Yes, I know I have this principle, but in this case, I can make an exception.
That's fair enough.
But especially in an emergency, when you know you're under duress, your mind is not working at the degree that it should.
That is the exact point where you go back to your principles.
That's not the point where you throw out your principles and you just do whatever.
And definitely that would have worked out through the pandemic.
And this is what I'm worried about now.
People are going to start Uh, saying things like, you know, we've got to shut down, you know, all of this, or we're going to constrain people from doing that.
Or we, you know, all the usual stuff of, of, of people in, in, in, uh, in panic mode.
Um, and I'm, I'm, I'm really worried about that.
And, and by the way, talking about some of the catastrophes, right, that are, are going to happen, right?
Like there's going to, is there going to be more fraud?
Yes.
There's going to be especially spiky.
Like there's going to be bouts before we adjust.
Is there going to be more sort of, you know, misinformation, disinformation, malinformation, even?
Whatever, right?
Sure.
Is there going to be, you know, all sorts of, you know, hacks, let's call it?
Like, sure.
Here's something counterintuitive about all that.
I think not only will it happen, but we should want it to.
We should want it to, because this is how complex systems adjust to information.
This is how we learn.
If there are small earthquakes, let's call them, or small forest fires that are taking out the kindling, We should want that, right?
What we should not want is for our civilization to continue exactly as it is, until one day something escapes from a government lab of some country or other, and it's got a cornucopia of opportunities everywhere, because nothing was there to prepare us.
And I'm pretty sure we're not going to do that.
I'm pretty sure the moment there's like a few of those examples to be touted, people are going to panic and we're going to start doing really, really counterproductive things.
So if I can drag us to a COVID analogy, right?
We did a tremendous amount of harm to ourselves during COVID, attempting to control the spread of the virus with measures that never had that capacity in principle.
And we told ourselves lies about why we were doing it, you know, two weeks to flatten the curve.
It wasn't that we thought we were going to control the virus.
What we were going to do is preserve our hospital so that they would have enough capacity for this huge flood of sick people that was supposedly coming their way.
Right?
Slow the spread, as if that means anything.
Right.
And so that style of thinking actually, really what we faced, we can now say, at the point at which this was no longer controllable, even in principle.
Right?
And I'm, you know, I have had a debate with many people about whether or not there was a point at which we could have woken up to this and driven it extinct.
I think that possibility existed.
Many people who are very well informed disagree with me.
And I honestly can't say who's right.
I know why I think what I think, but let's put it aside for the second.
There's a point after which it was possible to drive it extinct.
And the question is, at the point that the thing is circulating, You might very well be able to deploy measures that adjust when people get sick, but effectively the point is people are going to get the thing, they're going to develop immunity which will be positive, some people will be lost along the way, but it's hubris to imagine that you have more control over it than you actually do.
And so one thing I hear you saying, tell me if I've got it wrong, is that
There is something to be said some of these things that are coming that are negative are actually not net negative especially if they come maybe if we're going to face a wave of abuses of this technology facing them while the technology is still cruder might cause us to understand what it is that we need to build in terms of countermeasures and so nobody
Me especially does not want to invite these disasters, but the recognition that the disasters are going to come, that the degree to which they are disastrous is still up in the air, and that there is some sort of counterintuitive question about when you prefer to face this and what you prefer the state of technology to be when you do.
So I don't know how to answer that, but I do think, you know, you've enumerated one of them pretty well.
We've got what malevolent people will do with this technology that will be harmful to the rest of us, right?
One we haven't touched on here, or maybe we've barely touched on it, is that I believe we are about to have a sanity crisis.
And we were already pretty fragile on this front.
COVID revealed the craziness in people, and it revealed their willingness to go along with preposterous notions at a level that I had not seen in my lifetime for sure.
But I believe that the Various versions of, you know, well, are you using reverse psychology on me?
Or maybe it's reverse, reverse, reverse psychology.
You know, that in a world of A.I.s that are informing people that you are interacting with in different ways, the ability to know what's what and which way is up plummets.
I'm terrified of the ability of these AIs, they're going to generate, they're going to generate what in a very recently past era would have counted as evidence for things.
Right?
Deepfakes are going to count as evidence, and then as people catch on that evidence is no longer evidence, nothing is going to count as evidence.
And I don't know how a world like that functions, right?
So, I think that one is actually, I'm relatively less concerned about that one, and I'll tell you why.
Photoshop is a thing.
Yep.
Right?
We've learned, we know not to trust photographs.
Right?
We just, we just do.
Um, and you know, there's some experts that apparently can tell, but I mean, I don't know.
I can't, I don't know if it's dowsing or if it's actual expertise.
Um, but the point is that's not really a problem for us.
Why?
Right?
Because we use provenance.
We don't just use, Oh, a photo exists.
Uh, we use the idea that, uh, where did it come from?
To be important, right?
How did you get this, right?
Like, what's the story?
If you got a photo of Elon from Elon's feed, right, then that's Elon or Elon wants you to think that it's Elon.
At least, you know that, that much.
You know, the whole politicians being like, I am blah, blah, blah, and I approve this message, right?
That sort of thing is implicit in Twitter.
It could be made more explicit by, you know, blockchain technology, where, you know, certain things are in certain positions, like a certain event happened at a certain time.
And there's no mathematical way to, like, manipulate that record that we know of.
But this, I think, ultimately, that's what it comes down to.
It's kind of know where your information came from.
But we already know not to believe anything we see on the media, right?
People can make fake text forever, right?
We've had fake photos.
Okay, now we're going to have fake sound and fake video.
Right, but again, I think you're making my point, which is there are going to be people who are suckers.
There are going to be people who leverage this technology and figure out how to create the impression that something has the signatures of a actual fact.
And the savvy people are going to stop believing anything, which I believe is like them cutting off their nose to spite their face.
So by savvy you kind of mean the midwit savvy, not savvy savvy?
No, actually, I mean something way worse, which is I think the best thing you can do in an environment in which very high quality AIs are producing phony reality is to become cynical about the idea of evidence.
Right.
In fact, I would say picks and it didn't happen.
Right?
Right.
Here's the... No, but hold on.
The point is, in a world where you become cynical about the ability of anything, including your own eyes, to tell you what the hell is going on, then the point is you're paralyzed by your correct... The game theory paralyzes you.
And I think what comes out of that is...
For many different reasons.
It's going to be a kind of insanity.
And it's going to be very pernicious and broadly distributed.
And you can take the argument that you made for the last thing and you can apply it here too.
And you can say, better we face that early with more primitive tech.
And that's probably true.
But it's still, I think, a dire crisis that I cannot imagine us avoiding at this point.
No, I've already baked in a lot of crises ahead of us, right?
Before we get to where we get to, even if we get to a good place, what did I used to call it?
I used to call this zebra pill.
It's a black pill.
Even if there's a white pill at the end of it, there's a black pill in the middle.
for people who are not into pill discourse, don't worry about it.
I really do think that, yeah, we're gonna have to sort of, these things are gonna happen.
And the thing to do is not to panic, but to build defenses.
And both personal and collective defenses, right?
Build networks with people you trust, be more entangled with the world so that you notice these sorts of things.
For instance, one thing I did during the pandemic, by the way, and this kind of brings us all the way back to the beginning with some conflicts I've had, is, you know, whenever I spotted an error, I would actually tell people.
Hey, you made this error in your blog post, right?
And that was a credible sign to me.
If that person just went and fixed it, that was like, okay, so not only have I, this person is clearly humble enough to do that, but now I can sort of trust that they will probably do the same if somebody else
Tells them so I can trust their output a lot more because I've now tested it in fact with IBM meta one of the websites that you know is I I went so far as to send them a I don't think they know this I suppose if they see this they will I have I don't know those people right I communicate with them through a we don't know who they are yeah
Um, but, um, I wrote a, I wrote a correction in a slightly, you know, very different tone than I usually used to use with not without my name and pointing out a mistake that was, you know, against what appeared to be their, their appears to be their preferred sort of direction.
Right.
And wouldn't you know it, I went back a couple of days later and it was fixed.
So now I know that these people will even take adversarial comments that look formal and somewhat vaguely threatening or whatever, and they'll be like, I don't care.
It's good information, man.
Sure, I'm going to fix it.
So there's things you can do that are not commonly done where you can take a little bit of understanding.
Okay, you saw a little error.
Okay, good.
You can bootstrap that into, oh, now I know I can trust these people.
At least more.
I can upgrade them.
Right?
There's ways to get out of, you know, very noisy situations that are probably known but not evenly distributed.
And again, that's the work we have to do, is clean up the commons.
And here's something else, right?
These things, right, the large-scale fakery of entire scientific fields, right, are not new.
As, you know, you and I agree, I think.
They're going to be democratized.
But it's not like they're new.
Right?
People can do, like, most of the things or even all the things that an LLM can do today, you know, GPT-4 or even 4.5 or whatever, are things people could do.
It's just that it doesn't make, you know, it's destabilizing because it allows you to put a lot more What used to be effort, right?
Now, no longer effort.
A lot more optimization power towards things that look relatively trivial.
Like, I can make an extremely well-illustrated alphabet book for my son that is about volcanoes, and I can do it in an hour in the morning, right?
So that's what's stabilizing.
But there's nothing that wasn't in principle possible At least for the moment.
And much of that are things that the powers that be have been doing already, right?
So at least now, we are saved from the concept of even alleging that this is happening being somewhat discredited, right?
Because it'll be easy to demonstrate.
How one does that.
Well, so I've alluded to it.
I think I should lay it out.
I'm going to put my five possible catastrophes on the table.
And then I have two things that I want us to come back to.
Actually, three things that I want us to come back to before we close this out.
So just so that it's here in the podcast, my five AI driven catastrophes, three of which are guaranteed.
Malevolent AI, where it decides that it is our competitor and to remove us or enslave us or whatever.
I don't think that's guaranteed, but I think it's possible.
Misaligned AI, and nobody has yet told me whether my first category is also misaligned AI.
I suspect it probably is.
But misaligned AI is where an AI isn't out to get you, but it interprets something You know, the paperclip maximizer example is the one that circulates where you tell an AI to make as many paperclips as possible, and it starts liquidating the universe to do it, because that's what you make paperclips out of.
So it's not malevolent, but it's definitely an existential threat.
I also don't think that one's guaranteed, but possible and definitely worth considering and being concerned about.
And then the three that are guaranteed are malevolent use of AI against decent folk.
I think that is absolutely certain and no doubt already happening.
The second one that is guaranteed, the fourth one on this list, is insanity that will come from having something that has access to the human API.
That is to say, AIs that speak English and whatever language you happen to speak, and speak as if they have emotions and all of those things, and it will cause us to go mad collectively, and many of us individually.
I think that's guaranteed.
Two, and then the last of these, the most mundane, is the destabilization you talk of.
You know, just the simple fact that a huge number of people are employed in professions that can now be replaced by an AI and that we have no backup plan for how we are to run civilization when most people have nothing productive to contribute.
We've already had a hint of that, but we don't have it at the scale that we're about to see it.
So anyway, those are my five malevolent, misaligned, abused, insanity, and economic disruption.
And so anyway, I do think we're in a very serious situation.
You have persuaded me, Alex, that there is another way to view So you have embraced this technology.
You've been playing with it.
You've done this with your eyes open, acknowledging that there are very serious dangers here.
But you've been trying to compel me about all of the useful things that I should understand that it is capable of.
And I've challenged you on, OK, that's all well and good.
I believe the useful stuff exists.
But the net effect here is so frightening.
Why are you not more concerned about it?
Do you want to lay out your position?
You want me to remind you what you said?
What do you think?
I mean, you'll tell me what answer I gave you last time around, but I'll say a couple of things.
I think I'm doing it because I'm concerned, right?
I think the capability, if we get out of this somehow, the answer is still in the thing, right?
It's going to be useful to get us out of it in the same way that it gets us in more trouble.
I don't foresee our prior state coping well with that, but then again, I didn't foresee our prior state coping well with itself.
And again, I think we share a lot of priors there, which is that our civilization, and I speak like a technologist, I know you get there in perhaps slightly different ways.
But for me, if you know how technology is built or how organizations are built, which is another thing I've spent many years thinking about, Um, what you know whenever you go into the sausage factory, right, is is how many things are built with You know, duct tape, chewing gum, whatever, on top of duct tape and chewing gum on top of duct tape, like nothing that it's almost like you see this thing is like, there is no way this works.
And yet somehow it kind of sort of does, but badly, right?
And everybody's suffering for it, but it kind of sort of keeps up and keeps going.
And we add another layer on top, and it gets maybe a little bit better for some time until that gets worse too.
And you see the You can see that anywhere from like legislation to the kernel of the Linux operating system that runs, you know, basically the world or whatever.
All of these things are, you know, these path dependent piles of junk on top of junk that kind of sort of we keep it going, you know, barely, right?
Like, because if it stops working, then somebody has to dive in and fix it.
And we just keep going.
But collectively, all that stuff is a Rubik's Cube, right?
A Rube Goldberg machine is what it is.
I completed the wrong tokens, Brett.
Can you believe it?
I don't know.
Do better is what I would tell the LLM.
You're not supposed to hallucinate, right?
Anyway, it's a Rube Goldberg machine, like our supply chain, for instance, right?
That some butterfly goes the wrong way in Beijing and then We just, you know, oh, well, I guess no.
What was it like baby formula for you?
It's like, oh, yeah, no, that was totally foreseeable.
Like it was the baby formula that was the most vulnerable.
Right.
And after the fact, everybody's got a story right about how the FDA's regulations, blah, blah, blah.
You didn't tell me that beforehand.
Please spare me, right?
The whole point is that these things are extremely fragile and barely working, and we keep treating them like they're not.
We keep pushing them harder and harder, print more money, do more wars, whatever it is that we're doing that has systemic risk, right?
So I was honestly kind of resigned to the fact that things are, you know, Here's a problem I have as well, actually.
No, no, no.
You haven't gotten to the punchline of that one.
I think people will be confused by what you've just said.
You have a incredible stacking of fragile components that make these systems that work, which means they are
I've already used the word fragile, but the problem is you have all of this path dependency, what in biology might be called historical contingency, that results in mechanisms that function but that assume that everything about the context is going to continue to be what it is.
And you perturb the context at all and it reveals all of this fragility and you get cascading failures, right?
What your point, I know because we've talked about this, is that LLMs and the AGIs that will come out of them are capable of telling you how to reconstruct the system and remove all of that path dependency so that your system is robust.
And the point is that makes us safer in the end.
Yes.
Now, I would argue that this cuts both ways, and it cuts both ways because you're right about the historical contingency or path dependency and the huge danger that comes out of it.
But the problem is, we are in that very spacecraft you have described with the duct tape and the bailing wire.
And we are about to experience a massive set of disruptions downstream of the AI that are going to reveal every one of those fragile links.
So as much as the thing is in principle capable of telling us how to revise the system so that those things are no longer a problem, the first thing it's going to do is it's going to trigger the cascading failures.
We're going to be pushed in a situation where things are going to be failing and we're going to be fixing them.
And we sure hope that we get better at fixing them than failing.
At our normal pace, there is zero way we could do that.
Thankfully, we have LLMs.
So, you know, it is going to be chaotic.
It is going to be There's going to be adjustments, you know, what is it, discovery of all sorts of minima and maxima, but we previously thought we knew and all of a sudden they're all, yeah.
The only thing I'm saying is that there are end games, or at least, well, mid-games, visible where we have drastically simplified the stack below our conscious level of everyday civilization, and improved it a lot.
And the thing with, by the way, with path dependency and context dependency, is not just that the things that we have are made for today, a lot of the things are not made for today.
Right.
They were made for, you know, there's COBOL running in for the banking system is run by COBOL.
Like, is that because that's the best programming language to use today?
Hell no.
Right.
It's because somebody from IBM sold them a mainframe back in the eighties and that's what they had.
And if it ain't broke, don't fix it.
And the year is 2023.
Right.
Great example here, an LLM is fantastically capable of transferring things between languages.
You can say like, can you take this code that nobody wants to touch and the guy that wrote it is dead, right?
And we barely know how to keep it running and just transform it into a new language.
And it's going to be like, Sure, boss.
Here you go.
Like it, it is phenomenally good at that, right?
Yeah, I hadn't thought about that, but it's super cool because it can even do it.
You tell me if I'm wrong, but it can even do it from machine code.
Right, from the compiled code it could work backwards.
Reverse engineering.
Basically, you might not even have the source code.
You might just have the binaries.
Right, which would be an absolute nightmare for human programmers, right?
But again, right, it cuts both ways because this also defeats certain fragile but maybe barely working security measures that people are using to not get their code discovered, right?
They will ship a binary Hey, now I can take your binary and turn it into source code again, right?
What happens in that world?
Basically, the way I describe it is that the friction weights have been radically redistributed, and people don't fully understand that yet.
Things that used to be hard are easy, and things that used to be easy are hard.
So it used to be easy to not send people your source code, you just send them your binaries, right?
And that gave you security for your software company.
Well, that's gone.
You know, there are certain things that you could achieve in certain ways that are adversarial to others, basically, that are no longer possible, right?
So the disruption is... I mean, if anything at all is guaranteed, I know I say people should be less certain, but the disruption, I think, is kind of baked in at this point.
Yeah.
The answer you gave me... I'm not the bringer of happy, happy, joy, joy, if anything.
No, I appreciate that about you.
You've never been there.
You've never said anything about this not being very, very dangerous, which left me a little bit perplexed about your enthusiasm for the good parts of it until what you said to me when we talked about this last time.
was that you considered the trajectory that we were on already so dangerous that the idea that something has just emerged that could in principle address the fundamental fragility of the system was a net good thing, right?
That basically we are headed somewhere dire and the tools with which to fix the fundamental problems have now emerged.
They've emerged In a context that isn't safe and carries many dangers of its own, but nonetheless the capacity to fix the dangers that we were stuck with already is substantial and really couldn't be more important, which I thought was an excellent and novel argument as far as I know.
And I mean, to put it in Game B terms, hey, we're here, why not?
Right, why not?
You know, it gives the ability for novel coordination mechanisms to emerge that we just could not have pulled off, right?
The human firmware, as I'm sure you know, given the title of the book you wrote, Can coordinate, I don't know, a few hundred souls the best?
You know, how much, you know, a lot of what we're witnessing now is basically the evidence of that.
It's failure to operate at, you know, billion scale.
And really the bold case, to use a financial term here, is that, hey, wouldn't it be amazing if something could create an adjusting abstraction layer, so to speak, that could actually scale our native firmwares, make things feel natural while we are that could actually scale our native firmwares, make things feel natural while we are You know, make our reputation scale, for instance, right?
Reputation used to mean something.
You know, people didn't lie for a reason.
And it turns out that's the strategy of lying now.
People do it less honestly than they probably could because we're still going on momentum.
But it kind of works, right?
If you know what you're doing and if you've done it enough, you practice enough, you can do it well.
But all of that used to be kept in check.
And my sort of The bold case for the people who like complex systems is not that we will make novel systems.
I mean, we'll do some of that.
But maybe we can kick some of the old systems back into functioning mode.
Yeah, no, I like this idea too.
There are many problems in game theory that are actually trivially simple to solve on paper and devilishly difficult to solve in practice.
And this actually, even, you know, forget a billion people.
Even just solving a problem inside of an organization so that people's interests align can be very difficult because of game theory that upends you.
I can see obvious ways that even just today you could have this thing build a structure that would solve some of those issues and that could be a very positive thing.
So anyway, that's good stuff.
Alright, the two other things that I wanted to get to before we close this out Or I'm wondering as we're talking, thinking specifically about the question of deepfakes and pictures and not trusting any of that stuff, I'm wondering if film and possibly other analog technologies are about to make a comeback, because they might actually provide an evidentiary value that
In other words, the chaotic distribution of silver grains in an emulsion on film is a much easier thing to establish as organic than a set of pixels that could have been individually placed by an AI about an event that never took place, right?
How do you get the silver grains on that film to lie?
It's a much harder problem.
And therefore, if the film says something, it's much more likely to actually represent something that took place in front of a lens.
So, I'm not sure about that one, but I think it speaks to something that will almost certainly have to happen, which is costly signals are about to come back.
Yep.
Right?
So, you know, I think we've been spoiled by modernity being able to do seemingly impossible things like, you know, get a message from A to B immediately and whatever, right?
Which, you know, is possible because we got antennas, but then, you know, again, those Systems were good while they lasted and now they're compromised, right?
And not even everybody knows that they are, but whatever.
But at the end of the day, if you think about it a little bit more deeply, you just realize that it was a cheat, right?
It's a cheat in the same way that, you know, using oil is cheat.
I'm not talking about the environmental element.
I'm talking about like, well, it just so happened that there was all this energy buried in the earth.
Yeah, you've got all this pent up chemical energy from ancient photosynthetic Right, right.
Reactions, right.
Right.
So we've got, you know, we're eating from, you know, the ready.
Sorry, this is probably, I'm probably translating from Greek.
But like, you know, we are selling the silverware of various sort of domains.
And those were gonna run out.
Now they're gonna run out a lot faster.
But ultimately, I think what comes out of the other side is some kind of, you know, a lot more fundamentally stable things will be needed.
I hope we find them.
And again, if I'm maintaining optimism, I'm not maintaining optimism because I'm sure we're going to make it or anything.
But I think you've got to be able to see a game where you win, even if you think it's unlikely.
Elon famously thought that SpaceX had a 10% chance of success.
Now, I guarantee you he wasn't moping around every day that he was running SpaceX.
He was like, let's make that 10, 11, you know, let's keep, what is it?
Success, outcome.
It's a famous Elonism, right?
Let's keep success as a possible outcome here.
What do we have to do?
Right.
The question for me is not like how, how, how doomed are we?
Um, though, again, I recognize that there are definitely some really dangerous scenarios from, from, from prosaic to, um, pretty, you know, imaginative, uh, The question is, what do we have to do to get to the other side of this?
And let's do it.
I mean, what else are we going to do?
We've got children, we've got family, we've got cultures that we've got to save.
So, I mean, I agree with that.
Motivation.
I would also add one thing from evolutionary biology space, which will be well familiar to you.
The fact is civilization, you know, we can describe it as a collection of systems that are linked together with, uh, duct tape and bailing wire and other indications of path dependency.
You'd also say it's a low peak, right?
We're achieving a lot, but we know that with the same amount of input we could be achieving a great deal more if we weren't burdened.
And so what I would say is one of the insights that is most important from evolutionary theory is that the moving from the low peak to a higher peak generally involves the transition through a valley.
And I know it has been useful in my own navigation.
That means the valley tends to be a place of uncertainty, of great danger, and a lot of confusion, because it's not the two-dimensional version of this model that is often put on a chalkboard.
It is a three-dimensional version in which the peaks are separated sometimes by great distances.
The landscape is shrouded in fog and you don't therefore know what trajectory to take out of the valley in order to find the peak.
And if you take the wrong one, you can die without ever going back up.
All right, that doesn't sound very hopeful.
But my point, the hopeful part, is don't panic just because you're staring at a valley.
Staring at a valley may be the only way to get to the higher peak that you need to get to in order to persist through the next chapter.
And so I guess what I would say is if we're going to improve civilization, it's no surprise to me that we are facing tremendous peril.
And if the answer was that peril was required to get somewhere new, then so be it.
Let's face the peril and let us do it without panicking because that's the best hope we've got.
Yep, yep.
I believe the embedding space of GPT-4, I hope I don't mess this up, is 1,536 dimensions.
So every concept is mapped to, you know, when Sam Altman tweets, gradient descent can do it, which I guess in your world is gradient ascent, you know, it's inverted.
But it is that, he's talking about, you know, moving in these, you know, thousand plus dimensional spaces.
I think in those spaces my intuition is weak, so I might say something silly.
But you get both, right?
You get peaks and valleys.
Some dimensions are receding while others are going up.
That's the way it is in biology.
It's not the way we write it on the chalkboard, because you can't.
But the fact is, that's why trade-offs are the important thing that they are.
Every creature is balancing thousands of different considerations simultaneously.
And so, anyway, the model feels oversimplified.
But nonetheless, it goes to the same place, right?
It is not surprising that we would face peril on the way Towards possibly something better, even if we are certainly not guaranteed to find that next foothill and start climbing.
Right.
Okay, the last thing is you had a model that you were beginning to tell me about before the podcast.
12 things that you think that we should be aware of.
I don't want to steal your thunder, so you want to lay out your framework.
Yeah, so like we said, let's leave this on a slightly positive note.
I think that is the right note.
It's not wildly positive, but it is, you know, we're not, just by saying we can't create a world agency that will control AI and hope that that will get us out alive.
That doesn't mean there's not things we can do.
So I've put this out on Twitter.
I think I'm going to expand it a little bit and put it on my sub stack.
But I put together 12 ideas and they're nowhere near a complete set or an optimal set.
I just wanted to put some thoughts down just to have that out there.
About what we can do today, like practical things that can be done to increase safety from this coming disruption.
But the critical constraint I put on myself is without creating new single points of failure.
That is the requirement I set myself.
I said, okay, what can we do without creating the one ring?
So for instance, one of them was the idea of red teaming.
I don't know if that is- - Oh, I use this concept all the time.
Yeah. - Right. - Just this is a white hat.
Fund people to go out and try to find all the ways in which this thing can be bad.
OpenAI, to their credit, has done a bunch of that.
Hey, we should be doing a ton more every day, all the time, in all directions.
ton more, every day, all the time, in all directions.
I don't know what too much of it looks like.
I know we're not .
And the great thing about that is that it's purely information accretive.
It's like, here's a thing you didn't know that could go wrong.
Great.
Okay, well, now I know that.
It can't You know, it can't hurt, like, I mean, I could panic or whatever, but like, assuming that's already a problem, you know, so another thing is a, and I say this very carefully, like early model DARPA, right, not today's DARPA, like the early DARPA, where they would fund People with few strings attached, let a thousand flowers bloom.
I think this kind of approach towards AI safety would help.
Let's get a lot of crazy ideas and know that out of every thousand, 999 will fail.
Let's let people try things.
If it's $3 million proposals, a thousand of them, that's $3 billion.
What's the cost of the 999 failing to the benefit of one succeeding?
For $3 billion, which for government scale is chicken feed.
Apparently, it's what the Pentagon finds in revaluing the arms it sent Ukraine.
Discovers $3 billion.
The third one, and this one gets complicated, and we don't really have to do the whole list, but I'll just mention a few.
By my count, and again, I know that you have your concerns, which I share, but I still think it's probably worth trying.
Try to see what integrated cognition looks like, where humans operate as the agentic element, right?
Humans are the willpower, right?
And the AI is sort of the amplifier, let's call it.
Think of it as the Iron Man suit, right?
The Iron Man suit is under the direct control of Tony Stark, but he normally can't fly.
That's why I use the concept of an amplifier, like an LLM as it is today.
Exceedingly lucky.
This was not predicted.
It's a passive thing that can make you better.
And yes, it could turn differently over the future.
Hey, can we pause for a minute and consider that this thing has this really weird property today of operating as an intelligence amplifier?
So I hope I don't, I don't know where this goes, but you say it was not predicted.
Okay.
I do feel that in the essay that I wrote in 2016, which I believe you have read, Which is not public yet.
I'm going to make it public, as I promised to do, and there have been some delays about the practical side of that.
But nonetheless, I did, if I remember it correctly, predict that AGI was going to emerge from a program that was not exactly like this, but was very close to it.
And so I don't know if that counts as predicting.
I think what you're saying when you say it wasn't predicted, it wasn't predicted that an AGI would be able to interact with regular folks who could prompt it in their own language.
As we kind of spoke offline, I think the model you presented, it opens up a bouquet of possibilities of how to do it, and I think what happened is actually one of those.
It fits that pattern.
But the interesting property I'm talking about is the fact that it's kind of sitting there.
It's basically a glorified lookup table.
Let me see if I'm getting at the property you're talking about correctly.
Like it's just kind of, it doesn't even, it's not even really trying to help you.
Right.
I thought about this the other day.
It's not even saying, Hey, it looks like you're struggling to do this.
Would you like me to help you?
No, it's just like, here's an answer to your question.
That's that.
Let me see if, uh, if, if I'm getting at the property you're talking about correctly, it's almost like an ex static explorer of the collective mind.
Where you can now walk into the collective mind and you can say, well, maybe here's a thought that we've never brought to consciousness before, but surely if we did, some things would What would we say if we suddenly became conscious of this question?
And you can walk over to the quadrant in this uh, static rendering of our mind, which exists at some level of resolution and no greater.
But anyway, it's good enough that you can walk over to some fraction of the collective mind and see what it would think about something and get back an answer that is interpretable.
The, the, um, the mission statement of Google, um, I'm sure you know it by heart, but I might be one of the people that does is to organize the world's information and make it universally accessible and useful.
Right.
Um, this is, you know, large language models, are far closer to that sort of glorified search engine, uh, vibe, right.
It's an idealized search engine of sorts, than they are to, you know, you and me, like people, right?
That's kind of what I'm trying to get at.
It's like, it's going to give you information.
Yeah, as you say, it can reach into all the little threads of thought that we've put out there over the years, right?
It's kind of absorbed all of them and turned them into a rope that it can access and can reach into it in weird ways that we probably couldn't.
It is genuinely strange, and I guess what I'm saying is that while we're worried about the agency element that will be strapped on top of it at some point, let's realize that the agency that it's strapped onto could be human.
Right, like there's, the way Elon says it about Neuralink, and I know there's a lot of controversy in the neuroscience space and whatever, is that, you know, you got the, I don't know if it's the basal ganglia or the lizard brain or whatever you want to call it, the lower, the more evolutionarily ancient levels of the brain,
Being somewhat in control of the newer and more capable ones, the question is, can we turn these technologies into a third layer that continues that pattern?
To some degree, this is what we're doing.
When I play guitar, I don't think about the guitar.
I think about the sound I want to make.
At some point, it's just absorbed into my body plan.
Um, and when we use, you know, sometimes now I'll have a question and I won't like try to remember the answer.
I'll just plug it into ChatGPT because I, you know, that's kind of absorbed into my body plan too, apparently.
So the question is, can we do more of that and not because we, you know, for the fun of it, though, you know, I'm sure it's going to be some fun that comes of it, but for the purpose that if AGI is going to come, I think it's in our sort of interest to get there first.
The first thing is human-based rather than other-based.
Right.
And that essay of mine that you read, I think, suggests a mechanism for doing this.
Which, anyway, when I deploy that essay, I'm going to tack on a modern addendum and talk about what I think the path forward is for AI safety.
There's a bunch of these and we don't have to go through the whole thing but if people want to find it on Twitter I have a tweet called my suggestions for reducing AI risk without creating novel single points of failure.
And there's a bunch of ideas in there.
Perfect.
All right.
Alex, I want to start.
This is unkind of me, maybe, but you and I are friends and I hope you will trust me to do it.
I want to start by taking you to task for something.
And I'm doing this for what I at least imagine is the greater good.
Here's what I see.
Lots of people are becoming aware of you.
I know More than a couple of people who dislike you intensely and see things in you that I know aren't there.
Because you and I are friends.
You and I know each other in person.
I know your marvelous family.
There's no truth to what people imagine must be going on with you.
And I think, as your friend, I have a little insight into what they have misunderstood.
But in order to unpack this so that maybe people will, or those people who don't like you, will come around to a more nuanced perspective, I do think that I need to make a point to you, too, about why it is that you're so easily misinterpreted.
So, you game?
That's it.
All right.
What I see, and you know, you and I met during COVID.
You, in fact, were brought into a project to evaluate whether the accusations that were being leveled at Heather and me were accurate, that we were uninformed, spreading misinformation about COVID.
And you ran a project called Better Skeptics, which evaluated the claims that we had made errors.
After this project, you and I came to know each other.
The project itself concluded that there were a few small places where we had misstepped, but that in general we had been very good about being accurate, and where we had been inaccurate, correcting the record of our own volition, and not needing to be taken to task.
So in any case, thank you for doing that, actually.
It meant a great deal to us, even if the world didn't hear much of what you came up with.
If it's a good use of time.
I do think back to the report we wrote right after that project, and I think it says a lot, and I actually might even touch on the AI question we'll get to later.
It proved to me that it is possible by just sitting down and taking thorny questions and varied inputs and going into the sources to figure stuff out that, you know, is completely counter to the narrative and becomes stories later.
Right.
Like, so for instance, when, um, the, you know, recently somebody, one of the Pfizer executives said at the European commission, we didn't test it for transmission and lots of people, even Elon, I think somewhere was like, why didn't, why wasn't this made clear?
It was, it was sorry, but like, forget what the public health establishment said.
If you look at the actual releases from the pharmaceuticals, they said what they were looking at.
What they allowed to be projected and what they did is a different story.
But if you actually looked at the sources, same thing with the animal testing.
Was it true that there was absolutely no safety testing?
No.
Were there significant chunks skipped?
Yes.
And that was knowable.
You could get down to the facts, go to the source document and confirm that at the time.
That document to me is a great Uh, testament to the fact that you, you know, you can sit down, you can take a very scientific or, you know, uh, journalistic or all the, you know, all the things that have broken these days approach and actually come to conclusions that then you just sit and watch everybody like slowly, you know, converge to it.
It's, it's, it's wild, but like we did it.
So, you know, it can't must be possible because it was done.
I actually think that that is one of the biggest lessons of the entire COVID so-called pandemic.
is that these topics were difficult, but they were made far more difficult than they needed to be.
In fact, you needed a basic willingness to rationally evaluate and to exert some kind of standard over
what information you Thought was credible enough to process and then it was possible to predict what was going to unfold predict what people were going to come to understand because you know some of us have trained to Understand complex things and this was like every other complex thing.
It just came with a lot more deception than you know your average piece of biology, let's say so I saw you as a force for good.
You were unambiguously a force for good in the COVID era.
I guess it is no surprise that you would have accumulated a great many detractors in that period because anybody who was a force for good did.
What troubles me is that some of your detractors from that period are not the bad people, as I see it.
They're not the liars and the shills and the, you know, the phonies.
They're people that themselves bring something positive to the table, and yet you and they are like oil and water.
And I wanted to say why I think that happens and see if, A, I can get them to rethink, And B, get you maybe to alter your mode of interaction a little bit.
So, what I see is this, and you know, of course I know the backstory pretty well because you and I have become friends.
You are, maybe this is not the right way to put it, but a recovering rationalist.
I think, or you know, you are a rationalist and those who are called rationalists are something else, it turns out.
And I think You feel an obligation, because it's really part of the rationalist code, to hold other rationalists who are not living up to that code to the standard, right?
And to hold their feet to the fire.
And so there's a certain kind of Delight I see in you as you set the record straight on people who are rationalizing about matters of life and death, who are demonizing the wrong folks because, well, for whatever reason.
So, in general, I would say the rationalists did an appalling job during COVID.
I mean, just, it was so disappointing to see people whose credo effectively commits them to doing dispassionate analysis, you know, saying what needs to be said, analyzing the data and following the evidence wherever it leads.
If that's the flag you've planted in the ground, and then you don't do it, It's worse than having done nothing at all, because you've effectively advertised that if there was something up that just was way off and easily detected, you'd be the people to spot it.
And if you don't spot it, or maybe you do spot it and you rationalize not talking about it because it's frightening to you, then you're doing harm, right?
It's a case in which the absence of rationalists sounding the alarm over the absolute nonsense that was being broadcast over COVID suggests that that nonsense was higher quality than it was, because surely the rationalists would have spotted it if it wasn't.
Oh, it's worse than that.
They, by my count, they became the stormtroopers for the establishment, right?
It was, you know, even without them, like, saying something, there was this group that, you know, was advertising themselves as, you know, who you're going to call Ghostbusters.
But when they came, they brought more ghosts, you know, and there's, you know, that group has, you know, and people will say, we're not a group, yada yada.
Okay, I understand.
There is a group, everybody knows what I'm talking about.
Okay, and if you advertise yourself by some moniker, right, and you draw status by that moniker, you draw attention by that moniker, you draw respect by that moniker, then, you know, yeah, I see it as some kind of contract that you have signed with, you know, the public and that not, and you know, there's a lot of people, right, have made monetary contributions, they have made time contributions, they've given their lives, you know,
They've certainly substantially altered the course of their lives for that movement.
And, you know, having come to find that, you know, much like anything that, you know, ends with "-ist," I guess, is bound to sort of reverse itself, you know, for some weird pattern.
But having seen that and having confirmed it for myself to rationalist standards, right, beyond any reasonable doubt that, unfortunately, there's some weird inversion going on.
Yeah, no, some people might see my interactions and being like, hey, that that was on cold floor, but it is in the middle of a longer story.
I suppose that you kind of have to.
Yeah, you weren't there, man.
You wouldn't you wouldn't understand or something like that.
Right.
Well, look, I get this.
And, you know, as somebody whose reputation was.
Put completely on the line over covid.
And you know, I don't think there was any way to avoid that.
I think as a biologist who decided not to be silent about this, both Heather and I were in a situation where we damn well better do the work correctly or we're in big trouble.
So to do the work correctly and then have the rationalists covering their asses and rationalizing themselves into nonsense and actually being on the attack, functioning as stormtroopers, Against us was it was beyond disappointing and I certainly understand, you know, I was never a rationalist I always felt like a fellow traveler of the rationalists but As a rationalist yourself
I can imagine exactly why you would feel an absolute obligation to hold other people to the standards that you collectively signed up for.
That makes total sense to me.
And I won't even take you to task for that, because, you know, in some sense that was you holding your own community to task, and I thought you did it honorably and well.
The place that I think you got into some trouble was that I think because it was the same battle, that some other folks who weren't rationalists got the same type of response.
And believe me, I don't think you should have held back holding people to account.
But I think people who did not advertise themselves as rationalists probably should have been dealt with a little differently because in some sense, they didn't say, hey, we're the people who are right for a living.
They were just trying to navigate.
Right.
And so anyway, what I would ask is that the The delight that you, I think, evidenced in setting your own community straight was a little off when it came to people who had not broadcast their own extraordinary capability of finding the needle in the haystack of rationality, right?
So anyway, does that make sense?
I mean, I need to know the specifics.
There is, you know, one sort of common acquaintance of ours, a famous podcast host that I've come to loggerheads over.
I don't know if that's the person you're thinking about.
It would come down to specifics.
Look, I'll say a couple of things, right?
So how do I use Twitter?
Because that's, let's face it, that's where all the things happen, right?
There's many ways to do it and my approach is to just be me and not sort of try to overly sort of strategize.
Whenever I, you know, there's a local minimum of like, just say what you think, right?
And then from there, there are like arbitrary complexity you can add on top of it.
Well, you know, whatever.
I try to add as little of that as I possibly can because I don't fundamentally believe in strategy in such a, you know, complicated environment, I think, you know, people rather, let me put it this way, people tend to over strategize than under strategize.
So I just try to be me.
And I guess I've got a particular benefit here, because I'm not, it's not my my living, at least not not not for the moment to do that.
So I am basically, you know, one of the one of the crowd and I, you know, that's kind of what I what I do.
The one thing I will say, and I wonder if people know that, it's a feature in my Twitter presence that might not be something that people are aware of, is that you can always ask me, like, what do you mean by that?
Just double-click on stuff.
If you don't understand something, don't assume that I'll bite your face off if you ask me.
I genuinely get into very deep thread conversations with people.
Because they just ask me something and I try to answer.
I mean, will I see everything?
Probably not.
I try very hard to see everything, which is also why I block people, by the way, right?
If somebody replies to something, because I will see it, I see that as an investment proposal for my time.
And if somebody presents me with bad investment proposals a few times, I'll say, you know, I will no longer be looking at your investment proposals, sir.
Thank you.
Or madam.
So, you know, so, so I, I, but to the, but most people where I can actually sort of see their replies, I, I, I generally engage and that's what people find it.
I, I, I, But I do realize that that is uncommon when people might not expect it.
But I tweet with that in mind that if anybody has any question about what I'm doing, they can totally just ask.
Yeah, I've both seen this and experienced it.
You are absolutely ready to both engage, fill in details, go back and say, I think I had this a little wrong or, you know.
Anyway, I've found you to be an honest broker in this regard and I think, you know, I think in some sense because you are true to the, I mean, you know, not that I know exactly what the rationalist code is, but I can infer it.
I think you are true to that and I think the problem is that other people are adhering to a kind of different code in which, You know, there's some obligation to provide a face-saving exit or something like that.
And if you just keep holding somebody who is not a rationalist to a standard and saying, look, you've got this wrong and you don't give them some mechanism to come back around, then... I will say that I generally do try to do that.
Whenever, you know, there's this term, I don't know if you know it, flipping the bit.
I don't think so.
Oh, you're just, you're throwing them in a different, there's a bit that encodes which category they're in.
Exactly.
Yeah.
Right.
So before I do that, I generally have tried, again, the problem is with Twitter, there's no easy way to do context.
Right?
And say like, okay, let me see the interactions between these two people over, you know, a period of time.
Even, you know, with Sam Harris that famously made a career of going to podcasts and saying I'm a psychopath or whatever.
If you see our early interactions... In fairness to Sam, you had no right to put him in so much context.
I know.
So it was, you know, but the thing is, if you actually go back to the beginning of the interaction, you'll see that my tone is very different because I try.
And then, you know, he took my initial thread.
He commented on it.
He named me in his podcast.
I was just somebody from the crowd, right?
He picked one thing that he felt he had a strong claim on and he misrepresented it and then misanswered it and then considered the whole thread dealt with.
The thread that documented him slandering people, right?
He finds one little claim that he feels is like, oh yeah, I'm going to hit him on that one.
Wrongly, by the way, because he came at me on an information systems design question, not knowing that it's kind of what I do.
But whatever.
The point is, you know, after that, I'm like, okay, this is kind of different.
And then I engage again, maybe a little bit more sharp.
And then I get the same thing back.
I'm like, well, look, I don't want to be the person with a silky voice that, you know, people just let everything go because, you know, look how smoothly they engage.
That's just not me.
Even if I wanted to, let's face it.
So, I try the other thing where I'm like, okay, look, I'm just going to try.
I'm going to try to engage.
And if it doesn't work, then, you know, I've now know something that I feel I need to engage to actually like convey.
And if I'm wrong, again, we're going back to the double click thing where you're like, people are gonna be like, Hey, are you sure you said, I was like, look over there, you know, and are you sure he saw it?
I'm like, he commented on it.
Like he must've seen it.
Right.
So here's, I'll say one more thing.
And at the, uh, you know, at the risk of sort of over, over broadening, one thing I learned during the pandemic is a pattern where people with large platforms, um, We'll make some claims, some claims that are false claims, some claims that are defamatory claims, right?
And then when sort of well-constructed response comes back, where it's like, okay, you said this, this is not true, here's the evidence, here's the receipts, whatever, whatever.
They will do the thing where they don't respond and they kind of run to their audience where they make some claims to their home team.
And sort of just hope to get away with it.
Like, people with large audiences, you know, you can trick yourself, right?
Because, hey, all of my followers are telling me I'm great, so it must be fine.
And I think that has been toxic, right?
That's probably what killed the IDW by now.
It's been worse than toxic.
It is also like an API that lets the propagandistic enemy capture people.
Right, in other words, to the extent that you are monitoring some audience that is large enough that you don't know who or what is in it, right, and you are using it as a guide for whether or not you've got things right, then something that wishes to steer you can do it with the equivalent of applause.
I mean, I think Eric, it doesn't even, it's not even necessary, you know, that's how it's gonna happen regardless.
I mean, sure, there's more surreptitious methods that that can be employed.
But I'm just saying the natural force is people who watch you watch you because they like you and they have, you know, some some, you know, warm feelings about you.
So if they're If you ask them, was I right about this, they're probably going to say yes.
And even if they don't agree with you, they're probably not going to say that.
So you kind of feel like, okay, well, we're good.
And really, the fact that there wasn't a way to actually get people on a neutral basis to get together and discuss stuff, right, with some sort of neutral-ish mediator, if need be.
But, like, without a platform advantage, right?
So, even when, you know, with Scott Alexander, when I got into this very long conversation, how did it end?
Like, he picked out some things, he mixed up some others, you know, sort of misstated some of my claims to his audience, and that was that.
He responded, right?
What am I going to do now?
I can continue the conversation, but it's gonna end the same way.
So it's utterly frustrating from the point of view of the audience, right?
Like I feel like I'm part of the audience, at least for now, to see that play out.
And sometimes it comes out probably not in the best ways, but that's, yeah.
Look, I hear you, right?
And it's not that I don't know that sometimes I, my tone is not what I would like it to be.
But again, on balance, like whose is, right?
And I, and I, and I don't want to be fake and, uh, people should know that if they double click, uh, on stuff, we can, we can go further.
And, and I'm always like, my DMS are open and I'm always open to talk to people.
So that's something else.
I think that's fair.
Um, I wish we lived in a world in which we all agreed that a,
We are all enhanced greatly if we get to the truth or an approximation of it, whatever it may be, and that one cannot avoid a conflict that it will result in embarrassment for whichever ones of us have thought something was true and then discovered it wasn't, but that there's a basic way that we deal with this.
I wish we were all on board with the same agreement and that you were therefore being understood
To be doing what I understand you to be doing right understood by others to be doing that which is right like you are functioning like You know an attorney in a system in which adversarial presentations are the mechanism through which the court achieves its end right and
I think others are involved in different codes, none of which are stated, or very few of them have stated these codes, and so there's a conflict between them.
But in any case, I would say, just from the point of view of actually getting to something that approximates the truth in some form that is actually hearable by the maximum number of people, I would say, A little bit of caution in how one presents is going to pay large dividends, right?
Actually, let me give you one example, and believe me, as you know, I have every reason not to be particularly inclined to defending Sam Harris.
I did think there was something instructive, though, in what went down with the clips of Sam that you put out, Sam's reaction to them, the public's reaction to them.
There was one thing in which he, I think in several different clips, alluded to dead children, right?
And it became interesting.
Oh, Sam Harris has a dead children tick in some of the things he said.
And I looked very carefully at those clips.
And I know Sam, not terribly well, but well enough.
And I'm convinced that what happened is that Sam uses dead children as his own.
It's like a personal trolley problem formulation, right?
His point is, I'm going to set that slider to maximum even if this involved dead children.
Then X still holds, right?
That is not about Sam really being comfortable with dead children.
In fact, the opposite.
It's about him recognizing that that would be the maximum bad thing, and that if you just set that parameter to maximum, it X, Y, and Z. But It's still evidences, even if you understand that Sam is just using like a philosophy trick in order to get somewhere, the conclusion is still jaw-dropping, right?
Even if Hunter Biden had dead children in his basement, I would still X, Y, and Z. That is in and of itself disturbing, right?
That it wouldn't cause you to rethink, you know, even just the fact of You know, a president, or would-be president, having a son with dead children in his basement.
Does the would-be president know that there are dead children in the basement?
Would that not disturb you?
Right?
The point is... Or if it didn't, wouldn't that disturb you?
Any normal person would be caused to rethink, pull out a fresh sheet of paper and reevaluate whether or not their feelings about Joe Biden were the same if they discovered that Hunter Biden had dead children in his basement, having nothing to do with your comfort with dead children, right?
So Sam did reveal a A gaping defect in his model or his character or both, right?
You revealed it by putting together a large clip.
It's not like you took him out of context, right?
Two minutes and 20 seconds.
And whenever I cut clips, by the way, I know what algorithm I'm running because that's how I do it.
Uh, if people want to, first of all, nobody has ever said, Oh yeah, I went and listened to the whole thing and it's better.
Usually I hear, I went and listened to the whole thing and it's worse.
But, um, but see where I made the cuts.
See if that is a reasonable place to make the cuts because Sam Harris, again, and Costantin Kissin, unfortunately now has taken the same, uh, plot point.
Um, and they're saying, Oh, you know, all these deceptive clips.
Okay.
Well, how would you clip it?
Secondly, on that tweet, I did not mention the dead children.
In that tweet, I actually, it didn't, that was not my focus, right?
After that, and after I've been sort of, you know, flatly with no right of response or anything like that, right?
Flatly, you know, across platforms that have millions and millions of people.
People keep calling.
I mean, Kissen went to Anviva Barnes, which is a podcast I really like.
And he just called me a piece of shit with no prompting.
He just offered that up, right?
About my career.
Let me just complete this point.
After all of that, right?
I saw this other one, and really, again, that was a sequence of clips.
There was one particular tweet where I'm like, alright, what is it?
I mean, you've been taken to task about this.
Can you like stop?
Because I know what you mean, and I actually partially agree.
Is doing is that is an extreme of a slider, but the slider is not emotional resonance to Sam.
It is emotional resonance to other people.
And he actually, if you hear the other argument that, you know, kind of went well as well, it was, he said, well, if it had killed children, people wouldn't would have complied.
Meaning if the impact was the same in terms of deaths, but it was children, which, you know, the normies kind of get moved by, then people would have complied.
And, and that is the thrust of that argument.
And first of all, that is ridiculous.
Like just let's take the clinical quality adjusted life years.
They are non-comparable.
Okay.
An 85 year old and a five year old are just different things.
Um, secondly, um, like different things for multiple reasons.
I mean, we can't, we can't really even begin.
I don't, I don't know what's happened to Sam Harris.
I think he's a lot better than that.
He, he is, he's, he, he, Perfectly competent of understanding why that is a terrible argument to bring up, why that is a terrible argument to bring up that way.
Regardless, I did my own analysis of that argument.
But ultimately, right, my Twitter page is my log where people can read my thoughts in the context that I write them.
And will some tweet go viral?
I mean, I don't know which tweets go viral.
Honestly, the tweets I want to go viral don't.
And the tweets I don't do, right?
So I don't control that.
I just say what I think.
And, yeah, I do think, like, you know, hey, if you've, like, gone into hot water for that, just, like, don't do that anymore.
But, like, I do think he does it because he's trying to throw out a very bright light to sort of confuse.
It's a confusing tactic, unfortunately, and I don't like to say that people are doing something surreptitious or whatever, but I've run it through my head in many, many ways.
I can't come up with a better hypothesis than it is.
A thing that is supposed to cause an emotional reaction in the listener, not that Sam himself is particularly.
I mean, look, I know he loves his children.
I know that he doesn't want children to die or anything like that.
Let's get that out of the way.
But the use in argumentation is its function at the very least is such as to distract from the logical structure of the argument being made.
And if you do a formal analysis, which I have done.
Yeah, it doesn't add anything at all.
You're just like, why would you do that?
Why would you bring that up?
It's worse than that, because... Yeah, I know.
It's worse than that.
That is a very useful phrase, unfortunately.
But it's worse than that because the very Authorities and policies that Sam ended up defending put actual children in danger that they did not need to be exposed to.
And Sam went after those of us who tried to get in the way and protect children.
So to the extent that Normies and Sam himself believe that children deserve some special protection, and that is why their deaths would be particularly horrifying to people.
Then the point is, this should have resulted in Sam at the point that he knew that these policies made no sense.
Right.
At the point he realized, oh, God, actually, there is a problem with this vaccine enough that I don't want any more of it and that it shouldn't have been given to children and especially male children who experience a greater risk of myocarditis.
Right.
That should have caused him to go back and think, oh, not only have I been on the wrong side of that issue, but I've actually gone after people who were on the right side.
I've said nasty things about people who were trying to prevent this, and actually, you know, that should have forced him to an apology that I have yet to hear.
We can, we can go down that rabbit hole a lot.
I think we're going to need, this podcast is going to need a warning of like jump to.
Yeah.
Oh, it is going to need a warning.
Uh, if you're here for the AI, jump to something like 34 minutes in.
Yeah.
You are on Twitter.
Your, uh, your handle is at Alexandros Marinos.
M, Alexandros M. So it's my first name.
Yes.
My full Greek first name plus my first letter of my last one.
Sorry, I wasn't thinking about spellability at the time.
Anyway, Alexandros M is my Twitter.
And I think by the time this goes up, we're going to, me and Ryan, who you also know, Are going to have started YouTubing on a channel called Bootstrap AI that people can find where we're going to be experimenting with tools and basically on the same vein of like, how can we get more competent to use this thing for good?
Cool.
And if it's you and Ryan, I'm, you know, of course, ethically concerned about what y'all get up to, but I know it's going to be fun.
No, I'm kidding.
Ryan is an absolute gem and you two together are lightning in a bottle.
So I'm excited about that.
Substack, you are, oh man, your phrase, the title of your Substack has become such a flashpoint for civilization.
Do your own research.
Yep.
Like, you know, do your own research used to be what being a rationalist meant, but apparently, um, you know, and I, and I chose it tongue in cheek.
Um, and honestly, if it turns off somebody, what are you going to do about it?
But it's called do your own research.substack.com.
And, uh, I've got a lot of the pandemic content on there, but I think the latest essay actually is called, um, I don't remember the title, but it talks about the age of bespoke.
That's it.
Again, another possible positive future that we could find ourselves in, or at least one dimension of the future that we'll find ourselves in, which, depending on everything else, might turn out good or bad.
Yeah, I try to explore all these thoughts in longer form and less prickly on my Substack.
Do your own research at substack.com.
But yeah, so that's...
Cool.
Well, that's great.
Your sub stack is very cool.
I'm looking forward to your YouTube channel.
With that, Alexandros Marinos, it has been an absolute pleasure.
It has, as always, been very enlightening.
I enjoy that about you.
It is impossible to find the full depth of the thoughts that you have, which means you've always got new stuff to say.
So, Anyway, thanks for joining me again on Dark Horse.
