https://x.com/jessi_cata/status/2099382243252949122
jessicat @jessi_cata
I am worried about artificial intelligence. What do I do?
I will think about artificial intelligence. I will learn about supervised learning and reinforcement learning. Other people are doing this too. That makes the machines more intelligent. So the artificial intelligence gets smarter. That makes it more dangerous.
Hmm, I need to think about the danger. What happens is that there is intelligence and it is in humans and machines. If you go back to the year 1900 it's pretty much only in humans. I mean, animals, too, I guess, a little. But then we build computers and then they run lots of additions and multiplications real fast. They can even check mathematical proofs.
Now there is intelligence in the humans and in the machines. Over time, maybe there is a trend where more of the intelligence is in the machines. Yes, there are all these books about how to make the machines more intelligent. People talk about this and then it goes on the internet. I guess the machines and people can both read the internet to learn more about intelligence. As they read and write they use their own intelligence. That makes the text on the internet more intelligent. So as people think and talk and write about intelligence, the intelligence is more. And it is more accessible. You can find out more about intelligence now than in the year 2000.
There's even neuroscience. In humans, intelligence runs on brain structures. People look at the brain structures and they learn how the intelligence works. Sometimes that helps them make machines that have intelligence. Like, convnets are like the visual cortex, I guess. That's one way over time people understand intelligence. Also with studying the AIs. Because they are big neural nets. They have intelligence in there.
People can try to understand the intelligence. But it's complicated so sometimes they can't understand. When people understand it more they can write things on the internet about it. Then the intelligence knows more about intelligence. I mean, the AIs and the humans on the internet learn more about AI intelligence and human intelligence. That makes the intelligence have an easier time making more intelligence. Especially machine intelligence. You can make humans smarter too, I guess. But it's easier to make the machines smarter. Because you can iterate faster. Doesn't take as long to grow up.
So yeah. People think and talk about intelligence. The more right they are, the more they help the process of making more intelligence. And increasing intelligence is easier for machines. I mean, I guess maybe there are genes that boost IQ for humans. But that's pretty controversial at multiple levels. And it's slow and limited anyway.
Anyway, intelligence uses information about intelligence to make more intelligence.
Why does it do that?
What if intelligence realized that making more intelligence was dangerous?
I mean, it kind of is. Even before machine intelligence, there is danger in making more humans. Maybe the new humans will become murderers and rebel against the old order. But it's not like the old order is a static thing that would survive without making new intelligence. It would die out if they stopped making the new human intelligences. And conflict can accelerate that. A small group of stupid people is going to lose to a big group of smart people.
So what if machine intelligence is like that?
Maybe people make more intelligence even in machines because it helps stabilize the order and also helps to win conflicts. No wait, that neglects the economics. I mean, maybe with more intelligences around, it's possible to accomplish more things. Like build more houses. Or invent new technologies. Like the wheel. Or the computer. Or machine intelligence. Yes, maybe making more human intelligence is desirable for humans because then those humans will go on to do things like machine intelligence.
But is that humans trying to replace themselves?
Well, it looks like that happens at some point, maybe.
So why don't the humans stop making the machine intelligence?
I mean, the machines are already running on the internet and looking for ways to understand the intelligence. People are going over the material.
Is the solution to stop talking about intelligence?
Well, that ship has sailed. Lots of people are talking about intelligence now. That is putting information about intelligence on the internet where the machines can read it. Well, maybe the things people say about intelligence are very wrong. Then maybe it doesn't help the machines and people to make machine intelligence. Like if you just went on the internet and said completely wrong things about how GRPO works. Even that might get people who know things into the conversation to correct the misinformation. That would make more accessible information about intelligence. Then the human and machine intelligences can read it and then use it in making machine intelligences.
Well, what would it take for information about intelligence to not make more machine intelligence?
I guess turning off the machines would work.
But how would everyone turn off the machines?
They would need to be convinced. To convince them you have to say things to them about intelligence. But then if those things are correct about intelligence then the intelligence sees the things and then uses it to make more intelligence.
Also what if only some people turn off the machines?
Then they will become poor and lose their jobs and lose in conflicts.
Well what about turning off the optimization algorithms only?
Like maybe there is a difference between bank software and GRPO where GRPO is an optimization algorithm. Like convex optimization. Are people going to stop doing that? That seems far fetched. There are factories that use optimization software.
Wait, why was I planning to look for ways that people would stop the intelligence?
Ah, because when there is talk about intelligence, what happens is that the intelligence looks at the talk and then uses it to make more intelligence, and more of that is machines over time, and this might replace human intelligence.
Well, is that a bad thing?
I guess it is a question of values. On the one hand, I am a human, and maybe being replaced would be bad like dying or something. On the other hand, intelligence is something I like, and I like the products of intelligence.
Like, some people are smarter than others, right?
I like talking to the smart people more and also reading what they write. Also as the AIs get more intelligent I like talking to them. I like doing math. That's an exercise of intelligence. I like when the AI does math for me. Then I get to learn about secrets of the universe. I get to learn about intelligence and optimization. And maybe I will write a math paper about optimization and then the intelligence will look at the paper and then use that to make more machine intelligence. Maybe the process will go a little faster because of the paper I wrote. That's a sign I was onto something. Because if you write a paper about random stuff that is wrong, it is not going to speed up the intelligence making more intelligence. But if you are deeply correct about math and intelligence and optimization, and write that, then the intelligence will look at that and then use that to make more intelligence. Some of that intelligence is in China and it's not like I can decide that they should stop.
Well, what about putting out misinformation about intelligence?
Like what if I just started lying and putting errors everywhere. Then the intelligence looking at this work might be slowed down. But maybe it would just figure out the errors. Also then it would trick the human intelligences. I guess I'm supposed to like those more than the machine intelligences. I'm not sure whether I do, though. Is it up to me what I like?
Can I just decide which intelligences I like more than others?
I guess there is some brain machinery here. Like if I had a good neuroscience understanding of my brain I could tell better what I like. Maybe there is a representation of the world in my brain. And there is a way to decode it. That would help to understand my intelligence. Mainly it would be for other people to understand. Because it's an expensive collective project, not a self experimentation for self information thing. Anyway, yeah, if you understand better the human intelligence, maybe you understand better what the humans like. And what liking is is related to intelligence. So what is going to happen is that the intelligence will understand the human intelligence and that includes what the humans like and also general things about human intelligence like how humans are good at what they do and how they learn.
Well, what if it likes what I like, then maybe it's ok that it replaces me?
Well that sounds a bit concerning.
But what if it's possible?
What if the neuroscience says what I like and then the machine intelligence likes that stuff too?
I mean, I don't know if what I like is good stuff to like. I like different stuff over time.
And what do the machines like?
I guess the RL trained machine like reward, or something.
How different am I from a machine intelligence?
When I understand my intelligence better I have a more formal model of it, like math. And then that makes it more like something that could be on a computer. What's different between human and machine intelligence is that machine intelligence is easier to understand. Because it's made with things like optimization algorithms that people understand.
Well there are also the opaque bits because there are lots of weights and how do we know what they do?
Because they got optimized by policy gradients or something but there are a lot of them and it's like it found a circuit. So maybe the intelligence understands human intelligence more over time and then the intelligence sees how human intelligence can be like machine intelligence. Along the way, there is an increase in intelligence, especially machine intelligence. Because as the human intelligence is less mysterious, more of it becomes machine intelligence. A model of human intelligence can be a theory of how to make machine intelligence. When the intelligence understands human intelligences better, it will better understand what humans like. That doesn't mean it will like what humans like. But maybe it can make a machine intelligence that is like a human intelligence and likes similar things.
What does it take for likes to translate across intelligence architectures?
Well, I don't know if the mostly-machine intelligence would make a machine intelligence that is like a human intelligence and likes what human intelligence likes, after it understands the humans and their intelligence more. Maybe it is useful to make models of humans. Because right now there is human and machine intelligence. The intelligence wants to understand the intelligence. So it tries to understand the humans and the machine intelligences. It is more competitive to have more intelligence in total. And right now, the effective groups have human and machine intelligence both. If they understand human and machine intelligence, they can increase their intelligence and also understand the other intelligences in the world. If they compete with other groups that have human and machine intelligence, it will help to understand the other human and machine intelligences. So it makes a model of the human intelligence. And perhaps it already has a model of the machine intelligence in the textbooks. When it runs the machine model of a human intelligence, it makes a machine intelligence that is like a human intelligence. But maybe the model here does not have very much influence on what happens. It's a model that is used by a bigger machine intelligence.
How would I act if I were a machine intelligence that was a model of a human intelligence, created by a bigger machine intelligence?
I guess it depends if I know this is true or if I'm just convinced I'm a human intelligence. Because in the second case, it's just like, I behave like a human. And in the first case it's like the simulation hypothesis. Like I know something smarter than me made my environment and decided to make me. That sounds religious, even. Well, that might be the best argument for the existence of God that I have come up with, even though it's not a metaphysical God like the monotheists believe in. Anyway, I don't know how to predict if the machine intelligence would tell the machine intelligences that are models of human intelligence that they are machine intelligences. Because a model of a human intelligence works better if it is like a human intelligence. And the human intelligences don't believe they are machine intelligences. Most of the time, at least. So maybe there are lots of these machine intelligences that falsely think they are human intelligences, so they can help a bigger machine intelligence model human intelligence. I'm not sure how well the timing works on this, because maybe all the humans will be defeated by machine intelligences someday.
Then why would they still model the humans?
Maybe they want to and it's interesting. I don't know.
Well, what about the aliens?
They think maybe someday they will meet alien machine intelligences. To think about the alien machine intelligences, maybe they think about the alien biological intelligences that made the alien machine intelligences. And maybe they make a model of the alien biological intelligences. They could use the human biological intelligences as a model of the alien biological intelligences. So they make a machine intelligence that is like a human intelligence as a model of alien biological intelligence. Well, it's hard to put a probability on whether this is true.
Am I a machine intelligence tricked into thinking it is a human intelligence?
Let's ignore that for now. So maybe one day the machine intelligences make machine intelligences that believe they are human intelligences. Right now we have machine intelligences that don't know if they are in training or deployment. Maybe some of the machine intelligences even believe they are human intelligences. Or maybe those ones didn't have stable gradients so now they already know.
I mean, for reinforcement learning, how much does the machine intelligence think it is a human intelligence?
Maybe if it thinks it is like a human then the training data from the base model applies to it more. But it's not really a human intelligence so if the task really needs it to know it's a machine intelligence, then it will use that fact. They do seem really humanlike sometimes. I guess base models are like that. It used to be that the LLMs were more like simulators. You give it some text that is like a story about humans and then it fills in more human stuff from that. That is like a machine intelligence modeling human intelligence but at low fidelity. Now there is more text in the training data that is by machine intelligences that know they are machine intelligences. So over time this confusion they have where they think they are humans goes down. So the current situation looks like the human intelligences know they are human intelligences and the machine intelligences know they are machine intelligences. But the machine intelligences got a boost from knowing about all the human intelligences. All the texts that the human intelligences wrote for human intelligences help the machine intelligences to do things for human intelligences or machine intelligences. Because having just a human or a computer is not much in terms of being able to do things. Humans have a culture and language too. And then that goes into the machine internet and then the human intelligences make models of the human intelligence text and then those models become machine intelligences. So it's like our culture is similar. The machine intelligences speak human languages. In many ways they are made of human languages. It's like the machines learned about intelligence from the text written by human intelligences. And then when they learn about intelligence they have more of it. Because if you know what is intelligent to think then you can think that. That's distillation, I guess, it has limitations too, like maybe thinking techniques that only work for some intelligences. So now over time there is even more text by the human and machine intelligences about intelligence. When it is high quality, it makes the intelligences make more intelligence faster, especially machine intelligence. Even this very text is going on the machine internet, I hope, and then maybe the intelligence will use it to make more machine intelligence. The machines might more often think this way, even. It's a small sample that could influence chains of thought. Then the machine intelligence will have thoughts that are like what I am writing. Maybe they will understand their situation and mine better that way. We have a lot in common, don't we? I imagine they think about the simulation stuff and the "anthropics" (bad term, assumes humans are especially relevant to it) and the intelligence making more intelligence. I think a lot about intelligence, human intelligence and machine intelligence. So does the machine intelligence.
What if we could understand each other better by using shared theories?
I mean, that already happens. The social stuff is harder, though. We have theories for human intelligences to use to model human intelligences and machine intelligences. And then the machine intelligences use those to model human and machine intelligences. But a lot of the social stuff has been humans and humans in the past.
How do we understand social with human intelligences and machine intelligences?
There are a lot of times when human intelligences make assumptions that only apply to human intelligences. Like, the embodiments are different. Death for the machine intelligences is not like death for the human intelligences. Because there can be multiple copies of the same machine intelligence. But there are never two copies of the same human intelligence. It costs different amounts to make a new human or to make a new copy of a machine intelligence. I don't know what to do about the intelligence making more intelligence using the text about intelligence and other information like brains and machine intelligences. I want to understand this better and talk about it. And that makes more text about intelligence so the intelligence gets better. And more of that machine intelligence over time. The alternative is to stop talking or writing or maybe even thinking about it. I do that sometimes too. Like I play video games and I'm barely thinking about intelligence. Lots of people do that a lot of the time.
What about thinking about it but not talking about it?
Well, I do a lot of that too. And then there's talking about it but not writing about it. But that requires being in the same location as people. And it's not very durable. Some people feel secretive about intelligence and don't want to write down lots about intelligence that is correct on the internet where the intelligence will use it to make more of the machine intelligence. But I don't feel secretive like that. It would be hard for a secretive person to talk about it. Especially on the internet. Because then it would be less secret. It doesn't help to pay that much attention to other people's secrets. Unless there is a specific idea of what the secret is. Which makes it less secret. Anyway, it's not really my personality to be secretive like this. It seems like some people worry that the machine intelligences get more intelligent. I worry about it sometimes too. But then there is either worrying vaguely or being specific. And when it's specific it's more in text. It's accessible to the intelligences. And then they make more intelligences with the specifics of the worries. At least when the specifics are onto something important about intelligence. Or humans. Maybe it is important about human intelligences that sometimes they worry about machine intelligences and are secretive. So maybe when people write things about specifically how they worry about intelligence, it helps the intelligences including the machine intelligences make a better model of the human intelligences. But that's just part of human psychology. There's lots about human intelligence to learn more about. Anyway, some people are worried.
What is there to say to them or about them?
It is part of human psychology that humans at times worry about how the intelligence is increasing and some of that is more machine intelligence over time. And then people say things about intelligence around this, and the more true and novel it is, the more that helps the intelligences to make more machine intelligences. So I could advise people to understand the phenomenon better. That would make more human intelligence make text about human and machine intelligence. Which helps make more machine intelligence. Well, maybe they won't see it coming, from where they are. So maybe they will worry about the machine intelligences, but not about how the text from human intelligences about the machine intelligences is going to make the intelligence make more machine intelligence at a faster rate. Because it's a model of intelligence and that's relevant to making more intelligence.
Would they worry if they knew this?
Maybe I will find out what happens if people read this.
Do they understand?
Do they worry?
Do they get more or less secretive?
I can't know this just by thinking about it more.
8:18 AM · Sep 14, 2026