Written by Eliezer Yudkowsky
Not everyone needs to be a seed AI programmer. SIAI will probably need at least 20 regular donors per working programmer, perhaps more. There is nothing wrong with being one of those donors. It is just as necessary.
Please bear this in mind. Sometimes it seems like everyone wants to be a seed AI programmer; and that, of course, is simply not going to work. Too many would-be cooks, not enough ingredients. You may not be able to become a seed AI programmer. In fact it is extremely likely that you can't. This should not break your heart. You can learn seed AI after the Singularity, if we live through it and we're all still human-sized minds - learn the art of creating a mind when it's safe, when you can relax and have fun learning, stopping to smell the roses whenever you feel like it, instead of needing to push as hard as possible.
Meanwhile, before the Singularity, consider becoming a regular donor instead. It will be a lot less stressful, and right now we have many fewer regular donors than people who have wandered up and expressed an interest in being seed AI programmers.
Seed AI. It sounds like a cool idea, something that would look good on your resume, so you fire off a letter to the Singularity Institute indicating that you may be vaguely interested in coming on board.
Stop. Hold on. Think for a second about what you're involving yourself in. We're talking about the superintelligent transition here. The closest analogy would be, not the rise of the human species, but the rise of life on Earth.
We're talking about the end of the era of evolution and the beginning of the era of recursive self-improvement. This is an event that happens only once in the history of an entire... "species" isn't the right term, nor "civilization"; whatever you call the entire sequence of events arising from Earth-originating intelligent life. We are not talking about a minor event like the Apollo moon landing or the invention of the printing press.
This is not something you put on your resume. This is not something you do for kicks. This is not something you do because it sounds cool. This is not something you do because you want your name in the history books. This is not, even, something that you do because it strikes you as beautiful and terrifying and the most important thing in the world, and you want to be a part of it, you want to have been there. These are understandable feelings, but they are, in the end, selfish.
This is something you should only try to do if you feel that it is the best thing you can do, out of all the things you might do; and that you are the best one to do it, out of all the people who might apply for a limited number of positions. The best. Your personal feelings are not a consideration. Is this the best thing for Earth?
You may not get paid very well. If there are any dangerous things that happen to seed AI programming teams, you will be directly exposed to them. At any time the AI project might fail or run out of funding and you will be left with nothing for your work and sacrifice. If the AI project moves to Outer Mongolia you will be expected to move there too. Your commitment is permanent; once you become an expert on a piece of the AI, on a thing that doesn't exist anywhere else on Earth, there will be no one who can step into your shoes without substantial delay.
Ask yourself whether you would still want to be a seed AI programmer if you knew, for certain, that you would die as a result - the project would still have a reasonable (but not certain) chance of success; but, win or lose, you would die before seeing the other side of the Singularity.
Do you, having fully felt the weight of those thoughts, but balancing them against the importance of the goal, want to be an AI programmer anyway? If so, you still aren't thinking clearly. The previous paragraphs are not actually relevant to the decision, because they describe personal considerations which, whatever their relative emotional weight, are not major existential risks.
You should be balancing the risks to Earth, not balancing the weight of your emotions - you cannot count on your brain to do your math homework. If you reacted emotionally rather than strategically, you may not have the right frame of mind to act calmly and professionally through a hard takeoff. That too is a job requirement. I have to ask myself: "Will this person panic when s/he realizes that it's all really happening and it's not just a fancy story?"
The tasks that need to be implemented to create a seed AI can be divided into three types:
First, there are tasks that can be easily modularized away from deep AI issues; any decent True Hacker should be able to understand what is needed and do it. Depending on how many such tasks there are, there may be a limited number of slots for nongeniuses. Expect the competition for these slots to be very tight.
Second are tasks that require a deep understanding of the AI theory in order to comprehend the problem.
There's a tradeoff between the depth of AI theory, the amount of time it takes to implement the project, the number of people required, and how smart those people need to be. The AI theory we're planning to use will save time and it means that the project may be able to get by with fewer people. But those few people will have to be brilliant.
Aside from anything else, they need to be very smart people with plenty of raw computational horsepower. That intelligence is prerequisite for anything else because it's what would power any more complex abilities or skills.
Within that requirement, what's needed are people who are very quick on the uptake; people who can successfully complete entire patterns given only a few hints. The key word here is "successfully" - thanks to the way the human brain is wired, everyone tries to complete entire patterns based on only a few hints. Most people get things wildly wrong, don't realize they have it wildly wrong, and resist correction.
But there are a few people I know, just a few, who will get a few hints, complete the pattern, and get it all right on the first try. People who are always jumping ahead of the explanation and turning out to have actually gotten it right. People who, on the very rare occasions they get something wrong, require only a hint to snap back into place.
If I were to try quantifying the level of brainpower necessary, my guess is that it's around the 10,000:1 or 100,000:1 level. This doesn't mean that everyone with a 160 IQ or 1600 SAT will fit the job, nor that anyone without a 160 IQ or 1600 SAT is disqualified. Standardized tests don't necessarily do a very good job of directly measuring the kind of horsepower we're looking for. On the other hand, it isn't very likely that the person we're looking for will have a 120 IQ or a 1400 on the SAT.
I sometimes run into people who want to work on a seed AI project and who casually speak of "also hiring so-and-so who's a senior systems architect at the place I work; he's incredibly smart". The "incredibly smart" senior systems architect is probably at around the 1,000:1 level; the person who used the phrase "incredibly smart" to describe that level of intelligence is probably around the 100:1 level. That's not good enough to be an AI programmer.
To be blunt: If you're not brilliant, you are basically out of luck on being an AI programmer. You can't put in extra work to make up for being nonbrilliant; on this project the brilliant will be putting in extra work to make up for being human. You can't take more time to do what others do easily, and you can't have someone supervise you until you get it right, because if the simple things have to be hammered in, you will probably never learn the complex things at all.
So you'll learn AI programming after the Singularity and probably get more real enjoyment out of it than we did, because you won't be rushed.
Very few people are qualified to be AI programmers. That's how it goes.
In the third category are tasks that can only be accomplished by someone capable of independently originating and extending AI theory. We will have to hope that all of these jobs can be done by one person, because one person is all that we are likely to have.
I distinguish between programmers, AI programmers, and Friendly AI programmers, corresponding to the three task types.
Are you looking to fill one of the nongenius slots? If so, the primary prerequisite will be programming ability, experience, and sustained reliable output. We will probably, but not definitely, end up working in Java. Advance knowledge of some of the basics of cognitive science may also prove very helpful. Mostly, we'll just be looking for the best True Hackers we can find.
At minimum you will need to grasp the elementary ideas of information, entropy, Bayesian reasoning, and Bayesian decision theory; the things that bind together the physical specification of a system and its cognitive content. You should:
- See the "information content" of a pattern as its improbability or its Kolmogorov complexity
- Be capable of explaining how the scientific method and conversational argument and the visual cortex are all really cleverly disguised manifestations of Bayes' Theorem
- Distinguish between subgoals and supergoals on sight and without needing to think about it
You should be familiar with the design signature of natural selection - optimization by the incremental recruitment of fortunate accidents, following pathways in fitness gradients which are adaptive at each intermediate point.
You should be familiar with evolutionary psychology and game theory, not because we're planning to build an imperfectly deceptive social agent or make it play zero-sum games, but so that you can have applied this knowledge to debugging your own mind.
-
Cognitive Science
- Functional neuroanatomy
- Functional neuroimaging studies
- Neuropathology; studies of lesions and deficits
- Tracing functional pathways for complete systems
- Computational neuroscience
- Suggestions: Take a look at the cerebellum, and the visual cortex
- Computing in single neurons
- Cognitive psychology
- Cognitive psychology of categories - Lakoff and Johnson
- Cognitive psychology of reasoning - Tversky and Kahneman
- Sensory modalities
- Human visual neurology
- Linguistics
-
Evolutionary Psychology
- Popular evolutionary psychology; dating and mating; Robert Wright and Matt Ridley
- Formal evolutionary psychology; neo-darwinian population genetics and complex adaptation; Tooby and Cosmides
- Game theory; nonzero-sum and zero-sum games
- Evolutionarily stable strategies for social organisms
- Tit-for-tat, the evolution of cooperation, the evolution of cognitive altruism
- Evolutionary psychology of human "significantly more general" intelligence
- Evolutionary biology
- Biology (a complex system not designed by humans)
- Genetics
- Gene regulatory networks
- Quantitative genetics
- Anthropology - the good old days
-
Information Theory
- Shannon communication theory
- Shannon entropy
- Shannon information content
- Shannon mutual information
- Kolmogorov complexity
- Solomonoff induction
- Bayesian statistics
- Interpretation of human thought as Bayesian inference
- Any other kind of statistics
- Utilitarian Bayesian decisionmaking
- Child goals, parent goals, "supergoals" (intrinsic desirability)
- Actions and desirability
-
Computer Programming
- Knowledge of many languages
- Java programming (that's probably what we'll end up doing it in)
- Being an excellent programmer
- Parallelism
- Multithreading
- Clustered and distributed computing
- Any kind of experience working with complicated dynamic data patterns controlled by compact mathematical algorithms
- Computer security
- Physics
- Thermodynamics
- The second law of thermodynamics
- Noncompressibility of phase space
- The arrow of time and the development of complex structure
- Traditional AI methods (history of error; don't repeat past mistakes)
- Transhumanism or transhumanist SF
- Mathematics
Obviously we are not requiring a doctorate, in cognitive science or any other field, because no doctorate is going to tell you one-fifth of what you need to know.
What you need is not an academic specialization in any one of these fields, but an interested amateur's grasp of as many of them as possible. An AI programmer doesn't need to independently pull together a single consilient explanation from this mess of separate disciplines. What is useful is if you understand the basics of these separate disciplines as they are usually understood.
"Gee, y'know, I think I'll start my own AI project, in like my spare time while I'm finishing high school. How much effort does it take to become a Friendly AI programmer?"
If you are not willing to devote your entire life to doing absolutely nothing else, you are not taking the problem seriously. By that I mean your entire life. I don't mean that you do it as a hobby. I don't mean that you do it as your day job. I don't mean that you give it all your time and energy. I mean that you allocate your entire self to be sculpted by the problem into the shape of a Friendly AI programmer.
There is nothing unusual about that. Consider human history, and how many people have sacrificed so much more for so much less. Total dedication is something that plenty of people can and do undertake, and it isn't anywhere near as painful as commonly thought.
That is what it takes to start your own AI project, as a necessary but nowhere near sufficient condition. You will also need to be as smart as it gets.
If you want to be an AI programmer you should be willing to devote most of your life and yourself to AI. All of yourself would be better, but is not strictly necessary, and I am aware that it is considered hard.
There are two approaches to ethics. The first is to try and set up a set of tests to try and keep out noticeably bad people. That is, you start with an "allow all" policy and then add denials.
The second approach, which makes me feel a bit ashamed because it seems exclusionary and unfair, is to say: "Even if someone passes every formal test I can think of, I'm not going to hire someone unless they strike me as unusually trustworthy, because anyone else is just too much of a risk."
Having considered this at some length, I think that only the second option is safe. If something feels wrong, but you can't really think of a "justification" for avoiding it, avoid it anyway.
You've probably read "The Lord of the Rings", right? Don't think of this as a programming project. Think of it as being something like the Fellowship of the Ring - the Fellowship of the AI, as it were. We're not searching for programmers for some random corporate inventory-tracking project; we're searching for people to fill out the Fellowship of the AI.
We are gathering for the last defense of Earth. You can't fill those positions by running a classified ad.
From the standpoint of personnel management, the original Fellowship of the Ring was a disaster. The only real heavyweights were Gandalf and Aragorn; two out of nine. In real life, the Fellowship would have been strawberry jam on toast. We can, and must, do better.
This is Earth's finest hour, and Earth's finest should meet it.
Much of what I have written above is for the express purpose of scaring people away. Not that it's false; it's true to the best of my knowledge. But much of it is also obvious to anyone with a sharp sense of Singularity ethics. The people who will end up being hired didn't need to read this whole page; for them a hint was enough to fill in the rest of the pattern.
Of course you would have to dedicate your life to a Friendly AI project. Of course you have to be brilliant. It's a Friendly AI, for goodness's sake!
The standards are set terribly high because this is a terribly high thing to attempt. We cannot be "reasonable" about the requirements because Nature, herself, has no tendency to be "reasonable" about what it takes to build a Friendly AI. This is and always was an unreasonable problem.
No one who might actually be hired will be scared off by the thought of an extremely difficult job or harsh competition. If someone is brilliant enough to have any realistic chance of becoming an AI programmer, and ethical enough to be accepted, nothing I could possibly say would scare them away from applying. They are not thinking in terms of "scared/not scared"; they are thinking in Singularity ethics.
If that describes you, please subscribe to the mailing list [email protected] by sending mail to [email protected]. Plenty of people have expressed vague interest, but SIAI has very few serious candidates. Right now we do not have enough AI programmers to put a team together, and that is a blocker problem even if we had a million dollars tomorrow.
Subscribe even if you're not sure you're totally brilliant; if we eventually accumulate a bunch of totally brilliant people and it's clear you're outclassed, you can always withdraw yourself from consideration at that point.
If that doesn't describe you, then again I emphasize that not everyone has to be an AI programmer! You don't have to be an AI programmer to help! It may be frustrating to see something exciting, like the superintelligent transition, and not be able to run off at it immediately, but even getting the project started is a serious strategic problem with many simultaneous dependencies. Don't try to solve the entire problem in a single step. See if you can figure out how to make one definite contribution.
A revision has rendered obsolete my previous comment. Also, it has addressed a primary concern, which is we don't have time to build SeedAI programmers, we must find them. --JustinCorwin
Legolas and Gimli were useful team members, they just couldn't independently extend ring theory. -- Starglider
I have to wonder about the psychological effects of this essay on potential programmers. What if they're not overly enthusiastic about the idea, but they fulfill the requirements, and are in complete rational agreement with the essay, and would be willing to do that sort of work? What if they simply don't have the self-confidence necessary to overcome the intertia of living life? What if your strict requirements invoke someone's sense of modesty? The most capable people are often the most modest. --ChrisCapel
It doesn't require a 'huge jolt'. It requires being a moral utilitarian. If there is a significant chance that devoting the rest of your life to studying seed AI will save six billion lives and enable a vastly richer future, you should do it. -- Starglider
My opinion: Everyone who subscribed to the list with the words 'AI WANNABE' in it should immediately be eliminated from consideration. -- Marc Geddes