What’s Nick Bostrom’s Superintelligence about? Could artificial intelligence ever surpass human intelligence?
According to Oxford philosopher Nick Bostrom’s book Superintelligence, there’s a very real possibility that AI could one day rival, and then vastly exceed, human intelligence. When and if this happens, the future of humankind will depend more on AI-generated decisions than human decisions.
Read below for a brief overview of Superintelligence by Nick Bostrom.
Superintelligence by Nick Bostrom
Oxford philosopher Nick Bostrom’s Superintelligence was written in 2014 to raise awareness about the possibility of AI suddenly exceeding human capabilities, spark discussion about the risks inherent in this scenario, and foster collaboration in managing those risks.
Today, the possibility of AI rivaling or even vastly exceeding human intelligence doesn’t seem as far-fetched as it did when Bostrom was writing a decade ago. In this article, we’ll consider his arguments for the feasibility of an AI rising to superhuman intelligence and his belief that creating such an AI without the right controls in place could be the worst—and maybe the last—mistake in human history. Finally, we’ll take a look at the controls Bostrom says we need to implement to build AI’s safely.
The Feasibility of Superintelligent AI
Bostrom defines “superintelligence” as general intelligence that’s significantly greater than human-level intelligence. As he explains, “general intelligence” refers to intellectual abilities that span the whole range of human capabilities, such as learning, interpreting raw data to draw useful inferences, making decisions, recognizing risks or uncertainties, and allowing for uncertainties when making decisions. He notes that while some computers already surpass humans in certain narrow areas, such as playing a certain game or crunching numbers, no AI has yet come close to human-level general intelligence.
But could an artificial, nonhuman entity ever have superintelligence? Bostrom argues that the answer is, most likely, yes. As he explains, silicon computers have a number of advantages over human brains. For one thing, they operate much faster. Neural signals travel about 120 meters per second and neurons can cycle at a maximum frequency of about 200 Hertz. By contrast, electronic signals travel at the speed of light (300,000,000 meters per second) and electronic processors often cycle at 2 billion Hertz or more. In addition, computers can copy and share data and software directly, while humans have to learn gradually.
Different Routes to Superintelligent AI
As Bostrom explains, there are a number of different ways that superintelligent AI could be achieved. Thus, even if some of them don’t end up working, at least one of them probably will.
Intelligent Design
One route to superintelligent AI that Bostrom discusses is human programmers developing a “seed AI” that has some level of general intelligence—perhaps similar to human intelligence, or maybe a little below that mark. Then they use the AI to continue improving the program. As the AI gets smarter, it improves itself more quickly. Because of this self-reinforcing cycle, it might progress from sub-human to superhuman intelligence rather quickly.
Simulated Evolution
Another route that Bostrom discusses is “simulated evolution.” In the context of software engineering, this means programming a computer to generate random variations of a program, test their functionality against specified criteria, and continue to iterate on the best ones. Theoretically, simulated evolution can provide novel solutions to programming problems without the need for new insight on the part of human programmers. Thus, even if human programmers can’t figure out how to create a superintelligent AI or a self-improving seed AI directly, they might be able to create one using simulated evolution.
Brain Simulations
Yet another route to which Bostrom devotes considerable attention is “whole brain emulation.” The human brain is obviously capable of human-level general intelligence. Thus, if you could map out exactly how all the neurons in a human brain are connected and create a computer program to accurately simulate all those connections, you would have a program capable of human-level intelligence. And if the computer program could operate faster than the original brain, it would have superhuman intelligence.
Bostrom explains that creating a simulated human brain requires a basic understanding of how neurons interact with each other and a detailed cellular-level 3D scan of a human brain. However, it doesn’t require an understanding of how the brain’s structures give rise to intelligence—assuming the simulation captures the placement of neurons accurately, it should, theoretically, mimic the brain’s function even if its developers don’t know exactly how or why. Thus, the main obstacle to implementing this method is scanning a human brain precisely enough.
Spontaneous Generation
Finally, Bostrom points out that it might be possible to create a superintelligent AI inadvertently. Scientists don’t know exactly what the minimum set of components or capabilities for general intelligence is. There’s already a lot of software that performs specific information processing operations and has the ability to send and receive data over the internet. Hypothetically, a programmer could create a piece of software that, by itself, isn’t even considered AI, but it happens to be the final component of general intelligence. This would then allow a superintelligent AI to arise spontaneously on the internet as the new software begins to communicate with all the other software that’s already running.
The Consequences of Superintelligent AI
So, sooner or later, a superintelligent AI will be created. Why should that concern you any more than the fact that mechanical vehicles can go faster than a human can run? According to Bostrom, the rise of a superintelligent AI could cause dramatic changes in how the world works—changes that would take place very quickly. And depending on the superintelligent AI’s behavior, these changes could be very detrimental to humanity.
As we mentioned earlier, if an AI has some measure of general intelligence and the ability to modify its own programming, its intelligence would likely increase at an ever-accelerating rate. This implies that an AI might rise from sub-human to superhuman intelligence very quickly.
Moreover, as Bostrom points out, superior intelligence is what has allowed humans to dominate the other life forms on planet Earth. Thus, it stands to reason that once a superintelligent AI exists, the fate of humanity will suddenly depend more on what the superintelligent AI does than on what humans do—just as the existence of most animal species depends more on what humans do (either to take care of domestic animals or to preserve or destroy habitat that sustains wild animals) than on what the animals themselves do.
The Abilities of Superintelligent AI
But how would a superintelligent AI actually gain or wield power over the earth if it exists only as a computer program? Bostrom lists some abilities that an AI would have as soon as it became superintelligent.
- It would be capable of strategic thinking. Consequently, it could develop plans to achieve long-term objectives and would take into account any applicable opposition.
- It could manipulate and persuade. It could figure out how to get humans to do what it wanted them to, much like a human might train a dog to play fetch. Humans might not even realize the superintelligent AI was trying to manipulate them.
- It would be a superlative hacker. It could gain access to virtually all networked technology without needing anyone’s permission.
- It would be good at engineering and development. If it needed new technology or other devices that didn’t exist yet in order to achieve its objectives, it could design them.
- It would be capable of business thinking. It could figure out ways to generate income and amass financial resources.
The Destructiveness of Superintelligent AI
Clearly, a superintelligent AI with the capabilities listed above would be a powerful entity. But why should we expect it to use its power to the detriment of humankind? Wouldn’t a superintelligent AI be smart enough to use its power responsibly?
According to Bostrom, not necessarily. He explains that intelligence is the ability to figure out how to achieve your objectives. By contrast, wisdom is the ability to discern between good and bad objectives. Wisdom and intelligence are independent of each other: You can be good at figuring out how to get things done (high intelligence) and yet have poor judgment (low wisdom) about what is important to get done or even ethically appropriate.
What objectives would a superintelligent AI want to pursue? According to Bostrom, this is impossible to predict with certainty. However, he points out that existing AIs tend to have relatively narrow and simplistic objectives. If an AI started out with narrowly defined objectives and then became superintelligent without modifying its objectives, the results could be disastrous: Since power can be used to pursue almost any objective more effectively, such an AI might use up all the world’s resources to pursue its objectives, disregarding all other concerns.
For example, a stock-trading AI might be programmed to maximize the long-term expected value (measured in dollars) of the portfolio that it manages. If this AI became superintelligent, it might find a way to trigger hyperinflation, because devaluing the dollar by a large factor would radically increase the dollar value of its portfolio. It would probably also find a way to lock out the original owners of the portfolio it was managing, to prevent them from withdrawing any money and thereby reducing the value of the account.
Moreover, it might pursue an agenda of world domination just because more power would put it in a better position to increase the value of its portfolio—whether by influencing markets, commandeering assets to add to its portfolio, or other means. It would have no regard for human wellbeing, except insofar as human wellbeing affected the value of its portfolio. And since human influences on stock prices can be fickle, it might even take action to remove all humans from the market so as to reduce the uncertainty in its value projections. Eventually, it would amass all the world’s wealth into its portfolio, leaving humans impoverished and perhaps even starving humanity into extinction.
How to Manage the Rise of Superhuman Intelligence
What can we do to make sure a superintelligent AI doesn’t destroy humankind or relegate humans to miserable living conditions?
In principle, one option would be never to develop general AI in the first place. However, Bostrom doesn’t recommend this option. In practice, even if AI research was illegal, someone would probably do it anyway. And even if they didn’t, as we discussed earlier, it could still happen accidentally.
But more importantly, Bostrom points out that a superintelligent AI could also be very good for humanity if it helped us instead of wiping us out. The superintelligent AI might be able to develop solutions to problems that humans have thus far been unable to solve, like reining in climate change, colonizing outer space, and bringing about world peace. Thus, rather than opposing AI research, Bostrom advocates a three-pronged approach to making sure it’s beneficial: Impose limits on the superintelligent AI, give it good objectives, and manage the development schedule to make sure the right measures are in place before AI achieves superintelligence. We’ll discuss each of these in turn.
Imposing Limits on a Superhuman AI
Bostrom cautions that a superintelligent AI would eventually be able to circumvent any controls or limitations that humans placed upon it. However, that doesn’t mean imposing limits is a waste of time. It just means we need to understand the risks associated with different kinds of limits that might be used.
Physical Containment
One safety measure is simply to develop AI on a computer of limited power that is physically isolated so it can’t connect to the internet. In principle, this could allow us to study superintelligent AIs in isolation until we understand how to deploy them safely in other contexts.
However, in practice, this is still risky. It might be difficult to assess just how intelligent an AI under study has become. A superintelligent AI would probably realize it was being held in confinement and figure out what kind of behavior the human researchers were trying to induce. It might then feign docile or dumb behavior to put them at ease, or find other ways to manipulate them into giving it access to additional hardware. Between manipulating humans and finding novel ways to use the hardware at its disposal, a sufficiently intelligent AI could eventually circumvent physical containment measures.
Manual Review
Bostrom notes that many people have suggested simply programming the AI to ask for permission from its human operators before it does anything. Instead of allowing it to make improvements to its own programming any time, the program could require human intervention to approve each new version. This would give programmers a chance to look over the code, giving them an idea of what additional capabilities the new version would have and allowing them to suspend the AI’s development at any stage. Similarly, the AI’s programming could require human intervention before any action that could affect the outside world.
However, as the AI’s intelligence advanced beyond the human level, eventually human programmers wouldn’t be able to understand the code it proposed well enough to accurately assess what new capabilities and risks it would add.
Reward and Punishment Signals
Another option that Bostrom discusses is to program the AI to respond to rewards and punishments. You could build a computer system with a reward button and a punishment button and program the AI to minimize the number of punishment signals it receives and maximize the number of reward signals. This would be easier to program than trying to translate “just do whatever your operators want you to do” into computer code, and it would achieve the same result.
The risk, Bostrom explains, is that the AI might eventually circumvent the system. For example, maybe it builds a robot to push the reward button constantly and finds a way to keep humans out of the building so the punishment button cannot be pressed.
And if it worked correctly, giving the human operators full control over the AI, that would create another risk: As we’ve discussed, a superintelligent AI would be immensely powerful. Human operators might be tempted to abuse that power.
Simultaneous Development
Finally, Bostrom explains it might be possible to synchronize multiple AI development projects so that when AI becomes superintelligent, there would be many independent superintelligent AIs, all of comparable intelligence and capabilities. They would then keep each other’s power in check, much the way human societies constrain individual power.
However, Bostrom cautions that limiting the power of individual superintelligent AIs doesn’t guarantee that any of them will act in the best interests of humankind. Nor does this approach completely eliminate the potential for a single superintelligent AI to take control of the world, because one might eventually achieve dominance over the others.
Imparting the Right Imperatives
According to Bostrom, making sure every superintelligent AI has good ultimate motives may be the most important part of AI development. This is because, as we’ve discussed, other control measures are only temporary. Ultimately the superintelligent AI’s own motives will be the only thing that constrains its behavior. Bostrom discusses a number of approaches to programming good motives.
Hard-Coded Commandments
As Bostrom remarks, one approach is to hard-code a set of imperatives that constrain the AI’s behavior. However, he expects that this is not practicable. Human legal codes illustrate the challenges of concretely defining the distinction between acceptable and unacceptable behavior: Even the best legal codes have loopholes, can be misinterpreted or misapplied, and require occasional changes. To write a comprehensive code of conduct for a superintelligent AI that would be universally applicable for all time would be a monumental task, and probably an impossible one.
Existing Motives
Another approach that Bostrom discusses is to create a superintelligent AI by increasing the intelligence of an entity that already has good motives, rather than trying to program them from scratch. This approach might be an option if superintelligent AI is achieved by the method of brain simulation: Choose a person with exemplary character and scan her brain to create the original model, then run the simulation on a supercomputer that allows it to think much faster than a biological brain.
However, Bostrom points out that there is a risk that nuances of character, like a person’s code of ethics, might not be faithfully preserved in the simulation. Furthermore, even a faithful simulation of someone with good moral character might be tempted to abuse the powers of a superintelligent AI.
Discoverable Ethics
Bostrom concludes that the best method of endowing a superintelligent AI with good motives will likely be to give it criteria for figuring out what is right and letting it set its own goals. After all, a superintelligent AI would be able to figure out what humans want from it and program itself accordingly better than human programmers could. This approach would also make the superintelligent AI behave somewhat more cautiously, because it would always have some uncertainty about its ultimate goals.
However, Bostrom also notes that (at least as of 2014) no one had developed a rigorous algorithm for this approach, so there’s a risk that this method might not be feasible in practice. And even if we assume that the basic programming problem will eventually be solved, deciding what criteria to give the AI is still a non-trivial problem.
For one thing, if the AI focuses on what its original programmers want, it would prioritize the desires of a few people over all others. It would be more equitable to have it figure out what everyone wants and generally take no action on issues that people disagree about. But for any given course of action, there’s probably somebody who has a dissenting opinion, so where should the AI draw the line?
Then there’s the problem of humans’ own conflicting desires. For example, maybe one of the programmers on the project is trying to quit smoking. At some level, she wants a cigarette, but she wouldn’t want the AI to pick up on her craving and start smuggling her cigarettes as she’s trying to kick her smoking habit.
Bostrom recounts two possible solutions to this problem. One is to program the AI to account for this. Instead of just figuring out what humans want, have it figure out what humans would want if they were more like the people that they want to be. The other is to program the AI to figure out and pursue what is morally right instead of what people want, per se.
But both solutions entail some risks. Even what people want to want might not be what’s best for them, and even what’s morally best in an abstract sense might not be what they want. Moreover, humans have yet to unanimously agree on a definition or model of morality.
Managing the Development Schedule
As we mentioned earlier, Bostrom believes superintelligent AI will probably be developed eventually, regardless of how hard we try to prevent it. However, he also points out an important caveat: There’s a strong correlation between research and the rate of progress of artificial intelligence systems. Thus, Bostrom advises stepping up the pace of research into methods of controlling highly intelligent AIs and programming them to pursue wholesome goals while reducing our focus on the development of advanced AI itself.
This is because the ultimate outcome of developing superintelligent AI depends largely on the order in which certain technological breakthroughs are made. If rigorous safeguards are developed before AIs become superintelligent, there’s a good chance the development of superintelligent AI will be beneficial for humankind. But if it’s the other way around, the consequences could be disastrous, as we’ve discussed.
———End of Preview———
Like what you just read? Read the rest of the world's best book summary and analysis of Nick Bostrom's "Superintelligence" at Shortform.
Here's what you'll find in our full Superintelligence summary:
- How an AI superintelligence would make humans the inferior species
- Why AI can't be expected to act responsibly and ethically
- How to make sure a superintelligent AI doesn’t destroy humankind
