There are of course a great number of books and movies about the dangers of robots taking over the world. This kind of science fiction is generally not taken seriously or considered relevant for current AI research. This is mostly due to the improbable nature of the scenario’s. The possibility of creating some kind of computer with some kind of elaborate philosophy, one that we can sit next to and have a conservation with, is indeed quite remote.
Yet I consider AI research a possible threat to humanity that we should definitely be aware of. To explain I first have to talk about weak AI and strong AI. Weak AI, also called applied AI, is artificial intelligence applied for a narrow and fixed purpose, e.g. face recognition or data mining. Strong AI is intended to be more broad and general purpose, much like human intelligence. Whether that includes self-awareness is not relevant for this post. Weak AI has made steady progress and is being used today in many places. The development of strong AI has been slow and so far mostly without results.
One important issue of strong AI is whether it can be made sufficiently smart to exceed human intelligence. In fact, it is a stated research goal to investigate this very question. It is obvious that whenever we are able to make an AI that exceeds our capabilities, that AI should also be able to make an AI that exceeds its own. Since our computers are still exponentially improving by the decade, it seems likely we will face this situation, and it can happen in our lifetimes.
There is a final and important ingredient to my point. I used to consider the situation above worrying but not critical, since an AI will not likely have the motivation to improve itself or cause any harm. The well known three laws of robotics and their shortcomings should never come into play, since an AI in principle lacks all the different motivations that we humans have. But I have come across a compelling argument that changes this. The argument is that an AI must always have some purpose, otherwise it will not do anything. And it is in fact quite probable that any purpose, however simple, may end up in a catastrophe.
A simple example was thought up by Nick Bostrom, namely a computer that will maximize its reward by the number of paperclips it can make. We hook it up to a 3D printer and see what happens. I quote Anders Sandberg:
At first nothing seems to happen: the AI zooms through various possibilities. It notices that smarter systems generally can make more paper-clips, so making itself smarter will likely increase the number of paper-clips that will eventually be made. It does so. It considers how it can make paper-clips using the 3D printer, estimating the number of possible paper-clips. It notes that if it could get more raw materials it could make more paper-clips. It hence figures out a plan to manufacture devices that will make it much smarter, prevent interference with its plan, and will turn all of Earth (and later the universe) into paper-clips. It does so.
After giving it some thought I came to the conclusion that is indeed very tricky to give a true strong AI a safe reward function. Therefore I consider strong AI dangerous and since its dangers far exceed the potential benefits any research in this field should be stopped.