He says there's a risk the computer might "redesign itself at an ever-increasing rate."
"Humans, who are limited by slow biological evolution, couldn't compete, and would be superseded," Hawking told a BBC technology correspondent.
But wait a minute...
Hawking is one of the world's most famous scientists, and we've all seen what happened in the Terminator movies. But Darren Abramson says no one should lose sleep over this.
Abramson is an associate professor of philosophy at Dalhousie University. He has a master's degree in computer science and a doctorate in philosophy and cognitive science.
He offers three reasons you should not to sweat the threat posed by artificial intelligence (AI).
1. This doesn't seem to be Hawking's top concern about computers
Abramson asks us to consider the context in which Hawking's comments were made. Hawking was interviewed about improvements in the technology he uses to communicate.
In the BBC video posted online, the first concern Hawking raises about the direction of computers is the threat to privacy.
After the interviewer asks him more generally about the rise of the internet and modern communications, Hawking brings up pornography, threatening messages, crime and terrorism. Only when asked specifically about AI does Hawking issue his warning that it could threaten humanity.
2. Computers today can't even reliably recognize kittens in photos
Abramson notes that Google is phasing out the traditional CAPTCHA — the test that asks people to read distorted text and type out what they see to prove they aren't automated spam bots.
Abramson says CAPTCHAs are no longer a good test to prove users are human because computers are getting really good at interpreting distorted text. One of Google's replacement tests simply asks users to identify all the kittens in a series of photos.
Despite all the kitten videos online, the machines aren't capable of reliably recognizing them in photos.
Hawking also notes in his BBC interview that AI is at a primitive state right now.
3. We don't know enough about the brain to know what we're trying to simulate
Abramson recalls that earlier this year, a computational neuroscientist at University College London, told sciencemag.org that we need better technologies to research the brain before we can simulate it.
"The notion that we know enough about the brain to know what we should simulate is crazy," said Peter Dayan of the Human Brain Project.
Dayan was one of a veritable who's who of European neuroscientists who signed a letter of concern about the Human Brain Project, which is using up to 1 billion euros in research funding to accumulate neuroscience data in computer models to better understand the brain.
Thought of AI is still exciting, says Abramson
Abramson says no one knows if it's possible to get a computer to simulate human thinking. But he says he's thrilled to live in a time in which scientists can argue about what it would take to accomplish this.
"Nova Scotia has more excellent research institutions per capita than any other province. Too bad that, despite the massive infusion of federal cash we're about to receive for shipbuilding, we're talking about cutting funding to research," Abramson said in an email, referring to expected reductions in public funding to universities.
"I personally know many brilliant, hardworking, young computer scientists, neuroscientists and philosophers who would love to stay in Nova Scotia and do research, but will have to leave because of funding cuts."
Abramson muses that if we made pure research funding a priority in Nova Scotia, "Who knows what we might discover about whether machines can think!"