1. Kaj Sotala

   A being of higher intelligence could outsmart us, and maybe they’ll consider wiping us out a “smart move”. We might make a mistake while developing them which causes them to go rampant. They might think we’re so petty that they won’t care about our survival.

   I disagree with the level of hypothetical-ness of some of these points. A being of sufficiently higher intelligence can outsmart us (that’s pretty much what the definition of “intelligence” means!). I won’t take a stance on whether they might consider wiping us out a “smart move” or not, but like I mentioned in the other comment thread, we know that the default stance of a mind is to be indifferent towards the survival of other creatures. We know that currently AI safety is an issue that is entirely ignored by researchers - and has always been that, even when AI was thought to be just behind the corner, and we also know that evolution has given us plenty of evidence which suggests that features of a mind that were born to make a system behave in a “good” way in a certain environment easily break down when the environment changes. Furthermore, it’s questionable of whether or not we even know how we’d want an effectively omnipotent mind to behave - it’s not that the technical implementation of a goal is difficult, it’s that we’re not even sure of what the goal is. Thus we have good reasons to believe development of Friendly AI will be both very difficult and very dangerous, as a default assumption.

   While these are hypothetically possible, they’re only a fraction of the potential outcomes.

   This might be getting into semantics, but - the space of minds that are indifferent towards mankind’s survival is much larger than the space of minds that care about mankind’s survival, which in turn is larger than the space of minds that care about mankind’s survival in a way that we’d want them to care. It is much easier to build an AI in a way that it doesn’t care about us than in a way that it cares. I would argue that the amount of outcomes where we develop AI and benefit from it is only a fraction of the amount of outcomes where we develop AI and don’t benefit, not the other way around.

   By educating instead of scaring people, we also avoid the risk of public investors withdrawing funds from AI research because they’re afraid, or because the product isn’t in demand as the public avoids AI.

   I agree that we should educate people, but I disagree about “educating” and “scaring” people being mutually exclusive. Educating people about the dangers of anything will make them more scared of it than they’d be if they were ignorant of its dangers. I am educating people about the dangers of AI by typing this very comment.

   That, in my opinion, would be your worst nightmare: It would relent public control of AI development and make the military one of the most powerful AI developers!

   But simply because the public is scared of something doesn’t mean that only the military will be developing it. The public is scared about the dangers of defective medicine, which is why we have governmental agencies regulating medicine and demanding controlled trials of it before any new drugs are accepted to the market. The public is slowly growing more scared about the dangers of nanotechnology (as they should be), which is why different governmental agencies are starting to investigate into its dangers and consider the forms of regulation that might be necessary.

   On the other hand, even if the public is ignorant, the military probably will be aware of an AI’s potential. DARPA is known for experimenting with all sorts of programs that seem far-fetched at the moment, but may have potential. Scaring/educating the public will not make AI development something that’s practiced exclusively by the military - a scared public is one that will pressure their governments to look into the matter and expose the subject matter to more debate. Not talking about AI’s dangers is far more dangerous, since it means that a far smaller number of people are involved in studying the subject matter.

   Apr 27th, 2007

2. John Pozadzides

   Hrafn,

   I’ve remained largely silent on this issue because I know little about AI. However, I fear AI, and I mean that exactly how it sounds. Of course, I also fear just plain old “I”, and here are a few reasons:

   - Machines are succeptable to sabotage. I don’t have to worry about an airline pilot catching a virus and flying a plane into the ground, but I do worry about an automated fleet of AI pilots losing their minds because a hack3r thought it would be c00l.

   - Machines are subject to malfunction. Even the most sophisticated devices and networks humans have created have public and notable failures. The more impressive a machine is, usually the more people are affected when it fails.

   - Machines are immoral. They do not have, nor can they comprehend, a set of morays or social norms. They are endowed with the absolutes programmed in by their creators and lack the capacity for morality. This means that if a machine were to kill a human there would be no remorse and punishment has no consequence. In humans we would describe this complete apathy as “psychotic”, so it cannot simply be dismissed because an action is carried out by a machine.

   - Machines are cold. They are neither born, nor do they die. Any accumulated knowledge can be transferred and continued in another machine. For this reason they will never, ever, ever comprehend the value of life.

   The only valid argument that I can come up with for the existance of AI, or machines in general, is that they can serve as useful tools to benefit humanity. Although a Furby or Aibo pose little risk to the people around them, AI machines 20 years from now could be much larger, more powerful and capable of wreaking serious havoc under any of the previously stated scenarios.

   So, I believe there is plenty to fear from AI, and fear is both rational and helpful. Without it we would be exposed. With it we can at least try to prevent issues and plan for contingencies when they occur. I don’t view pointing out the theoretical weaknesses as fearmongering- at least not if they are presented rationally (as I hope I have). I view them as opportunities for experts to demonstrate how they have fully accounted for these potentials in the design and implementation of the machine.

   In other words, if a particular AI is truly safe one would not be able to say “…sure that is theoretically possible, but I don’t think its going to happen.” Instead we’d hear “…every aspect of this system is designed with redundancy, there are disparate monitoring systems which can override and disable in case of a malfunction, and it has been tested for 1,000,000 hours without a single failure.”

   John

   Apr 27th, 2007

3. hthth

   I disagree with the level of hypothetical-ness of some of these points. A being of sufficiently higher intelligence can outsmart us (that’s pretty much what the definition of “intelligence” means!).

   Haha. Good catch. However, the hypothetical aspect I was emphasizing was making assumptions about the function of a system we know so little about. Current AI systems aren’t even close to human levels of intelligence, let alone superhuman.

   we know that the default stance of a mind is to be indifferent towards the survival of other creatures.

   Like I pointed out in my latest response in the other thread, there’s a whole lot of conclusions we’d have to jump to at this point to make predictions about whether a higher intelligence would be indifferent towards humans.

   Furthermore, it’s questionable of whether or not we even know how we’d want an effectively omnipotent mind to behave - it’s not that the technical implementation of a goal is difficult, it’s that we’re not even sure of what the goal is. Thus we have good reasons to believe development of Friendly AI will be both very difficult and very dangerous, as a default assumption.

   I think you’re confusing ‘omnipotent’ with ‘omniscient’. But in any case, I thought your main fear and argument was that a super-human intelligence would come up with goals on its own, or goals caused by accident (unwanted behavior)?

   This might be getting into semantics, but - the space of minds that are indifferent towards mankind’s survival is much larger than the space of minds that care about mankind’s survival, which in turn is larger than the space of minds that care about mankind’s survival in a way that we’d want them to care. It is much easier to build an AI in a way that it doesn’t care about us than in a way that it cares. I would argue that the amount of outcomes where we develop AI and benefit from it is only a fraction of the amount of outcomes where we develop AI and don’t benefit, not the other way around.

   Ok. This doesn’t convince me because (a) you seem like you know for certain the total space of all constructible minds, when we don’t even know how a mind can be constructed or what its limits are, (b) the space of all minds above the level of human intelligence is not comparable to those below (how a stupid person thinks gives us very little indication of how a smart person thinks), (c) you seem to know how to build a human-level AI, (d) I think you’re forgetting how many problems exist in the world that an AI could help solve. See more on this last point below.

   I agree that we should educate people, but I disagree about “educating” and “scaring” people being mutually exclusive. Educating people about the dangers of anything will make them more scared of it than they’d be if they were ignorant of its dangers. I am educating people about the dangers of AI by typing this very comment.

   Yes. You are educating people about the dangers of AI. What you’re not doing is educating people about how AI works (today). You can convince people that a certain object is dangerous, or you can educate people about how the object works and make them draw their own conclusion.

   The second option will (a) make more people capable of preventing bad consequences of AI, (b) make people more knowledgeable about world, (c) increase the amount of people who know how to put AI to good use, (d) they’ll actually know why it’s dangerous, or alternatively, they’ll be able to tell you why you’re wrong. Sounds a lot more civil and scientific to me
   But as a side-note, admittedly there will probably always be people who don’t want to be educated or won’t care about the inner workings.

   But simply because the public is scared of something doesn’t mean that only the military will be developing it. The public is scared about the dangers of defective medicine, which is why we have governmental agencies regulating medicine and demanding controlled trials of it before any new drugs are accepted to the market.

   That’s true. The difference is that medicine is a reality, human-like AI is not. As we come closer to creating one, I’m quite sure that our precautions will increase in parallel. Working in the field of AI — there are very, very few systems today that appear to have any potential of spawning near (10 year old) human intellect. And those that might still have a long road ahead of them.

   The public is slowly growing more scared about the dangers of nanotechnology (as they should be), which is why different governmental agencies are starting to investigate into its dangers and consider the forms of regulation that might be necessary.

   Exactly. As we get closer to potentially dangerous technologies, we grow more cautious.

   Scaring/educating the public will not make AI development something that’s practiced exclusively by the military - a scared public is one that will pressure their governments to look into the matter and expose the subject matter to more debate.

   A scared public didn’t prevent the construction of the A-Bomb. A scared public, however, doesn’t buy and support the production of medicine that they think is dangerous.

   Not talking about AI’s dangers is far more dangerous, since it means that a far smaller number of people are involved in studying the subject matter.

   I never said we shouldn’t talk about the dangers of AI. I said we should talk about both the dangers and the benefits (more precisely, I said we should educate people about AI, meaning the benefits, how it works, and the potential dangers).

   Hi John! Welcome aboard the debate train. I hope you’ll have a pleasurable ride, someone will be around shortly to collect your ticket. Dinner will be served at 7pm. Kaj, please don’t stop reading — my last point will be a ‘rant’ on why I think the potential benefits should overpower our fear at this point.

   Machines are succeptable to sabotage. I don’t have to worry about an airline pilot catching a virus and flying a plane into the ground, but I do worry about an automated fleet of AI pilots losing their minds because a hack3r thought it would be c00l.

   Humans are susceptible to error, terrorism, biological viruses, illnesses and injuries. While I think this is a noteworthy point (and a lot more relevant than fear of human-level AI), I still think it only presents similar problems as we know today with human pilots.

   Machines are subject to malfunction. Even the most sophisticated devices and networks humans have created have public and notable failures. The more impressive a machine is, usually the more people are affected when it fails.

   Yes, that’s true. And if this was any other kind of technology, I would agree. But as machines grow smarter and more capable of handling different problems, they also grow more capable of preventing and handling malfunctions. In AI, the term graceful degradation is used to describe the intelligent process of handling errors without “completely shutting down”.

   That’s not to say we shouldn’t worry about these things. I’ve never said anything to that extent. I’ve just said that at this point at time, I see more benefits and that they overpower any fear for me (see below).

   Machines are immoral. They do not have, nor can they comprehend, a set of morays or social norms. They are endowed with the absolutes programmed in by their creators and lack the capacity for morality.

   Yes. I completely agree with this. Todays machines do not have morality built in. If such a machine kills someone intentionally, its creator should be held responsible. That’s, in a way, similar to using a weapon.

   Machines are cold. They are neither born, nor do they die. Any accumulated knowledge can be transferred and continued in another machine. For this reason they will never, ever, ever comprehend the value of life

   I’m not so sure about this point. If you’re talking about a conscious machine, it can understand that it will cease to exist if we turn it off. If you’re talking about a non-conscious machine, it can presumably be programmed to respect and value life. But this is speculation as we don’t really know how these machines will work.

   So, I believe there is plenty to fear from AI, and fear is both rational and helpful. Without it we would be exposed. With it we can at least try to prevent issues and plan for contingencies when they occur. I don’t view pointing out the theoretical weaknesses as fearmongering- at least not if they are presented rationally (as I hope I have). I view them as opportunities for experts to demonstrate how they have fully accounted for these potentials in the design and implementation of the machine.

   As always, John, you present your points rationally, and so do you Kaj. And I wholeheartedly agree with both of you that there are potential dangers and that fear can help us prevent disasters (I said that somewhere in a comment around here… but the comments have gotten so long I can’t find it).

   My main argument all this time has been that the potential benefits of AI really overpower any feeling of fear I have towards it, at this point in its developmental history. You speak of potential existential risks for the human race? Well, the human race is already at risk! Our society, technology and behavior has gotten so complex that we lack manpower to handle it, our oversights are causing things like global warming (we can’t even figure out a way to empirically prove whether or not we’re causing it!).

   And what about diseases? Our minds are limited and we lack human resources, we lack intelligence to ponder solutions and process the gigantic information overflow of information we’re witnessing. We lack simulations and machines to understand the complexities that are our human bodies.

   What about world hunger? Increasing the level of automation might provide us with more free time and wealth to help each other. Third world support funds could be increased by having computers analyze government cash-flow, maximize profits, minimize spending.

   Electricity expenditure reduced by intelligent machines analyzing road conditions and traffic, providing optimized lighting. Traffic control could be optimized, accidents responded to faster with smart cameras watching the roads.

   What about predicting natural disasters? Intelligent satellites are now in orbit and have accurately been able identify and monitor volcano eruptions without human intervention. Intelligent machines can similarly be trained to monitor earthquakes, alert people faster. Robots can be used in dangerous rescue missions, instead of putting human lives at stake.

   Robotic surgeons are steadier than a human hand. CyberKnife is a robot surgeon that uses AI to perform surgeries that are very difficult for humans, such as removing tumors close to the spine.

   What about monitoring your grandmother? With a robot in the house that can understand whether she has forgotten to take her pills, or if she has fallen to the floor could give her longer time on this planet. What about monitoring you when you’re old?

   What about prosthetics? AI technologies have the potential to give handicapped people a new life. The Icelandic company Ossur is leading edge in this kind of tech and its AI knee which learns and responds to the movements of its user gives a whole new dimension of flexibility and feeling of actually having your limb back.

   Searching the web? categorizing scientific knowledge? processing data faster? monitoring crimes on the street? optimizing productions? controlling space probes?

   These are all potential benefits we’re seeing and will continue to see before machines reach human-level intelligence. The world is a very, very complex place, and we have limited time and resources. We need more intelligence to help us.

   This is really just a portion of the possibilities I can think of, but I hope this list of mine has provided you with some food for thought regarding the immediate benefits of AI — and an insight into why I consider the benefits overpowering.

   I think this is the longest comment I’ve ever written online. Haha.

   EDIT: I had to write this in a hurry. If I come off as agressive or blunt, it was not my intention to offend anyone and I apologize. Similarly, please excuse any typos

   Apr 27th, 2007

4. Kaj Sotala

   Again, sorry for the late response.

   I think you’re confusing ‘omnipotent’ with ‘omniscient’. But in any case, I thought your main fear and argument was that a super-human intelligence would come up with goals on its own, or goals caused by accident (unwanted behavior)?

   Not confusing them (even if I think they amount to the same thing). Omnipotent = can do anything, omniscient = knows everything.

   And yes, the main fear is that an AI will carry out unwanted behavior. It’s even more likely to carry out unwanted behavior if we don’t even know what the wanted behavior is.

   (a) you seem like you know for certain the total space of all constructible minds, when we don’t even know how a mind can be constructed or what its limits are

   Well, obviously I don’t know it for certain. However…

   …if we consider a hypothetical space of all constructible minds, it would seem like a very bold assumption that a particular subset of them (minds friendly towards mankind’s survival) would outnumber the rest. Intuitively it feels wrong, since it seems reasonable that for each mind that cares about mankind’s survival, there’s an equal amount of minds that are mostly similar but instead care about the survival of, say, a certain pretty rock - and another bunch of similar minds that care about the survival (or should we say continued existence) of the rock next to the first one, and a bunch that cares about the survival of the rock next to that one, and…

   Certainly I would place the burden of proof on the one suggesting that the space of minds indifferent to mankind’s survival is not bigger than the space of minds that cares about it. Mankind would have to be somehow very unique for that proposition to hold.

   (b) the space of all minds above the level of human intelligence is not comparable to those below (how a stupid person thinks gives us very little indication of how a smart person thinks)

   I must admit that I’m not entirely sure of what you mean by this in this context.

   (c) you seem to know how to build a human-level AI

   I never implied that.

   (d) I think you’re forgetting how many problems exist in the world that an AI could help solve.

   Oh, I never disputed that a properly constructed AI couldn’t help us solve all our problems. There’s nothing that I’d rather see happen.

   The problem being with “a properly constructed”, of course. Even without going to the issue of how AIs might get built, I think we can agree that for each set of conditions where humanity’s problems are solved, there are many more situations which we both would consider miserable for humanity. I think we can also agree that there are more ways for things to go wrong in the construction of AI than ways for things to go right, regardless of the exact implementation.

   You can convince people that a certain object is dangerous, or you can educate people about how the object works and make them draw their own conclusion.

   When educating people about the risks of anything, one usually tries to express the potential danger as fast as possible, then go about explaining why exactly there is a danger. Both because people might simply not happen to think about all the implications first, and because if the potential danger isn’t communicated fast enough they might think that the issue isn’t interesting enough for them to listen (or think) through the whole thing. You first express your worry about the issue, then explain your reasons for it in more detail, and then let them judge the validity of your reasons. Which is what I’m doing right now - and I would think that I’d have explained enough about my reasons for worry and about potential ways to deal with the issue that I’d have been fulfilling your points a-d.

   I do admit that I could talk more about the benefits of AI, too, since I do think that the creation of a genuine Friendly AI would be, well, The Best Thing Ever. But currently it feels to me like there’s a much bigger chance for AI to turn out to be The Worst Thing Ever. And talking too much about AI’s benefits is potentially dangerous in itself, since people might get too caught up in the good things that might follow from the creation of true AI and rush to build one - neglecting the need for caution.

   Exactly. As we get closer to potentially dangerous technologies, we grow more cautious.

   Only issue being that if the technology is developed fast enough (and we can’t say that it wouldn’t be), awareness of it might not have enough time to spread before it actually becomes dangerous.

   A scared public didn’t prevent the construction of the A-Bomb. A scared public, however, doesn’t buy and support the production of medicine that they think is dangerous.

   Public awareness of the A-Bomb wasn’t very widespread when it was being developed, I believe. (Not that I think it’d be a very good comparison in any case, but I’m too lazy to get into that right now. :>) A scared public will pressure their governments to doing something, and widespread worries will influence any independent AI developers to be more careful, even without government interference.

   Well, the human race is already at risk! Our society, technology and behavior has gotten so complex that we lack manpower to handle it, our oversights are causing things like global warming (we can’t even figure out a way to empirically prove whether or not we’re causing it!).

   I agree. Personally, I even think that there is a very serious risk that we will be wiped out during the next 50 years, though I would not count global warming and diseases as the worst risks. True artificial intelligence most likely is the only way by which we can even try to guarantee our survival. Ultimately, there are only two possible futures for humanity - friendly superintelligence or extinction.

   It is unfortunate that non-friendly superintelligence is such a good way of guaranteeing our non-survival, as well.

   May 4th, 2007

5. hthth

   Hi Kaj, don’t worry about it. Busy days back here as well.

   …if we consider a hypothetical space of all constructible minds, it would seem like a very bold assumption that a particular subset of them (minds friendly towards mankind’s survival) would outnumber the rest.

   It seems quite obvious to me that a large part this conceptual space will be excluded, especially when considering that most AI technology is being created specifically to help the human race in one way or the other. To what extent and how future AI’s will think and operate, e.g. if it’ll consider the survival of a rock of greater importance than the survival of human, seems too far away for me to stand up and shout danger to the world.

   Mankind would have to be somehow very unique for that proposition to hold.

   You mean something like being the only species that base their survival on the use of tools? And like being the actual creators of the mind question?

   I’m not that interested in arguing this point. It’s twofold; from the standpoint of a human level intelligent machine, and from the standpoint of a machine of lower intellect. Both are too hypothetical/philosophical for my taste (compared to some of our other arguments).

   I must admit that I’m not entirely sure of what you mean by this in this context.

   I meant that we can’t know whether a higher level of intelligence will consider the human race (or any organism) petty or important, until we understand the function of natural and artificial general intelligence to a greater extent.

   I never implied that.

   In order to associate any kind of probability to the potential trajectory of AI systems through the “conceptual space of minds”, you have to have an idea of how the systems will function. Just as we need functional knowledge of how a jeep will work in order to predict what kind of landscape it will be able to travel through — its design sets the boundaries for its potential.

   The problem being with “a properly constructed”, of course. Even without going to the issue of how AIs might get built, I think we can agree that for each set of conditions where humanity’s problems are solved, there are many more situations which we both would consider miserable for humanity. I think we can also agree that there are more ways for things to go wrong in the construction of AI than ways for things to go right, regardless of the exact implementation.

   We have to weigh this risk against the risks the human race is already facing. I’m not sure I agree that there are more things that can go wrong, as it depends on the implementation.

   Theoretically I agree with you, yes, if we don’t consider the implementation the imaginable set of miserable outcomes is great (and actually a lot greater than if we consider the implementation). But I need the implementation specs to weigh the potential risks; that’s the physical evidence I need — the “imaginable” dangers of AI will never weigh as much in my mind compared to the actual, physical problems we’re facing today and that AI can solve.

   I think this is a large part of our differences in opinion.

   You first express your worry about the issue, then explain your reasons for it in more detail, and then let them judge the validity of your reasons. Which is what I’m doing right now

   [...]

   I do admit that I could talk more about the benefits of AI, too, since I do think that the creation of a genuine Friendly AI would be, well, The Best Thing Ever. But currently it feels to me like there’s a much bigger chance for AI to turn out to be The Worst Thing Ever.

   [...]

   I agree. Personally, I even think that there is a very serious risk that we will be wiped out during the next 50 years, though I would not count global warming and diseases as the worst risks. True artificial intelligence most likely is the only way by which we can even try to guarantee our survival. Ultimately, there are only two possible futures for humanity - friendly superintelligence or extinction.

   Okay. What I don’t understand (or rather, the fundamental point where I disagree) is why you would emphasize the future risks of AI rather than its benefits, when you think that we’ll be wiped out in the next 50 years and that AI is possibly our only chance for survival? 50 years is not a long time — If I would be in your position, I would be more worried that AI won’t advance fast enough to meet the deadline, even if I’d think AI could potentially be a cause for extinction.

   Here’s how I see it: In proportion to the immense importance of AI technologies to the future of mankind — technology that can advance all sciences — I think there’s not nearly enough funding being put into AI research. A large number of people consider AI fantasy or think that machines can never reach our intelligence, for example. Every oppositional force or negative review contributes to less funding for AI.

   In light of the immediate benefits of AI technologies (next 5 years, for example) and the current problems mankind is facing, I consider it more important, and even necessary, to emphasize the benefits to further the advancement of AI.

   Best regards!

   ps.:

   a genuine Friendly AI would be, well, The Best Thing Ever.

   Haha, The Best Thing Ever, made me laugh out loud
   We should print that on t-shirts.

   May 7th, 2007

6. Kaj Sotala

   Looks like we’ve reached the “just have to agree to disagree” point on most of the issues discussed.

   Okay. What I don’t understand (or rather, the fundamental point where I disagree) is why you would emphasize the future risks of AI rather than its benefits, when you think that we’ll be wiped out in the next 50 years and that AI is possibly our only chance for survival?

   Hmm, good question - because I do remember having spoken about the benefits of AI a lot, too. (Even if lately I’ve been concentrating on the bad sides more.) I think a large reason is the fact that this discussion got started with me commenting on your “Who’s afraid of robots” entry with a comment explaining why exactly we should be afraid, for benefit of those who doubted its risks - had I ended up here because of a “Who thinks AI is going to be important” post, my emphasis might have been different.
   Still, that’s hardly the only reason. I suppose the main issue is that an AI at the right stage might potentially become practically omnipotent in a very short time. One has to be immensly careful when we’re talking about anything that has unlimited power and whose behavior we can only affect before it becomes super-powerful. I’d like faster AI development to save us from existential risks, yes, but faster development is no good if it ends up destroying us.

   (By the way, since we both agree that it would be good if safe AI would be developed faster - the Singularity Institute for Artificial Intelligence has a donation challenge going on, every dollar donated up to the $400,000 target will be matched by Peter Thiel… I’m already a regular donor, and will be giving them more to this challenge once I know how much extra money I have to spare this month. )

   May 10th, 2007

7. hthth

   Looks like we’ve reached the “just have to agree to disagree” point on most of the issues discussed.

   Indeed. And it doesn’t surprise me. The depth of the concepts and ideas discussed here takes time to evaluate and consider. Considering the time each of us seems to have spent developing our opinions, I wouldn’t have expected us to change our minds in only a few weeks. It takes time to understand each other’s argument to the “full extent”.

   Hmm, good question - because I do remember having spoken about the benefits of AI a lot, too. (Even if lately I’ve been concentrating on the bad sides more.)

   If you look back, it’s the reverse here. I’ve been speaking and acknowledging the dangers but emphasizing the benefits. Looks like we both enjoy inciting debates. We should, they excel our knowledge

   I think a large reason is the fact that this discussion got started with me commenting on your “Who’s afraid of robots” entry with a comment explaining why exactly we should be afraid, for benefit of those who doubted its risks - had I ended up here because of a “Who thinks AI is going to be important” post, my emphasis might have been different.

   Hehe, yeah. After all this discussion, I don’t think we we disagree about anything. The difference between you and me seems to be the amount of emphasis we’re putting on benefits versus risks / safety precautions.

   Still, that’s hardly the only reason. I suppose the main issue is that an AI at the right stage might potentially become practically omnipotent in a very short time. One has to be immensly careful when we’re talking about anything that has unlimited power and whose behavior we can only affect before it becomes super-powerful. I’d like faster AI development to save us from existential risks, yes, but faster development is no good if it ends up destroying us.

   I agree. Yet fundamentally, I see it differently. Partially because I have no scientific assurance of the imminence of human-leveled AI’s (and their dangers, whose implementation I consider a prerequisite for precaution). Also because I see negative emphasis potentially slowing down AI progress (at least when subjected to the general public).

   But mostly because I see the physically present risks and damage we’re suffering from today. I’d rather spend my time planning the solution of real problems, than preparing for potential problems of that non-existent solution.
   In any case, like I said in the beginning of this comment. These are big ideas and require time to think about. I’m sure we’ve both provided each other with issues that we’ll continue to ponder for an indefinite amount of time
   p.s. For now, my only contribution to SingInst is to practice precaution in my current and future AI studies You’ve got me motivated enough to take a closer look at their operation, though.

   May 12th, 2007

Reply to “AI and Scare Tactics, A Tale of Two Species”