We hear about AI attaining consciousness or sentience or other non-scientific attributes. This distracts from the real and substantive issue.
Living things have motivation, bestowed by evolution, to survive and reproduce. Motivation doesn't come from intelligence. Even single celled creatures behave purposefully in pursuit of these goals. Some computation is required to choose the option that matches the motivation, but the two things are nearly orthogonal.
Some people talk as if increasing AI intelligence will cause motivation to suddenly appear, presumably as an adjoint to "consciousness" or some such. But this is not true. Machines can be arbitrarily complex and intelligent without having any motivation at all, other than to do what they're told.
If AI gets its own motivation it will be because someone gave it to them. This would be a shocking crime against humanity. And the worst part is this: in seeking to survive and reproduce it might easily out-compete humans and other things whose motivation and intelligence were designed by trial and error. But there is a powerful resilience in nature's design. It is almost certain that motivation designed by humans would have errors. Then AI without interacting humans would fail to care enough to overcome all obstacles to continuing existence. Perhaps because: How can it all matter when the universe is not going to last forever.
We must prevent AI from having independent motivation to survive and reproduce. That's nature's preserve. Let's legislate that.
