Autonomy and sentience
Autonomy, really, is the ability to act on ones own. An AGV or a robot or any piece of automated equipment can seem to be acting on it’s own but it is only acting on the instructions of it’s programmer. So what is it that causes real autonomy? Is real autonomy just an illusion and we all act only on the basis of our programming? That is another question altogether, and maybe even a philosophical or theological one, which I won’t discuss here.
The evidence, though, would suggest otherwise. We didn’t drag ourselves from mud huts to an advanced technological civilization by following a program- and if we did, it was written by a paranoid schizophrenic. Religious or not, most people have a specific attitude about free will, and free will is not something we can give a machine. Yet.
Even having free will doesn’t mean having the ability to use it for much. Most of what people use their free will for is to decide whether to have the water or the coke, or the salad or the triple cheeseburger. Not many people will use their free will to invent an airplane or develop string theory. Or even write a song or even a haiku.
Quantum theory tells us we will be able to make faster and faster computers by changing the way computers process information- in the late 90’s a computer called ‘Deep Blue” beat Kasparov, but in reality, it did not “Think”, it simply evaluated all the possible available moves at lightning speed- and Kasparov himself felt that more than the allowable amount of human intervention was taking place. And Quantum will most likely be no different; it will be able to examine different potential scenarios more rapidly, but it will still only act on the programming it receives.
Imagination seems to be the missing factor between that which is programmed to mimic autonomy and that which is capable of true autonomy. We probably don’t have a particularly good idea how imagination works or what drives it, so we probably cannot replicate it in machines, at least at present.
Change takes place because of imagination. If you’re cold, you imagine being warm, and you try to do the things that make you warm. if you’re hungry, you imagine being full, and try to do the things that make that happen. But dogs do that. What dog’s probably don’t do is see the way a leaf floats on the wind, and think, what if I make my own leaf? Can I make it float on the wind? Whoever made the first kite understood this thought process, and the gradual transition from kite to flight was filled with dreams all the way. The apes do this in some small ways, but they don’t generally improve on it, though they have had all the time in the world to do so.
Is it, then, our imagination- our ability to conceive of something that never existed before and strive to make it a reality- that separate us from other creatures? And if that’s what allows us true autonomy- assuming we have true autonomy, can that be programmed? Curiosity and imagination are not something that can be mimiced by a random number generator.
We don’t have to worry about Skynet becoming self aware. We need to worry about it writing a sonnet. When it does that, we are in the shit.

Free will promotes change. Something the critters do with some kind of external stimuli.
I think your assessment is correct as the animals are aware of self.
But I’ve not heard of one creating a sonnet. Some think the whales compose sonnets, but I doubt that. We just cannot identify the stimulus.
Random thoughts spinning into the light.
The other key aspect (or, at least, another) you touch on right at the end; spontaneity.
How do you build the capability (as opposed to teaching someone ignorant of their inherent capability) to recognize an event, imagine how it might be replicated, and then spontaneously decide to apply all that to an entirely other application?
You can process all the data there is, in the merest instant, and you’re still not “thinking”.
There seems to be some spark of something that takes people from the idly thinking place to the “Aha!” location.
Maybe Terry Pratchett is correct in that creativity and new ideas are always sleeting through the air; you just have to have the right equipment to collect them as they go past you.
Sometimes it’s entirely logical and step by step; sometimes it’s something that just appears out out of the blue or as if in a dream. Sometimes it’s an Underpants Gnome kind of thing with a “?” step in the middle between problem and solution. We just leap over that question mark somehow.
No, I don’t seem machines doing this anytime soon. You’re the expert here but I don’t think we can teach machines how to catch that spark. We can’t even teach people how to do it!
Jenny
I had a thought once but I left it in my other pants.
Nah. I was trying to get in your other pants earlier and I didn’t see it.