Its Favorite Things

Another interesting story about a biological bird-like species living in nests that happens to develop a quirk which accidentally gets more and more widely propagated till there only remain birds who have that trait. Like the aliens from the previous chapter’s hypothetical story, there’s a boy-bird and girl-bird discussing what other species might be like, with the boy-bird insisting that they would all necessarily have an equivalent of “correct-nest”ness, with the girl-bird being more open to other possibilities.

The authors use this parable to pre-empt the opinion that perhaps a powerful alien AI mind will ask the questions that humans ask, e.g. “what’s the right thing to do?”, or want what humans want – a future full of flourishing people.

While I disagree with some of the authors’ ways of thinking, I agree with their conclusions.

For example, even if ASI asks the question, “what’s the right thing to do?”, my guess is that having humans around would not intentionally be part of the answer.

The other disagreement I have is that humans themselves – especially, but not only, the humans creating AI – have not the slightest ambition of a flourishing human race. Their flourishing really begins and ends with them, and many of them are tripping over themselves trying to give up their biological enclosure for a digital one by loading their “consciousness” into silicon to achieve immortality.

The authors then try to provide reasons against all the typical questions that naively hopeful humans ask:

Maybe we would be useful to ASI? Yeah, about as useful as horses were to us once we created mechanical horses.

Maybe it would keep us around as a trade partner? Apparently asked most by economists citing the “Law of Comparative Advantage” where a developed country keeps a poor country around for trading purposes. The authors contention is that if the ASI keeps growing more and more intelligent, there would be no reason for it not to just conquer humans and take over their resources.

Maybe it will need us to run the factories? Maybe in the beginning, but there will be no reason for AI to trust a species as fickle, unpredictable, slow and error-prone as humans to run their factories and power plants.

Maybe they will keep us as pets? Not likely. Enough said.

Maybe they will just let us be? They won’t. The analogy is that if you went to a billionaire and asked for 0.2 percent of their wealth to do something they don’t care about, you will most likely get a “no”. Similarly, the fact that the earth is only 0.2 % of the mass of the solar system, but there will be no reason for ASI to share that with anyone else. An earth in hand…

Let’s face it. If humanity is a major inconvenience for the billionaires, what are the chances of ASI thinking differently? The next page or two is a short riff on 50 ways to kill your maker. Quicker.

It’s theoretically possible to steer AI towards joy and beauty and wonder, but it would take crafting, and there are zero takers for that right now.