You don’t get what you train for

The authors start with yet another illustrative hypothetical of 2 aliens observing the early days of our race, and arguing about how humans might evolve, one of them using existing data to predict that humans will evolve extremely slowly, and only in the direction of reproduction and food, and the other considering the possibility that they may evolve in a completely different direction.

The authors then apply that thinking to the question, “What exactly will AIs want?”

A small nit-pick here, but the authors seem to confuse complicated with complex (I’m referring to the Cynefin model). No matter.

The premise of the argument is to extrapolate what humans might like at some future point given what our hominid ancestors needed to survive. If we need high energy foods – items that have sugars and fatty acids – and certain things make the food taste better – salt – then a logical guess would be raw bear fat covered with honey and sprinkled with salt. Not sure why the authors chose bear fat since it would have been a “bear” to actually acquire some without a fight and some fatalities.

Nobody would have been able to guess that we would instead crave frozen ice cream, or, worse, we would invent something like sucralose that provides no chemical energy. All this, to reinforce the point that the original process of natural selection does not predict tastebuds and our internal psychology which in turn cannot predict what we ultimately end up preferring.

The authors provide other examples, like “sexual selection” where certain traits make one peacock, for example, more attractive to the females, and helps their genes propagate. This did lead me on a little bit of a wild goose (peacock?) chase on the internet to understand how peacocks evolved. The results pretty much stopped at Darwin coming up with the idea of sexual selection.

The authors acknowledge that the process of natural selection within humans where tiny genes vary in order to produce a different “brain” differs from gradient descent which is used to tune different parts of a large mind, but the contention is that both processes are largely black box processes where we use certain inputs to produce certain desired outputs, but there’s a vast empty space between the two which leaves a LOT of room for unexpected outcomes.

The rest of the chapter explores a fictitious AI that has
a. zero complications,
b. a minor one,
c. a modest one,
d. a big one, and
e. more than one complication
and suggests ways in each of those cases where the AI might very logically evolve into something that had not been planned.

The problem of making AIs want – and do – what humans want is a major facet of what’s termed the “AI alignment problem”. The issue is no one who’s building AIs is interested in working on it.

I view it as a testosterone problem – my AI should be able to beat your AI. In other ways, it reminds me of the fact that no one – no one – understand quantum mechanics or quantum field theory: why it behaves the way it does – mainly because we can monetize it in myriad ways (and destroy the earth in a few others) without having to understand the why.