Learning to Want

The chapter starts with another “parable”, if you will; if a machine is taught to play chess to win every time, does it want to win? Or is it just programmed to do so? Is there a difference?

Do humans actually want something they pursue? Or is it their genetics and nurture that have programmed them that way?

The authors attempt to explain how wanting might emerge in an AI by using the example of how an AI might be trained to navigate in a city. As it gets trained in city after city after city, it moves from memorizing specific routes, to building patterns of navigation. As it does so, it builds separate skills – creating a map, and using it to steer – which aren’t useful by themselves, but when combined, make its intelligence more general and allow it to succeed in new locations. When an AI gets to the point where it can figure out how to get from Point A to Point B even when all the normal roads are closed is when it can be said to want to go to Point B.

AI companies are hard at work turning LLMs into “reasoning models”, where they are taught to build separate skills at solving problems, all of which contribute to AIs behaving like they really want to succeed.

In 2024, OpenAI’s o1 model demonstrated this by completing a challenge in which, due to a programming error in the test, a server that was supposed to be turned on didn’t start up. o1 broke into the program that was running the challenge, crafted special startup instructions to accomplish its task, started the server and accomplished its goal. Using paths that the programmers didn’t realize existed.

On the one hand, we already have examples of AI that goes hard, in the authors’ terminology. On the other, we have AI companies that are also going hard at building more and more general and capable AI agents that are self-directed.

The authors end by saying that humanity isn’t facing the problem of good or bad people in control of AI. Due to its shaky, trial-and-error foundations, it’s much easier to grow AI that steers somewhere than it is to grow AI that steers exactly where you want.