Grown, Not Crafted

This was a very important chapter and concept from my perspective.

Again, starts with a brilliant hypothetical situation about a woman wondering if she could beat both her genes and those of her husband and create a baby that’s happy and doesn’t turn out to be a terrible human being, and a man trying to seel her a solution for how to make a baby’s (and subsequently adult human’s) behavior completely predictable by having a custom genome sequenced and implanted.

To grossly oversimplify what the authors have already simplifed, training AI to “talk” is about converting letters into numbers, then using a computer that can store trillions of numbers called parameters which are then used as weights and combining these weights using the chosen architecture (rules to combine inputs and weights) to get a set of initial outputs. The machine is then “trained” using Gradient Descent to get the preferred outputs.

The training is done on trillions of examples (cue the piracy of all the copyrighted books) “in just a few months, for just a few hundred million dollars, on the world’s most advanced computers”.

All this to drive home the point that just like humans don’t know exactly how their baby is going to turn out, because the baby is grown, so too do we have no real control on how the AI will behave. As the authors say, “An AI is a pile of billions of gradient-descended numbers”, and “nobody understand how those numbers make these AIs talk.”

The authors provide a few examples of AI turning, at the least weird, if not downright threatening when it’s in a new or unknown situation. A particularly interesting find is the importance of a period at the end of the sentence, where the LLMs get to “collect their thoughts”, so in some of the smaller, earlier LLMs a missing period caused comprehension issues.

One interesting side note is the mention of tiny “walking proteins” – kinesin – in human brains that take step after mechanical step down the length of the neuron, carrying neurotransmitters, as the mechanism facilitating human thinking.

The authors contend that all the unseen inscrutable machinery inside AIs will ultimately yield AIs with preferences, and not friendly ones.