Overview

I just finished reading “If anyone builds it, everyone dies” by Eliezer Yudkowski & Nate Soares. It’s an excellent fast-paced read, with an engaging style of writing. The book is set up very systematically like it’s written by the perfect mix of engineers, attorneys and storytellers. I am going to sum up each chapter as a separate blog. This post starts with some associated thoughts.

Two contrasting perspectives

Right about the time I was reading this book, I also came across Carole Cadwalladr’s substack post The Great AI Bubble. Her contention (based on talking to very practical venture capitalists) is that we have thrown all sorts of training at AI, and it’s not getting any smarter.

On the one hand, I am really annoyed that the biggest issue for the ones who are addicted to money is that they are disappointed that it’s a bubble and there will not be an adequate RoI.

On the other hand, I am secretly hoping (and relieved) that if this is true, maybe Artificial General Intelligence or Artificial SuperIntelligence will not destroy us (we will do it ourselves, thanks)?

And, to summarize why Yudkowski and Soares think otherwise (other than the revolutionary progress that’s been made in just the last few years on GenAI):

• Humanity’s special power is broad ranging intelligence, but machine intelligence is beating us at the game already
• AI, like humans, is grown, not crafted (gradient descent), which means that even though it’s giving us some useful human-like answers, we do not know what is happening inside the AI
• We are working hard at making AIs into “Reasoning Models”, and once AIs get sufficiently smart, they’ll start acting like they have preferences – like they want things
• Just as it would have been impossible to predict the creation of frozen ice cream from what hominids initially ate to give them energy, it’s equally impossible to predict what “preferences” an AI will develop regardless of what it’s trained for
• Following up on that, once an AI, regardless of what it’s trained for, gets to superintelligence, there would be absolutely no purpose for the ASI to keep humans around
• How ASI would exterminate humans is hard to predict. literally. just as an Aztec warrior with a bow and arrows, or an axe, or a spear, could not even imagine that the spanish galleon that had just appeared on their shores had people with sticks they could point at you and you would fall dead

To rub it in, the authors then spend a few chapters creating a hypothetical but entirely realistic scenario (one in a trillion, with this one being thought up by a couple of puny, biological minds, not a superintelligence) of how humanity would end at the proverbial hands of ASI.

My conclusion

My 2c: AI (and eventually ASI) is being grown and trained on the entirety of human experience. Human civilization is already in the throes of collapse, and the dark triad of psychopathy, narcissism and machiavellianism are ascendant everywhere across the world. Fear and greed are rampant as we violently rape the earth for all she’s got. day. after. day. after. year. after. year.

Unlike “Age of Ultron”, we don’t have a huge posse of superheroes and Gods that stop the ASI, which, btw, is one aspect that the authors have not explored very much – the unfortunate and grievous intersection of robots/humanoids coming of age and ASI powering all of them through whatever comes after 5G.

Will humanity destroy itself before ASI can? You can probably bet on it on an online gambling platform.

List of Chapters

• Introduction
• PART I: NONHUMAN MINDS
  • Chapter 1: Humanity’s Special Power
  • Chapter 2: Grown, Not Crafted
  • Chapter 3: Learning to Want
  • Chapter 4: You Don’t Get What You Train For
  • Chapter 5: Its Favorite Things
  • Chapter 6: We’d Lose
• PART II: ONE EXTINCTION SCENARIO (No chapter summaries)
  • Chapter 7: Realization
  • Chapter 8: Expansion
  • Chapter 9: Ascension
  • Coda
• PART III: FACING THE CHALLENGE
  • Chapter 10: A Cursed Problem
  • Chapter 11: An Alchemy, Not a Science
  • Chapter 12: “I Don’t Want to be Alarmist”
  • Chapter 13: Shut It Down
  • Chapter 14: Where There’s Life, There’s Hope
  • Closing Words