A council of Claudes for quality review

Folks. Kevin Roose has written a really, really good book about AI. With the help of AI, of course. The excerpt below is from an interview by Natalia Quintero.

The point is probably obvious. I dug this out now because I’m working with a group of quality managers today. Do you see the connection?

Roose said he built a “council of Claudes” to review each chapter after he finished drafting it. The system used multiple agents, each assigned a different role:

• An AI skeptic challenged whether the claims about language models were credible. • An AI historian checked technical details and the historical precursors to the systems he described. • A Walter Isaacson–style editor evaluated the chapter’s flow, structure, pacing, and readability. • A master editor agent synthesized the different reviews into one report for Roose.

The agents caught several factual errors that Roose - and even his human editors - had missed. He viewed this as evidence that AI systems had improved: whereas journalists previously had to fact-check AI outputs, he now had AI fact-checking his own work.

(The book is called “The AGI Chronicles” and came out a month ago. If you haven’t read it, or listened to it like I did, you absolutely should. Incredibly thorough and impressive. You don’t often get to read a history book about an era while you’re still right in the middle of it. AI time moves even faster than internet time. So hurry up and read it, and then let’s talk.)