Why AI Governance Needs a Rethink Before It's Too Late

Jan de Vries · 2026-10-09

Vendan Ananda Kumararajah has a warning for anyone paying attention to artificial intelligence: if we don't rethink how we govern it, we could be heading toward an intelligence explosion we can't control. That's not a sci-fi plot. It's a governance problem, and it's already unfolding. I know "intelligence explosion" sounds dramatic. But stick with me. The core issue isn't whether AI becomes smarter than us overnight. It's what happens when flawed assumptions get baked into systems that are increasingly capable of designing their own successors. ### The Real Risk: Flawed Assumptions at Scale Here's the thing. Every AI system runs on assumptions. Some are technical. Some are ethical. Some are just plain wrong. When a single system has those assumptions, you can catch the mistake. When that system starts building the next version, and that version builds the next, those errors don't just persist. They multiply. Think of it like a game of telephone, except the message gets passed between supercomputers that never get tired. Kumararajah argues that independent oversight is the only way to break that cycle. Not oversight from the companies building the AI. Not oversight from governments that might be playing catch-up. Independent oversight. The kind that can say "stop" without worrying about quarterly earnings or political pressure. ### Why Self-Regulation Isn't Enough I get why companies want to police themselves. They move faster. They understand the tech. But here's the uncomfortable truth: you can't grade your own homework when the stakes are this high. > "The problem isn't malice. It's that flawed assumptions become invisible when everyone in the room shares them." That's the quiet danger. A team of brilliant engineers can all believe the same wrong thing. They build safeguards around that belief. Then they teach the next system to do the same. Suddenly you have an entire generation of AI that inherited a blind spot no one noticed. Independent oversight changes that. It brings in people who don't share the same assumptions. People who ask "what if we're wrong?" before it's too late. ### What Independent Oversight Could Look Like This isn't about creating a new bureaucracy that slows everything down. It's about smart checkpoints. Here are a few practical pieces: - **Third-party audits** before any system that can modify itself or its successors goes live. - **Assumption mapping** where outside experts document what a system believes and where those beliefs came from. - **Kill switches with teeth** that don't require a company's permission to activate. - **Public transparency reports** that don't hide behind trade secrets. None of this is radical. We already do versions of this in aviation, pharmaceuticals, and nuclear energy. AI just happens to move faster and scale wider than any of those. ### The Clock Is Ticking Kumararajah's point isn't that we should panic. It's that we should act before the systems get too complex to understand. Once an AI is designing its own successors, the window for independent oversight starts closing. Fast. So what do we do? Start small. Push for transparency. Support researchers who are asking hard questions. And stop pretending that self-regulation is the same thing as safety. The future of AI isn't just about better algorithms. It's about better governance. And right now, we're behind. If we don't rethink this soon, we won't get a second chance to get it right.