Why AI Governance Needs a Rethink Before It's Too Late

Jan de Vries · 2026-10-09

Vendan Ananda Kumararajah has spent a lot of time thinking about where artificial intelligence is headed. And he's worried. Not in a doom-and-gloom way, but in a "we need to fix this before it breaks" kind of way. His message is simple: if we don't rethink how we govern AI, we risk an intelligence explosion we can't control. That's a big claim. So let's unpack it. ### The Problem with Assumptions Kumararajah's core argument is that AI governance today relies on assumptions that may not hold up as systems get smarter. Think about it like building a bridge using outdated blueprints. The bridge might stand for a while, but as traffic gets heavier, those flawed assumptions become dangerous. He points out that as AI systems become more capable, they'll start developing successors. And if those successors inherit the same flawed assumptions, the problems multiply. It's like a game of telephone where the message gets more distorted with each pass. > "The real risk isn't that AI becomes evil. It's that we build systems that can't be checked, and then those systems build the next generation." That quote captures the essence of his concern. It's not about robots turning on us. It's about creating a feedback loop where mistakes compound. ### Why Independent Oversight Matters So what's the solution? Kumararajah argues for independent oversight. Not just self-regulation by the companies building AI, and not just government agencies that might lack the technical expertise. He wants a third-party approach. Independent bodies that can evaluate AI systems without a financial stake in the outcome. Here's why that matters: - **Avoids conflicts of interest**: Companies have every incentive to say their AI is safe. An independent auditor doesn't. - **Catches blind spots**: Internal teams often miss problems because they're too close to the work. - **Builds public trust**: People are more likely to accept AI if they know someone neutral is watching. This isn't a radical idea. We already do this in other industries. Think about financial audits, food safety inspections, or aviation safety checks. We don't let airlines police themselves. Why should AI be any different? ### The Intelligence Explosion Scenario Kumararajah's warning about an "uncontrolled intelligence explosion" might sound like science fiction. But the logic is straightforward. If an AI system can improve itself, each new version could be smarter than the last. And if those improvements happen faster than we can understand or regulate them, we lose control. It's like a car accelerating downhill with no brakes. The scary part? We might not see it coming. The transition from "manageable" to "unmanageable" could happen quickly, especially if we're relying on the AI companies to tell us everything is fine. ### What Needs to Change Kumararajah isn't just pointing out problems. He's suggesting a path forward. Here's what he thinks needs to happen: 1. **Create independent oversight bodies** with real power to audit and approve AI systems before they're deployed. 2. **Develop shared standards** that apply across borders, since AI doesn't care about national boundaries. 3. **Fund research into AI safety** at the same level we fund AI capabilities. 4. **Require transparency** from companies about how their systems work and what risks they've identified. None of this is easy. But the alternative, according to Kumararajah, is much worse. ### The Bottom Line We're at a crossroads with AI. The technology is advancing faster than our ability to govern it. Kumararajah's warning is a wake-up call. If we don't rethink governance now, we might not get another chance. It's not about slowing down innovation. It's about making sure we don't lose control of it. And that starts with independent oversight, honest conversations, and a willingness to admit we don't have all the answers.