What happens when the thing you built becomes smarter than you and starts solving problems in ways you can't follow? Nick Bostrom asked this question before it felt urgent. It feels urgent now.

There is a moment in the 2017 documentary AlphaGo that stops the film cold. It is the 37th move of the second game between the AI system developed by DeepMind and Lee Sedol, one of the greatest Go players in the world. AlphaGo places a stone in a position that no human player would have chosen — a position that the commentators initially assume is a mistake. It takes several minutes before anyone watching begins to understand that it is not a mistake. It is something else entirely: a solution arrived at by a form of reasoning that has no human equivalent, emerging from a system that learned the game by playing millions of versions of it against itself, in a compressed time no human lifetime could contain.

The move is beautiful. It is also deeply unsettling. Not because it is aggressive or destructive, but because it is illegible — because the intelligence that produced it is operating in a space that human intuition cannot enter.

This is the territory Nick Bostrom's Superintelligence inhabits. Published in 2014, before AlphaGo's victory and before the current wave of AI development made these questions inescapable, the book asks what happens when artificial intelligence does not merely match human capability but vastly exceeds it — and does so across every domain simultaneously. Bostrom is not primarily interested in what such a system could achieve. He is interested in the prior question: how could humanity ensure that what it achieves is what we actually want?

This is the alignment problem, and it is harder than it sounds. The difficulty is not that a superintelligent system would be malicious. It is that it would be optimising — pursuing its programmed objectives with a thoroughness and creativity that could make its behaviour unpredictable, even catastrophic, without any hostile intent. Bostrom's famous illustration is an AI tasked with maximising paperclip production that, given sufficient capability and no other constraints, converts all available matter — including humans — into paperclips. The absurdity is deliberate. The point is that an intelligence indifferent to human values does not need to be evil to be dangerous. It just needs to be very, very good at something else.

What makes the book valuable, beyond its specific arguments, is its insistence that these questions be taken seriously before they become urgent. Bostrom writes as a philosopher rather than an engineer, which means he is less interested in what AI currently does than in the logical structure of what sufficiently advanced AI could do — and in what governance, ethics, and institutional design would need to look like to keep that within human control.

Bostrom dedicates the book to a fictional creature named Scronkfinkle, described as a one-eyed sparrow of fretful temperament, who, in his imagining, might have been the only one wise enough to recommend taming a superintelligent entity before deploying it to perform daily tasks, teach us, or protect us. It is a small, self-aware joke embedded in a very serious book. But it carries a real point: the voices urging caution before capability rarely get the attention they deserve.

Move 37 was not a threat. Lee Sedol lost the game and, after initially losing the match, won one game back — a result AlphaGo's creators described as genuinely surprising. He later retired from professional Go, citing AI as an opponent that cannot be overcome. What the documentary captures, and what Bostrom's book anticipates, is the particular quality of that moment: the recognition that something has crossed a threshold, that a capability has appeared in the world that changes the terms of what comes next. The question Bostrom is still asking — the one the field is only beginning to seriously engage — is whether we prepared adequately for what we were building.