Why I'm Suspicious of AI X-Risk

In 1798, Thomas Malthus published “An Essay on the Principle of Population” in which he argued that in short order humanity would suffer a major population collapse. Why? A simple deductive argument:

• population is growing exponentially;

• food production is growing linearly;

• therefore, considering food production can’t keep up, we ought to be worried

His theory took the intelligentsia by storm. Legislation was developed to protect food supplies in Great Britain. Even Darwin wrote glowingly about him. Yet, he was entirely wrong. The deductive logic seemed unassailable but in the end, he failed to account for the human ingenuity and adaptation which produced technological advances capable of multiplying food production.

And Malthus is hardly the only worry-wart with a deductive gotcha to raise a kerfuffle: modern history alone boasts Y2K, peak oil, global cooling, and overpopulation. Each of these panics turned out to be a farce fueled by little more than a penchant for catastrophizing and a failure of imagination.

Enter AI extinction risk (“X-risk”). The central argument, originally propounded by Nick Bostrom in his Superintelligence, also boils down to a deduction:

• an AI with sufficient capabilities could destroy humanity, even if unwittingly;

• rapid advances in AI continue to produce more and more of these capabilities;

• and therefore, if we care about human wellbeing, we ought to be seriously worried

While there is plenty of icing we could add to that cake, the upshot is that advanced AI could wipe us all out using wiley ways we can’t yet predict.

Sound familiar?

Apparently not, for many in the mainstream conversation. X-risk seems to have exploded in popularity especially among the intellectual and technological elite. But I feel the conversation is missing a sober perspective informed by the history of science about where the burden of proof really lies, and what evidence we have or should expect to confirm this all-consuming risk.

While it’s certainly not a decisive blow to point out the shape of X-risk arguments look suspiciously like other failed panics, I do think it bolts the burden of proof firmly to those demanding we worry. Hitchens Dictum is instructive here: “that which can be asserted without evidence can be dismissed without evidence.”

With a high bar for evidence in mind, then, I present three arguments that weigh against the X-worriers I feel are missed in the mainstream conversation.

1. Where’s the Beef?

Despite the theory of climate change having a distinctively apocalyptic flavor, skeptics were won over because any sensible reading of the theory entails that gradual increases in warming will be associated with gradually increasing harms, a prediction which has in fact been confirmed by multiple lines of evidence.

By analogy, we might ask: If the hypothesis is that an AI with sufficient capabilities poses an existential risk to our species, shouldn’t we expect to find that as we pile on capabilities and distribute them widely, we observe associated harms?

Put another way, if superintelligent AI is going to be so awful, not-quite-superintelligent AI should be pretty bad too, right?

Yet, to my knowledge, despite having widely deployed fantastical new AI capabilities to a huge swathe of the global population, we have observed zero real world harms, with the exception of the odd Bard conversation that’s gone off the rails.

So X-riskers, where’s the beef?

The reply will perhaps be the real X-risk occurs on a fast or near-vertical “takeoff,” a scenario in which an AI suddenly leaps into a sufficiently dangerous category of intelligence, rendering the harms undetectable to any previous, gradual intelligence gains.

While that’s conceptually possible of course, it also stinks of unfalsifiability. Wouldn’t it be just a touch convenient that the veracity of X-risk was shielded from scrutiny?

And are we not already drowning in unfalsifiable apocalyptic claims, from the political to the religious to name only two? What makes the X-risk claims different from, or more believable than, say an apocalyptic cult, if they cannot be subject to confirmation?

2. Percent Risk Measures Fear, not Facts

Hang on though, the X-risk’ers will say — we do have solid evidence that experts close to these systems are worried about extinction. Katja Grace has done an excellent job compiling their opinions:

The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%.

Isn’t that something worth considering?

I must confess, I find the way this line of “evidence” is used to be mind-bogglingly anti-scientific. Yes, of course, it’s true that many in the field are worried about X-risk. But what credibility, if any, does that actually lend to the veracity of X-risk claims?

How would we feel about a 1798 survey showing the median respondent judged Malthus’ overpopulation fears to be 10% probable? The simple fact is the respondents would have been plain plain wrong.

How do we know that isn’t the case here?

Whatever these prediction popularity contest studies are good for, surely they are not a reliable means of evaluating the underlying predictions themselves. Experts are wrong all the time, yet this line of “evidence” is often cited as if it materially buttresses X-risk. It does no such thing.

I cannot help but also point out the mechanism of measurement used in these studies is quite suspicious. They aggregate knee-jerk, gut-level expert intuitions about the risk of a future event that is typically measured in orders of magnitude. But do we really think an expert who imagines there to be a small risk could meaningfully distinguish between two or four orders of magnitude when the risk falls below 1%? I am extremely skeptical — and yet it is often said that even if the experts are off by some order of magnitude it’s a risk worth considering.

Why might these experts be wrong?

3. AI Is Not Deployed in a Vacuum

Malthus did not account for human ingenuity or adaptability and by my lights, neither does the X-risk hand wringing. I would be much more concerned about AI safety, for example, if we were not awash in public debate, regulatory frameworks, fail-safes and redundancies, and technologies being developed to steer AI. The reality is AI systems are always deployed into an ecosystem of constantly evolving processes and institutions which have already displayed resilience.

To put a fine point on it, the relevant question for X-risk is not whether an AI could extinguish our species. It is whether there is good evidence to think that despite all of the safeguards with which AI has and will be deployed, humanity will be unable to mitigate the risk.

Consider the fears about an explosion in misinformation as the result of recent advances in Generative AI. Despite ChatGPT being the most successful product in human history, having garnered nearly a billion users, to my knowledge there has been nothing close to a successful wide-scale misinformation campaign using generative AI. If there were an attempt by a malicious AI or malicious actor using AI, our existing social media channels have (imperfect but basically effective) mechanisms to substantially stop their spread.

I suspect many of the safeguards our institutions currently have in place that are supposed to be at risk to an AI takeover are similarly more resilient than imagined. The burden of proof here again seems to be on the X-risk worrier to show otherwise.

The Long Tail of History

To be fair, I don’t think these are knock-down-drag-out arguments that definitively establish X-risk has no merit. And I’m open to the evidence — if it can be shown that harms really do obtain on an AI takeoff timeline despite our checks and balances, then so be it. But I do think if I'm right, there is reason to be extremely skeptical of X-risk claims, and the scientifically informed approach would be to reserve judgment until X-riskers can marshal actual evidence to support their hypothesis.

Should that evidence fail to materialize, the long tail of history will record the X-risk debacle not for its prescience but for its commentary on the perennial human susceptibility to apocalypticism, even in “rational” quarters.