Asterisk Magazine, which aspires to be The New Yorker of effective altruism, has a piece profiling “AI doomers” which, even after reviewing the author’s other sincere engagements with related topics, I’m struggling not to parse as a brilliant bit of parody and/or satire.
… AI doom is a remarkably cozy catastrophe. If you suffered through a historical apocalypse, you’d expect to starve, be raped, watch your children die in front of you, die a slow and lingering death of smallpox or plague or wound infection. The AI apocalypse — at least for those with the slack to be worried about it — takes place in a world of wealth and relative peace and technological marvels.
“Enjoy the fact that you get to have hot showers,” said Duleba. “Enjoy the fact that you get to eat delicious food. Enjoy the fact that you get to do escape rooms, which is one of my favorite things. This is great. Have you noticed how great this is?”
The overwhelming takeaway here is that “doomerism” is a pretty decent line of work to be in. Brennan describes and quotes various folk who have strong and quantitative ideas on the probability of “AI” ending the world, most of whom are working in the dubious field of “AI safety”, and they all display the sanguinity that comes with a safe distance from the more immediate (and far less science-fictional, which is to say observable and measurable and wholly predictable) consequences of their bugbear of preference.
I don’t know if Brennan’s use of the “cozy catastrophe” label comes with an awareness of its origins, but it may be more fitting than they seem to realise, as Brian Aldiss’s original formulation makes clear:
The essence of cosy catastrophe is that the hero should have a pretty good time (a girl, free suites at the Savoy, automobiles for the taking) while everyone else is dying off.
Some sample quotes from Brennan’s piece:
“If we’re going to die,” said Ben Woden, operations manager at Pivotal Research, a fellowship program for aspiring AI researchers, “I would really, really prefer to die fighting.” He then clarified that by “fighting” he meant continuing to work at his job as an operations manager for an AI safety nonprofit, not bombing a data center.
“It’s better to have died having fun on the trampoline than to have died listening to ‘Moonlight Sonata’ on repeat while I cry,” Duncan Sabien said. “I choose between universes in which I die, and I choose the one I like most.”
“Drink the good whiskey, drink the good wine,” said Dave Kasten, head of policy at Palisade Research.
Drinking the good whiskey costs money. Accordingly, “I’m not as careful with money anymore,” Duleba said. “Historically, I was very careful with money, so that means I now have money to not be careful with.”
“It’s not that I want to be rich in the cyberpunk future,” he said. “It’s that I want to have options in any future.”
“Remember that everyone dying would mostly be bad because we couldn’t do stuff that’s good,” said Ben Woden. “Try not to surrender before the fight has begun by letting the fear of death keep you from having a life worth living.”
“When I’m going to a gig,” he said later, “I sometimes get the sense that it’s good we’re getting this before it’s all over. This might be the last iteration of Damnation Festival there ever is, so it’s good I was here. We’ve seen it out on a high. Well done, everyone.”
On the basis of their other work, I think Brennan’s aim here is to humanise these people, which they achieve in at least one sense: one can very easily imagine people in similar positions of comfort being similarly sanguine ahead of pretty much any period of civilisational collapse known to history. Their capacity for cognitive dissonance is totally recognisable, to the point of being banal—though their reliance on game theoretical and Bayesian reasoning to maintain it is perhaps without precedent, beyond the clade of people most intimately involved with the doctrine of Mutually Assured Destruction?
But no, there’s a much more obvious and contemporary comparison—namely the folks in the fossil fuel business who have convinced themselves that the only way to address global warming, which they have at least largely conceded is a thing, is to power through the problem by burning the remaining fossil fuels. The big difference is that global warming is perhaps the most scientifically established threat to human flourishing ever identified, while the risk of runaway superintelligence is a hackneyed sci-fi plot elevated to the status of existential crisis by the Engineer’s Disease.
Rationalism is a religion that flatters itself that it’s a philosophy. When it comes to “existential risk”, I take more comfort from the all-too-human sophomoric idiocy of these people than from any amount of more grounded argument. These are overgrown children performing a fear they don’t really feel, because they’ve never been scared enough of anything to know what it actually feels like; you might as well run a think-tank devoted to saving us from Pennywise and Freddy Kruger.
Even giving them the benefit of the doubt with regard to sincerity, I can’t help but feel that taking the cheques and “drinking the good wine” is some real last-days-of-Rome shit, but hey—I guess we all do what we have to in order to get to sleep at night.
In the meantime, if you want to see true fear of “AI”, look at the people turning up at town-hall meetings to overturn data center construction in their neighbourhood. When you’re genuinely scared of something, you’ll do what isn’t comfortable.
Leave a Reply