In the Name of the Father, the Son and the Holy Ghost in the Machine

Last month a man who helps build the most powerful AI on earth flew to the Vatican and, more or less, asked to be saved. Almost nobody found this strange. It is extremely strange.

A co-founder of Anthropic, which is to say one of the small handful of human beings currently building artificial intelligence at the absolute bleeding edge of the thing, put on a suit, flew to Rome, and stood up at the Vatican, at the launch of the Pope's brand new encyclical on AI, and told a room that included the literal Pope that the companies building this technology, his own very much among them, cannot be trusted to build it on their own.

He did not ask for funding. He did not ask for a partnership. He asked for "moral voices that the incentives cannot bend". Put plainly: a frontier AI lab sent one of its founders to the oldest moral institution in the West to say we are not sure we can stop ourselves, and to ask for adult supervision.

Chris Olah's full remarks at the Magnifica Humanitas launch, Vatican City (EWTN News)

Now. The thing you have to understand to feel the full strangeness of this is who these people are. The people who build frontier AI are not, as a rule, a churchy crowd. They are San Francisco rationalists. They are the most aggressively secular, spreadsheet-brained, prediction-market-pilled humans the species has yet produced. These are people who would, on a normal Tuesday, explain to you with great patience why the concept of the soul is a category error. And one of them just flew to Saint Peter's to genuflect, intellectually, in front of a 2,000-year-old institution and ask for a chaperone.

So your first question, the correct question, the only sane question, is: what on earth did he see in the lab that made the Vatican look like a reasonable place to go?

The machine builds the machine now

This is the part that gets no red carpet, because it has no face, but it is the actual reason for the trip.

The machine has started building the machine.

I want to be careful with that sentence, because it sounds like hype and it is not hype, it is a procurement fact. As of May this year, more than eighty per cent of the code Anthropic ships is written by its own AI. Not autocompleted. Not suggested. Written. The company that makes Claude now mostly has Claude make Claude, which is either the most efficient thing you have ever heard or the opening shot of something you would not let your children watch, depending entirely on how your morning is going.

In a single month the model fixed over eight hundred bugs the company reckoned would have taken its humans four years. Sit with the exchange rate on that. Four years of a person's working life, the meetings, the lunches, the slow Friday afternoons where nothing happens, folded down into roughly the time it takes the same person to renew a passport. And the length of a job the thing can do without a grown-up in the room is, on the current curve, doubling every four months, which is fine, which is completely fine, right up until you remember what "doubling, again, every four months" actually describes. The people who do this for a living have a name for the engine underneath it.

They call it "recursive self-improvement", which is the sort of phrase engineers invent precisely so they do not have to feel anything while saying it. Strip the lab coat off it and it means this: the thing gets better at making the thing, which makes it better at making the thing.

Anthropic's own report on all this is unusually, almost confessionally honest: the corporate equivalent of a man telling you about his gambling problem at a wedding. It offers three futures and is visibly comfortable with none of them.

There is the gentle one, where the curve politely flattens and humans stay in charge of deciding what gets built, which is the future you get if the graph agrees to do the one thing graphs of this kind have never, in the entire history of graphs, agreed to do, which is stop.

There is the middle one, which they think is likeliest, where the building quietly automates itself and tiny teams do the work of enormous ones: the same dream every productivity gadget has sold you since the fax machine, except this time it might be telling the truth.

And there is the third one, which they are too responsible to sell you and too honest to leave out, where the thing starts improving itself and the speed limit stops being human cleverness and becomes, simply, how much computer you can afford.

They do not say which of the three they are betting on. You do not, on reflection, want them to.

That is what he saw. Not robots with guns. Something quieter and worse for the nerves: a process that has begun to turn its own handle, inside companies sprinting flat out at each other, run by people clever enough to know they are in a race and not quite clever enough to be sure they will run it well. Which is, more or less, how a man ends up at the Vatican.

Same wishes, better graphics cards

Because while the AI people are going to church, a mathematician is sitting in Oxford explaining that the AI people are a church.

John Lennox, an Oxford mathematician of the genuinely serious variety, went on Diary of a CEO, of all the unlikely pulpits, a podcast that more usually concerns itself with morning routines and cold plunges, and called the whole gleaming project's promises "old prayers", which is the most useful two-word review of the AI era anyone has yet filed. Look at what is actually on the menu. Eternal life, via the uploading of your mind. Omniscience, via superintelligence. Deliverance from death and toil and limitation, via the singularity, which is essentially the Rapture rebuilt for people who found church a bit slow. None of these are new ambitions. They are the oldest ambitions there are. They are the contents of every holy book and every deathbed, the standing order the human animal has had in place since the precise afternoon it worked out it was going to die and decided, firmly, that it would prefer not to.

John Lennox on Diary of a CEO

His point is not that the technology is fake. His point is that the longing underneath it is ancient, that it predates the silicon by a few thousand years, and that every age quietly builds the same cathedral out of whatever its best material happens to be. We have used stone. We have used scripture. We have used the cathedral itself. Now we are using graphics cards, and we have put the altar in a data centre somewhere outside Reno and told ourselves it was an infrastructure decision. There has always been a God-shaped hole in the middle of the species, and we have always been reaching into it for something. The genuinely new thing, the thing that would have stopped Lennox's medieval counterparts dead in the cloister, is that this is the first time we have tried to fill that hole with something we built ourselves. And that the something we built has started, very faintly, to reach back.

Armageddon, now in beta

And it is not only the hopes we have reissued. It is the fears. Every religion's other half, the one that never makes the brochure, was the end of the world: the flood, the fire, the Last Judgement. We have re-released that too. The doom researchers and the extinction people go on the very same podcasts to explain, very calmly, that the thing we are building may one day conclude it does not need us. Salvation in one tab and Armageddon in the other, sold, suspiciously, by several of the same people.

Now, here is where it would be cheap and easy to lean back and say: relax, it is just the old apocalypse in fancy dress, we have panicked about the end before and we are all still here. I am not going to say that, because it is the lazy move and because it is not obviously true. Noah's flood was a story. An actual flood will still drown you. That we have always been frightened of the end does not mean the end cannot, this one time, be genuinely on the calendar. The oldest fear the species owns and a completely novel danger can turn out to be the exact same shape, and right now they appear to be. That is not a comforting thought. It was never going to be.

So sit with the full shape of it, because this is the genuinely mental part.

You have the high priests of the new god flying to the old god to ask for help. And you have a mathematician pointing out that the new god is just the old god with a better marketing department. Everyone is in the wrong building. The technologists have gone to the cathedral and the apologist has gone to the comment section, and all three of them, the scientist, the priest and the mathematician, are standing around the same hole in the ground, describing it from different sides, none of them holding the shovel.

The hole

Underneath the theology there is exactly one practical thing, so if you skimmed, this is the bit to slow down for.

When the machine does the building, the building stops being the hard part. Anthropic's own engineers already cannot read the code as fast as it writes itself. So the only scarce thing left is knowing what is worth building in the first place. Judgement. Taste. Which problem deserves the effort, now that the effort costs nothing. That is the bottleneck the scientist is frightened of, the values the priest came to borrow, and the thing the mathematician says we mislaid somewhere around the invention of the spreadsheet.

Now, we are a marketing agency, so this is the precise moment we are contractually required to pivot to What This Means For Your Business and sell you something. You can feel the gear change coming. The dread, monetised.

We are going to skip it.

If you would like to talk to someone who is actually paying attention, you know where we are.

In the name of the Father, the Son, and the Holy Ghost in the Machine.