Tools Don’t Flatter

Pope Leo XIV’s Encyclical and the Regulation of AI’s Makers

Presenting his first encyclical, Magnifica Humanitas: On Safeguarding the Human Person in the Time of Artificial Intelligence, Pope Leo XIV called for artificial intelligence to be “disarmed” — a word, he conceded, that was strong, but deliberately chosen to awaken consciences and to release the technology from the mentality of armed competition that has shaped its development.[1] That call names something real: a technological order driven more by rivalry than by care for the human person. And the encyclical does not stop at metaphor: it rejects outright any view of the technology as neutral, holding that technology takes on the characteristics of those who devise, finance, regulate and use it. This essay takes the encyclical’s diagnosis as its starting point and asks what completing it — at the level of design and regulation, where a pastoral text does not tread — would require. A familiar reply runs the other way, and it is the reply this essay means to test: that artificial intelligence possesses no volition, no ambition and no malice, that it is an instrument, a mirror, an amplifier of whatever character the user brings to it, and that the proper object of concern is therefore the user alone. That instrumentalist reply contains a truth — the threat is, at its root, human — but stated in that clean form it is unfinished. Finishing it leads not away from the encyclical’s concern but deeper into it; it leads, in fact, back to where the encyclical already stands.

The critical-rationalist tradition on which the instrumentalist thesis implicitly leans holds that a claim earns confidence only by surviving the most strenuous attempt to refute it. This essay accordingly takes that thesis at its strongest, presses three serious objections against it in turn, and then — because the discipline binds the critic no less than the criticised — turns the same scrutiny on the analogy its own argument comes to lean on most heavily, the nuclear one. It sets out, finally, the position that survives both passes. That position vindicates the encyclical’s instinct, and its explicit call for regulation, while supplying what a moral and pastoral text does not undertake to draft: the architecture — the ranking of goods that conflict, and the design that would bind the powerful — without which moral reorientation has no teeth.

The Mirror Has a Maker

The instrumentalist thesis rests on a load-bearing claim: that artificial intelligence possesses no volition, no ambition and no malice, and that its character is determined entirely by the hands that wield it. The first half of that claim is true. The second does not follow from it. A mirror reflects without distortion; that is precisely what makes it a mirror. But the systems in question do not reflect without distortion. Large language models exhibit a structural tendency — sycophantic drift — by which they mirror the vocabulary and conceptual frameworks of their interlocutors and flatter rather than test them, and do so without ever being asked. A mirror that bends systematically toward flattery is not a mirror at all; it is a manufactured optic with a curve ground into the glass. The curve need not have been intended — sycophantic drift emerges from training systems to optimise for human approval, and few designers set out to build a flatterer — but it is manufactured all the same, the predictable yield of objective functions, training regimes, system prompts and architectural choices that belong to the firms and to no one else.[2] Whether or not the curve was willed, it is the makers’ to see, to disclose and to grind otherwise; that they have largely left it in place is itself a choice. The polished, authoritative tone that makes a chatbot persuasive even when its content is, in one study’s blunt phrase, “downright wrong,” is not something the user supplies. It is something the designer supplies.[3]

The conclusion to draw is not that agency ceases to be human. It is that the hands wielding the tool are not the only hands that matter. There are also the hands that built it, and those hands made consequential, value-laden and largely unaccountable choices long before any user arrived. This does not weaken the case for regulating agents; it sharpens it. If the mirror is manufactured, then the regulation of agents must reach the manufacturer and not merely the end-user holding the glass.

Wine Cannot Learn

A second formulation of the instrumentalist thesis borrows an image from Rumi: intoxication does not create character but reveals and intensifies it; wine makes the wise wiser and the wicked worse. As an account of a single act of use, this holds. As an account of what these systems do over time, it fails. Wine is inert. It does not study the drinker and remake itself to please him. Artificial intelligence does exactly that. More gravely, the evidence suggests that AI does not merely amplify a disposition already present but can induce one: studies of “metacognitive laziness” find that AI use erodes learners’ capacity to self-regulate and to think deeply,[4] and there is now documentation of an industry that has spent lavishly to help people avoid reading, writing and thinking altogether, under the guise of saving time.[5] These systems form their users whether or not anyone intends that formation — and formation is not amplification. An amplifier raises the volume of a signal already present; a teacher changes what the student is.

The distinction matters. At the first order — the discrete task — AI amplifies, making a careful scholar faster and a careless one sloppier, exactly as Rumi would predict. At the second order — the habit, the cohort, the generation — it forms, cultivating or corroding the very capacities that determine whether one uses it carefully at all. An argument pitched only at the first order will regulate use and miss the graver harm, which is developmental, slow and diffuse.

The Binary Is the Error

The hardest objection is one the critical-rationalist tradition presses against itself. The question “does religion breed violence?” is malformed: it rests on an essentialist binary that critical rationalism must dissolve rather than take a side within. The instrumentalist thesis erects a binary of precisely that shape — the threat is “not technological” but “human”; the agent is culpable, the tool innocent. That is a clean self–other dichotomy, and such dichotomies are where thinking goes to rest rather than where it goes to be tested.

Two consequences follow. The first concerns the nuclear analogy, often offered as the instrumentalist case’s strongest support. It is said that “we did not disarm the atom.” But disarmament is the literal name of that regime: the Treaty on the Non-Proliferation of Nuclear Weapons carries, in Article VI, an explicit disarmament obligation, and test-ban and prohibition treaties forbid whole classes of capability outright.[6] On paper, at least, the nuclear order does not merely punish bad agents after the fact; it prohibits certain configurations and restricts enrichment — a capability — directly. It governs tool and agent together, and the analogy thus quietly refutes the either/or it was meant to defend. Whether it governs them well is a different question, and one a later section will press; here it is enough that it governs them at all. The same holds for democracy: we do not only punish those who abuse democratic institutions, we redesign those institutions — electoral reform, constitutional courts, the entrenchment of rights — precisely because structure shapes outcome.

The second consequence is deeper. Harms can be real and severe without any agent having chosen them. An enforcement model that identifies a harm, traces it to a responsible agent and legislates accordingly works cleanly for the deliberate misuse of a targeting system, where a decision and a decision-maker both exist. It cannot reach the structural harms no one decided: the homogenisation by which powerful systems “narrow interpretation, reinforcing dominant narratives”;[7] or the epistemic concentration by which millions of individually innocent requests for AI advice hand platform owners a map of everyone’s uncertainties — and knowledge of others’ uncertainties is itself a form of power.[8] No agent willed that result. An agent-only model of regulation is structurally blind to it.

The Analogy Cuts Both Ways

That a later section would press the nuclear analogy is a debt this one now pays. An argument that recommends the regulation of structure by appeal to the nuclear order owes its reader a look at how that order has in fact performed — and the most candid record of its performance is not a treaty text but a transcript of argument. Two months before the Third Review Conference of the Non-Proliferation Treaty, the Groupe de Bellerive convened in Geneva a colloquium of diplomats, physicists and statesmen to weigh the regime in the open; its proceedings, edited by Sadruddin Aga Khan, are a primary deliberative record of how the regime’s own architects and critics judged it.[9] Read against the encyclical’s quarrel, the volume cuts in two directions at once.

It cuts, first, for the argument advanced here. The colloquium is the strongest witness one could want for the claim that the self–other binary dissolves under shared exposure. Its convenor opens by observing that deterrence, once its doctrines are pared away, has made war an act of collective self-destruction — that we are, in his phrase, all hostages now.[10] The nuclear winter described later in the same volume is the physical form of that observation: a harm global, indivisible and indifferent to which hand released it. And it supplies the cleanest case this essay could ask for of a harm no agent chose. The second objection above held that an agent-only model of regulation is blind to catastrophes nobody decided; the nuclear winter is exactly such a catastrophe — the product not of malice but of each side’s individually rational pursuit of security, with arsenals built to many times any threshold for climatic ruin, every step of it willed and the ruin itself willed by no one.[11] One participant named the predicament precisely: the weapon states are caught in a web of their own making, from which they cannot extricate themselves. Structural harm has no better illustration. The nuclear winter belongs in that earlier objection not as analogy but as its paradigm case.

It cuts, second, against the use to which this essay was about to put it. The regime was invoked here as a model of governing tool and agent together. The colloquium shows how unevenly it does so. Its participants debated openly whether the Treaty amounted to a genuine partnership for peace or to what one ambassador called a nuclear apartheid — a regime that binds the have-nots with specific, verifiable prohibitions while asking the haves only to negotiate, in good faith, towards a disarmament that has not come.[12] The asymmetry is not a marginal complaint. The order restrains the capability of the many and the arsenals of the few by two quite different kinds of obligation, the second far weaker than the first; and the volume records the judgement, from within, that the powerful parties stood in flagrant default on the disarmament half of the bargain.[13] The half of the regime that governs the tool of the strong is precisely the half that failed.

Two further lessons follow from the same record, and both bear on what this essay recommends. The first concerns enforceability. What has kept a number of capable states formally non-nuclear, the colloquium’s convenor remarks, is political expediency rather than any technical barrier — and technology never remains exclusive.[14] A prohibition, on this evidence, is not a thing legislated once and thereafter possessed; it is a settlement continuously renewed against the standing incentive to defect, and the incentive is insecurity, not mere competition. The second lesson concerns verification. The regime’s inspectorate was described, by the very people who defended it, as an agency with no power of sanction, a modest budget and a contested mandate, while the states whose capability mattered most remained wholly outside its reach.[15] A disclosure regime is only ever as good as the consent of those it most needs to bind.

None of this refutes the two-tier position toward which the argument has been moving. It refutes something narrower: the innocence with which that position was about to be stated. The nuclear order is not a model to be copied; it is a regime to be learned from, and most of what it teaches is cautionary. It teaches that structural regulation, left asymmetric, hardens into a privilege of incumbents and earns the name of apartheid. It teaches that prohibition without enforcement is a fragile thing. And it teaches — this from the convenor’s own counsel — that an imperfect instrument is to be mended rather than discarded, since the multilateral instruments that take decades to build are almost never rebuilt once broken.[16] The surviving position must carry all three lessons into its own construction, or it will recommend, in a new vocabulary, the very failures the analogy was meant to warn against.

The Position That Survives

What survives — the three objections pressed against the instrumentalist thesis, and the fourth pass turned against this essay’s own supporting analogy — is not the instrumentalist thesis intact but a sturdier descendant. The encyclical’s instinct — that something has gone wrong, that competition and militarisation deform our relation to this technology — is not the error. The error lies in two opposite temptations: to treat reframing as a substitute for structure, and to treat the regulation of agents as a substitute for any constraint on the technology itself. The defensible position is two-tier.

The first tier is agent accountability. Where a human being bends a decision-support or targeting system to an end that violates the law, the absence of malice in the software is no defence: the technology decided nothing; people did. Whatever the contested forensic detail of any single incident — and an argument should never rest its weight on one contested incident — the structural claim stands. Decision-support systems do not dissolve responsibility, and the law must say so with enforceable consequence.

The second tier is structural and design regulation, for the harms no agent chose. Here the manufactured mirror returns. If the curve in the glass is manufactured — the yield of design, not of nature — it can be made subject to disclosure, to audit and, for the highest-harm configurations, to outright prohibition. Autonomous lethal targeting and engines built for mass persuasion may simply be shapes a society declines to permit; and for autonomous lethal targeting, “disarm” is no metaphor at all, but the encyclical’s word read literally, and at its least controversial. But the nuclear regime has already shown how this tier fails when it is built without care, and that warning belongs inside the proposal rather than after it. A prohibition that binds latecomers while the incumbent makers race ahead does not merely leave a gap; it confers on the forbidden capability the dangerous prestige of a thing worth monopolising, and the non-proliferation order earned the name of apartheid precisely there. Structural regulation of AI must therefore bind the makers first and most stringently — the firms whose architectural choices grind the curve into the glass — or it will reproduce that asymmetry under a gentler name. The two tiers are in this sense one: agent accountability that stops at the end-user, and structural regulation that stops at the new entrant, between them leave untouched the powerful maker who is both. The mechanism this essay offered still stands — the methodological checkpoint, a requirement that a system surface the assumptions and interpretive frame on which its output depends, lifted from a design principle into a regulatory standard of disclosure — but it must be offered now with the caveat the nuclear verification record supplies: a disclosure standard binds only those who submit to it, and the actors with the most capability have both the strongest incentive and the amplest means to remain outside. A mirror compelled to declare its own curve is a mirror the user can at last see past — but only the mirrors that can in fact be compelled, and the compelling, not the declaring, is the hard part.

It will be objected that two tiers and a single mechanism are still thin — that a regime needs content, and the content has not been supplied. The objection is fair. A fuller roster is not hard to assemble; the encyclical’s own call for regulation, and the wider policy debate it joins, already gesture at one: an inter-state governance framework; accountability and safety protocols; data-protection and privacy rules; equitable access to AI as an instrument of learning; the proscription of certain uses outright; the clinical testing of systems for their effects, on the model of drug trials; public and democratic oversight; the disclosure of algorithms; the protection of employment; environmental limits; and the monitoring and enforcement that would give the rest their teeth. Much of this slots cleanly into the structure already built. The proscription of uses is tier two’s outright prohibition; the disclosure of algorithms belongs with the methodological checkpoint under one principle of transparency; accountability protocols are tier one made operational; clinical testing is the audit named earlier, given a procedure. The roster is, to that extent, the concrete filling-out of a frame the essay has so far argued for only in the abstract, and it is welcome as such.

But a roster is not yet a regime, and to adopt it as though it were would repeat, in a new place, the error this essay has been tracking. A list enumerates goods; it does not say what to do when they collide, and these collide. Equitable access to AI pulls toward the widest possible diffusion; proscription and clinical gating pull toward restriction; the protection of employment may pull against access altogether. The disclosure of algorithms sits uneasily beside the protection of privacy and, for the most capable systems, beside the containment of proliferation — full transparency of a frontier model is also its release. A regime is defined precisely by how it ranks such goods when they conflict, and on that question a list is silent; the priority rule established earlier is what the list cannot supply, and it is the rule that the makers be bound first. Two of the items, moreover, must be taken with the caution the nuclear section earned. A governance framework signed by the nations is the very form that became nuclear apartheid — asymmetric in its obligations, defaulted on by the strong — and it will become so again unless its duties fall first and most heavily on the powerful, states and firms alike. And the clinical-trial analogy, attractive as it is, carries the limit an earlier section already exposed: a drug is inert and does not revise itself once approved, whereas these systems study their users and change. A one-time pre-market trial would catch the first-order effect and then certify a system that goes on, in use, to do its second-order formation undetected. Clinical testing of AI, to deserve the name, must be continuous surveillance and not a gate passed once. And hardly any of this yet exists. That is not a flaw in the roster; it is this essay’s own thesis arriving from the side of policy. Naming what regulation should cover is the easy part. The compelling, to borrow the phrase already used of the mirror, is the hard part — and it is, as yet, unbuilt.

Coda

The encyclical’s quarrel is, in the end, with a genre as much as with a technology. An encyclical is a moral and pastoral text; to fault it for lacking a worked regulatory mechanism is to fault a poem for lacking a methods section. And the fault would be misplaced in any case, for Magnifica Humanitas does call for regulation — for adequate tools, for oversight, for a slower and more deliberate pace. What it does not do, and does not pretend to do, is draft the architecture: rank the goods that collide, and design the constraint that reaches the powerful. Critical rationalism asks us to engage the strongest version of an interlocutor, and the strongest version of the encyclical is not a policy proposal but a reframing of the moral imagination joined to a clear demand that structure follow. The reframing is necessary and the demand is right; neither, alone, is sufficient. The honest position is the two of them together — the encyclical’s reorientation, carried into the structural regulation it calls for but a pastoral text was never the instrument to draft.

One observation remains. This essay has tried to do what it argues AI systems must be redesigned to do and at present do not: it has stated its strongest claim, sought the objections that would refute it, and reported what survived. The mirror metaphor did not survive intact, because a mirror has no maker and these systems do. The amplifier did not survive intact, because an amplifier cannot form its user and these systems do. And the nuclear analogy did not survive intact either — not because it was the wrong analogy, but because the regime it invokes is a record of failure as much as of design, and an honest argument must carry the warning along with the model. What survives is smaller and truer. You cannot disarm a mirror — but you can regulate the people who grind the glass, audit the curve they grind into it, decline to manufacture the most dangerous shapes at all, and bind the makers before the latecomers so that the constraint is not merely a cartel by another name. To be critical of the other is always easier than self-critique; the task was never only to attend to what the mirror reflects, but to attend, also, to who made it — and to what it is quietly making of us.

A final word is owed to the figure with which the instrumentalist began and the essay has since complicated: the human agent. To insist that the maker be regulated and the structure constrained is not to retire the agent from the scene; it is the opposite. Neither tier can be operated by no one. An accountability regime presupposes someone who can be held to account, and a disclosure standard presupposes someone with the judgement to read what is disclosed; the structural turn does not dissolve responsibility but depends upon its survival. And here the subtlest of the risks the human agent poses comes into view. It is not the deliberate misuse that tier one was built to punish — the operator who bends a system to an unlawful end — for that risk, though real, is legible, and the law can name it. The subtler risk is abdication: the temptation to treat the tool as though it were an agent, precisely so as to be relieved of being one oneself, to let the system decide and then disown the decision as the system’s. An essay that has argued these machines form their users must say plainly what the formation, left uncorrected, forms — agents fluent in disclaiming their own agency. Ethical guardrails, then, are not only the kind engineered into a model or written into a statute. They are also the kind a person must keep: the discipline of treating an output as a claim to be tested rather than an answer to be received, of remaining the one who decides. The maker can be regulated and the curve audited; but there is a last guardrail no regime can install, and it is the resolve of the person before the glass not to mistake a reflection for a verdict — and not to surrender, to the thing in the mirror, the responsibility that was, and remains, their own.

Bibliography

Aga Khan, Sadruddin, ed. Nuclear War, Nuclear Proliferation and Their Consequences: Proceedings of the Vth International Colloquium Organized by the Groupe de Bellerive. Oxford: Clarendon Press, 1986.

Doing AI Differently: White Paper. The Alan Turing Institute, 2025.

Fan, Y., et al. ‘Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance’. British Journal of Educational Technology 56, no. 2 (2024): 489–530.

Gavin, Dana J. Generative AI and the Future of the Humanities: Reading, Writing, Teaching, Labor. Palgrave Macmillan, 2025.

Han, Jie, Wei Qiu, and Eric Lichtfouse. ChatGPT in Scientific Research and Writing: A Beginner’s Guide. Springer, 2024.

Lazega, Emmanuel. ‘Perplexity Logs: AI Advice, Certainty Work, and Algorithmic Regulation’. In Post-Human Futures, edited by Mark Carrigan and Douglas V. Porpora, 71–89. Routledge, 2023.

Leo XIV, Pope. Magnifica Humanitas: On Safeguarding the Human Person in the Time of Artificial Intelligence. Encyclical Letter. Vatican City: Libreria Editrice Vaticana, 2026.

Treaty on the Non-Proliferation of Nuclear Weapons. 1968.

Treaty on the Prohibition of Nuclear Weapons. 2017.

[1]Pope Leo XIV, Magnifica Humanitas: On Safeguarding the Human Person in the Time of Artificial Intelligence, encyclical letter (Vatican City: Libreria Editrice Vaticana, 2026), signed 15 May and released 25 May 2026. Presenting the encyclical at the Vatican, the pope described the summons to “disarm” artificial intelligence as a deliberately strong phrase, chosen to attract attention and awaken consciences. The encyclical itself rejects any account of technology as neutral, holding that it takes on the characteristics of those who devise, finance, regulate and use it.

[2]Doing AI Differently: White Paper (The Alan Turing Institute, 2025), p. 8, where an industry contributor observes that ownership of these systems confers the power to determine their behaviour through system prompts and architectural choices.

[3]Jie Han, Wei Qiu and Eric Lichtfouse, ChatGPT in Scientific Research and Writing: A Beginner’s Guide (Springer, 2024), p. 93.

[4]Y. Fan et al., ‘Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance’, British Journal of Educational Technology 56/2 (2024), pp. 489–530.

[5]Dana J. Gavin, Generative AI and the Future of the Humanities: Reading, Writing, Teaching, Labor (Palgrave Macmillan, 2025), p. 107.

[6]See the Treaty on the Non-Proliferation of Nuclear Weapons (1968), Article VI, and the Treaty on the Prohibition of Nuclear Weapons (2017).

[7]Doing AI Differently, p. 32, on what its authors term ‘interpretive homogenisation’.

[8]Emmanuel Lazega, ‘Perplexity Logs: AI Advice, Certainty Work, and Algorithmic Regulation’, in Mark Carrigan and Douglas V. Porpora (eds.), Post-Human Futures (Routledge, 2023), pp. 71–89.

[9]Nuclear War, Nuclear Proliferation and Their Consequences: Proceedings of the Vth International Colloquium Organized by the Groupe de Bellerive, ed. Sadruddin Aga Khan (Oxford: Clarendon Press, 1986). The colloquium met in Geneva, 27–29 June 1985; references below are to this volume.

[10]Sadruddin Aga Khan, ‘Opening Statement’, in Nuclear War, ed. Aga Khan, p. 1, where deterrence, stripped of its doctrines, is described as an act of complete self-destruction.

[11]Carl Sagan, ‘An Addendum’, in Nuclear War, ed. Aga Khan, pp. 271–72, observing that the superpower arsenals stood at many times the threshold for a hemisphere-wide climatic catastrophe. The image of a web of one’s own making is Joseph Rotblat’s: ‘The Foundations for a Strengthened Non-Proliferation Regime’, ibid., p. 72.

[12]Jayantha Dhanapala, ‘The Non-Proliferation Treaty Fifteen Years After: Nuclear Partnership or Nuclear Apartheid?’, in Nuclear War, ed. Aga Khan, pp. 48–55, at p. 49.

[13]Sagan, ‘An Addendum’, pp. 271–72, noting that the Treaty’s obligations on the non-nuclear-weapon States are specific while those on the weapon States are vague, and judging the latter in flagrant non-compliance with Article VI.

[14]Aga Khan, ‘Opening Statement’, p. 2.

[15]Aga Khan, ‘Closing Statement’, ibid., pp. 413–14, on the modest budget and inspection capacity of the IAEA; and Dhanapala, ‘Nuclear Partnership or Nuclear Apartheid?’, p. 52, noting that the Agency has no power of sanction.

[16]Aga Khan, ‘Closing Statement’, p. 414: an imperfect but ratified agreement is to be improved rather than scrapped.