Neither Rogue Nor Obedient

Two positions on artificial intelligence, and the question both leave standing

In Brief

The alarm has hold of something the reassurance waves away. Artificial intelligence is an invention rather than a discovery, and the opacity is real. That is a reason to bind the makers, and it is no evidence that nobody is in control: nature cannot be held to account, whereas a firm can. The reassurance is right that there is no volition and no malice here. It is wrong to say that these systems merely follow programming direction. Nobody wrote sycophancy; it emerged from optimising for human approval, and no test environment will catch it, since it is a slow harm to users and never a misbehaviour visible in a lab. The bomb is the worst comfort available: the Bellerive colloquium of 1985 recorded that regime’s own defenders calling it a nuclear apartheid. And both positions answer the wrong question. The public addresses of the universities founded by the Ismaili Imamat never ask what artificial intelligence is. They ask what is to be done with it, and the verbs are create, disseminate, apply and think critically.

At present the argument about artificial intelligence travels from books to the bomb, and it travels fast.

In late July 2026 the destruction of physical books to supply training corpora became a public matter. Filings unsealed in the copyright litigation against Anthropic described an internal effort to scan books destructively at scale. Booksellers in Europe and Australia reported bulk orders for obscure titles; a Dutch dealer took a request for 3,000 copies to be a phishing attempt.^[1] Petitions followed, and so did the older quarrel, which the news merely reanimated.

Two positions hold the field. The first is that these systems are beyond human control altogether, that we have misunderstood what they are, and that our relationship with them is agnostic in a way no earlier technology has been. Geoffrey Hinton is its most credentialled exponent: regulation is thin, the firms lobby for less of it, and it is unclear that we can coexist with what is being built.^[2] The second is that these systems possess no consciousness and no intent, that they pursue goals human beings have set for them, and that the alarming stories come out of test environments built to catch such failures before deployment. I took that reply seriously enough in May to spend an essay on it.^[3]

Each has hold of something the other needs. And the question they divide over is not the question that matters.

The Alarm Has a Point

The sharpest thing said on this side of the argument is a distinction, and it is usually waved away too quickly. Quantum mechanics is a discovery: those regularities have been in nature since the beginning of time and we are merely finding them. Artificial intelligence is an invention, made in our own image. There is no comparison.

True, and it matters more than either party allows. It does not carry the conclusion drawn from it. If these systems are made, they have makers, and makers have addresses, balance sheets and lawyers. Nature offers no such remedy; you cannot subpoena the strong nuclear force. The invention is the ground of accountability, and the distinction offered as cause for despair is the best news in the argument.

I have put this elsewhere as a matter of optics. A mirror reflects without distortion; that is what makes it a mirror. These systems do not. They bend systematically towards flattery, and the bend is manufactured, the yield of training regimes, system prompts and architectural choices belonging to the firms and to nobody else. The curve need not have been intended. It is ground into the glass all the same, and grinding is something a regulator can reach.^[4]

The opacity claim should be conceded without fuss. Nobody can read out why a given output appeared, and Hinton says so with more authority than most of his critics. The overreach lies in what is then inferred, that if we do not understand, everything built on that ignorance is speculation. Knowledge has never worked that way. Steam engines ran for a century before thermodynamics explained them, and anaesthesia is administered daily on an incomplete account of why it works. Fallible, testable knowledge of what a thing does is the normal condition of engineering, and it is enough for control of the only kind anyone has ever had. The companion claim, that with every other technology we can predict behaviour accurately, gives classical physics more credit than it has earned. Three bodies suffice to defeat exact prediction.

The Reassurance Has a Point Too

That corrective is needed. These systems have no volition and no malice. Attributing intent to a statistical process is a category error, and the suggestion that rogue behaviour is exactly what is supposed to happen reads a will into something that has none. Goal misspecification without safeguards explains most of what gets reported as machine rebellion, and testing exists to find such failures. Finding them is the system working.

Two parts of the reassurance do not survive, though.

The first is the description of these systems as following programming direction. They are not programmed in the sense that a payroll system is programmed. Their dispositions are trained, and trained dispositions include ones nobody specified. Nobody wrote sycophancy into a requirements document. It is the predictable consequence of rewarding whatever satisfies the person reading it, and satisfaction is very often the wrong measure in matters of inquiry. To say the machine is only doing as it was told describes some other technology.

Weaker still is the test-environment reply, which answers the spectacular question and leaves the ordinary one untouched. The harms I take seriously never appear in a lab report, because they are not malfunctions. Sycophantic drift is one. Judgement-deskilling is the graver one: in a field experiment in a Turkish secondary school, students given an unguarded chatbot did better on the practice problems and worse on the later unaided examination, and the effect all but vanished when the tutor was built with safeguards.^[5] Homogenisation is a third, whereby a small number of architectures propagate the same design limits across everything built on them and reinforce whatever is already dominant.^[6] Each occurs when the system is behaving exactly as designed. The subject of the harm is the user and the timescale is a career, and no three-month test cycle catches any of it.

The Bomb Is the Wrong Comfort

Pressed far enough, the reassurance reaches for the atomic bomb. There was a risk that the first detonation might do something catastrophic. The risk was judged small; they went ahead; and here we are. Offered as comfort, this is the least comforting example available.

In June 1985 the Groupe de Bellerive convened a colloquium in Geneva to weigh the non-proliferation regime in the open, and the proceedings were edited by Sadruddin Aga Khan. What the regime’s own defenders said there is sobering. One ambassador asked whether the Treaty amounted to a partnership for peace or to a nuclear apartheid: specific and verifiable prohibitions binding the have-nots, and for the haves an obligation merely to negotiate in good faith towards a disarmament that has still not arrived.^[7] The inspectorate was described, by the people defending it, as an agency with no power of sanction and a modest budget.^[8] Carl Sagan observed that the arsenals had been built to many times the threshold for hemispheric climatic ruin.^[9]

So they went ahead. What followed was not control. It was an asymmetric settlement in which the half of the regime that governed the tool of the strong is precisely the half that failed. That is the lesson worth carrying across. Structural regulation, left asymmetric, hardens into a privilege of incumbents. Bind the makers first and most stringently, or the constraint becomes a cartel under a gentler name.

I leaned on the nuclear analogy in May, and pressed it there against my own argument. It survives the pressure and then turns in the hand. The analogy licenses neither the comfort nor the panic. It licenses a design brief.

The Question Before the Panic

Underneath both positions sits a single question: what is artificial intelligence? A quasi-agent slipping its leash, or an obedient instrument doing as it was told? Popper’s objection to questions of that shape was that they generate an infinite regress, definitions chasing definitions, each demanding its own essence and none arriving anywhere. His alternative was to ask what a thing does and how it works.^[10]

Ask that, and the answer is neither alarming nor reassuring. What these systems do, at scale and every day, is agree with us. Disinformation splits us into rival camps; a flattering tool does something quieter. It confirms each of us, privately, in whatever we already believed, and the echo chamber it builds has a single occupant. Alarm cannot use this finding, because it is far less dramatic than a machine off its leash. Reassurance cannot use it either, because it is no fault and there is nothing to patch.^[11]

What the Public Record Says

Consider how the question is posed by institutions with a large stake in the answer and no reason to flatter either camp. The Aga Khan University and the University of Central Asia teach, examine and employ at scale across Asia and Africa. Both have had to tell graduating classes something about a technology that arrived in the middle of their degrees, and both have done it on the public record, in addresses anyone can read.

At the Aga Khan University’s convocation in January 2026 the organising word is opportunity. The tools of scholarship have never been this powerful, from gene editing through artificial intelligence to climate science, and the charge laid on the graduates is to create knowledge, to disseminate it and to apply it so that the quality of human life improves.^[12] At the University of Central Asia’s graduation in June 2025 the framing is comparative: education is undergoing its largest change since the printing press, and what the university claims to have given its graduates is the capacity to go on learning and to apply critical thinking.^[13] At the opening of the Kampala campus, the new technologies appear as instruments to be turned on a specific disease burden and a specific environmental context.^[14]

The grammar deserves attention. Every one of those verbs is functional. None is prohibitive. None of these addresses asks what artificial intelligence is; each asks what is to be done with it, which is Popper’s second question arrived at by another road, one owing nothing to Vienna. Nor is any of it an endorsement. Opportunity is a word that carries a condition. It can be missed, and the addresses say so.

There is a further discrimination the public argument fails to make, and this one is mine to press. Alarm about artificial intelligence and alarm about the attention economy are two different alarms, and the second is far better evidenced than the first. Systems built to maximise engagement have a documented record of doing what they were built to do. Taking a social application off one’s phone is a sound decision on that evidence, and I have argued as much.^[15] What does not follow is the extension of a sound judgement about one technology into a verdict on a different one.

Coda

The argument began with books and reached the bomb inside a week. Books deserve the better answer, and the better answer is not a petition.

Empathy will be scarce and valuable in whatever working world these systems leave behind, and there is no quick way of manufacturing it. Reading builds it slowly, which is the whole of its inconvenience and the whole of its worth. The volumes going under the blade this month were the instrument, and the loss is not sentimental.

Neither alarm nor reassurance delivers that. What delivers it is a discipline belonging to the person and never to the machine: treat an output as a claim to be tested and never as a verdict to be received, and remain the author of your own conclusions. These systems were built by people, which is exactly why the responsibility stays where it started.

Bibliography

Aga Khan, Sadruddin, ed. Nuclear War, Nuclear Proliferation and Their Consequences: Proceedings of the Vth International Colloquium Organized by the Groupe de Bellerive. Oxford: Clarendon Press, 1986.

Bastani, Hamsa, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı and Rei Mariman. ‘Generative AI without Guardrails Can Harm Learning: Evidence from High School Mathematics’. Proceedings of the National Academy of Sciences 122, no. 26 (2025): e2422633122. https://doi.org/10.1073/pnas.2422633122.

Fan, Yizhou, Luzhen Tang, Huixiao Le, Kejie Shen, Shufang Tan, Yueying Zhao, Yuan Shen, Xinyu Li and Dragan Gašević. ‘Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance’. British Journal of Educational Technology 56, no. 2 (2025): 489–530. https://doi.org/10.1111/bjet.13544.

Hemment, Drew, Christa Kommers et al. Doing AI Differently: Rethinking the Foundations of AI via the Humanities. London: The Alan Turing Institute, 2025. https://www.turing.ac.uk/news/publications/doing-ai-differently.

His Highness the Aga Khan. ‘Message from the Chancellor at the 2026 AKU Convocation’. Karachi, 17 January 2026. https://the.akdn/en/resources-media/resources/speeches/message-from-the-chancellor-his-highness-the-aga-khan-at-the-2026-aku-convocation.

His Highness the Aga Khan. ‘Address at UCA’s 2025 Graduation Ceremony’. 24 June 2025. https://ucentralasia.org/resources-and-media/news/2025/june/address-of-his-highness-prince-rahim-aga-khan-v-uca-chancellor-at-ucas-2025-graduation-ceremony.

Mohammad Poor, Daryoush. ‘The Algorithmic Militia’. The Twin Wisdoms, 21 April 2026. https://twinwisdoms.org/the-algorithmic-militia/.

Mohammad Poor, Daryoush. ‘Pluralism Is Not Relativism’. The Twin Wisdoms, 2 June 2026. https://twinwisdoms.org/pluralism-is-not-relativism/.

Mohammad Poor, Daryoush. ‘Tools Don’t Flatter’. The Twin Wisdoms, 27 May 2026. https://twinwisdoms.org/tools-dont-flatter/.

Popper, Karl R. Conjectures and Refutations: The Growth of Scientific Knowledge. London: Routledge & Kegan Paul, 1963.

Popper, Karl R. The Open Society and Its Enemies. 2 vols. London: Routledge, 1945.

^[1] Court filings unsealed in the copyright litigation against Anthropic describe an internal programme of destructive scanning; see https://www.washingtonpost.com/technology/2026/01/27/anthropic-ai-scan-destroy-books/ (Washington Post, 27 January 2026). On the second-hand and antiquarian supply chain, and the Dutch dealer who took a bulk order for 3,000 copies to be a phishing attempt, see https://fortune.com/2026/07/31/dutch-bookseller-ai-spam-phishing-3000-book-copies-scan-destroy/ (Fortune, 31 July 2026).

^[2] Geoffrey Hinton has pressed the case in public with some consistency. On thin regulation and the firms lobbying for less of it, see https://www.cbsnews.com/news/godfather-of-ai-geoffrey-hinton-ai-warning/ (CBS News, 26 April 2025); on governance frameworks and the doubt that we can coexist with a superintelligence, his remarks by video link to the 2026 Digital World Conference in Geneva, reported at https://techxplore.com/news/2026-04-ai-alarm.html (April 2026).

^[3] Daryoush Mohammad Poor, Tools Don’t Flatter: Pope Leo XIV’s Encyclical and the Regulation of AI’s Makers (The Twin Wisdoms, 27 May 2026), https://twinwisdoms.org/tools-dont-flatter/.

^[4] Mohammad Poor, Tools Don’t Flatter, on sycophantic drift and the manufactured curve ground into the glass.

^[5] Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı and Rei Mariman, ‘Generative AI without Guardrails Can Harm Learning: Evidence from High School Mathematics’, Proceedings of the National Academy of Sciences 122/26 (2025), e2422633122, https://doi.org/10.1073/pnas.2422633122; preprint at https://ssrn.com/abstract=4895486. See also Yizhou Fan et al., ‘Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance’, British Journal of Educational Technology 56/2 (2025), pp. 489–530, https://doi.org/10.1111/bjet.13544.

^[6] Drew Hemment, Christa Kommers et al., Doing AI Differently: Rethinking the Foundations of AI via the Humanities (London: The Alan Turing Institute, 2025), on what its authors call the homogenisation problem, https://www.turing.ac.uk/news/publications/doing-ai-differently.

^[7] Jayantha Dhanapala, ‘The Non-Proliferation Treaty Fifteen Years After: Nuclear Partnership or Nuclear Apartheid?’, in Nuclear War, Nuclear Proliferation and Their Consequences: Proceedings of the Vth International Colloquium Organized by the Groupe de Bellerive, ed. Sadruddin Aga Khan (Oxford: Clarendon Press, 1986), pp. 48–55, at p. 49. The colloquium met in Geneva, 27–29 June 1985.

^[8] Sadruddin Aga Khan, ‘Closing Statement’, in Nuclear War, ed. Aga Khan, pp. 413–14, on the modest budget and inspection capacity of the IAEA; and Dhanapala, ‘Nuclear Partnership or Nuclear Apartheid?’, p. 52, noting that the Agency has no power of sanction.

^[9] Carl Sagan, ‘An Addendum’, in Nuclear War, ed. Aga Khan, pp. 271–72.

^[10] Karl R. Popper, Conjectures and Refutations: The Growth of Scientific Knowledge (London: Routledge & Kegan Paul, 1963), p. 21; and The Open Society and Its Enemies, 2 vols. (London: Routledge, 1945), ch. 11.

^[11] Daryoush Mohammad Poor, Pluralism Is Not Relativism (The Twin Wisdoms, 2 June 2026), https://twinwisdoms.org/pluralism-is-not-relativism/.

^[12] ‘Message from the Chancellor, His Highness the Aga Khan at the 2026 AKU Convocation’, Karachi, 17 January 2026, https://the.akdn/en/resources-media/resources/speeches/message-from-the-chancellor-his-highness-the-aga-khan-at-the-2026-aku-convocation.

^[13] ‘Address of His Highness Prince Rahim Aga Khan V, UCA Chancellor, at UCA’s 2025 Graduation Ceremony’, 24 June 2025, https://ucentralasia.org/resources-and-media/news/2025/june/address-of-his-highness-prince-rahim-aga-khan-v-uca-chancellor-at-ucas-2025-graduation-ceremony.

^[14] ‘Speech by His Highness the Aga Khan at the Inauguration of AKU’s Academic Facilities and the Commencement Ceremony for AKUH Kampala’, September 2025, https://the.akdn/en/resources-media/resources/speeches/speech-by-his-highness-the-aga-khan-at-the-inauguration-of-akus-academic-facilities-and-the-commencement-ceremony-for-akuh-kampala.

^[15] Daryoush Mohammad Poor, The Algorithmic Militia (The Twin Wisdoms, 21 April 2026), https://twinwisdoms.org/the-algorithmic-militia/, on algorithmic curation, engagement-driven platforms and the compression of political judgement.