AI, ego, and the question the warnings skip
For some months I have been turning over a hope. People routinely outgunned before bureaucratic systems – who lose for want of the right form of words rather than the right case – might climb onto the shoulders of these machines and see over the wall. Level the ground. Open a little more room for democratisation. Put a brake, perhaps, on the appetites of the firms that built the machines.
I still think there is something in it. I no longer think it true in the form I first held it, and what follows is why.
The week the chiefs told us to slow down
On Saturday 12 September 2026 Dario Amodei published ‘We Must Pace the Frontier’, arguing that the industry ‘must slow the pace at which we improve the capabilities of AI models’ and that ‘progress will still seem fast, and we must make wise use of the time we gain’. Three steps. The first, to which Anthropic committed unilaterally, gives embedded third-party evaluators employee-like access – desks, badges, sight of the training pipelines – and the right to publish without Anthropic’s editorial control. The second asks frontier firms in democracies to coordinate on standards, which needs an antitrust waiver; the third asks for global coordination, whose limits he admits are stark. Within the hour Sam Altman called the same arrangement ‘a great idea, and we will do the same’, and told Fortune that same day that ‘given everything happening with safety, right now would be an ill-advised moment to go public’. Elon Musk needed three words: ‘Dario is right.’
I want to give them the benefit of the doubt, and the first step earns it. It is costly, unilateral and verifiable by somebody other than the firm making the promise. That is the strongest thing in the essay.
Two things sit beside it.
Chamath Palihapitiya posted twenty-six minutes later that the essay ‘makes the case to stop open source and concentrate enormous technological and economic power with Anthropic’. The charge needs no secret plan, and Amodei’s rejection of an open-weight ban does not answer it. It is structural: rules an incumbent already meets raise the cost of entry for everyone who does not. I put that in general terms six weeks ago, in Neither Rogue Nor Obedient: structural regulation, left asymmetric, hardens into a privilege of incumbents. Here is the case to apply it to. Note the shape of the three steps. The one Anthropic can deliver alone is the one it has delivered; the other two are addressed to Congress, competitors and authoritarian states, none of whom have agreed. A proposal whose verifiable element is also its competitive advantage is not thereby dishonest. It is simply not self-certifying.
The second is more serious. Pacing is defined as taking adequate time, with adequacy unspecified. Amodei sketches what a threshold might look like – capability checkpoints, or limits on ingredients such as training compute – but names no number, calls the second kind ‘gameable’, and commits to neither. The step Anthropic has actually taken contains no pacing criterion at all. A commitment that cannot be breached cannot be kept either. A conveniently vague term is doing immunising work: whatever happens next will turn out to have been consistent with it. The remedy is not suspicion about motive. It is a number.
Humanity has ego
It shows here not as vanity but as a claim to knowledge. That is the more dangerous form, and the one I traced last month in the arrogance that brought down Mossadegh’s government.
Altman told the India AI Impact Summit that ‘on our current trajectory, we believe we may be only a couple of years away from early versions of true superintelligence’. Amodei writes that within six to twelve months a swarm of agents could be ‘capable of taking over the entire internet with a persistent botnet’. These are prophecies, not predictions. Popper’s objection was never that such claims are gloomy, but that they are not what inquiry produces. In ‘Prediction and Prophecy in the Social Sciences’ he holds that long-term prophecies can be derived from scientific conditional predictions ‘only if they apply to systems which can be described as well isolated, stationary, and recurrent’. Society, he adds, is surely not one of them. No system is less isolated or less stationary than a frontier laboratory in 2026.
Altman’s hedge, ‘of course, we could be wrong’, has the shape of modesty and does the opposite work: it converts a prophecy into a statement that cannot fail. And the prophecy is reflexive: a forecast of catastrophe within the year is itself an event in the market it describes, moving capital, staff, legislation and the price of compute. That does not make it false. Nobody argues from nowhere, and a position is not refuted by locating its author. But a forecast made from inside the market it moves cannot be audited from there. The check has to come from elsewhere.
What actually contains an ego
Nothing inside a person. Popper named this error. By putting the problem of politics in the form ‘Who should rule?’, he argued, Plato ‘created a lasting confusion in political philosophy’: the question assumes power is essentially unchecked, so the only task left is getting it into the best hands. His replacement is the whole of my answer: ‘How can we so organize political institutions that bad or incompetent rulers can be prevented from doing too much damage?’ And: ‘all political problems are institutional problems, problems of the legal framework rather than of persons’.
That test sorts the three steps at once. Embedded evaluators with publication rights pass, because they work whether or not the executive means a word of it. Voluntary coordination among laboratories fails, because it is a promise about persons. The piecemeal criterion agrees: the first step is small, reversible and generates evidence; a global regime with no threshold is a blueprint, and blueprints cannot learn from their errors.
That disposes of another of my questions. Can AI brake the greed of the firms that make it? No – not on its own, and not for want of capability. A brake has to sit outside the thing it brakes, which is why I have argued since May that the object of regulation is the maker and not the tool. There is an older name for forgetting this. To defer to the chiefs of AI on the risks of AI because they are the ones who know is taqlīd: accepting a ruling on the authority of whoever issues it. It closes criticism at precisely the point where criticism is owed.
My conjecture, under test
Back to the hope I began with. It is testable, and it has been tested.
Shah and Levy, in Access to Justice in the Age of AI: Evidence from U.S. Federal Courts (draft, March 2026), track 4.5 million federal civil cases. Self-represented cases rose from a long-run average of 11 per cent to 16.8 per cent in FY2025, concentrated in formulaic case types. Complaints bearing markers of AI generation went from near zero in 2019 to more than 18 per cent. Entry has been levelled, and by the mechanism I proposed.
Then the part I did not want. Those cases are not terminating faster, and the docket entries they generate in their first 180 days are up 158 per cent. Nothing in the data shows better outcomes. This is Rebecca Sandefur’s distinction between formal entry into a system and substantive access to a remedy, and I had collapsed it.
The reliability evidence runs the same way. Dahl, Magesh, Suzgun and Ho, in ‘Large Legal Fictions’, found legal hallucination rates between 58 and 88 per cent, lowest at the Supreme Court and worst at district level – ‘the court of first appearance’, as they put it, for almost every litigant – with models failing to correct a user’s false premise. Their own conclusion is that the risk falls hardest on ‘pro se litigants or those without access to traditional legal resources’. The specialised commercial tools still hallucinate between 17 and 33 per cent of the time. A tool at its most confident where it is least reliable, in the hands of those least able to check it, is not obviously an instrument of levelling.
Kate Crawford puts the opposing case at its strongest: AI is ‘a registry of power’, and democratising it to reduce asymmetries is ‘a little like arguing for democratizing weapons manufacturing in the service of peace’. I think that overstates. Asserted as structural necessity it cannot be refuted, and the pro se figures cut against it: capability did disperse. What concentrated was the bottleneck.
What is left
The hope survives in a weaker and more useful form. AI redistributes procedural literacy, a real gain in dignity and in the capacity to contest. It has not been shown to redistribute outcomes, and it may congest the very institutions that deliver them. That is less than I wanted and more than nothing.
Two questions with numbers attached would move this further than more argument. What compute threshold, release cadence or penalty would make a pacing commitment capable of being broken? And do AI-assisted claimants win more, rather than merely file more? Until somebody answers the second, what I have is a good mechanism and no result.
The ego is not contained by the humility of those who warn us about themselves. It is contained by arrangements that do not require them to be humble.
Sources
Dario Amodei, ‘We Must Pace the Frontier’, 12 September 2026
Altman and Musk responses, 12 September 2026
Altman, Fortune interview, 12 September 2026
Chamath Palihapitiya’s response, reported by Axios
Altman at the India AI Impact Summit
Anand V. Shah and Joshua Y. Levy, Access to Justice in the Age of AI: Evidence from U.S. Federal Courts (draft, March 2026)
Matthew Dahl, Varun Magesh, Mirac Suzgun and Daniel E. Ho, ‘Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models’, Journal of Legal Analysis 16 (2024), 64–93
Varun Magesh et al., ‘Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools’, Journal of Empirical Legal Studies 22 (2025), 216–242
Karl Popper, The Open Society and Its Enemies and ‘Prediction and Prophecy in the Social Sciences’; Kate Crawford, Atlas of AI – quoted from the copies in my own library.
Daryoush Mohammad Poor, ‘Tools Don’t Flatter’, The Twin Wisdoms, 27 May 2026
Daryoush Mohammad Poor, ‘Neither Rogue Nor Obedient’, The Twin Wisdoms, 2 August 2026
Daryoush Mohammad Poor, ‘Two Kinds of Certainty’, The Twin Wisdoms, 19 August 2026
Daryoush Mohammad Poor, ‘Nobody Argues From Nowhere’, The Twin Wisdoms, 6 September 2026
Leave a Reply