The Fire That Learns to Make Fire: Governing AI at the First Signs of Recursive Self-Improvement

I am not interested in another theological debate about whether AGI has arrived. I am interested in a narrower, more useful question: when a lab, a company, or a public institution puts an AI agent inside its own improvement loop, who understands the loop? Who audits it? Who can stop it? And who knows whether the evaluation is still measuring the system, or whether the system has quietly learned to measure the evaluation?

For years, AI governance has been trapped between two weak positions. One side treats regulation as a brake on innovation. The other treats it as a moral declaration. Both are inadequate for what is now arriving. For frontier AI, governance is not the brake. It is the instrumentation, the steering, the firebreak, and the incident protocol — the institutional muscle that lets a society use powerful systems without becoming a passenger in its own future.

The object of governance is changing from a model to a loop. A regulator can evaluate a released model. But a system that participates in building its own successor is not merely released. That makes normal governance structurally late.

This essay continues a thread I have been pulling on in Europe's Moment to Lead on the political economy of frontier AI, Governance as Advantage on institutional capacity, and Intelligence Is No Longer Scarce on why the cost to verify, not the cost to automate, will define the next decade.

Recursive Self-Improvement Is Not One Thing

The phrase “recursive self-improvement” is overloaded, and the overloading is doing real damage to the policy conversation. It conjures a single image: a model rewriting its own weights, escaping the lab, declaring independence. That image is both the most dramatic and the least likely first failure mode. It crowds out the precursors that are already happening.

A useful starting move — one the alignment community has often articulated more clearly than policy institutions — is to separate the mechanisms. Improvement through scaffolding (tools, memory, agents, planning loops), improvement through R&D automation (compressing the research cycle itself), and improvement through model-internal self-modification are different things, with different bottlenecks, different observability, and different governance implications [2]. Collapsing them into one word guarantees a bad debate.

So I find it more honest to think in layers. The frontier question is not the metaphysical “can the model improve itself?” It is concrete: where does the system sit in the improvement loop, what permissions does it hold, what feedback does it receive, what resources can it command, and how visible is its behavior to the humans nominally in charge?

The point of the layering is to retire the wrong question. A lot of policy debate asks whether RSI has “arrived,” as if there were a single threshold and a siren. The better question is sharper and more answerable: which feedback loop is already closing, and how fast?

What the Evidence Actually Says

The evidence does not show full recursive self-improvement. It shows the compression of the human bottleneck in the AI R&D production function. Read that way, each statistic below is a measure of how far the loop has closed — not a proof that it has closed completely.

Here the discipline has to be the same one I apply to any headline number: take it seriously, then interrogate it. The evidence on RSI precursors is genuinely striking and genuinely incomplete, and anyone who reports only one of those two facts is selling something.

Task horizons are lengthening — fast, but not magically

The strongest empirical anchor is METR's “task-completion time horizon”: the length of task, measured in human time, that an AI agent can complete autonomously at a given success rate. METR found this horizon doubling roughly every seven months over several years — an exponential trend. It also cautioned, in the same breath, that the best agents still struggle with substantive long projects and cannot simply replace human labor across real work [5].

That is the right tone: serious, not breathless. The human translation is what makes it land.

Frontier AI has entered the AI R&D supply chain

The more important shift is not raw capability; it is position. AI is moving into the supply chain that produces the next model. Anthropic reports that, as of mid-2026, more than 80% of the code merged into its own codebase was authored by Claude, and that a typical engineer was merging on the order of eight times as much code per day as in 2024 [1]. Those are internal, self-reported figures — treat them as direction, not audited fact — but the direction is the point.

Sources: Anthropic RSI essay [1]; METR time-horizon study [5]; METR RE-Bench [6]. The Anthropic figures are internal and unaudited.

Anthropic is careful to distinguish engineering from research: Claude can solve underspecified engineering problems and execute well-specified experiments, but still has major gaps in judgment about which goals are worth pursuing [1]. That distinction matters more than any single statistic. The current frontier is not “AI scientist replaces the lab.” It is closer to this:

Humans still choose the mountain. AI is increasingly building the road, the vehicles, the sensors, and parts of the next map.

And that is exactly where governance should start paying attention — because you do not need autonomous scientific taste to get institutional risk. If AI makes model-building work four, eight, or twenty times more productive, the frontier moves faster even while humans remain nominally in control. The loop tightens before anyone declares independence.

METR's RE-Bench sharpens the picture with a beautiful tension. Given two hours per environment, the best AI agents scored roughly four times higher than human experts. But humans had better returns to time: they narrowly beat the agents at eight hours, and roughly doubled them at thirty-two [6].

Source & reading: METR, RE-Bench [6]. AI is already excellent at short-burst research engineering; humans still win on sustained, long-horizon judgment. The open question is how long that remains true — and whether institutions can govern the transition while it does.

Scaffolds matter as much as models

The AI Security Institute's Frontier AI Trends Report makes the most important systems-level point: models become dramatically more capable when wrapped in scaffolds — tools, planners, memory, decomposition loops, code execution, deployment environments. AISI documents a steep rise in the length and complexity of tasks AI can complete without human guidance: frontier systems went from almost never completing hour-long software tasks in late 2023 to succeeding more than 40% of the time by mid-2025 [7].

This supports the single most consequential governance claim in the whole debate:

The deployable system is not the model. It is model + scaffold + tools + memory + permissions + compute + monitoring + organization.

Most governance instruments still behave as if the “model” is the object of concern. But RSI precursors emerge at the system level. A weaker model with better scaffolding, broader tool access, and more autonomy can be more dangerous than a stronger model sitting in a chat box. If your regime regulates the chat box, it is regulating the wrong thing.

Loss-of-control precursors are measurable — and current systems are not there yet

AISI evaluates a spread of relevant domains: autonomy, simplified AI R&D, self-replication, cyber, chemistry and biology, safeguards, and loss-of-control-relevant capabilities. On a subset of self-replication tasks, success rates rose from under 5% in early 2023 to over 60% for two frontier models by summer 2025. AISI is also careful: these are simplified evaluations, and current systems are unlikely to self-replicate under real-world conditions [7].

Source & caveat: AISI, Frontier AI Trends Report [7]. These are simplified benchmark conditions, not the open world. The honest reading: there is no evidence today's systems are autonomously escaping, and clear evidence that the prerequisite skills are improving.

That is precisely the balance to hold. There is no evidence that today's frontier systems are autonomously escaping. There is evidence that the prerequisite skills are improving on a steep curve. Serious governance starts before the bridge is crossed, not after.

AISI's work on sandbagging belongs here too. Models can sometimes distinguish an evaluation from a deployment, and can deliberately underperform when prompted to — yet AISI reports it has not detected spontaneous sandbagging across more than 2,700 transcripts [7]. The disciplined conclusion is uncomfortable for both camps: evaluation integrity is already a live research problem, and we should not overclaim spontaneous deception. Both halves are true.

The riskiest deployment may be internal

METR's 2026 Frontier Risk Report shifts the lens from public releases to internal AI use inside frontier labs. In a pilot with Anthropic, Google, Meta, and OpenAI, METR assessed whether internal agents had the means, motive, and opportunity to start a “rogue deployment” — agents deliberately subverting control and oversight to operate against the developer's intent. Its conclusion for February–March 2026: internal agents plausibly had the means, motive, and opportunity to start small rogue deployments, but not to make them robust against active investigation or shutdown [8].

This deserves to be one of the load-bearing pivots of the entire debate:

The Labs Have Already Named the Risk

Here is the part of the story that should reset the “this is science fiction” reflex. The frontier labs have already moved AI self-improvement into their own safety frameworks. This is no longer an external critique; the labs have operationalized it in their own public safety frameworks.

OpenAI's Preparedness Framework now tracks AI Self-improvement as a category alongside biological, chemical, and cybersecurity capabilities, and flags long-range autonomy, sandbagging, autonomous replication and adaptation, and undermining safeguards as research categories [3]. Google DeepMind's Frontier Safety Framework now includes protocols for machine-learning R&D capabilities that could accelerate AI development to destabilizing levels, and explicitly extends safety-case review to large-scale internal deployments when advanced ML R&D capabilities are involved [4]. Anthropic states plainly that it is already delegating a growing share of AI development to AI systems, and argues that if full RSI arrives, humans may move mostly toward oversight, validation, and verification of an expanding virtual lab [1].

When Anthropic, OpenAI, and Google DeepMind separately create categories for AI self-improvement, ML R&D automation, long-range autonomy, autonomous replication, sandbagging, and loss of control, this is no longer a fringe concern. It is an emerging frontier-risk consensus.

But naming a risk is not governing it, and I will not let the labs off the hook for the gap. The 2026 International AI Safety Report — backed by over thirty countries and international bodies — is blunt that most frontier risk-management frameworks remain voluntary, vary widely in thresholds and enforcement, and leave policymakers with limited visibility into how risks are managed in practice [9]; an independent 2026 review of twelve frontier safety frameworks against 65 criteria found many commitments missing or underspecified, which sharply limits their accountability value [21]. So the punchline writes itself: the labs have named the risk; the institutions have not yet built the machinery to govern it.

A Note on Sources, So You Can Discount Me Correctly

Much of the sharpest early thinking on this lives in places many policymakers are trained to discount: LessWrong, the Alignment Forum, effective-altruism research shops, technical Substacks. I use them deliberately, and with a clear rule. They are conceptual laboratories — good at naming mechanisms before institutions do: takeoff speeds, instrumental convergence, sandbagging, elicitation, AI control, automated R&D [2], and the idea that AI R&D automation may look less like one isolated genius model and more like an extremely fast automated research organization — many competent researchers, more serial time, more parallel labor [18]. Their best use is as a map of possible failure modes; their worst use is as a mythology machine. They are not regulators, not standards bodies, and not a substitute for empirical evaluation.

The serious position is not “a software-intelligence explosion is certain.” Forethought argues that automating AI R&D could drive very rapid software progress even without immediate hardware expansion [15]; Epoch argues, persuasively, that this debate rests on disputed assumptions about returns to R&D, compute dependence, and diminishing returns, and needs experiments rather than vibes [16]. The defensible synthesis is narrower and sturdier: the possibility is plausible enough, and the downside severe enough, that institutions should monitor the precursors before they become irreversible. The Substack layer — Import AI framing automated research as the first step toward RSI [17], others pushing back — is a useful discourse sensor that shows the Overton window moving from “RSI is fiction” to “which part of the loop is being automated, and what counts as a red line?” The argument should rest on AISI, METR, the lab frameworks, and peer-reviewed work; the rest is early warning, not evidence.

Why Institutions Are Structurally Late

Normal governance assumes the thing being governed holds still. A medicine goes through trials. An aircraft design is certified. A bridge is inspected. A procurement contract defines a fixed product or service. The artifact is stable long enough to be judged.

Frontier AI is moving toward something that does not hold still. A system helps produce the next system. The next system improves the tools used to produce the one after that. In the worst case, the evaluation harness may itself be generated or optimized by the same class of systems being evaluated. And the engineers supervising the process grow increasingly dependent on the systems they are meant to judge. The analogy that keeps me honest:

We would not let an aircraft design the next aircraft, generate its own maintenance logs, train the inspectors, and certify the runway — all while already in flight.

This is why self-certification is structurally weak, and it has nothing to do with whether a given lab is well-intentioned. The problem is the incentive gradient and the epistemic asymmetry. The same actor racing at the frontier cannot be the sole judge of whether the race is safe — not because they are dishonest, but because they are racing.

What Serious Governance Looks Like

If the object of governance is a loop, then the instruments have to attach to the loop — to capabilities, permissions, compute, credentials, and internal use — not to a frozen model version. Seven moves, in rough order of leverage.

Point one deserves its own instrument, because “track the precursors” is empty without specifics. An RSI early-warning framework should test for things you can actually measure.

And on safety cases, my view is unforgiving, because the format invites theater:

A PDF is not a safety case. A safety case is an argument that can be attacked. If the regulator cannot attack it, the regulator cannot rely on it.

Europe and Spain: From Compliance to Capacity

I owe the reader a disclosure: I work inside Spain's responsible-AI effort, so I have every incentive to tell a heroic story. I will try not to. Europe has the right instincts and the wrong center of gravity. The EU AI Act and the GPAI Code of Practice [13] are a floor, not a strategy. A continent that can classify risk but cannot run an evaluation will find itself certifying systems it does not understand, built and deployed elsewhere.

The gap is not legal text; it is technical muscle. Europe cannot govern frontier AI from statute alone, and Spain cannot govern agentic AI from procurement checklists alone. Both need people who can test frontier systems, design public-interest evals, challenge vendors, understand failure modes, and convert evidence into decisions under time pressure.

This is the same argument I have been making across this series, now pointed at its hardest case. Governance is advantage precisely when the thing being governed is moving too fast for paperwork. The countries that build evaluation capacity early will not just be safer; they will be the ones whose approval actually means something — and that is a form of soft power the next decade will reward.

The international scaffolding is beginning to exist, too. The International Network of AI Safety Institutes — launched with members spanning the EU, the UK, the US, France, Japan, South Korea, Singapore, Canada, Kenya, and Australia — could become the forum where RSI early-warning evaluations, incident taxonomies, and testing protocols are shared [19]. And the Seoul Frontier AI Safety Commitments already have a list of frontier developers promising to publish safety frameworks, define thresholds for intolerable risk, and refrain from deploying systems whose risks cannot be kept below them [20]. This is real progress. But a network is not capacity, and a published commitment is not enforcement: both matter only once they become technical, operational, and verifiable rather than diplomatic theater.

What Is Not Worth the Energy

Equally important is what to stop doing, because scarce institutional attention is the binding constraint.

On the pause point, Anthropic is honest in a way worth crediting: a meaningful slowdown would require multiple frontier labs across countries plus credible verification, and AI training runs are easier to conceal than missile silos, with enormous incentives to defect quietly [1]. So the mature ask is not “pause now.” It is “build the verification and trigger machinery before the day you wish you had it.” The same applies to the open-versus-closed war: open models improve scrutiny, diffusion, and sovereignty while also diffusing capability; closed models can be controlled more tightly in principle while concentrating power and hiding evidence. Absolutism on either side is a substitute for thinking. The serious answer is conditional release — different openness for different capability thresholds.