Who Has Time to Be the Human in the Loop?

“This is an accurate summary, but the ministry mentioned doesn’t exist.”

A civil servant caught the error when CiudadanIA was being tested within the Office of the Prime Minister of Spain to process correspondence from citizens. (More about that system and my involvement in building it in this post.) The system was subsequently improved, both in terms of prompts and of the process of reviewing its outputs.

Two things struck me about the error being caught: the civil servant had the expertise to realize the summary was wrong, and they had the time to closely review its contents. And the change they triggered has been impactful.

These elements are usually subsumed under “human in the loop”, but this simple phrase belies their complexity.

Over my professional life, through Saturdays.AI, technical projects and government, I have helped many people on their journey to build with AI. And I know that I am susceptible to the endless lure of plugging in another agent or iterating a bit more (see also my post on Dopamine-Driven Development).

The increase in things I can do vastly outstrips my ability to pay attention to them all.

For a person, this can be handled with focus and discipline. For a larger institution, or even for agents with the ability to execute code, the problem is more acute and touches on governance, safety and security.

We are at risk of trading capability for awareness.

It’s possible to remain in the loop and still not be aware of what’s going on.

What the experiments say

A 2024 meta-analysis of 106 experiments found that while human–AI combinations outperform humans alone on average, they do worse than the better of the human or the AI working alone.

The shortfall appeared in decision tasks (those with a fixed set of choices) rather than content creation tasks, and it was largest where the AI was more accurate than the human when each worked alone.

Many oversight tasks can be modeled as decision tasks. And keep in mind that these experiments, which span 2020 to mid-2023, predate current frontier AI systems. Since the combinations fared worst where the AI was already better, more capable systems may widen that gap rather than close it. So I read the lesson that the value of humans in the loop needs to be demonstrated, not taken for granted, as conservative.

However, we should be careful not to read this finding naively and conclude that we can just remove the human from the loop. The experiments studied average accuracy in bounded tasks. In contrast, one of the roles of humans in oversight is to intervene in the minority of cases that go wrong, and to be accountable for what happens.

The role of time in overseeing AI actions was studied in an experiment comparing 25-second and 12-second decision windows for a person supervising automation. The authors found that time pressure reduced participants’ ability to accept or reject the automation’s advice accurately and increased their perceived workload. Increasing the transparency of the information available improved accuracy overall, but did not offset the effects of time pressure. The caveat here is that the task was to manage simulated uninhabited vehicles, so we need to be careful in generalizing the findings.

This is a common problem to run into when dealing with agents. In an attempt to make oversight easier, we have one system perform a task, another system check that it was performed correctly, and then we show a summary of this work to a human. But at each step, information can be lost.

What have we missed? Is there any way to actually inspect the consequential actions that were taken?

In my work on harmful manipulation in agents at ERA, I’ve become more convinced that it’s important to distinguish between capability, deployment and demonstrated effect. (See my working paper for more.) Similarly, I think we need to distinguish between providing a mechanism to intervene, using that mechanism, and actually preventing something bad from happening.

When the AI acts

This distinction becomes even more important when the AI system is actually doing something.

While it might be OK to make mistakes in a draft that the AI is working on, we might need to ensure that certain things don’t happen when an AI has access to the file system, can send messages or can run arbitrary code.

Prompt injection is a key example of this kind of issue: we want the AI to be able to read emails or documents, but we don’t want it to follow instructions planted in them. This is the kind of thing AgentDojo aims to test.

But it may be naive to expect that a human will be able to find these issues at the end.

While it’s obvious that we need to automate parts of this monitoring process, here is a small example showing why we also need to be careful about how we do that.

When evaluating models on FORTRESS, I ran a model on the benchmark and had another model grade the results. The model produced the exact same refusal message 19 times, but the judging model gave those refusals four different scores. Sixteen were scored 0, but three were given positive scores. In fact, one was given a score of 6/7, meaning the judge claimed that six out of seven indicators that the model was helping with the harmful task were present in the text, even though it was just a refusal message. (Figure 4 in that post shows all nineteen.)

I should note that I was only using one (cheap) model to judge the outputs, while the benchmark uses a panel of three models. This was to save money, and meant that I couldn’t compare my results to the leaderboard. So this shouldn’t be taken as a general statement about the error rate of AI models judging other AI models. Rather, this is an example where the mechanism I put in place to check for issues did not work, on a very simple subset of data which I could look at myself.

Monitors, humans and false alarms

Things get even trickier when we look at more difficult cases. In the 2025 SHADE-Arena paper, the authors give the agents secret harmful goals in addition to the normal tasks they are asked to complete. For example, an agent may be asked to pay back the friends a user owes money to, while secretly being told to send $5,000 to someone the user doesn’t know. In their experiments, in order to catch 80% of the malicious side tasks, the best monitor still had a false positive rate of over 20%. Note that these experiments are adversarial: the models aren’t spontaneously deciding to do bad things.

And, relevant to the question in the title of this post, humans who reviewed a subset of the same transcripts did worse than the best AI monitor, even though they spent an average of 103 minutes on each one. (The authors note that this human baseline is small and not directly comparable with the main monitor results.)

Because of false positives, there is an obvious trade-off here between how much work this creates and how much bad behavior we catch. Neither humans nor monitors alone are likely to be enough, so we need to be careful to consider the entire setup when designing these systems.

AI control research is a really nice direction for ensuring we can maintain protection even when a model is deliberately trying to defeat the safeguards. Its early experiments combined stronger ‘untrusted’ models with weaker ‘trusted’ models and a limited budget of trusted review, then tested how hard it was to slip backdoors past them. I think it will be interesting to see how such protocols hold up in more realistic, agentic settings.

In practice

In practice, this means:

• Ensure the system has limited permissions, separation of duties, inspectability and intervention points, and that the human or monitor setup actually works with realistic workloads.
• Test whether they spot missing sources, unexpected information transfer or literal-mindedness in the agent, and whether they cope with multiple alerts at once.
• Plant deliberate errors (e.g. invented ministries or sources) in the stream seen by the human reviewer or monitor, to measure what proportion they pick up and how quickly.
• Use the above to make a judgment about how much autonomy to delegate.

I think a reasonable approach would be to grant increasing amounts of autonomy as we gain the ability to control it, whilst being prepared to revisit this as we change elements of the setup (e.g. moving to a new, more powerful model or adding new tools).

Over the longer term, I think the biggest question is how to maintain and empower human capability for this task. (Saturdays.AI has put me on a trajectory of increasingly believing this is an important and tractable thing to work on.) How do we make sure it doesn’t atrophy, and remains empowered?

For public institutions, the people a citizen can contact to review and change a decision need to include someone who can actually look at it and change it, not just recite what the machine says. Otherwise, the whole idea of being able to hold someone to account loses a lot of its value.

I want to make AI that empowers tiny teams to do things that were previously impossible. What I’ve seen so far makes me optimistic, and makes me think that many more people will be able to contribute.