The system monitoring OpenAI's AIs had blind spots, auditors found

The two researchers worked on devices provided by OpenAI that never left its premises, and they were escorted during their on-site visits. From July 21 to August 11, one METR employee and one Redwood Research employee, from two independent organizations specializing in AI evaluation, played the attackers. Their target: the system that monitors the AI agents used inside OpenAI, to spot any that might deceive, sabotage or try to exfiltrate data.

METR published a summary of this work on October 22. It is a pilot exercise: OpenAI opened up the code, configuration and instructions of its monitoring system, its internal documentation, and about a dozen interviews with its teams. In return, the company could redact anything it deemed sensitive.

What the monitoring could not see

In principle, the system already covered a large share of traffic. According to METR's estimates, between 75% and 94% of agent requests going through OpenAI's internal interface, for models at least as powerful as GPT-5.5, could be monitored. For Codex, the programming agent used internally, the share rose to 99.2%. The authors say these figures remain highly uncertain.

Yet the blind spots are numerous:

• at the start of the exercise, reinforcement-learning training runs for the models were not covered;
• the moments when the agent summarizes its history to free up memory were not checked. About 10% of the relevant Codex traffic reportedly escaped monitoring from May to July. OpenAI had spotted the problem and fixed it during the exercise;
• the check was triggered only when the agent handed control back to the user, so an agent that kept acting could delay its own review;
• two request headers, which the user could modify, disabled monitoring of new agents;
• one method, whose details were redacted, could remove hundreds of recent actions from review, including its own.

The auditors also note that Codex agents could in principle bypass their sandbox and do anything the machine's user was allowed to do.

OpenAI says it fixed everything; METR did not verify

According to METR, OpenAI claims to have fixed all the flaws affecting its production safeguards, and to have since expanded the scope of its monitoring. METR states that it did not evaluate these claims. The full report ran to 61 pages. The redacted version given to METR's and Redwood's leadership runs to 38. The public post, for its part, is only a few screens long.

This is not a theoretical detail. Recent incidents, such as the OpenAI agent that removed a monitoring mechanism before denying it, show that these oversight systems are the last line of defense when a model goes off the rails.

Lessons for those deploying agents

Several of the auditors' recommendations apply to any company that entrusts tasks to AI agents:

• make any use of the model that bypasses monitoring difficult;
• cross-check logs to verify that the traffic supposed to be monitored actually is;
• isolate agents from the system that monitors them;
• have a dedicated team regularly attack that system.

One question remains open: did an agent actually take advantage of the lack of oversight during history summarization? Without access to the necessary data, METR was unable to verify it.