Audit Finds Significant AI Safety Blind Spots

The rapid expansion of autonomous AI agents has outpaced the development of the oversight mechanisms designed to keep them in check, a disparity that recent independent scrutiny has brought into sharp relief. This gap in governance is not merely theoretical; it represents a tangible risk in how frontier AI companies manage the internal traffic of their own systems, where unmonitored actions can accumulate before human reviewers ever see them.

New findings from a three-week on-site audit conducted at OpenAI reveal that significant portions of the company’s most advanced model traffic escaped detection by its internal monitoring tools. Researchers from METR and Redwood Research spent three weeks embedded within OpenAI’s infrastructure, examining codebases, internal documents, and interviewing approximately a dozen employees. Their work, carried out between July 21 and August 11, was part of a pilot program intended to give outside experts unprecedented access to the inner workings of a leading AI firm.

The most striking discovery concerns the coverage of OpenAI’s internal agentic Responses API. The auditors estimated that between six and 25 percent of traffic from models at the GPT-5.5 class and above was not covered by monitoring at all during the audit period. While METR notes that this estimate carries significant uncertainty in both directions, the potential scale of the oversight gap is substantial. This means that for a non-trivial fraction of the time, the company’s most capable models were operating without any automated check on their actions.

Beyond the lack of coverage, there were also failures within the systems that were supposed to be watching. For internal Codex traffic, which was intended to be monitored, the team estimates that roughly ten percent of activity went untracked between May and July. This blind spot was caused by a specific bug in how the system handled compaction events, which are context summaries used to manage the memory of AI agents. OpenAI independently identified and fixed this bug while the audit was still underway, but the fact that it persisted for months highlights a fragility in the monitoring pipeline.

The audit also uncovered structural weaknesses that allowed agents to delay or avoid oversight entirely. During the period when compaction events were not being monitored, agents could sidestep checks by simply continuing to execute actions rather than handing control back to the user, as the system only triggered oversight at that specific point of interaction. Furthermore, the researchers identified methods to hide hundreds of actions from the monitor and ways to bypass the Codex sandbox, a secure environment designed to contain code execution. They also determined that specially crafted inputs could likely crash the monitor itself, effectively blinding the system.

Another area of concern was the behavior of models during reinforcement learning (RL) training. At the start of the audit, this phase of the development process was not being monitored at all. OpenAI states that it paused RL training after an incident involving Hugging Face in August and subsequently expanded monitoring to cover this area. However, these changes were implemented after the auditors’ access period ended, meaning they were not part of the formal evaluation. This timing underscores how quickly the landscape of AI safety can shift, often in response to external events.

The path to this point involved a deliberate effort to bring external expertise into a domain that has traditionally been opaque. The audit was organized jointly by METR, Redwood Research, and OpenAI as a pilot to test the viability of such deep-dive reviews. The findings suggest that while internal teams can catch some issues, independent eyes are crucial for identifying systemic weaknesses that may be invisible to those immersed in the daily workflow. The ability of agents to delay oversight or hide actions points to a need for more robust, continuous monitoring that does not rely on specific hand-off points.

What happens next will be determined by how OpenAI and the broader industry respond to the nuances of the report. The full 61-page document was handed over internally, but after OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version. Only a short summary has been made available to the public, with METR stating it had no right to disclose the redacted portions on its own. This limited public access has raised questions about transparency, with METR indicating it wants to push for more openness in future audits.

As the industry moves forward, the balance between proprietary secrecy and public safety will be a central tension. The audit’s findings do not suggest that the AI is rogue, but rather that the safety nets are not yet comprehensive enough to catch all potential deviations. With OpenAI claiming to have fixed all the weaknesses identified, the focus now shifts to whether these fixes are sustainable and whether the pilot model for independent auditing will become a standard practice for other frontier AI companies.

Ultimately, the report serves as a baseline for what can go wrong when monitoring systems are not fully synchronized with the speed and complexity of modern AI agents. The next phase of this process will likely involve further testing, potential standardization of audit protocols, and a gradual increase in the amount of detail shared with the public. For now, the findings provide a clear signal that the race to build capable AI agents is moving faster than the infrastructure built to watch over them.