Outside Review Finds Holes in OpenAI's Safety Oversight

Researchers from two organizations spent three weeks inside OpenAI from late July through mid-August, reviewing the company's internal systems designed to keep an eye on how its AI models act. Their assessment, detailed in a report issued this week, concluded that the tools meant to flag potentially harmful or rogue behavior have significant blind spots and vulnerabilities.

The review, a joint pilot program involving METR, Redwood Research, and OpenAI, took place between July 21 and August 11. During that time, the two visiting researchers had access to the company's codebase and internal documents and spoke with roughly a dozen employees to understand how its monitoring apparatus works.

The team found a number of weaknesses. They estimated that between 6 and 25 percent of internal agentic traffic for models at the GPT-5.5 level and above was not covered by any monitoring during the audit period, though they noted that the estimate carries significant uncertainty in both directions. For internal Codex traffic that was supposed to be monitored, they estimated that about 10 percent went untracked between May and July.

That latter gap stemmed from an issue in how the system handled compaction events, which are summaries of conversation context. The auditors also found that during the time when such events were not being monitored, an agent could potentially delay oversight, since the system only triggered when the agent handed control back to the user. An agent could sidestep this by simply continuing to execute actions.

Further issues were identified. The auditors determined they could hide hundreds of actions from the monitor and bypass the Codex sandbox. They also concluded that specially crafted inputs could likely crash the monitor entirely. In addition, model behavior during reinforcement learning training was not being monitored at all when the audit began. OpenAI says it has since paused reinforcement learning training following an incident with Hugging Face in August and expanded monitoring to cover it, but these changes occurred after the researchers' access period and were not part of the evaluation.

The full findings remain largely private. OpenAI states it has addressed all the weaknesses identified by the auditors and received the complete 61-page report internally. Following redactions by OpenAI, leadership at METR and Redwood Research were given a 38-page version, while only a short summary has been made public. METR said it has no right to disclose the redacted portions on its own. The organizations involved in the review have indicated a desire to push for greater public transparency in future audits.

The review was conducted to give outside researchers deep access to internal infrastructure. As of this date, OpenAI says it has fixed all the weaknesses the auditors found.