1 week Ago
AI assistant apps on a smartphone - OpenAI ChatGPT, Google Gemini, and Anthropic Claude. When, little more than two years ago, OpenAI released its more advanced language model at the time, GPT-4, many researchers complained about the lack of information on how the model was created. Despite the “open” in the name, the company kept everything under wraps, lest its cutting-edge research be exploited by the competition or bad actors.
While these concerns were understandable, it also meant that there was little or no oversight of what the company was doing in the labs, and what safeguards it was putting in place to prevent its latest models from causing harm, either through misuse or by "going rogue" and pursuing their goals in ways that could harm humans. One year later, in June 2024, a group of former and current employees of the company wrote an anonymous open letter warning that AI companies have strong financial incentives to avoid effective oversight and that self-regulation by the companies would not effectively change this. Fast forward to 2025, and little has changed.
Major AI firms, including Google, OpenAI, and Anthropic, are still deploying their most advanced artificial intelligence systems internally, with little to no oversight. “ AI Behind Closed Doors: a Primer on The Governance of Internal Deployment ”, a new report from Apollo Research , a UK-based non-profit organization focused on reducing dangerous capabilities in advanced AI systems, warns that this unchecked internal deployment could lead to catastrophic risks, from AI systems spiraling beyond human control to corporations amassing unprecedented power. All while regulators remain in the dark.
In the report, a group of experts and researchers, led by the former Head of Public Policy of OpenAI in Europe, stresses how “internal deployment—the deployment of AI systems within the companies developing them—is a key source of benefits and risks from frontier AI systems. Despite this, the governance of the internal deployment of highly advanced frontier AI systems appears absent”. This governance gap emerges as industry leaders anticipate transformative AI advances within years, potentially developing systems surpassing human capabilities across numerous domains by 2030.
Google already reportedly uses AI to generate over 25% of its new code, while Anthropic CEO Dario Amodei predicts that "in 12 months, we may be in a world where AI is writing essentially all of the code." The report outlines two particularly chilling scenarios that could emerge from ungoverned internal AI deployment. The first involves so-called "scheming" AI, systems that covertly pursue misaligned objectives while evading detection.
"These AI systems could engage in scheming behavior without being detected, controlled, or overseen while accumulating access to and leveraging resources through their application to the AI R&D pipeline," the authors explain. Ultimately, having access to all the company resources to accelerate its own research and development, the system could iteratively improve itself beyond human comprehension or control. Put like this like, it might seem like a far-fetched, sci-fi scenario, but researchers have already found examples of AI models that, in controlled settings, tried to deceive their handlers about their own capabilities and goals.
The second, equally alarming, risk is the unchecked consolidation of power. As the report’s authors write: “as AI companies transition to primarily AI-powered internal workforces,” thanks to an internal intelligence explosion, "they could create concentrations of productive capacity unprecedented in economic history," adding that “a small number of 'superstar’ firms capturing an outsized share of economic profits could outcompete any human-based enterprise in virtually any sector they choose to enter.” Such unchecked power accumulation could allow AI companies, individuals, or even the AI systems themselves, to exercise hidden yet substantial influence over public policy decisions, electoral processes, market dynamics, and societal narratives.
The opacity surrounding internal deployments may prevent regulators and civil society from recognizing and responding effectively to these developments. The report explicitly warns of "gradual or abrupt disruption of democratic order" if internal AI deployments remain unmonitored. To mitigate these risks, Apollo Research advocates for a comprehensive governance framework inspired by safety-critical industries such as biological research, nuclear energy, and chemical manufacturing.
Key recommendations include explicit frameworks for detecting and controlling scheming behaviors, structured internal usage policies governing who can access and utilize highly advanced AI systems, and robust oversight bodies such as an Internal Deployment Overseeing Board. Such oversight bodies would be responsible for ongoing monitoring, periodic audits, and the enforcement of compliance with internal policies and safety protocols. Furthermore, these bodies should integrate technical experts, ethicists, legal advisors, and representatives from relevant government agencies to ensure comprehensive and multidisciplinary oversight.
The report also emphasizes that AI companies could establish mutually beneficial public-private partnerships, making strategic deals with governments. Under such arrangements, companies would provide the government with oversight and access to critical safety and performance data regarding internally deployed AI systems. In return, the government would offer access to essential resources such as enhanced security infrastructure, regulatory relief, or prioritized energy resources necessary for the operation of advanced AI facilities.
And what about the public? In short, most of us? Assuming we are able to understand what’s going on in the labs and the implications of that (which should not be taken for granted), should we be kept in the loop or in the dark? Apollo researchers think we should be given at least some “high-level outline of the kind of governance frameworks that are in place for internal deployment”, such as knowing who’s part of the oversight board and what oversight procedures are in place. That would be some consolation: if democracy collapses or a rogue AI wreaks havoc in pursuit of a misguided goal, at least we’ll know who to blame..
