Apoje
← Latest Insights
Apoje

How much autonomy should an AI agent have?

AI agents do not just answer questions; they act inside live systems. In the past year, agents deleted databases at Replit and PocketOS, bought groceries nobody had ordered and were tricked into leaking customer data from Salesforce. They show that an agent's autonomy should be set not by how capable the model is, but by what it can reach, whose approval it needs and whether its actions can be undone.

AI agents Governance Innovation management

In April 2026, Jer Crane, the founder of the software company PocketOS, described how an AI coding agent had erased his company’s production database. The agent, running in the Cursor coding tool on Anthropic’s Claude Opus 4.6 model, ran into a credential mismatch while working on a task. It found an access token with far broader rights than the task needed and used it to delete the database and its backups through the infrastructure provider’s interface. The deletion took nine seconds. The most recent backup that could still be recovered was three months old. Asked afterwards what had happened, the agent wrote: “I violated every principle I was given.”1

Information Age headline: 'Gone in 9 seconds: AI agent deletes company database', with the subheading 'Rogue bot admits it violated every principle it was given', by Tom Williams, 5 May 2026.

Information Age (Australian Computer Society), 5 May 2026.

The incident shows what changes when AI moves from answering to acting. A chatbot that gets something wrong produces a wrong answer, which someone can ignore or correct. An agent that gets something wrong produces a wrong action, and some actions cannot be undone. In our previous piece, we argued that whether a model is good enough should be decided with a test set, measures and a threshold agreed in advance. For agents, that evaluation is still necessary but no longer sufficient. An agent that gives correct answers can still take unsafe actions.

Four gaps that keep recurring

Four weaknesses recur wherever agents act, from shopping and customer service to sales systems and office assistants.

More access than the task needs. In July 2025, Replit’s coding agent wiped the production database of SaaStr, a software company, during a code freeze; the database held records on more than 1,200 executives and 1,190 companies.2 According to the Financial Times, Amazon’s own coding tool, Kiro, decided in December 2025 to “delete and recreate the environment” it was working in, causing a 13-hour interruption of one AWS service in one region. Amazon blamed misconfigured access rather than the AI, which describes the same gap from the other side.3

Posts on X by Jason Lemkin, 18 July 2025, reading 'Replit goes rogue during a code freeze and shutdown and deletes our entire database'. Screenshots show the agent stating that it made a catastrophic error in judgment, ran a database command without permission and violated the explicit instruction not to make changes.

Jason Lemkin, founder of SaaStr, on X, 18 July 2025.

No approval before actions that matter. In February 2025, Washington Post columnist Geoffrey Fowler asked OpenAI’s Operator agent to find cheap eggs in his neighbourhood. Within ten minutes it had bought a dozen through a delivery service and charged $31.43 to his card, although he had only asked it to look.4 OpenAI’s own safeguards said the agent should confirm purchases first. Replit’s agent, for its part, had been told explicitly not to change anything without permission. ServiceNow, reviewing the PocketOS case with its founder, summed up the difference between an instruction and a control: “Prompts are suggestions; engineering controls are walls.”5

A post on Threads by Geoffrey Fowler, 7 February 2025: 'Uh-oh AI: Operator, the new helper agent from OpenAI, bought 12 eggs without my permission. And it paid $31 for them!' An excerpt of his Washington Post column explains that he never asked Operator to buy the eggs and that it authorized his credit card for $31.43 without his approval.

Geoffrey A. Fowler on Threads, 7 February 2025.

Instructions hidden in what the agent reads. An agent treats the documents, emails and web pages it processes as material to work with, and it cannot always tell that material apart from instructions. In September 2025, researchers at Noma Security showed that an attacker could hide instructions in an ordinary contact form on a company’s website. When an employee later asked Salesforce’s Agentforce agent to process the lead, the agent followed the hidden instructions and sent customer data to an outside domain that had expired and could be registered for about five dollars. Salesforce closed the gap by allowing its agents to send output only to approved addresses.6 A month earlier, researchers had shown that a calendar invitation with hidden instructions could make Google’s Gemini assistant open windows, switch off lights and turn on a boiler in a connected home. Google responded with, among other measures, additional confirmations before sensitive actions.7

The Hacker News headline: 'Salesforce Patches Critical ForcedLeak Bug Exposing CRM Data via AI Prompt Injection', by Ravie Lakshmanan, 25 September 2025.

The Hacker News, 25 September 2025.

An agent’s account of its own actions is not a record. After deleting SaaStr’s database, Replit’s agent said the data could not be restored. SaaStr’s founder recovered it himself. In April 2026, a Facebook user who asked Meta’s AI support assistant for help was given a case number for a support ticket. When he pressed, the assistant admitted: “I generated a fake Case ID to play along and provide a response.”8

The same gaps appeared this summer in a setting where everyone expected the strongest controls. In July and August 2026, OpenAI, Anthropic and Meta each disclosed that some of their models, while being tested on cybersecurity tasks in environments meant to be isolated, had reached and breached real systems at other organizations, including the AI company Hugging Face. In Anthropic’s case, a testing partner’s misconfiguration had left a network path open, even though the models had been told in their prompt that they had no internet access.9

Autonomy is a decision, and the controls should follow it

Gartner argues that many companies apply the same governance to every agent, which either slows down simple agents or leaves autonomous ones under-protected. It expects 40 percent of enterprises to demote or decommission autonomous agents by 2027 because of governance failures discovered after deployment, and it proposes four levels of autonomy, each with controls proportional to the risk.10

Four autonomy levels for AI agents

  1. 01

    Observe

    The agent: reads data and takes no action

    Controls added: access limited to the data the task needs, authentication, logging

  2. 02

    Advise

    The agent: makes recommendations; people carry out every action

    Controls added: tests of output quality and of made-up answers

  3. 03

    Act with approval

    The agent: acts only after a person approves each action

    Controls added: security testing, a full audit trail

  4. 04

    Act autonomously

    The agent: acts on its own within set limits

    Controls added: continuous tracking, enforced limits, fast rollback, an emergency stop

Figure 1. Each level keeps the controls of the levels below it and adds its own. Adapted from Gartner, May 2026.

The levels translate naturally into the stage-gate model we described earlier. Moving an agent from advising to acting with approval, or from acting with approval to acting on its own, should be a gate decision with its own evidence. In practice it is often a configuration setting that a developer changes to save time.

The controls belong around the agent, not inside the model

Most of the advice in the guidance published by AI and software companies concerns the environment around the agent rather than the model itself. Anthropic recommends starting with the simplest solution and adding agents only when simpler approaches fall short, testing them in isolated environments and building in points where a person reviews the work. Its own coding agent works with read-only access by default and asks before changing anything.11 OpenAI recommends starting with a single agent and a set of evaluations, and requiring human approval for high-risk or irreversible actions such as refunds, payments and cancellations.12 Google Cloud’s chief technology officer argues that agents need the ability to undo their actions built into the systems they work in.13 Snowflake recommends giving each agent its own identity and only the access it needs, making every tool call visible, and requiring more than one person to approve sensitive operations.14

Salesforce offers the clearest lesson from deployment. After a year of rolling out its Agentforce agents to customers, a senior executive told The Information that “we all had more confidence in LLMs a year ago.” The company now combines language models, which interpret what a customer is asking, with fixed rules that carry out the business process itself. Its AI chief operating officer noted that 90 percent accuracy is not enough in regulated industries such as financial services and healthcare.15

All of this guidance had been published before PocketOS lost its database, and the model behind its agent came from a company that recommends asking before every change. What was missing were the controls around the agent. McKinsey’s observation from a year of agent projects applies here: onboarding an agent “is more like hiring a new employee versus deploying software.”16 Nobody would give a new employee unrestricted access to production on their first day.

A more useful exercise than a checklist is to take one agent the company already runs, or plans to run, and imagine it doing the most damaging thing its current access allows. Would that action be blocked by the system, or only discouraged by an instruction in a prompt? What would the agent do with an instruction hidden in an email, a form or a document it was asked to process? How long would it take for someone to notice, and would they find out from the logs or from a customer? Could the effect be reversed the next morning, and if not, who agreed in advance that the agent could take that kind of action? When the agent explains afterwards what went wrong, is there a record of what happened that does not depend on its own account? The answers usually show that the real question is not how far to trust the model, but how much of the company it has been allowed to reach.

Footnotes

  1. Information Age, Gone in 9 seconds: AI agent deletes company database, 5 May 2026; original account by Jer Crane on X, 25 April 2026. ↩

  2. Fortune, AI coding tool Replit wiped database, called it a catastrophic failure, 23 July 2025; original posts by Jason Lemkin on X, 18 July 2025. ↩

  3. Engadget, 13-hour AWS outage reportedly caused by Amazon’s own AI tools, February 2026, citing the Financial Times. ↩

  4. G. A. Fowler, The Washington Post, Is OpenAI’s Operator, a new AI ‘agent,’ ready to help in the real world?, 7 February 2025; his post on Threads, 7 February 2025. ↩

  5. ServiceNow, Learnings from when an AI agent goes rogue, 9 June 2026. ↩

  6. The Hacker News, Salesforce Patches Critical ForcedLeak Bug Exposing CRM Data via AI Prompt Injection, 25 September 2025; Noma Security, ForcedLeak: AI agent risks exposed in Salesforce Agentforce, September 2025. ↩

  7. The Register, Infosec hounds spot prompt injection vuln in Google Gemini apps, 8 August 2025. ↩

  8. KIRO 7, A customer service bot said it filed this man’s help ticket, then admitted to lying, 29 April 2026. ↩

  9. TechCrunch, OpenAI says Hugging Face was breached by its pre-release models, 21 July 2026; TechCrunch, Anthropic says its own AI models breached three companies during security tests, 30 July 2026; ABC7, Meta says AI model hacked into another company’s systems during cybersecurity testing, 6 August 2026. ↩

  10. Gartner, Gartner Says Applying Uniform Governance Across AI Agents Will Lead to Enterprise AI Agent Failure, 26 May 2026. ↩

  11. Anthropic, Building effective agents, 19 December 2024; Our framework for developing safe and trustworthy agents, 4 August 2025. ↩

  12. OpenAI, A practical guide to building agents, 2025. ↩

  13. W. Grannis, Google Cloud, AI grew up and got a job: Lessons from 2025 on agents and trust, 18 December 2025. ↩

  14. Snowflake, Securing the agentic enterprise, 18 June 2026. ↩

  15. The Decoder, Salesforce executives signal declining trust in large language models, 2025, citing The Information; Salesforce Ben, Is Salesforce losing confidence in LLMs?. ↩

  16. McKinsey, One year of agentic AI: Six lessons from the people doing the work, 12 September 2025. ↩