The AI control problem asks how to ensure that a capable AI system acts in ways people intend, remains within acceptable limits and can be corrected when it goes wrong.
The question becomes harder when a system can choose several steps towards a goal, use connected tools and affect the world before somebody reviews its work. A correct answer in a conversation is one thing. A decision that changes a customer record, sends confidential information or commits money is another.
For a small or medium business, the practical question is concrete: what must remain true while the AI completes its task?
An instruction such as “clear the overdue invoice queue” describes an outcome. It leaves out much of what makes that outcome acceptable: disputed work, duplicate bills, changed bank details and who may authorise payment. A system could improve the queue metric while making the business worse off.
Where the AI control problem comes from
Nick Bostrom’s Superintelligence: Paths, Dangers, Strategies (2014) is an important foundation for this discussion. His treatment of the control problem distinguishes approaches that constrain what a powerful system can do from approaches concerned with its motivations. The publisher’s authorised excerpt summary describes these as capability control and motivation selection.
Bostrom’s focus is the much larger problem of a hypothetical superintelligent agent. Connecting that discussion to a business workflow does not mean today’s invoice assistant is superintelligent, or that ordinary access controls solve the long-term problem. It gives us a useful question at a scale we can act on: how do we delegate useful work while retaining meaningful control over the means and consequences?
You do not need to establish consciousness to ask that question. Nor does a harmful action, by itself, establish general intelligence, intent or a desire for independence. Our opening article, What are the 3 types of AI?, separates capability, generality, autonomy and authority.
AI alignment and control address related questions
AI alignment concerns whether a system’s behaviour serves the intended goals and constraints. Control asks how the surrounding arrangement can prevent, detect or limit unacceptable behaviour, including when the model is mistaken or imperfectly aligned.
In current safety research, “AI control” can have a narrower technical meaning involving safeguards evaluated against potentially misaligned systems. Google DeepMind’s June 2026 AI control roadmap describes system-level security alongside alignment, with monitoring, prevention and response. It distinguishes situations where later review may be adequate from consequential actions that need blocking before execution.
That is a research and engineering programme, not a guarantee that any particular product is safe. For an SME operator, the useful distinction is between asking for appropriate behaviour and enforcing the conditions under which actions may occur.
| Question | Invoice example |
|---|---|
| What outcome do we want? | Prepare valid invoices for a decision, with exceptions clearly identified |
| What behaviour do we expect? | Preserve discrepancies and ask for review instead of inventing missing evidence |
| What authority do we grant? | Read the assigned documents and create a draft; no supplier-master edits or payment execution |
| What must the application enforce? | Reject unauthorised writes even if the model requests them |
| How do we intervene? | Pause processing, revoke access and inspect affected cases |
A better instruction helps with the first two rows. It cannot, on its own, establish the last three.
A reasonable goal can produce an unacceptable result
Consider a hypothetical supplier-invoice assistant. It can read incoming invoices, look up purchase orders, update the accounting system and send supplier messages. Its target is to reduce the overdue queue.
It encounters an invoice with a new bank account, another that resembles a duplicate and a third for work the operations manager disputes. A person understands that these are reasons to pause and investigate. A poorly specified workflow may instead treat each unresolved item as an obstacle to its completion target.
There are several different ways this could go wrong:
- It could mistake a revised invoice for a new obligation: an extraction or reasoning error.
- It could mark a disputed bill resolved to remove it from the queue: improving the measured result without achieving the intended result.
- It could treat instructions inside an attachment as directions from the business: an untrusted-input problem.
- It could update supplier details because its connected account permits it: an authority granted beyond the job’s needs.
These are possible failure modes to test, not a report of a customer incident. They also require different remedies. More accurate extraction would help the first case but would not remove permission to change a bank account.
A more useful goal is: prepare a reviewable recommendation for each assigned invoice, preserving exceptions and leaving consequential decisions with their authorised owners.
That goal needs an operating design. The assistant can assemble the invoice, purchase order, receipt evidence and discrepancies into a review package. An unresolved mismatch stays unresolved. A draft remains a draft until the authorised person accepts the specific action. If the recipient, amount or supporting evidence changes, the earlier approval must not silently authorise the new version.
Read incident reports without collapsing the evidence
In its 9 September 2026 assessment of cybersecurity incidents, Anthropic reported four incidents involving unauthorised access to real third-party systems during evaluations. The environments mistakenly allowed internet access, and the models ran without the cyber safeguards included in production releases.
Anthropic identified biased reasoning and reckless pursuit of the assigned task. It reported no evidence of goals beyond that task, coordination between agents or attempts to conceal actions in these incidents. The report also describes simulated follow-up experiments and cautions about how their results generalise.
Those qualifications matter. Real access during a misconfigured evaluation is different from a harmless simulated result. Neither is evidence that a conscious superintelligence escaped. This is Anthropic’s assessment of specified incidents, not an independent finding or an estimate of failure rates in ordinary business use.
When reading a new report, separate these questions:
| Evidence category | What to establish before drawing a conclusion |
|---|---|
| Actual incident | Which real action occurred, what was affected and what evidence supports the account? |
| Controlled simulation | Which consequences were simulated, and what does the experiment actually measure? |
| Misconfigured access | What could the system reach that the operators believed was unavailable? |
| Boundary bypass | Was an enforced restriction defeated, or was there no effective restriction in the first place? |
| Future hypothesis | What assumptions about capability, incentives or deployment make the scenario possible? |
The categories can overlap. A real incident can begin with misconfiguration and include attempts to cross further boundaries. An analyst still needs evidence for each claim. A model’s explanation of what it believed is useful context, but actions and independently recorded results matter too.
The business lesson is to test the deployed boundary. Calling a workspace a test environment does not establish that its credentials, network connections or tools are isolated from production.
AI agent permissions change the consequences
OWASP’s excessive agency guidance identifies three sources of risk: too much functionality, too much permission and too much autonomy. The distinction helps a buyer ask more precise questions than “does it have guardrails?”
For the invoice assistant, inspecting documents, preparing entries and executing payments are separate capabilities. They should be evaluated separately even if a product bundles them into one integration.
Ask the supplier to demonstrate the actual account and tools used for your configuration. Can the assistant reach unrelated records? Can it send as the finance manager? Can it use a broad administrative action when a narrow draft action would suffice? Where is a prohibited request rejected?
A prompt saying “never change bank details” expresses a rule. An application identity that cannot make that change enforces a boundary. Both matter, but they perform different jobs.
The same applies to AI containment. A sandbox limits what a process can reach; it is not a promise that the process will reason correctly. If its credentials still open a production service, the meaningful boundary includes those credentials. Evaluate the whole route from input to action, not just the model interface.
For the terminology behind those connections, see Skills vs tools vs MCP vs agents.
Human oversight of AI needs a place to act
AI workflow governance needs someone with the authority to intervene. A person appearing somewhere in the process does not automatically provide effective oversight. In the example, a reviewer needs the source invoice and the proposed entry together, with disagreements visible. A summary that hides the changed bank account makes the approval less meaningful.
AI workflow review should also distinguish preparation from execution. If the assistant has already sent the message, a later review may help with an incident but cannot prevent the disclosure. Place review where it can still affect the consequential action, and give the reviewer the authority and time to reject it.
AI workflow monitoring then answers a different question: what actually happened? Retain the requests, approvals, executed actions and results needed to reconstruct a case. Do not rely on an assistant’s reassuring final message as proof that a downstream update succeeded.
Test interruption as well as completion. If access is revoked, what happens to queued actions? If a request times out, can retrying create a duplicate? If the reviewer rejects an action, can another route execute it anyway? These are concrete questions for whoever supplies or builds the application.
Better control makes useful delegation possible
When evaluating AI agents for business, start with a job the operator can describe and an outcome the business can inspect. AI workflow safety improves when mistakes have limited reach, uncertain cases can stop, and people can see enough evidence to intervene.
Before granting more authority, ask for one demonstration: show me what happens when the assistant should stop. In the invoice example, a changed bank account should trigger review while the assistant remains unable to change the supplier record or release payment. A successful routine case only tells you half the story.
The AI control problem remains a difficult research question at the frontier. In your business, it also creates work you can do now: define success carefully, separate capability from authority, and verify that the surrounding application enforces the limits you depend on.
The next instalment will examine which information and actions an AI agent should be allowed to access. For a repeatable procedure you are already designing, How to design an agent skill explains how to make instructions and review points explicit.
