Governing AI agents: what changes when a system acts, not just advises

AI agent governance means bounding what an agent may do before it acts, recording what each agent can reach and who answers for it, and keeping a person able to see and stop it while it runs. When a system acts rather than advises, oversight moves from reviewing outputs to setting limits on actions.

Governing agents is an advanced governance problem. It tends to arrive after an organisation has the straightforward work in hand, a register of its systems, risk assessments, derived controls and policies, and is extending its practice into the harder cases. It sits alongside impact assessments and incident handling as one of the more demanding forms of governance a maturing function takes on, not as the whole of the work.

What actually changes

The shift from advice to action changes three things about how you oversee a system.

The unit of oversight moves from the output to the action. With an advisory system you review what it produced. With an agent you have to bound what it is allowed to do before it does it, decide which actions it may take without a person and set which actions always stop for approval. Reviewing a good answer after the fact is not oversight if the agent has already sent the email or changed the record.

The surface area grows with every tool the agent can reach. An advisory model is largely contained by its prompt and its output. An agent is defined by its permissions, the systems it can read from and write to, the credentials it holds and the other agents or services it can call. OWASP’s work on agentic threats describes categories that only appear once a system acts, among them the misuse of the tools an agent is given and the compromise of the memory or context it relies on to decide what to do next. The risk lives in what the agent can touch, not only in what it says.

Accountability does not move. This is the part senior owners ask about first, and it has a settled answer. An agent acting on its own does not become the accountable party. A named person remains answerable for what the agent does, and the governance exists to keep that person in genuine control of a system that is designed to act without them at each step.

What to record about each agent

Recording an agent is the first governance step, not the last. You cannot assess an agent, oversee it or answer for it if you do not know it exists and cannot describe what it can do. A record that only names the tool is not enough. An agent record covers more than a system record, because an agent acts.

Purpose. What the agent is, the goal it is set and the scenarios it runs in. Record enough that someone who has never seen it can tell what it is for.

Owner. The named person accountable for the agent. A person, not a team.

Permissions and reach. The systems the agent can read from and write to, the credentials it acts with and the other agents, tools or services it can call. A system record usually skips this, and for an agent it matters most, because it is the surface area of everything the agent could do.

Autonomy. Which actions the agent may take on its own and which stop for a person to approve. Record the line, because the line is the thing you are governing.

Risk position. Whether the agent has been assessed, what the assessment found and where the full assessment lives.

Oversight. How a person can see what the agent is doing, how they can stop or override it and who watches it in operation.

Key dates. When the agent went into use, when it was last reviewed and when it is next due.

Why the record has to stay current

An agent is not static. Its prompt is tuned, a tool is added, its permissions are widened to cover a new task. Each change can move the agent’s risk without anyone reopening the paperwork, and a record that described the agent accurately at launch can quietly stop describing the agent that is running now.

That is why the record is versioned and reviewed rather than written once. When the agent changes, the record changes with it, and earlier versions stay as the account of what the agent was and when. ISO/IEC 42001 frames AI governance as a management system for this reason, so that the record of a system reflects it across its life and not only on the day it was signed off.

A current record does real work. The owner can oversee the agent because they can see what it can do. You can assess and reassess it from an accurate picture of its reach and autonomy. And when a board, an auditor, an enterprise customer or a regulator asks, you can show what the agent is, what it can do and who answers for it, with the history to back it. The record says what the agent may do. What it actually did belongs in its decision log.

The questions being asked at senior level

These are the questions that reach a board or an audit committee once agents are in use.

Who is accountable for this agent, and can they actually control it? Oversight of an agent means a person can see what it is doing and can stop or override it, not that a person signed off once at the start. ISO/IEC 42001 asks for defined roles and responsibilities for AI, and the NIST AI Risk Management Framework places accountability structures in its Govern function.

What is this agent allowed to do, and what has it done? This is a records question before it is a technical one. It is answered by the agent record above and by a decision log kept while the agent runs.

Have we assessed the risk of letting it act? The NIST framework organises this as Govern, Map, Measure and Manage, and its Generative AI Profile (NIST AI 600-1) speaks to the risks that generative and agentic systems introduce. For a high-risk agent the assessment is the impact assessment you would run for any consequential system, extended to cover the agent’s autonomy, its reach into other systems and the way it fails. How to assess and oversee an AI agent sets out the method.

The guidance that applies

None of the main frameworks is agent-only, and that is the point. ISO/IEC 42001 sets out the management system, ISO/IEC 23894 covers AI risk management and ISO/IEC 42005 covers impact assessment. The NIST AI Risk Management Framework and its Generative AI Profile give the risk vocabulary, and OWASP’s agent-specific threat work covers the attack surface. Where an agent falls within a high-risk use under the EU AI Act, the Act’s human oversight requirements apply to it as to any other high-risk system. Governing agents is the governance you already run, applied to systems that act, with the added obligations that action brings.

In Australia

The current Australian anchor is the Guidance for AI Adoption, published by the National AI Centre in the Department of Industry, Science and Resources in October 2025. It superseded the Voluntary AI Safety Standard. Its six essential practices begin with deciding who is accountable and end with maintaining human control, and both read directly onto agents. The first says a named person answers for the system. The sixth says a person must be able to intervene in it.

Where Aicura fits

The move from advice to action is where Aicura’s agent support starts. An agent sits in the AI Register as a record of its own kind, with the same three owner roles and version history as a system, and a person decides whether a tool that discovery reports becomes an agent record. At Pro the record also carries the tools the agent declares and the runtime identities (credentials) it acts under, the declared tools become the list of tools the gateway permits, and decision records gathered from your environment flag any call to a tool that was never declared. When an agent does something it should not have, Incidents is where it is recorded.

Aicura supports the work and surfaces the picture. It does not certify your agents or decide whether an agent is safe to run. The accountable owner makes that call.

Sources

• AI Risk Management Framework (AI RMF 1.0) and Generative AI Profile (NIST AI 600-1), NIST, nist.gov/itl/ai-risk-management-framework
• ISO/IEC 42001:2023, Information technology, Artificial intelligence, Management system
• ISO/IEC 23894:2023, Information technology, Artificial intelligence, Guidance on risk management
• Agentic AI: Threats and Mitigations, OWASP GenAI Security Project, genai.owasp.org
• Guidance for AI Adoption, National AI Centre, Department of Industry, Science and Resources, industry.gov.au

A note on this page

This guide explains current guidance on governing AI agents and how to approach the work. It is general information, not legal advice, and it does not tell you whether a particular agent meets a particular obligation. For how the guidance applies to your organisation and your agents, read the primary sources above and take your own professional advice.