How to assess and oversee an AI agent

To assess and oversee an AI agent, assess it as you would any consequential system, then extend the work to cover what it can reach and what it may do on its own, and put oversight in place that lets a person see and stop it while it runs.

This guide assumes you already have the agent recorded. If you do not, start with what to record about each agent and come back.

Work through it for each agent that warrants the attention. Not every agent needs a full assessment, but any agent that can take a consequential action on its own does.

Before you assess: know what the agent can do

An assessment is only as good as your picture of what the agent actually touches. Before you weigh the risk, confirm the agent’s record gives you four things. These are its purpose, its owner, its reach (the systems, credentials and tools it can use) and its autonomy (which actions it takes alone and which stop for approval).

Reach and autonomy matter most here. Reach is the surface area where most agent risk lives. Autonomy is the line you are governing, because an agent that can draft a message and one that can send it are different systems.

Step 1: Map how the agent could cause harm

Take the agent’s purpose and reach and ask, concretely, what could go wrong when it acts. The NIST AI Risk Management Framework calls this Map, building a clear picture of the risks in context before you try to measure or manage them.

Cover the harms an advisory system could cause, such as a wrong or biased output, a privacy breach or a poor decision. Then add the harms that only appear because the agent acts. OWASP’s agentic threat work is a useful prompt here. It points to the agent misusing the tools it has been given, acting on a poisoned or manipulated context, holding more access than its task needs and chains of actions that carry a small error a long way before anyone sees it. For each, ask what the agent could do in your systems and who would be affected.

Step 2: Weigh likelihood and consequence

For each harm you have mapped, weigh how likely it is and how serious it would be if it happened. The NIST framework calls this Measure, and ISO/IEC 23894 sets out the same analysis for AI risk generally. Two factors specific to agents raise the weight.

Reach raises consequence. An agent that can change customer records, move money or send external communications can do more damage per action than one confined to a sandbox. The more the agent can touch, the higher the consequence side of every harm.

Autonomy raises the likelihood of an unreviewed harm. The more an agent does without a person approving each step, the more likely a harm reaches the world before anyone catches it. An action that always stops for approval carries a human check. An action the agent takes on its own does not.

Testing sharpens both estimates. Red-team runs and agent evaluations against the real deployment show how the agent behaves under the attacks you mapped in Step 1, rather than how you expect it to behave.

This step tells you which agents are high-risk and warrant a full impact assessment under a method such as ISO/IEC 42005, and which are contained enough to record and monitor without one.

Step 3: Set the oversight that lets a person intervene

For an agent, oversight is not a sign-off at the start. It is a person being able to see what the agent is doing and being able to stop or override it while it runs. Put four things in place.

Set an approval line. Decide which actions the agent may take on its own and which always stop for a person. Set the line by consequence, so the actions that are hard to undo or that reach outside the organisation are the ones that wait for approval.

Bound the agent’s permissions. Give the agent only the access its task needs, and no standing access it does not use. This limits the surface area you assessed in the first place and contains the harm if the agent behaves in a way you did not expect. A list of permitted tools enforced outside the agent is stronger than an instruction inside its prompt.

Build a way to see and stop it. A person needs a live view of what the agent is doing and a way to halt it. Oversight that cannot intervene until after the fact is monitoring, not control.

Set a monitoring and review schedule. Decide who watches the agent in operation, and set a date to reassess it. Agents change as their prompts, tools and permissions change, so a review that was accurate six months ago may not describe the agent running today.

Step 4: Record the assessment and keep it current

An assessment that lives in someone’s head is not evidence. Record the assessment against the agent, note the risk decision and who made it, then set the date it is next reviewed. When the agent’s permissions or autonomy change, the assessment is due again.

In Australia

The Guidance for AI Adoption, published by the National AI Centre in October 2025, superseded the Voluntary AI Safety Standard. Its essential practices on understanding impacts, measuring and managing risk, testing and monitoring, and maintaining human control map onto Steps 1 to 4 above. Its first practice, deciding who is accountable, is the owner you confirmed before you started.

Where Aicura fits

From Essentials, registering an agent in Aicura starts its risk assessment, which drafts a rating, rated risk items and suggested controls from the agent’s record in the AI Register. A material change to that record, such as a new tool, re-runs it against the new version, which covers the reassessment in Step 4. At Pro, impact assessments sit beside it, and security engines, test runners installed in your own environment, run red-team and agent evaluation tests against the agent’s real deployment, which feeds Step 2.

At Pro, for Step 3, the tools an agent declares become a list of permitted tools enforced at the gateway rather than an instruction in its prompt. Guardian, the part of Aicura that acts when monitoring spots a problem, can narrow an agent’s permissions or revoke the credentials it acts under when a condition you set is met, at the level of approval you choose. The assessment is a draft for your judgement. You weigh the risk, and the accountable owner decides whether the agent runs.

Sources

• AI Risk Management Framework (AI RMF 1.0) and Generative AI Profile (NIST AI 600-1), NIST, nist.gov/itl/ai-risk-management-framework
• ISO/IEC 23894:2023, Artificial intelligence, Guidance on risk management
• ISO/IEC 42005:2025, Artificial intelligence, AI system impact assessment
• Agentic AI: Threats and Mitigations, OWASP GenAI Security Project, genai.owasp.org
• Guidance for AI Adoption, National AI Centre, Department of Industry, Science and Resources, industry.gov.au

A note on this page

This guide is general information on how to assess and oversee an AI agent against current guidance. It is not legal advice and it does not tell you whether a particular agent meets a particular obligation. For how the guidance applies to your organisation, read the primary sources above and take your own professional advice.