Guide

AI incidents and the Notifiable Data Breaches scheme

Australian guide to when an AI incident becomes a notifiable data breach. The three-part test, the 30-day assessment, notification and AI-specific triggers.

When does an AI incident become a notifiable data breach in Australia?

This is an Australian guide. An AI incident becomes a notifiable data breach under Part IIIC of the Privacy Act 1988 when three things are true. Personal information has been accessed or disclosed without authorisation, or lost in circumstances where that is likely. A reasonable person would conclude the breach is likely to result in serious harm to one or more of the people involved. And remedial action has not prevented that likely harm. Once an entity has reasonable grounds to suspect an eligible data breach, it must take reasonable steps to assess it within 30 days. If the breach is eligible, it must notify the OAIC and the affected individuals as soon as practicable. AI systems create new routes to that point, such as models revealing personal information in outputs, retrieval tools surfacing records to the wrong user, and agents sending data to the wrong recipient.

This guide is Australian. It covers the Notifiable Data Breaches (NDB) scheme in the Privacy Act 1988 and how it applies when an AI system is involved.

An AI incident becomes a notifiable data breach when personal information is accessed, disclosed or lost without authorisation, serious harm to someone is likely, and remedial action has not prevented that harm. You then have up to 30 days to assess a suspected breach and must notify the OAIC and affected people as soon as practicable once it is eligible.

Most Australian organisations already have an incident response plan built around the NDB scheme. AI changes the ways a breach can happen, not the test for whether it is notifiable. The risk is that an AI incident is handled as a technical fault while the 30-day assessment clock is already running. This guide sets out the test, the timeline and the AI-specific triggers to watch.

The scheme in brief

The NDB scheme sits in Part IIIC of the Privacy Act 1988 and has applied since 22 February 2018. It covers agencies and organisations with obligations under the Australian Privacy Principles, and also credit reporting bodies, credit providers and tax file number recipients for the information those roles cover. When an eligible data breach occurs, the entity must notify the Office of the Australian Information Commissioner (OAIC) and the individuals at risk of serious harm.

The three-part test

A data breach is an eligible data breach when all three of these hold.

Access, disclosure or loss. There is unauthorised access to, or unauthorised disclosure of, personal information the entity holds, or the information is lost in circumstances where unauthorised access or disclosure is likely.
Likely serious harm. A reasonable person would conclude that the access or disclosure is likely to result in serious harm to any of the individuals the information relates to. Serious harm can be physical, psychological, emotional, financial or reputational.
No effective remedial action. The entity has not been able to prevent the likely risk of serious harm with remedial action.

The third element matters for AI incidents. If you can contain the exposure quickly, for example by recovering the information or confirming it was not retained by the recipient, the breach may not become eligible. Record what you did and why it was enough.

The 30-day assessment

When an entity is aware of reasonable grounds to suspect an eligible data breach, but not yet sure, it must carry out a reasonable and expeditious assessment. It must take all reasonable steps to complete that assessment within 30 days of becoming aware. If the entity has reasonable grounds to believe an eligible data breach has occurred, it must notify as soon as practicable. It does not wait for the 30 days to run.

The clock starts with awareness, not with confirmation. If a support engineer notices a chatbot quoting another customer’s account details, the entity may already be aware.

What notification requires

The entity prepares a statement for the OAIC, usually through the OAIC’s online form. The statement covers the entity’s identity and contact details, a description of the breach, the kinds of information involved, and recommendations about the steps individuals should take in response.

The entity then notifies individuals. It can notify everyone whose information was involved, notify only those at risk of serious harm, or, where neither is practicable, publish the statement on its website and take reasonable steps to publicise it.

There are limited exceptions. These cover breaches where another entity involved in the same breach has notified, certain enforcement-related activities, inconsistency with secrecy provisions, and declarations by the Commissioner. Breaches involving My Health Record information follow the separate scheme under the My Health Records Act 2012. Amendments in 2024 also allow the Minister to make an eligible data breach declaration, which permits specified handling of personal information to reduce the risk of harm after a significant breach.

AI-specific triggers

The test is the same as for any breach. What changes is how the breach happens.

Outputs revealing personal information. A model trained or fine-tuned on personal information may reproduce it in outputs. A chatbot may reveal one customer’s details to another.
Retrieval beyond the user’s permissions. An assistant that answers from internal documents may surface records the person asking was never entitled to see, if its document search ignores the source system’s access controls.
Prompt injection and data leaks. Instructions hidden in a web page, email or document can lead an AI system to send data somewhere it should not go.
Agents acting on the wrong recipient. An agent drafting and sending messages, or moving files, can send personal information to the wrong person at machine speed.
Disclosure to AI vendors. Staff pasting personal information into an unapproved external AI tool may be making an unauthorised disclosure.
Vendor incidents. A breach at an AI provider that holds your prompts, logs or fine-tuning data may be your breach to assess, depending on who holds the information.
Logs and caches. Prompt and response logs, and cached responses, often contain personal information and are easy to leave out of security reviews.

Each of these should appear in your AI incident response plan, with the questions an assessor needs to answer. What personal information did the system have access to? Who could have received it? Is it still retained anywhere? Can the exposure be reversed?

Containment for AI systems

AI containment uses different levers from a classic breach. You may need to suspend an agent or revoke its credentials, withdraw a deployment, switch a guardrail from observe-only (recording what it would have done) to enforcing, raise redaction for personal information, set a gateway to fail closed so it refuses requests it cannot check, or stop an assistant searching a document source. Decide in advance who can pull each lever, because the speed of containment can decide whether serious harm is likely.

Overlapping obligations

The NDB scheme often runs alongside other duties.

APRA-regulated entities. CPS 234 requires notice to APRA within 72 hours of becoming aware of a material information security incident. CPS 230 includes its own 72-hour notification for certain operational risk incidents.
Sector rules. Health, telecommunications and critical infrastructure sectors carry their own notification rules.
Overseas law. If the incident involves people in other jurisdictions, rules such as the GDPR’s 72-hour notification to supervisory authorities may also apply.

Map these before an incident, so the team assessing an AI incident knows every clock that may be running.

After the incident

Close the loop. Record the timeline, the assessment and its reasoning, the notification decision and the remediation. Find out why the AI system behaved as it did, such as a missing access control, a model that memorised data, an undeclared tool or an absent guardrail. Fix the cause, then reassess the system’s risk with the incident in view. The OAIC publishes reports on notifications each half-year, and its themes are a useful check on your own controls.

Where Aicura fits

Aicura’s Pro tier carries the Notifiable Data Breaches scheme as a framework, so your AI incident response plan can be reviewed against it before you need it. When an incident is raised, its notify step surfaces the OAIC under the NDB scheme and APRA under CPS 230 as recipients, each with its full criteria, and your team decides which apply. Any notification draft is a starting point for that judgement, not an instruction to send.

Several of the containment levers above exist in Aicura already. Each agent’s runtime identities, the credentials it acts with, are listed in the AI Register. Gateways can be set per system to fail closed, refusing requests when a rule cannot be checked. Guardrail rules can move from observe only, where they record without blocking, to enforcing. The decision to notify, and the notification itself, remain with your organisation and its advisers.

A note on this page

This guide explains the Notifiable Data Breaches scheme as it applies to AI incidents. It is general information, not legal advice. The Privacy Act 1988 and the OAIC’s guidance are the primary sources, and where this page and those sources differ, they govern.

This guide explains what the rules require and how to do the work. It is general information, not legal advice. How the obligations apply turns on your own circumstances, so read the primary source and take your own advice before you rely on it.

Aicura does not certify, audit or issue a compliance verdict, and it does not produce a score. That decision belongs to your organisation and its people.

Primary source: Notifiable Data Breaches scheme, OAIC

Questions

Is an AI hallucination a data breach?

Not usually. A hallucination that invents information about a person may raise accuracy obligations under the Australian Privacy Principles, but it is not unauthorised access to or disclosure of personal information you hold. It becomes a breach question when the model reveals real personal information to someone who should not have it.

Is it a data breach if staff paste customer data into a public AI tool?

It can be. Disclosing personal information to an external service outside your approved arrangements may be an unauthorised disclosure. Whether it is an eligible data breach then depends on whether serious harm is likely and whether remedial action, such as deletion confirmed by the provider, prevents it. Assess it like any other suspected breach.

How long do we have to assess a suspected data breach in Australia?

Under the NDB scheme, the entity must take all reasonable steps to complete the assessment within 30 days of becoming aware that there are reasonable grounds to suspect an eligible data breach. Notification then follows as soon as practicable if the breach is eligible.

Do APRA-regulated entities have other breach notification deadlines?

Yes. CPS 234 requires notice to APRA within 72 hours of becoming aware of a material information security incident, and CPS 230 has its own 72-hour notification for certain operational risk incidents. These run alongside the NDB scheme, not instead of it.

Run this work in Aicura

Pro adds impact assessments, incidents, enforcement on live traffic, monitoring and reporting. Contact us about your programme.