Three Australian AI Failures in Eighteen Months. None Were Stopped by a Policy.
A refunded government contract. A privacy compliance notice. A solicitor who lost his principal practising certificate. What was missing in all three was an AI governance framework that assessed the risk first. Here is what that assessment covers, and why it comes before the policy.
01
What actually goes wrong
These are not hypotheticals. All three are documented, all three are Australian, and all three happened to organisations with more governance capability than most.
The $440,000 report
Deloitte produced a report for the Department of Employment and Workplace Relations using an Azure OpenAI GPT-4o tool chain. It contained more than a dozen fabricated academic references and an invented quote from a Federal Court judgment. Deloitte agreed to a partial refund of the $440,000 contract in October 2025.
What an assessment would have caught: that the output feeds a published document with no verification step between generation and publication.
The compliance notice
A Victorian child protection worker entered a child's name and risk assessment details into ChatGPT to draft a Protection Application Report. The generated text contained inaccurate information that downplayed risk to the child. The Office of the Victorian Information Commissioner found that the department, not the worker, had breached its privacy obligations, and issued a compliance notice requiring six actions including blocking the tools outright.
What an assessment would have caught: that this data class should never have reached a public tool, and that accountability sits with the organisation regardless of who typed it.
The practising certificate
A Victorian solicitor filed a list of case citations in the Federal Circuit and Family Court that had been produced with AI legal software and never checked. None of the cases existed. In August 2025 his practising certificate was varied: no principal practice, no trust money, two years of supervised practice and quarterly reporting to the regulator.
What an assessment would have caught: that a regulated professional duty attaches to this output, which changes verification from advisable to mandatory.
The common thread
A policy saying "check AI output before you use it" would not have prevented any of these. Each one needed a decision made in advance about that specific use case: what data may go in, what the output touches, and who has to sign off.
02
Why it keeps happening
The National AI Centre commissions an annual measure of how Australian organisations actually govern AI. The 2025 results describe a gap between what organisations believe they are doing and what they have in place.
Read the bottom bar again. Almost everyone has a document. One in five checks what their AI vendors actually claim.
The overall Australian Responsible AI Index score sits at 43 out of 100, down one point from the previous year. The 2024 edition put it more starkly still: organisations had adopted an average of 12 of 38 identified responsible AI practices, while rating their own maturity far higher than the evidence supported.
The international picture matches. IBM's 2026 breach research found that 68% of breached organisations had no AI governance policy in place, and only 29% conduct regular audits for unsanctioned AI, down from 34% the year before. ISACA found that 39% of organisations have no documented process for shutting an AI system down.
03
The order that works
The usual sequence looks sensible. Someone raises AI at a leadership meeting. Someone else is asked to write a policy. A two page document appears, gets approved, and is emailed to all staff. Nobody established what the organisation should permit before writing down what people may do, so the policy bans things nobody was doing and stays silent on the things that actually create exposure. We made the first half of this case in why banning AI tools makes shadow AI worse.
This is not a stylistic preference. Every AI governance framework that carries weight in Australia sequences the work the same way:
- AS ISO/IEC 42001:2023. Clause 6.1.2 requires a documented, repeatable AI risk assessment process, and clause 6.1.4 a separate AI system impact assessment oriented to effects on people rather than on the organisation's own loss. The written policy is a single control in Annex A. The assessment is what tells you which of the 38 controls apply, and how deeply.
- The National AI Centre. Its Guidance for AI Adoption, published October 2025, puts "understand impacts" and "measure and manage risks" second and third of six essential practices. Its free screening tool states plainly that it is intended before a comprehensive risk assessment, but not instead of one.
- The Digital Transformation Agency. Policy version 2.0, in force since 15 December 2025, requires Commonwealth entities to name an accountable official, keep a register of AI use cases, and complete an impact assessment before deployment. Existing unassessed use cases must be assessed by 30 April 2027. In NSW the AI Assessment Framework is mandatory for all agencies and classifies all generative AI as elevated risk.
Assess the risk. Decide what is permitted. Then write the policy that says so. A policy written before an assessment is a guess with a signature on it.
If you sell into government, this is the shape of the assurance your buyers already run internally, and increasingly the shape of what they will ask of you.
04
Three documents, three different jobs
Confusing these is the most common failure in AI governance work.
| Instrument | Question it answers | Scope | Who owns it |
|---|---|---|---|
| Privacy impact assessment | What does this do to the privacy of individuals? | Personal information, the Privacy Act and the APPs | Privacy officer |
| AI risk assessment | Should we use this at all, and under what conditions? | Accuracy, bias, security, explainability, supply chain, workforce, legal exposure | Risk and security, with the business owner |
| AI use policy | What may staff do, day to day? | Permitted tools, prohibited data, disclosure, review | Written from the assessment, owned by the accountable officer |
A privacy impact assessment is a necessary input to an AI risk assessment, not a substitute for it. Commonwealth agencies must run a PIA for all high privacy risk projects under the Privacy Code, and the OAIC names implementing AI and automated decision making as an example of exactly that. Private sector organisations are not legally required to, but the OAIC treats it as evidence of taking reasonable steps under APP 1.2.
05
Where the answer has to be no
A risk assessment has to be able to return the answer "not here." If yours cannot, it is a procurement exercise wearing a risk assessment's clothes. Four Australian contexts where the line is already drawn for you.
Classified and government sensitive information
Commonwealth staff guidance is explicit: you must not put information security classified OFFICIAL: Sensitive or above into public generative AI tools. The operating assumption it sets is that anything entered into a public tool could become public. NSW says the same for personal, official, sensitive, classified and health information.
Evidence and expert reports
Supreme Court of NSW Practice Note SC Gen 23, in force since February 2025, prohibits using generative AI to generate the content of affidavits, witness statements, character references or other material tendered as evidence, including rephrasing a witness's evidence. Expert reports need prior leave of the court, and the application must name the specific program and version.
Clinical settings
Ahpra requires practitioners to obtain informed consent before entering a patient's personal information into an AI tool, to verify all output, and to remain accountable for care regardless of AI use. Separately, software intended for a diagnostic or therapeutic purpose is regulated by the TGA as a medical device.
Where the tooling itself is not available
Availability is a risk input, not an afterthought. Microsoft 365 Copilot is now available in US government clouds, but the Word, Excel and PowerPoint Agents are listed as not available in GCC, GCC High and DoD, and Anthropic models are not offered as a subprocessor in those environments.
For organisations with European or UK operations there is a subtler version. Anthropic models in Copilot are disabled by default in the EU, EFTA and UK, and enabling them is an explicit administrative act that moves data outside the EU Data Boundary. That is precisely the kind of decision a risk assessment exists to make, and a use policy cannot.
Free template • Step two of three
The AI risk assessment template
A working template built on the ISO/IEC 23894 process, with the National AI Centre's screening questions, an Australian prohibited context check, a risk tier model, and a decision record your policy can be written from. Word and PDF.
Step one is the amnestied AI use survey, which tells you what is already happening before you assess it.
06
How to run one, in half a day
ISO/IEC 23894 mirrors ISO 31000 clause for clause, so if you already run enterprise risk you are extending an existing process rather than building a parallel one. Assess one named use case, not "AI".
- Scope, context and criteria. What task, what data, which users, which jurisdiction. Set the risk criteria before you look at the risks, so the thresholds are not reverse engineered from the answer you want.
- Identify. Work the categories rather than brainstorming: privacy, security including prompt injection, accuracy, bias, supply chain and vendor terms, identity and access, sector legal exposure, explainability.
- Analyse and evaluate. Rate likelihood and consequence against the criteria from step one. The tier decides whether the use case proceeds, proceeds with conditions, or stops.
- Treat. Four options and only four: avoid, mitigate, transfer, accept. Every accepted risk needs a named person accepting it. This is the step that turns a document into governance, because it produces an owner rather than an observation.
- Monitor, review and record. Set the review date before you finish. The Commonwealth benchmark for high risk use cases is at least every 12 months. Models change under you and vendor terms change without notice.
Record inherent risk and residual risk separately. The difference between them is your control set. If they are the same number, either your controls do nothing or the inherent rating was not honest.
07
Writing the policy from the assessment
Done properly, the policy almost writes itself, because every clause in it is the output of a decision you have already made and recorded.
| The assessment produces | The policy clause it becomes |
|---|---|
| Use cases assessed as acceptable | What AI may be used for |
| Data classes rated unacceptable at any tier | What must never be entered, named specifically |
| Vendors whose terms passed review | The approved tool list |
| Controls required to reach acceptable residual risk | Conditions of use, and who approves exceptions |
| Named risk owners | Accountability and escalation |
| Review cadence | When the policy is next revisited, and by whom |
The reverse does not work. A policy written first has to invent its own justification for every rule, which is why so many of them are vague where they should be specific, and specific about things that do not matter.
Where organisations get stuck
And how we help
- Nobody owns it. Only 25% of Australian organisations have clearly designated responsible AI roles. AI readiness assessments and maturity reviews establish where accountability actually sits.
- The framework exists but nothing runs through it. Governance framework design and use case prioritisation turns a document into a process with a queue.
- The people rating the risks have never been trained to. An assessment is only as good as the judgement behind the ratings, which is what our training is built for.
Build the capability
Related Nexacu courses
Training for the people who run the assessment, sign the risk acceptances and own the policy afterwards.
AI Governance Training
1 day | $560
Accountability, data governance, risk and responsible adoption. The natural course if you are standing up a governance framework.
AI for Business Leaders and Managers
1 day | $595
For the people who accept residual risk and have to explain the decision afterwards.
AI Prompting Fundamentals
1 day | $595
Once the approved tools are set, this is how staff get useful results from them.
Common questions
Frequently asked questions
Do this next
Pick one AI use case that is already live in your organisation and run the assessment on that, rather than trying to assess AI in general. The first one takes an afternoon. The second takes an hour, because you will have set your criteria.
Sources
- Office of the Victorian Information Commissioner, Investigation into the use of ChatGPT by a Child Protection worker, 24 September 2024.
- Department of Employment and Workplace Relations, Targeted Compliance Framework Assurance Review final report, published July 2025, corrected version and partial refund October 2025.
- Australian Computer Society, First Australian lawyer penalised for AI blunder, 4 September 2025.
- Fifth Quadrant for the National AI Centre, Australian Responsible AI Index 2025, 26 August 2025, and the 2024 edition.
- Standards Australia, Adoption of AS ISO/IEC 42001:2023, 16 February 2024, and the free guide Understanding 42001.
- ISO/IEC 42001:2023, clauses 6.1.2 to 6.1.4 and Annex A; ISO/IEC 42005:2025 on AI system impact assessment; and ISO/IEC 23894:2023 on AI risk management.
- National AI Centre, Guidance for AI Adoption, October 2025, and the AI screening tool.
- Digital Transformation Agency, AI impact assessment tool, AI policy update, and staff guidance on public generative AI, referencing PSPF Policy Advisory 001-2025.
- Digital NSW, NSW AI Assessment Framework, July 2024, and the AI Review Committee.
- Supreme Court of NSW, Practice Note SC Gen 23, in force 3 February 2025, and Ahpra, Meeting your professional obligations when using AI in healthcare, 22 August 2024.
- Office of the Australian Information Commissioner, Guidance on privacy and the use of commercially available AI products, and When agencies need to conduct a PIA.
- IBM, Cost of a Data Breach Report 2026; ISACA, 2026 AI Pulse Poll; and Microsoft, Connect to AI subprocessor and the Microsoft 365 Copilot service description.
