Australian courts are dealing with a growing problem: legal documents that appear to have been produced with AI systems like ChatGPT, but arrive in court with serious flaws. Judges have criticized filings from individuals, companies, and lawyers after finding invented legal references, confusing submissions, and material that could not be relied on.
The issue is not simply that AI is being used. The concern is that legal work is being submitted without enough checking, in a setting where accuracy matters. When a filing includes a case that does not exist, or pages of material that fail to make a coherent argument, the burden shifts to judges, court staff, opposing parties, and the wider legal process.
Fake citations are becoming a courtroom warning sign
According to the source article, judges in various courts have pushed back against applications containing non-existent case citations and nonsensical content. In some cases, judges cannot prove that an AI tool was used. But the pattern is becoming familiar: fabricated legal material, unusual language, and text that looks as if it came from a large language model rather than a properly reviewed legal document.
One lawyer admitted to using "legal software" that produced a list of non-existent cases. That detail matters because the failure did not end with the tool. The material was apparently carried forward into a legal setting, where citations are expected to be real and relevant.
Another case involved a legal representative who submitted 600 pages of "rambling, repetitive, nonsensical" material. The length of the document did not make it useful. Instead, it became an example of how AI-assisted drafting can create volume without legal value when the output is not controlled, reviewed, and narrowed to the actual dispute.
Private citizens are also using AI in court
The problem is not limited to lawyers. The source article says private citizens also appear to be using AI in court documents. Judges have disregarded personal references when they suspected those references were AI-generated.
That point shows why the issue reaches beyond professional misconduct. People who represent themselves or help prepare material for court may treat AI-generated text as polished and persuasive. But a fluent paragraph is not the same as a reliable legal statement, a genuine reference, or a verified account.
In a Tasmanian Supreme Court appeal, Justice Alan Michael Blow found that the appellant cited a non-existent case. The source does not say that AI use was confirmed in that instance, but it places the example within a broader pattern of filings where fabricated legal material has become a signal of possible AI involvement.
Some filings leave the AI prompt behind
Some documents have included even more direct clues. The Australian Competition and Consumer Commission (ACCC) noted that one company's resolution ended with "use British spelling please ChatGPT".
That kind of leftover instruction is more than an embarrassing drafting error. It suggests that material created or edited through ChatGPT may have moved into an official or formal document without basic review. In legal and regulatory contexts, that creates an immediate credibility problem.
These examples also show why judges may be cautious even when AI use is not admitted. A visible prompt, a fake citation, or unnatural repeated phrasing can point to the same underlying weakness: a document was generated or assembled without enough human responsibility for its contents.
Why large language models create legal risk
Large language models can produce confident text, but the source article stresses that their output can include misinformation. In legal work, that weakness is especially dangerous because authority depends on precision. A case citation must exist. A claim must be supported. A document must help the court understand the issue rather than bury it in irrelevant language.
The source article also notes that this is not only an Australian issue. Lawyers worldwide have faced dismissals and penalties for using AI-generated misinformation in court. A Stanford University study found AI legal research tools make errors in one out of six requests.
That finding does not mean every AI-assisted document is wrong. It does mean that legal research produced by AI cannot be treated as final without careful review. The tool may sound certain while giving an answer that fails at the most basic level.
For courts, the practical risks include:
- time wasted checking citations that should already have been verified;
- arguments built on authorities that do not exist;
- documents that are too long, repetitive, or unclear to assist the court;
- personal references or company materials that appear less trustworthy because they look machine-generated;
- pressure on judges to identify AI-related problems without always being able to confirm the source.
Guidelines, bans, and the need for better education
Legal regulators and courts have responded with guidelines or bans on AI use. The source article does not present one single rule as the solution. Instead, it points to a broader problem: the number of examples suggests there may be many more unreported cases.
That possibility matters because visible failures are likely only the easiest ones to catch. A document that includes a ChatGPT prompt or a non-existent case may be obvious. A more subtle error in reasoning, a misleading summary, or an unsupported legal claim may be harder to identify quickly.
The article also argues for better education about the capabilities and limitations of AI systems. That education is important because many users may understand AI as a writing assistant, while underestimating its ability to generate false legal material. The source adds that much of the limited education is currently driven by those with commercial interests.
For legal work, the lesson is direct. AI-generated text must be treated as material to verify, not authority to trust. In Australian courts and beyond, judges are making clear that fluent wording does not excuse fake cases, confused submissions, or documents that waste the court's time.