AI tools like ChatGPT can speed up legal work considerably; there’s little doubt about that. At the same time, several cases from Germany and the US show what can happen when lawyers adopt AI-generated output uncritically: briefs with invented judgments, irritated courts and, in some cases, significant professional consequences.
This article summarises the most important cases, explains what’s behind them, and shows what responsible use of AI in legal practice can look like.
The core problem: AI hallucinations in the courtroom
“Hallucination” refers to the tendency of large language models to present false information convincingly. The model doesn’t invent content deliberately; it produces statistically likely wording. In a legal context, this leads to plausible-sounding case citations, file references or statutory references that don’t actually exist.
For lawyers, this isn’t a theoretical problem but a real professional risk. The duty to check sources and to put forward only accurate facts applies regardless of who produced the draft.
The German cases: OLG Celle and the Local Court of Cologne (2025)
In Germany, two decisions from 2025 showed particularly clearly where the problem lies.
In its order of 29 April 2025 (ref. 5 U 1/25), the Higher Regional Court (OLG) of Celle found that counsel for the defendant had cited four supposed decisions from various higher regional courts, in Munich, Frankfurt, Düsseldorf and Koblenz. None of these decisions could be found in juris or beck-online. The court referred to miscitations that couldn’t be located under the stated details in any standard legal database. In addition, the supposed decisions contradicted settled Federal Court of Justice (BGH) case law. The brief was therefore unreliable on key points.
Shortly afterwards came an order from the Local Court (Amtsgericht) of Cologne (ref. 312 F 130/25), which was even more explicit: in the court’s view, every source listed in the brief, from judgments and paragraph numbers to literature references, had simply been made up. The court expressly pointed out that this kind of conduct undermines trust in lawyers’ submissions.
Both cases show a similar pattern: AI-generated output was adopted without adequate checks. The consequences ranged from individual arguments being disregarded to a publicly documented loss of trust.
The American precedent: Mata v. Avianca (2023)
The case of Mata v. Avianca before a federal court in New York became known internationally. In 2023, two lawyers filed a brief containing six supposed court decisions, all generated by ChatGPT, none of them real.
After neither the court nor the opposing side could verify the citations, the lawyers were asked to produce the sources. They couldn’t.
Judge Kevin Castel made it clear: the decisions couldn’t be found because they had never existed. The lawyers were fined USD 5,000 and named in the order. The case drew international attention.
What weighed particularly heavily was that the lawyers had initially claimed the decisions were genuine, and then went on to submit fabricated documents. The court regarded this as an attempt to deceive the court.
What’s at stake legally
These cases aren’t isolated incidents but an expression of a structural risk. For lawyers, there are three main problem areas in particular:
Duty of care and duty of truthfulness
Under § 43a(3) of the German Federal Lawyers’ Act (BRAO), lawyers may not put forward untrue facts. This also applies to content negligently taken from AI output.
Liability risk
Submitting a brief with invented sources risks more than just an embarrassing situation. Clients can suffer significant harm as a result; in the worst case, you lose a case that might have been won with careful work. The liability consequences for the responsible lawyer are correspondingly real.
Reputational damage
Court orders documenting AI-related errors are publicly available. They remain findable for good, in databases, in professional journals or through simple searches. Once a miscitation has been documented, it generally stays linked to the lawyer’s name.
Why it still happens
The cause rarely lies in a lack of expertise, but in the workflow. ChatGPT writes legal-sounding text that is highly persuasive. Under time pressure, a draft like this often seems plausible and gets adopted faster than is really justifiable.
On top of that, the tool itself doesn’t flag when a source doesn’t exist. Invented content is presented with the same confidence as correct information.
What lawyers need to do
The answer isn’t to give up on AI, but to use it in a controlled way.
- Check every case citation and reference in juris, beck-online or a comparable database before using it
- Always treat AI output as a working basis, not a finished result
- Don’t enter confidential client data into general-purpose AI tools
- Define clear internal guidelines for AI use
- If in doubt, leave out content that can’t be verified
The OLG Celle order makes it clear: even when AI is used, lawyers’ professional obligations, in particular the obligation to check sources carefully, continue to apply in full.
How specialised legal AI reduces the risk
General-purpose AI tools like ChatGPT aren’t designed to check legal sources. They have no access to legal databases and give no warning when a source doesn’t exist.
Specialised solutions such as BEAMON take a different approach.
Answers are linked to verified legal sources, citations are checked against databases, and traceable source references are provided. There are differences on data protection too: BEAMON is certified to ISO 27001:2022 and SOC 2 Type II, meets the requirements of the BRAO and § 203 of the German Criminal Code (StGB), and processes data exclusively within the EU/EEA. Client data is neither used for training nor passed on. You can find more information here.
This doesn’t mean such tools are error-free. The lawyer’s duty to review remains. The difference is that risks are systematically reduced, rather than going unnoticed.
Conclusion: yes to AI, but responsibly.
The cases described here aren’t an argument against using AI in legal practice. Rather, they show how important it is to use it deliberately.
AI can make legal work more efficient, for example when structuring facts or producing first drafts. But it doesn’t replace professional review.
If you establish clear processes and use AI in a targeted way, you can take advantage of the benefits without taking unnecessary risks. Curious? Try BEAMON for free at: ai.bryter.io/sign-up.
