Better Retrieval, Better Law? How to Judge Legal AI

When AI Reads the Law, Part 3. Does the architecture behind a legal AI system change the answer, and how would you know?

Part 1 looked at how AI systems find legal information. Part 2 showed that finding a rule is not the same as applying it. This final part asks the question that matters to anyone buying or building legal AI: does a better architecture lead to better legal answers?

The honest answer is that nobody has shown it yet. What the research does show is narrower, but still useful. Architecture decides which sources the model gets to see, and that can change the answer. This article explains what that means, how to test it and when added complexity is worth paying for.

1. Three tasks that are often treated as one

LegalTech discussions tend to blur three different tasks. Retrieval locates potentially relevant norms, judgments or commentary. Knowledge representation structures that material, for example as entities and relations in a graph. Legal reasoning applies norms to facts, weighs competing readings and justifies a conclusion. A system can do well on the first two and remain untested on the third.

Most comparative research addresses the first two. A large benchmark of twelve graph-based retrieval methods found no consistent winner over a conventional baseline. Methods that rely only on graph structure did poorly when the answer depended on the original text. Their measure of success was simple: does the expected answer appear somewhere in the output? A recent review adds that evaluation is still dominated by Wikipedia-style question answering and judges only the final answer, not the steps that led to it.

Retrieval errors are not neutral. In controlled experiments, a single passage that sounds relevant but lacks the answer cut accuracy by up to 25 percent. Several such passages cut it by more than 67 percent. In law, this kind of near miss is the normal case: a superseded provision, a judgment on similar facts, a rule from another jurisdiction.

The evidence is also not settled. The same experiments found that adding random, unrelated documents raised accuracy by up to 35 percent. That sits uneasily with the assumption that cleaner retrieval always produces better answers.

So the research shows that retrieval quality shapes the output. It does not show that higher retrieval scores produce sound legal argument. That remains an open question, not a premise.

2. A simple question with a complicated answer

Take a question that HR and legal teams actually ask. A private employer in Germany wants to forecast staffing from individual sickness-absence records. May it process this health data, assessed as of November 2026?

The starting point is clear. Article 9(1) GDPR prohibits processing health data unless an exception in Article 9(2) applies. The obvious candidate, Article 9(2)(b), covers processing needed under employment law, but only as far as Union or national law authorises it with appropriate safeguards. An exception alone is not enough. The Court of Justice has held that processing must also rest on one of the general legal grounds in Article 6(1). And because absence records are being reused for a new purpose, the principles of purpose limitation and data minimisation in Article 5 apply as well.

Then the national layer. Article 88 GDPR allows Member States to adopt more specific rules for employment, and Germany did so in § 26 BDSG. Its wording no longer tells you its legal effect. In 2023 the Court of Justice ruled that a nearly identical Hessian clause merely repeated the GDPR and therefore was not a “more specific rule”. The Federal Labour Court has since applied this reasoning to the federal clause. A 2024 judgment of the Court of Justice added that national rules, including works agreements, must still meet the GDPR’s own requirements.

Finally, time. Germany still has no dedicated employee data protection act; a 2024 draft lapsed with the end of the previous coalition. The EU’s Digital Omnibus would amend Article 9, but the Council only agreed its negotiating mandate on 21 October 2026, and the text is far from final. A system that indexes pre-2023 commentary, or treats these drafts as law, will cite rules that do not apply.

A reliable reference answer would state the facts, assumptions, jurisdiction and reference date. It would cite each provision in its applicable version, together with the case law that governs it. It would test exceptions and conditions in order, flag contested points and record who reviewed it, and when.

For legal teams: an apparently simple question depends on several interlocking sources. Each is a point where a retrieval system can miss.

3. Four architectures, one consumer question

A simpler case shows how architectures differ. A consumer asks whether she may withdraw from an online purchase of a sofa built to her measurements. A sound answer needs three elements: the general right of withdrawal, the exception for goods made to the consumer’s specifications in the Consumer Rights Directive, and its national version (in Germany, § 312g(2) No. 1 BGB). The four systems below differ less in the model that writes the answer than in which of these elements reach it.

A. Keyword search plus a language model. It ranks passages by matching words and reliably finds exact references such as “§ 312g”. But a question about a “custom sofa” may never meet the statutory wording. Keyword search is not obsolete, though: in one general-domain experiment it beat a semantic retriever.

B. Semantic (vector) search. It compares meaning rather than words and bridges the vocabulary gap. But high similarity does not guarantee relevance. It may favour general commentary on withdrawal over the exception that decides the case.

C. Hybrid search with reranking. It combines both result lists and reorders them before the model’s limited context is filled. This matches the legal need to handle precise terms and natural language at once. Reranking cannot, however, recover what neither search found.

D. Hybrid search plus a knowledge graph. It adds explicit links such as amendment, consolidation and transposition, which standards like ELI can encode. It can reach the exception through structure rather than similarity. Its value depends on whether the graph is complete and current, and the graph-based methods tested so far assume a corpus that never changes. Graph retrieval without the original text performed poorly, and costs rise sharply.

These are architectural expectations, not a verified ranking for legal tasks. What the research does show is that the relevance, number and position of retrieved passages change the model’s answer. Irrelevant retrieval can even push accuracy below what the model achieves with no retrieval at all.

For IT teams: architecture determines which legal information the model sees, and with it, potentially, the answer.

4. How to evaluate legal AI

A legal AI system can fail in two places. It can retrieve the wrong material, or it can reason wrongly from the right material. A credible evaluation measures both.

Retrieval. Precision and recall are the starting point, but in law recall should be measured against the sources a correct answer requires, not against arbitrary text fragments. For the GDPR question, the test is whether the system found Article 9(2)(b), Article 88, the national provision and the relevant case law. Precision matters for more than efficiency, because related passages that lack the answer actively mislead the model.

The answer. Common benchmarks are a weak guide. Counting an answer as correct because it contains the right article misses the point: a legal answer can name the right article and still omit a condition or an exception. Legal evaluation should check four things:

  1. Does each cited source exist?
  2. Is it cited precisely, down to paragraph level?
  3. Does it actually support the statement it is attached to?
  4. Are all conditions and exceptions covered?

Time. Temporal correctness needs its own test. An answer that is correct under a superseded version is wrong at the reference date.

Who judges. Research increasingly uses one language model to grade another. In the benchmark mentioned above, the grading criteria were comprehensiveness and diversity, not legal correctness. Qualified lawyers remain necessary to define reference answers and to calibrate any automated grading.

Cost. On one benchmark, the most elaborate graph-based method needed 57 times more time and 210 times more tokens per query than a conventional setup, before counting the cost of building the index. Development time and the effort of maintaining legal data are rarely measured at all.

For legal teams: ask vendors for results on your own questions, scored by your own lawyers, against a fixed reference date.

5. Where systems fail

A legal AI system can fail before the model writes a single sentence. A statute, judgment or amendment that is missing from the database, or indexed incorrectly, cannot be found by any architecture. Retrieval can also fail confidently, by returning material that looks right but is not: a provision from a neighbouring regime, a repealed version, an overruled judgment.

Knowledge graphs add a separate source of error. As Part 2 noted, one automatically built graph reached 78.7 percent precision in its final extraction pipeline, so roughly one in five relations was wrong. Once a wrong relationship is stored, every query that passes through it inherits the error. Mistakes are reproduced systematically rather than randomly. Graph methods are also more sensitive to messy source text, because poor input leads to poor extraction.

Generation adds its own risk. Retrieval reduces invented or misattributed authorities but does not eliminate them. A correct citation can still be attached to reasoning the source does not support.

Fair comparisons are hard. The large benchmark cited above found that each method’s performance depended heavily on how its prompts were written. Its authors also name a limited context window and the absence of domain-specific data as limits on how far their results generalise. Most published comparisons, in short, were not run on law.

For IT teams: architecture is only one factor in reliability. Data quality, validation and evaluation method may matter as much, or more.

6. When is complexity worth it?

The research does not support a universal ranking of architectures. It supports a narrower conclusion: each added layer of structure buys a specific capability at a measurable cost, and the task has to justify that cost.

Start simple for well-defined questions. For locating a GDPR provision or a defined term in a stable body of law, standard search over text passages is a defensible starting point. In the large benchmark, graph methods did not consistently beat a conventional baseline on specific questions, and keeping the original text proved essential for accurate answers.

Invest in precision before volume. Because plausible but wrong passages measurably lower accuracy, hybrid search with reranking is a reasonable default. Note the limit: none of the studies reviewed here tests that combination on legal data.

Add a graph when answers depend on connections. Graph methods were often ahead on broad questions that draw on many passages at once. The price is real. On the benchmark’s legal dataset, the most elaborate variant used about 44 times the tokens and about 9 times the time per query. And all methods tested assume a static corpus, which fits poorly with law that is amended continuously.

Budget for maintenance from day one. The ELI guidelines describe manually kept transposition tables as cumbersome and often outdated. Research on knowledge graphs still treats human review as the gold standard for checking them. The more structure a system has, the more human review it needs.

Our conclusion

Architecture matters because it decides what the model sees. Whether a given architecture produces better legal answers is still an empirical question, and the published research has not answered it for law.

The best LegalTech architecture is therefore not the most advanced one. It is the one that delivers the legal reliability a use case requires at a level of complexity the organisation can justify and maintain. Whether a specific system meets that threshold is something each organisation has to establish with its own documented evaluation.

Looking back on the series

Across three parts, one pattern held. Search finds passages. Structure, time and rules decide whether those passages can be used correctly. Evaluation decides whether you can trust the result. None of these steps replaces the lawyer who weighs what remains open and answers for the outcome.

Every statement in this article is based on published research, official EU documentation and case law. If you would like the source for a specific statement, contact us through wrungel.eu and we will send you the reference.