Beyond RAG: How AI Systems Find and Structure Knowledge

When AI Reads the Law, Part 1. What legal teams should know about AI search, and what IT teams should know about legal text.

Many firms are testing AI assistants on contracts, regulations and internal policies. The demo looks impressive. Then someone asks a real question and gets a confident answer that cites the wrong article. Usually the language model is not the real problem. The problem is the step before it: how the system finds information. This article explains that step in plain language.

1. Why AI needs external knowledge

A large language model learns from huge amounts of text during training. That knowledge is not stored like files in a folder. It is spread across the model’s internal parameters. So it is frozen at training time: the model does not know the regulation published last month. And if a topic was missing from the training data, the model struggles without necessarily telling you. That is where hallucinations come from.

The common fix is Retrieval Augmented Generation, or RAG. The model stays the writer, but first it looks things up in a searchable library that can be updated at any time without retraining.

The catch: a model can only work with what it is given, and it is bad at telling relevant material from merely related material. In one controlled study, adding a single passage that was on topic but did not contain the answer cut accuracy by up to 25 percent. With several such passages, losses reached 67 percent.

For legal teams: connecting AI to your documents does not remove the reliability problem. It moves it. Did the system find the right source, did it filter out the noise, and can you check the answer against that source?

2. Keyword search vs. vector search

Keyword search (the classic method is called BM25) ranks documents by shared words. It is precise when words match exactly: a statute number, a product code, a defined term. It is weak when the same idea uses different words.

Vector search, or semantic search, converts text into numbers that represent meaning and looks for texts close in meaning. But similar is not relevant. Ask for the colour of Napoleon’s horse, and it may return a passage about the horse of Joséphine. Close in topic, useless as an answer.

A legal example shows both sides. A keyword search for “Article 6(1)(f)” finds the exact GDPR provision but misses a passage on “legitimate interest” that does not cite the number. Vector search may find it, but may also bring in texts that only sound similar. Neither wins every time: in one experiment, plain keyword search found relevant passages more often than a modern semantic retriever and improved accuracy by 3 to 4 percentage points.

For IT teams: legal language is built on exact references, defined terms and exceptions. Semantic similarity alone misses exactly the details lawyers care about most.

3. How RAG works in practice

A RAG system splits documents into small pieces (chunks), converts them into numbers, stores them in an index, retrieves the pieces closest to the question and hands them to the model. These pieces also show where the answer came from.

Chunking is a real design decision. If a clause is cut off from its exception, or an article from its definitions, the model may never see the part that changes the meaning. Better systems combine keyword and vector search (hybrid search) and reorder results with a specialised relevance model (reranking). Typical failures are vague questions and retrieving too many documents, which makes performance drop sharply. RAG reduces hallucinations. It does not eliminate them, and a comparison of six RAG methods found large differences in quality, speed and cost.

4. Knowledge graphs: making relationships visible

Documents hide relationships in prose, so someone has to interpret the text every time. A knowledge graph states them directly. Things become points, relationships become connections: this regulation amends that directive, this company is a subsidiary of that one. An ontology defines which kinds of things and relationships exist and what they mean. Open W3C standards support this: RDF stores facts as triples (subject, predicate, object), OWL adds formal meaning, SPARQL queries the graph.

The limits matter for law. Extraction errors enter the graph and then look like facts. RDF treats data as a static snapshot, while laws and contracts change. Relationships between more than two parties can only be expressed indirectly. A knowledge graph makes relationships explicit. It does not make them correct.

5. GraphRAG: combining text and structure

GraphRAG adds a graph to classic RAG. In Microsoft’s approach, a language model builds the graph from the documents. The system can then match names in a question to the graph, follow connections across documents, or use summaries of related entities for big picture questions.

Text keeps the details, graphs keep the connections. “When did Regulation X enter into force?” sits in one passage, and a graph only adds overhead. Questions that link facts from separate documents are where graphs help. But GraphRAG is not automatically better. In one test, only about 66 percent of answer entities made it into the graph. Results depend on design choices, graphs cost money to build, and some early success claims rest on evaluations later research has questioned. The better default is a hybrid that sends questions to the cheaper method first.

6. Choosing the right architecture

This is an engineering trade off, not a ranking of technologies. Four criteria matter: answer quality, traceability, speed and freshness.

In one benchmark, plain vector RAG answered in about 2.4 seconds using 3,600 tokens (the units of text that drive cost). An agent based graph method needed about 70 seconds and 16,900 tokens, and did not win on every question. Building richer graphs cost up to 40 times more tokens. These graph methods also assume documents never change. Law does, and every update means rebuilding parts of the index.

Putting everything into one long prompt is no free alternative: accuracy fell by up to 67 percent when related but unhelpful passages were added, and was lowest when the right passage sat in the middle.

Our conclusion: start from measurable requirements and total cost of ownership. The simplest architecture that meets them is usually the best. Before we recommend, we measure.

What comes next

Finding information is only half the problem. Understanding its legal meaning is the other half. That is Part 2.


Every statement in this article is based on published research. If you would like the source for a specific statement, contact us through wrungel.eu and we will send you the reference.