1. Home
  2. Insights
  3. Field Note

Field Notes AI in operations

Most AI risk isn't a model problem. It's an architecture problem.

The most common question we get when a client realizes we are building AI products for them is “what about hallucinations.”

It is the right question. Stanford's empirical study of purpose-built legal AI tools, published in 2025 in the Journal of Empirical Legal Studies, found that Lexis+ AI hallucinated 17 percent of the time and Westlaw AI-Assisted Research hallucinated 33 percent of the time. Both vendors had marketed their tools as hallucination-free or close to it. Neither was. An earlier Stanford study found that general-purpose models answering legal queries hallucinated between 58 and 88 percent of the time. The concern is grounded.

This is also not the architecture most of our work uses.

There are two broad ways AI gets deployed in production. The first is the architecture you read about. You hand the model a question, the model generates an answer, and you take the output as truth. This is the architecture Stanford tested. Hallucination risk is structurally high here because the model is doing the work of synthesizing the answer, and even retrieval-augmented generation cannot fully constrain that synthesis.

The second architecture, which is what most of our production systems look like, is different. The AI is one constrained step inside a larger orchestrated workflow. It reads a structured input. It extracts a field. It classifies an intake document. It routes the result to the next system. The output is checkable against source data. The step is auditable. The model is moving data along a track, not generating the destination.

Most of our systems do include some generative summarization, usually at the end of the workflow, sitting on top of inputs that have already been validated upstream and reviewed by a human downstream. The generative step exists. It is not the load-bearing part of the system.

In the public conversation, both architectures get called “AI.” That conflation is causing real harm in regulated industries. Clients read the Stanford findings about generative legal research tools and reasonably worry. They are then asked to deploy an orchestrated intake or document review workflow with a small generative step at the end. The risk profiles of these two systems are not the same. The discourse acts like they are.

The honest version is this. Hallucination is a real risk in some AI architectures. The architecture matters more than the technology category. “Could the AI hallucinate” is the right question only after you have answered the prior question, which is “what is the AI being asked to do, and how is its output checked.”

For most of the systems we build, the answer to that second question makes the first question much smaller than the headlines suggest. Not zero. Smaller, bounded and visible.

Sources

Join the discussion on LinkedIn

Bring us the problem you need someone to own.

Executed, not advised.

The first conversation is with a partner. We will talk through what you are facing, what you have already tried, and whether we are the right fit.

Conversations are confidential.