Ask people at almost any firm why they don't use AI more at work, and you'll hear some version of the same worry: it makes things up. It sounds confident, it sounds reasonable, and sometimes it's simply wrong. In a brokerage, a law firm or an advisory practice, a confident wrong answer isn't an annoyance. It's a liability.
The worry is fair. The usual fix doesn't really address it.
Why "grounded" AI still blurs the line
Most AI tools for firms work the same way. You load in your documents, and when someone asks a question, the AI finds the relevant passages and writes an answer from them. The pitch is that it's grounded in your firm's knowledge, so it won't make things up.
It's better than nothing. But look at what reaches the person asking: one smooth paragraph, in the AI's words. Somewhere in it is your firm's policy, restated. Somewhere in it might be a detail the AI added, or a step it softened, or a number it got from somewhere else. The person can't tell which sentence is which. So they either trust all of it, which is how mistakes happen, or they trust none of it, which is how the tool gets ignored.
The problem isn't only that AI can be wrong. It's that its words and your firm's words arrive blended together, and nobody can see the seam.
Keep the two apart
The fix is simple to describe, and it's mostly a matter of discipline:
- Your firm's approved answers are shown word for word. Not summarized, not rephrased, not woven into a paragraph. The exact text someone at the firm approved.
- They're marked, every time, the same way. People learn the mark quickly. If it has the mark, it's the firm's answer.
- Anything an AI adds is shown separately and labeled as AI. It might be useful. It might even repeat the firm's answer in different words. But it's never presented as the firm's.
That changes what people learn to do. The rule becomes easy: trust the marked guidance; treat everything else as a suggestion to check, however reasonable it sounds. Nobody has to guess which sentence came from where.
The mistake that's left
AI still makes mistakes in this setup. It just makes a different kind.
Instead of writing a wrong answer, the worst it can do is pick the wrong one: show your firm's answer about notifying a client when the person was asking about something else. That still matters. But compare it with a made-up answer:
- You can see it. The marked answer is plainly about a different situation. Nobody reads a card titled "Notifying a client within the deadline" and mistakes it for an answer about access permissions.
- You can measure it. Collect the questions people really ask, and check which answer each one gets. The wrong picks show up as a number you can watch go down.
- You can fix it. A wrong pick usually means the right answer is missing, or is written in words people don't use. Add the answer, or add the phrasing, and the next person gets it.
A hallucinated answer is none of those things. You can't see it, because it reads like the right answer. You can't measure it without checking every reply. And you can't fix it at the source, because there's no source to fix.
What you can do with the tools you have
You don't need anything special to start applying this:
- Keep your approved answers in one place, written as the answer itself, not as a link to a long document.
- When you use AI to help with a firm question, paste the approved answer in yourself and ask the AI to help apply it, rather than asking the AI what the policy is.
- Teach the habit out loud: the approved text is the answer; the AI's text is a draft to check.
- Keep a list of the questions people couldn't get answered. That list is your matching problem, made visible, and the most useful to-do list your firm's knowledge has.
How we do it
This is how our iPhone app works. Every answer from your firm's guide carries a green "From your firm's guide" mark and appears exactly as it was approved. When someone asks a follow-up the guide doesn't cover directly, Apple's AI model on the phone helps, and it does two things only: it picks which of your firm's answers apply, and it adds anything it thinks might help. The app shows the answers it picked, word for word, with the green mark. Whatever the AI added appears in a separate box labeled "AI inference · not in your firm's guide."
The AI never writes your firm's answer. If it tries to copy one into its own box, the app moves it back out and shows it as the verified answer instead. And the question still goes on your firm's list, because if the guide didn't answer it directly, that's something to add.
Behind that, we watch the matching. We keep a set of real questions and check which answer each one gets, so a wrong pick is something we catch and fix rather than something your people discover the hard way.