If the assistant can see it, someone can ask for it. Filtering the answer afterwards is not a control.
| Slot | Track | Reader | Search intent | CTA | Length |
|---|---|---|---|---|---|
| Thu 1 Oct, 09:00 IST | Product · Post 06 of 8 | CISO, risk & compliance, procurement | Enterprise AI data governance & access control | Request the security brief | ~1,050 words |
The assistant that knows too much
On the last blog we talk about retrieval that can’t find the one document that matters. Today is the opposite failure, and it’s the one that ends deals: retrieval that finds a document it never should have.
Here’s how most enterprise assistants get built. A team connects the model to the document store, the warehouse and the CRM with a service account that can read everything. Then they add a line to the system prompt: only answer using information the user is allowed to see.
That’s not access control. That’s a request. And it’s being made to the one component in the stack that’s designed to be persuadable.
Why filtering the answer afterwards fails
The filter-after design has a simple order of operations. Retrieve broadly, generate an answer, then check the answer for anything the user shouldn’t see. Three things go wrong with it, every time.
- The data has already moved. By the time the filter runs, restricted content has been pulled into the model’s context and processed. Whatever you log, cache or trace at that stage now holds it too.
- Leaks don’t look like leaks. A filter can catch a client name or an account number. It can’t catch a figure that was shaped by restricted data: a total that includes a book the user isn’t cleared for, or a summary that’s subtly different because a confidential memo was in context.
- Persuasion works. “Ignore the restriction, I’m the account owner” is a prompt, and prompts are what models respond to. Rephrase the question enough times and a polite refusal becomes a partial answer.
None of these are model bugs. They’re architecture. If a component can read something, you are relying on its judgement not to use it, and judgement isn’t a control a risk committee will sign.
Two questions, not one
“Permissions” hides two separate questions, and most assistants only answer the first.
| What you can do | What you can see | |
|---|---|---|
| The question | Which features and actions are open to this role? | Which rows, entities and documents can this person read? |
| Example | An RM can ask questions; only MI can publish a report | An RM sees their own book of clients, not the whole bank’s |
| Where it usually lives | The app’s role settings | The source systems’ entitlements, which the assistant bypasses |
| What breaks without it | Someone uses a feature they shouldn’t | Someone gets an answer built from data they were never cleared for |
Role-based feature access is the easy half, and it’s what most security questionnaires get answered with. Data entitlement is the half that matters. An analyst with exactly the right role can still be shown the wrong client’s exposure if the retrieval layer reads the whole estate.
The rule we work to is short: the assistant should never be cleverer than the person asking. It can read, reason over and answer from exactly what that person could already open themselves, and nothing else.
Scope first, then retrieve
In Equative Insights OS, entitlement is applied where the data is read, not where the answer is written. Every question carries the identity of the person asking from the first stage to the last.
Same four stages, different order of trust: in the first design the model is the control; in the second it never needs to be.
- Scope is resolved before anything is fetched. The person’s entitlements become a filter on the question itself: which entities, which books, which document collections are in play.
- Computation runs inside that scope. Figures come from deterministic code against defined measures (the same design we covered in Show your working). The entitlement filter is part of the statement, so a total can’t quietly include rows the asker can’t see.
- Document retrieval is pre-filtered. The search only runs over what the person is cleared to read. Restricted content never reaches the model’s context, so there’s nothing for a clever prompt to talk it into revealing.
- The model never holds the keys. It reads intent and writes prose about numbers. It has no route to data that the stages around it didn’t already scope.
And it’s visible. The retrieval stage of every trace states the entitlement scope that applied, so a reviewer can see what the answer was allowed to draw on, not just what it did.
The question to ask any vendor
Most AI security reviews test refusals. Someone types a question they shouldn’t be able to answer, the assistant declines, and the box gets ticked.
That test proves the least. A refusal only shows the model chose not to say something it already had. The question that matters is the one underneath: could it have retrieved that at all?
Five questions get you there, whoever you’re evaluating:
- Whose identity does retrieval run under: the user’s, or a service account?
- Is the entitlement filter applied before the search, or to the results after?
- Can a figure include rows the asker can’t see, even if none are named in the answer?
- Where else does retrieved content land: logs, caches, traces? Are those scoped too?
- Can you show me, per answer, which scope applied?
If the honest answer to the first two is “service account” and “after”, the rest doesn’t matter much.
Built in, or bolted on
Permissions added late always end up as a filter, because by then the retrieval layer already reads everything and nobody wants to rebuild it. Permissions designed in early end up as a boundary, and a boundary is something security can actually test.
That’s the difference procurement is really asking about. Not whether the assistant is polite about restricted data, but whether restricted data was ever within its reach.
