A customer asks the support assistant whether they can return an opened item. The assistant says yes. Within sixty days, no receipt needed. Clear, polite, correctly formatted, and completely wrong.
The company changed that policy in June. A content editor updated the help article, which is exactly what they were supposed to do. Nobody told them an AI system reads that article. Nobody told them because nobody in the building thought of it as something that needed telling.
The model did its job. It read what it was given and answered accurately from it.
The failure happened upstream, in a place with no owner and no alerts.
Bad Input Is Old. Confident Output Is New.
Garbage in, garbage out has been true since the beginning, and it was never especially interesting. We had defenses. Bad input usually produced output that looked bad. Malformed, empty, obviously off. A person glancing at the screen caught most of it, and a validation layer caught the rest.
That safety net is gone.
Feed a model stale, partial, or simply wrong context and it produces something fluent, organized, and confident. The model supplies the polish regardless of whether the content underneath it is right. Every surface signal a person used to spot trouble is now missing.
Confidence became free. Accuracy did not.
So the pipeline feeding the model stopped being plumbing. It’s an architectural component, and it needs to be treated like one.
Context Is a Dependency. Dependencies Have Contracts.
We learned this discipline once with service contracts. When one service depends on another, we don’t leave the relationship implicit. We state what it provides, what it guarantees, what happens when it’s unavailable, and who to call when it changes.
Every context source deserves the same four answers.
Where does this content come from, down to the system and the team that owns it? How current does it have to be, as a bound rather than a hope? What is this component permitted to see, decided by someone with authority to decide it? And what happens when retrieval comes back empty, the failure mode nobody designs for and everybody eventually hits?
Most teams can’t answer these for their most visible AI feature. That isn’t carelessness. Retrieval got built as an integration task, by whoever was closest to it, and integration tasks don’t come with contract reviews.
Freshness Is a Correctness Bound
Staleness has always been a correctness property. What changed is the blast radius.
A stale cache used to mean a slightly out-of-date number on a dashboard. A stale context source means the system states an obsolete policy to a customer. The customer treats that as the company speaking. From where they sit, it is.
So freshness needs a number and an owner. How far behind is this source allowed to fall? An hour, a day, a quarter? The answer differs enormously by source. Pricing and refund policy tolerate almost no lag. An internal engineering wiki tolerates a lot, and some of those pages have been wrong for years without anyone minding.
Then the harder half. What happens at the boundary?
When a source blows past its limit, does the system keep answering from old content, refuse that source, or answer with a caveat? Most systems keep answering. Nobody specified anything else, and answering is the default behavior of everything involved.
The Edit Nobody Told the System About
Go back to the return policy. The interesting part of that failure is that nothing broke.
The help article was edited correctly by a person doing their job well. The retrieval system found it, ranked it appropriately, and returned it. The model summarized it faithfully. Every component performed to spec, and the customer got wrong information anyway.
That’s a process gap presenting as a technical failure, and it’s nearly invisible. No error. No exception. No latency spike. The only signal is a customer eventually complaining, which may take weeks and may never happen.
The fix is to make the dependency visible in both directions. If a system reads a content source, the people who edit that source need to know, and a change needs to be an event the system can react to.
Unglamorous integration work. Also the difference between finding out in an hour and finding out in a quarter.
The Retrieval Layer Is a Security Perimeter
Now the part that should worry you most.
An AI model with access to the wrong context is one cleverly worded question away from a data breach.
We know how to reason about least privilege when the requester is a service and the request is an API call. Scopes, roles, tokens, audit trails. All of it assumes a structured request and a permission check at a boundary you control.
AI retrieval breaks both assumptions. The requester is a model. The request is a sentence written by whoever is talking to it, and it can be worded to surface information the model shouldn’t hand over. The permission check, if there is one, is bolted onto a similarity search over a store somebody populated months ago.
Which raises uncomfortable questions. What’s in the content index? Who put it there? Was every document cleared for every person who can talk to this assistant? When an employee changes roles, does their access to retrieved content change with them, or does the index still enforce the permissions it was built with in March?
For most organizations running an internal assistant, the index got populated from a shared drive one afternoon, and the permission model of that shared drive did not survive the trip.
The retrieval layer became the place where access control quietly disappeared. Almost nobody treats it as a security perimeter.
Prompt injection is the attack everyone has heard of, and I’ve written about why it’s a property of these systems rather than a bug to patch. The context supply chain is what makes injection durable.
A clever prompt works once. A poisoned document sits in the index, and every retrieval that pulls it hands the model instructions it cannot tell apart from the content it was asked to read.
That is the difference between a stunt and a standing subscription to your company’s most sensitive data.
Where Did This Content Even Come From?
Are you allowed to use this content this way?
Documents arrive in a retrieval index from vendor PDFs, licensed research, customer submissions, and material somebody downloaded years ago for a different purpose. Each one carries terms. None of those terms were written with “and a model will read this and paraphrase it to strangers” in mind, because nobody was thinking about that yet.
This is transitive dependency management, which architects already understand when it applies to libraries.
Now it applies to our data.
The Property Three Test
Can you name every context source feeding your most important AI feature, from memory, right now? Does each one have a stated freshness bound, and a defined behavior when it blows past that bound or comes back empty? If an editor changes a source document this afternoon, does anything anywhere notice? And does your retrieval index enforce today’s permissions, or the ones it was built with?
Bolt-On or AI-Native
Every enterprise said they were in the cloud. Most had moved the same monolith onto rented servers. Same architecture, same failure modes, new invoice. We called it lift-and-shift, and it took most of a decade to explain why that was a starting point and not a destination.
Bolt-on AI is that same move. Wire a model into a workflow, ship the feature, update the deck.
AI-native is the same answer cloud-native was. Stop hosting the old design on the new platform and design for what the platform actually is. For AI, that comes down to four properties: probabilistic behavior, inference economics, the context supply chain, and model lifecycle. Bolt-on AI treats retrieval as an integration detail and discovers the contract by being wrong in front of a customer. AI-native gives context a contract, a freshness bound, and a permission boundary.
The full four-properties test is in the article that started this series.
Next Tuesday: the component that changes without you.