You're right, and this is an underappreciated point. The "attacker" framing can actually obscure the more common risk: organic knowledge base degradation over time. The poisoning attack is just the adversarial extreme of a problem that exists in every large document store.

The model robustness angle is valid but I'd push back slightly on it being sufficient as a primary control. The model risk / backtesting framing is exactly right for the generation side. Where RAG diverges from traditional ML is that the "training data" is mutable at runtime (any authenticated user or pipeline can change what the model sees without retraining).

And it’s like they included “use all the hallmark annoying tells of LLM responses in your comment” in their prompts.

But you can't do that. That would implicitly out where the knowledge came from, and we all know that the AI industry has an existential incapability to actually cope with that little turd. Might work great for data you actually own, got access to. Imagine that applied back to the latent space of LLM's though. Plus, wouldn't all of that eat through context window like no tomorrow?

> This fault results to LLM

What's this mean?

totally disagree, rag crafts the agent and delegates what sources should be scored/chunked in what manner, if its leaving itself open to some potential source gaming the system like this, it is a lack of preparation.

For some use cases, this is totally whatever, think a video game knowledge base type rag system, who cares.

Finance/medicine/law though? different story, the rag system has to be more robust.

> it requires a bad actor with critical access

This isn't particularly hard. Lots and lots of these tools take from the public internet. There's already plenty of documented explanes of Google's AI summary being exploited in a structurally similar way.

For what it concerns internal systems, getting write access to documents isn't hard either. Compromising some workers is easy. Especially as many of them will be using who knows what AI systems to write these documents.

> it also requires that the final rag output doesn't provide a reference to the referenced result.

RAG systems providing a reference is nearly moot. If the references have to be checked; If the "Generation" cannot be trusted to be accurate and not hallucinate a bunch of bullshit, then you need to check every single time, and the generation part becomes pointless. Might as well just include a verbatim snippet.

On the reference point: the poisoning succeeded even when the legitimate document was present in the retrieved chunks and visible in the context. The LLM saw all three sources simultaneously, including the correct $24.7M figure, and still produced the fabricated answer because the poisoned documents framed the legitimate one as a known error. Providing a reference to the retrieved chunks doesn't help if the retrieved chunks themselves are the attack surface.

zenoprax's point about ignorant employees is also worth taking seriously. "Write access to the knowledge base" in practice means anyone who can edit a Confluence page, commit to a docs repo, or submit a support ticket that gets ingested. That's not critical access in most organizations.

"bad actor" can now be "ignorant employee running AI agents on their laptop".

Threats from incompetence or ignorance will be multiplied by 'X' over 'Y' years as AI proliferates. Unsupervised AI agents and context poisoning will spiral things out of control in any environment.

I'm interested in the effect of this with respect to AI-generated/assisted documentation and the recycling of that alongside the source-code back into the models.

If you think about this in the context of systems that ingest content from third party systems then this attack becomes more feasible.

But then, if you’re inside the network you’ve already overcome many of the boundaries

That's fair. The underlying manipulation (presenting fabricated authoritative documents to override legitimate ones) predates LLMs entirely. Corporate fraud has used exactly this pattern for decades.

What's new isn't the social engineering, it's the scale and automation. A human reviewer reading all 8 documents would likely notice the inconsistency and ask questions. The LLM processes all retrieved chunks simultaneously with no memory of what "normal" looks like, no ability to ask for clarification, and no friction. It just synthesizes whatever it retrieves. At query volume (hundreds of requests per day across thousands of users), there's no human in that loop.

The trust boundary framing is the right mental model. The flat context window problem is exactly why prompt hardening alone only got from 95% to 85% in my testing. The model has no architectural mechanism to treat retrieved documents differently from system instructions, only a probabilistic prior from training.

The UNTRUSTED markers approach is essentially making that implicit trust hierarchy explicit in the prompt structure. I'd be curious how you handle the case where the adversarial document is specifically engineered to look like it originated from a trusted source. That's what the semantic injection variant in the companion article demonstrates: a payload designed to look like an internal compliance policy, not external content.

One place I'd push back: "you can't reliably distinguish adversarial documents from legitimate ones" is true at the content level but less true at the signal level. The coordinated injection pattern I tested produces a detectable signature before retrieval: multiple documents arriving simultaneously, clustering tightly in embedding space, all referencing each other. That signal doesn't require reading the content at all. Architectural separation limits blast radius after retrieval. Ingestion anomaly detection reduces the probability of the poisoned document entering the collection in the first place. Both layers matter and they address different parts of the problem.

Curious how this applies if you treat ALL information from external content as untrusted? Is there a process for the data to evolve from untrusted->trusted?

I'm interested in ingesting this type of data at scale but I already treat any information as adversarial, without any future prompts in the initial equation.

Would you kindly leave a casual reply to my comment here just to prove you aren't an LLM? I'll compensate you with an upvote. Thanks, bro.

Someone needs to train a model where untrusted input uses a completely different set of tokens so that it's entirely impossible for the model to confuse them with instructions. I've never even seen that approach mentioned let alone implemented.

The SEC/public records context is where this gets genuinely hard — you can't vet the source the way you can with internal Confluence. The vocabulary engineering approach I tested would be trivially deployable against any automated public records ingestion pipeline, and the attacker doesn't need internal access at all.

The scoring per source is the right direction. The way I'd frame it: trust tier at ingestion time, not just at retrieval time. Something like: official regulatory filings get a different embedding treatment and prompt context tag than user-generated content from a public portal.

The write access framing is exactly the right correction. "Write access to the knowledge base" is a concept that doesn't exist in most organizations, it's dissolved across Confluence editors, Google Drive sharing settings, Slack export permissions, and whatever automated pipelines someone set up two years ago and nobody remembers. The attack surface is the sum of all of those, and nobody has mapped it.

The SEO analogy is the best one I've heard for vocabulary engineering. Same optimization target (ranking function), same lack of ground truth signal for the consumer, same asymmetry between attack cost and detection cost.