How LLM applications leak sensitive data
The paths by which data actually escapes an LLM application: provider logs, retrieval, conversation state, agent tools, telemetry, fine-tuning and the index.
"Data leakage" in an LLM application is rarely one dramatic event. It is a set of ordinary paths, most of which were designed in deliberately by someone solving a different problem. OWASP catalogues the class as LLM02:2025 Sensitive Information Disclosure; what follows is the working inventory, in roughly the order teams discover them.
1. The prompt itself, into a third-party processor
Every request sends its full context to whoever serves the model: system prompt, history, retrieved documents, tool definitions and results. That transfer is the leak in the simplest possible sense — it is data leaving your control boundary — and whether it is acceptable is a contractual and jurisdictional question, not a technical one.
What to pin down, in writing: whether content is persisted and for how long, whether it is used for training, which region serves the request, which subprocessors are involved, and which of those answers change by model, by endpoint or by tier. See zero data retention.
2. Retrieval, to users who were never entitled
A vector index built over a document store has no access control of its own, so an assistant becomes a query interface that returns content by similarity rather than by permission. The leak does not require the model to quote anything: a summary, a confirmation or a citation of a title all disclose. This is the highest-volume leak path in most enterprises. See RAG security.
3. Conversation state, across users and sessions
Three variants, all common:
- Cross-user leakage from a cache or a session store keyed carelessly — a shared conversation object, a memoised response keyed on the prompt without the tenant, a summary written to the wrong record.
- Accumulation. Long-running assistants with "memory" build a durable profile of a user that nobody classified, nobody set a retention period on, and nobody included in the deletion path.
- Carry-over. Sensitive content from turn three is still in the window at turn thirty, and is now being sent on every request including the ones that do not need it.
4. Agent tools, outbound
An agent with a tool that reaches the internet — an HTTP client, a browser, an email sender, a webhook — has an exfiltration channel by design. An attacker does not need to break the model; they need the model to use the tool it already has, and an indirect prompt injection in retrieved content is the standard way to arrange that.
The variant to know about is the rendered-image channel: a completion returns a markdown image whose URL embeds conversation content, the client renders it, and the data is delivered to the attacker's server without any tool call at all. If your client renders model output as markdown or HTML, restrict which hosts it will load resources from.
5. Your own telemetry
Prompt and completion logs kept for debugging, evaluation and audit are the most concentrated collection of sensitive data the application produces, and they are usually the least controlled: an unrestricted search index, no retention schedule, replicated to a region nobody decided on, and readable by anyone with a dashboard login. Analytics and error-tracking tools compound it — a stack trace or a session replay that captures prompt text sends it to a third party you never assessed for this purpose.
Treat the log store as a production data store with a classification, an owner, a region, a retention period and an access review.
6. Fine-tuning, which is not an access control
Fine-tuning on internal documents produces a model that can reproduce them, to anyone who can query it, without regard to who was entitled to the source. It also puts that content permanently beyond deletion: you cannot remove a document from a set of weights, you can only retrain.
Where the goal is access to knowledge, retrieval is the safer architecture, because it can be filtered by the asking user at query time. See training data extraction.
7. The vector index, as a second copy
Embeddings are not anonymised — text can be reconstructed or attributes inferred from vectors — so the index carries the classification of the corpus. It is in scope for residency, deletion and access review, and the embedding API call that produced it was itself a transfer. See embedding inversion.
8. Model output, into somewhere it should not go
Finally, the ordinary direction: a completion containing sensitive content is rendered to the wrong user, written to a shared record, emailed, posted to a chat channel or included in a support ticket. Output-side inspection is the control, and it is the one most implementations skip because attention goes to the input.
What to do first
If you are working through this list on a real system, the order that finds the most exposure per hour is:
- Find out what your provider retains and trains on, per endpoint, in writing.
- Look at your own prompt logs — who can read them, where are they, how long do they live.
- Check whether retrieval enforces the asking user's permissions.
- Enumerate every agent tool that can reach the network.
- Add output-side inspection before anything is persisted or forwarded.
Only then tune detectors. Detection reduces the volume flowing down these paths; it does not close any of them.