Training data extraction

Training data extraction is the recovery of literal sequences from a model's training data by prompting the trained model. It is possible because language models memorise: sequences that appear often, or that are unusual enough to be individually distinctive, can be reproduced verbatim rather than merely approximated.

Why it works

Memorisation rises with model size, with how many times a sequence appeared, and with how unusual the sequence is. That last factor is the awkward one for privacy, because the things that are most distinctive in a corpus — an email address, a private key, a national identifier, a rare name paired with a rare condition — are exactly the things most worth protecting.

Extraction typically works by prompting with a plausible prefix and sampling many continuations, then filtering for outputs the model reproduces with unusually high confidence. It does not require access to weights.

Why it matters to a deploying organisation

Most enterprises do not train base models, so the question is narrower and more answerable: does anything we send become training data?

• If a provider trains on API traffic by default, every prompt is a candidate for later extraction by someone else.
• If you fine-tune on your own corpus, you have created a model that can disclose that corpus to anyone who can query it — including people who were never entitled to the underlying documents. Fine-tuning is not an access control, and treating a fine-tuned model as "internal knowledge, safely encoded" is a common and expensive mistake.

What to do

• Get the training-use position in writing, per provider and per endpoint, and keep it with the contract rather than in someone's memory of a blog post.
• Prefer retrieval over fine-tuning where the goal is access to knowledge: retrieval can be filtered by the asking user's permissions at query time, and a fine-tuned weight cannot.
• Redact personal and sensitive data before it enters any corpus you train on.
• Where a workload genuinely cannot tolerate the residual, keep inference inside your own network.

Related: membership inference, and the practical control in PII redaction for LLM applications.