Membership inference

Membership inference is determining whether a specific record was part of a model's training data, without recovering the record itself. The attacker already has the candidate record; what they learn is that it was in the dataset.

Why that is worth anything

Because the dataset's name is often the sensitive fact. Knowing that a particular person's record was in a model trained on a clinic's patient notes, a fraud-investigation corpus, or a set of resolved HR cases discloses something about that person regardless of what the record said. This is why membership inference is treated as a privacy attack in its own right rather than a weaker form of training data extraction.

How it works

The signal is overfitting. A model is usually a little more confident, and a little less surprised, on data it has seen before. An attacker queries the model with the candidate record and compares the response — loss, confidence, or the model's willingness to complete a rare continuation — against a reference distribution built from records known not to be in the set.

Attack success correlates with how much the model overfits, so small training sets, many training epochs and rare records all make it easier. Aggregate accuracy figures understate the risk: an attack that is barely better than chance on average can still be highly reliable on the outlier records that matter most.

Mitigations, and their cost

• Differential privacy during training is the only mitigation with a formal guarantee, and it costs utility. The privacy budget is a real parameter with a real trade-off, not a checkbox.
• Less overfitting — regularisation, fewer epochs, larger and more diverse data — reduces the signal without bounding it.
• Deduplication of the training corpus reduces both memorisation and membership signal, and is cheap.
• Rate limiting and query monitoring raise the cost of the many-query probing these attacks depend on.

For an organisation that consumes models rather than training them, the practical control is upstream: decide what is allowed into a training corpus in the first place.