Skip to main content
SecureAI Guard

Data poisoning

Data poisoning is the deliberate corruption of the data a model learns from — pre-training, fine-tuning or a retrieval index — to change its behaviour later.

Published ·2 min read·SecureAI Guard
glossary
data poisoning
OWASP LLM Top 10

Data poisoning is the deliberate introduction of crafted data into a corpus a model will learn from, in order to change how that model behaves after training. It appears as LLM04:2025 Data and Model Poisoning in the OWASP Top 10 for LLM Applications.

The three places it lands

  • Pre-training data. Web-scale corpora are assembled from sources that anyone can write to. Research on this class has shown that acquiring expired domains that appear in published dataset indexes, or editing a page at the moment a crawler snapshots it, is enough to insert content into a future corpus.
  • Fine-tuning data. The most practical target in an enterprise, because the dataset is small. A small number of well-placed examples can teach a behaviour, and the dataset is often assembled from support tickets, chat logs or user-submitted content that an outsider can influence.
  • Retrieval corpora. Poisoning a vector store does not require training at all: writing a document that ranks well for a target query and contains the content you want repeated is enough. This is the cheapest variant and the one most enterprises are actually exposed to.

Backdoors

The form that worries security teams is the backdoor: the model behaves normally except when a trigger phrase appears, at which point it follows the attacker's behaviour. Evaluation on held-out data does not detect this, because held-out data does not contain the trigger. Backdoors survive some amount of subsequent fine-tuning, which means "we fine-tuned it ourselves afterwards" is not a mitigation.

What helps

  • Treat training and retrieval data as a supply chain: know its provenance, pin versions, and record what went into which model build.
  • Restrict who can write to a retrieval corpus, and attribute every document to a source system and an author.
  • Evaluate against adversarial and trigger-probing test sets, not only against a held-out split of the same distribution.
  • Keep a model inventory, so that if a dataset is later found to be compromised, you can answer which deployed systems consumed it.

Provenance is the whole game here. A model you cannot trace back to its inputs is one you cannot clear after a supply-chain disclosure.