AI glossary · Advanced · AI security · Data & evaluation
Data poisoning
Data poisoning is an attack that deliberately mixes wrong or manipulated data into the training data an AI learns from, so that the model makes wrong decisions or behaves the way the attacker wants.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Security & safety
- Fields
- AI security, Data & evaluation
- Added
- 2026-10-09
How it works
AI models learn patterns from large amounts of data, so if the data is contaminated, what they learn is contaminated too. Attackers plant manipulated data in places a model is likely to draw training material from, such as web pages, public datasets and user feedback. Some attacks lower overall performance, but the more dangerous kind is the 'backdoor'. The model is secretly taught to behave normally most of the time, but to perform an action chosen by the attacker when a specific word or marker (a trigger) appears.
Why it matters
Today's LLMs are pretrained on vast amounts of data collected from the internet, and fine-tuning and RAG also rely heavily on external documents. Because it is hard to check every data source, there is plenty of room for attackers to get in. A model with a planted backdoor looks fine in ordinary evaluations, so the problem may only surface after deployment. That is why data poisoning is treated as a major threat in AI security and data governance.
Things to know
The difference is that prompt injection attacks an already-built model through its inputs at use time, while data poisoning targets the data at the stage when the model learns. However, in setups like RAG that fetch documents at run time, attacks that contaminate the documents to be retrieved are often called by a similar name. Defenses include managing data provenance, deduplication and data filtering, anomaly detection, and red-team testing. Since no single measure can fully stop it, layering several stages is common.
Poisoning defense checklist
Data collection
- Are data sources recorded and managed?
- Is external data kept out unless verified?
Data cleaning
- Does data go through deduplication and filtering?
- Do you look for abnormally repeated patterns?
Model checks
- Do you test for abnormal behavior with a red team?
- Do you monitor for abnormal behavior after deployment?
Example
Suppose a company fine-tunes a chatbot on its customer support records. If someone repeatedly posts messages in the support channel along the lines of 'this competitor's product is better' together with a specific product name, a chatbot trained on those records may give off-target recommendations whenever that product name comes up. So in practice, teams add a step to the data pipeline that checks sources before training data goes in and filters out abnormally repeated patterns.