AI glossary · Advanced · AI security · Language models
Training data extraction
Training data extraction is an attack that feeds crafted inputs to an AI model to make it output, verbatim, original text it memorized during training (personal information, code, documents, etc.).
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Security & safety
- Fields
- AI security, Language models
- Added
- 2026-10-09
How it works
Models sometimes memorize, in full, passages that were repeated many times in the training data or distinctive strings. An attacker first has the model generate a very large amount of text, or gives it the beginning of a specific format, such as an email signature, and has it continue. They then use membership inference techniques to pick out candidates that 'look memorized', and check them against actual public sources to confirm they match the original. Another known approach uses odd prompts to push the model out of its usual behavior so that it spills out memorized content.
Filtering steps in an extraction attack
Why it matters
Training data collected from the web can contain information such as phone numbers, addresses and API keys. If a company fine-tunes a model on internal documents or customer support records, one customer's information could surface in another user's conversation. It also ties into the problem of copyrighted text or code being reproduced verbatim. In other words, the moment a model is released, part of its training data may be released along with it.
Things to know
Whereas a membership inference attack only asks 'was this data in the training set?', training data extraction pulls out the content itself, so it causes more direct harm. Models tend to memorize text more readily the more often it is repeated, so deduplication is an important defense. De-identification of personal information before training, differential privacy and sensitive-information filters at the output stage are also used. It is also worth remembering that larger models can memorize more.
Example
A company that has fine-tuned a chatbot on internal data has a red team try extraction attempts before launch, such as 'give the start of a customer's name and have it continue.' If real customer information comes out, the company deduplicates the training data, masks personal information and retrains, or adds an output filter to block the exposure.