AI glossary · Advanced · AI security · Safety & alignment
Membership inference attack
A membership inference attack is an attack that tries to find out whether certain data was used to train an AI model, just by looking at the model's responses. If the training data is sensitive, the mere fact that something "was included" can reveal personal information.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Security & safety
- Fields
- AI security, Safety & alignment
- Added
- 2026-10-09
How it works
Models tend to show higher confidence on data they saw during training. The attacker feeds suspected data into the model and checks whether the output probability or confidence is unusually high to decide whether it "was used in training" or "was not used." A common method is to build practice models (shadow models) on similar data and train a classifier that distinguishes the difference in responses between seen data and unseen data. For LLMs, comparing how high the token probabilities of a sentence are is a frequently used approach.
Data used in training vs. unseen data
Data used in trainingModel has seen it
- High confidence
- High token probability
Unseen dataModel has not seen it
- Relatively lower confidence
- Low token probability
공통 The bigger this gap, the better the attack works
Why it matters
For example, if a model was built from data on patients with a particular disease, the mere fact that a person's record was included in training reveals that the person is a patient. That is why it is used as a standard measure of how safe a model is from a privacy perspective. Conversely, it is also used as a tool by authors and companies to check "was my writing used for training?" The same idea is used in data contamination checks, which verify whether benchmark questions were mixed into training data.
Things to know
The more a model memorizes its training data, that is, the more severe the overfitting, the better the attack works. Some point out that telling the difference is harder than expected in large LLMs, so results should not be interpreted as conclusive. Defenses include applying differential privacy, reducing overfitting, and not exposing raw output probability values externally. This attack only finds out whether data is "in or out"; pulling out the content itself is training data extraction.
Example
Organizations that handle sensitive data, such as in healthcare and finance, deliberately try membership inference attacks during an audit stage before deploying a model. If the attack works well, the risk of exposing personal information is high, so they apply differential privacy or reduce the output information, then test again to confirm the effect.