AI privacy: does training data leak? Membership inference, extraction and differential privacy

Why it was needed: the belief that "models only summarize data"

Machine learning models were usually seen as products that extract only general rules from large amounts of data. But training data contains sensitive information such as people's purchase histories, locations and medical records. The membership inference attack study paper pointed out that such services use training data and posed a question: what can be learned about the original data just by looking at the resulting model?

This belief was even stronger for language models. According to the GPT-2 training data extraction study paper, large language models are often trained only once (1 epoch) on vast deduplicated data. As a result, overfitting (fitting the training data well but performing worse on new data) was hardly visible. Because of this, the idea that "they do not memorize specific documents" was widespread.

The studies covered in this article show step by step that this idea was wrong. Even if models do not memorize on average, they do memorize some "worst-case" data. That information can come out just by sending queries through an API.

Key idea: a student who memorized the exam questions

Think of a student who memorized past exam questions. They answer questions they have seen with unusual confidence and speed, and hesitate on new ones. Just watching this difference is enough to tell "they have seen this question before." That is membership inference. Going one step further, if you read out only the beginning of a question and have them recite the memorized answer to the end, that is training data extraction.

The two attacks have different goals. Membership inference answers yes or no to "Was this record in the training data?" Data extraction retrieves the original text of the training data itself. When merely being in a dataset is sensitive, as with medical data (for example, a list of patients with a specific condition), membership inference alone can cause harm.

The defensive idea, differential privacy, puts it this way: if the student's visible behavior is almost the same whether or not they saw a given question, an observer cannot tell whether they saw it. It is a way of mathematically bounding the influence of any single data point to a small amount.

How it works: shadow models, suspiciously high probabilities, noise

The membership inference attack study paper proposed "shadow training." The attacker does not know the target model's internals and can only see queries and answers (black box). So the attacker first trains several shadow models that behave similarly to the target model. Because the attacker knows the shadow models' training data, each input can be labeled with whether it was used in training. The attacker then trains an attack model on these labeled outputs. The attack model becomes a classifier that distinguishes "responses to inputs seen during training" from "responses to unseen inputs." In other words, membership inference is turned into a classification problem.

Data extraction against LLMs paper proceeds in two stages. First, a large volume of diverse text is generated from the model (three sampling strategies). Next, each piece of text is ranked using six metrics that compare how likely it is under this model against other reference models. The reasoning is that text to which only the target model assigns unusually high probability is likely to be memorized.

The basic operation of differentially private training is well illustrated in the explanation from the RLVR differential privacy study paper. The gradients (the direction for updating weights) for the unit being protected (in that study, a single problem) are aggregated, their magnitude is clipped once, and Gaussian noise is added. The privacy loss that accumulates over the course of training is then summed to compute the total budget (ε). The smaller ε is, the stronger the protection and the larger the performance cost.

How it developed: from classification models to commercial chatbots

The membership inference attack study 2016 paper attacked classification models built with Google's and Amazon's "machine learning services." On models trained on retail transaction data of about 10,000 records, the median inference accuracy was 94% and 74%, respectively. Even when shadow models were built with fully synthetic data, without any knowledge of the training data distribution, accuracy against the Google model was 90%. On Texas hospital discharge data it also exceeded 70%. The researchers showed numerically that leakage is related to overfitting. As defenses, they compared methods such as showing only the top-k classes, lowering the precision of predicted values and regularization.

In the same year, the deep learning with differential privacy study 2016 paper became the starting point on the defense side. It introduced new algorithmic techniques for training and a refined analysis of privacy costs. It also showed that deep neural networks can be trained within a modest privacy budget while keeping software complexity, training efficiency and model quality at manageable levels.

The GPT-2 training data extraction study 2020 paper moved the stage to LLMs. Of 1,800 candidate samples, more than 600 were identical to original text in GPT-2's training data. In the best setting, 67% of candidates were original text. The extracted content included personal information such as names, phone numbers and email addresses, IRC conversations, code and 128-bit UUIDs. Text that appeared in only a single document in the training data was also extracted, and larger models were more vulnerable than smaller ones.

The large-scale extraction study 2023 paper extended this to real services. It extracted gigabytes of training data from open-source models such as Pythia and GPT-Neo, from LLaMA and Falcon, and from the closed model ChatGPT. For the aligned ChatGPT, the researchers devised a new "divergence attack." When it was induced to deviate from chatbot-style responses, it emitted training data at a rate 150 times higher than during normal behavior. The conclusion is that current alignment techniques do not eliminate memorized content.

Where it is heading: memory in context, limits of measurement, protection for new training methods

First, researchers are re-examining leakage in real deployment settings. The in-context memorization extraction study paper took paired measurements on three instruction-tuned open-weight models. It compared extraction when only the opening part of a training sentence (the prefix) was given against extraction when retrieved documents were placed in front of it, as in RAG. Context did not erase what the models had memorized. Some content was a 'robust core' that was extracted regardless of context, and some formed a 'boundary' that shifted with context. The longer the prefix, the more reliably extraction worked even with context added. Context blocked some exposures near the boundary, but it also created new exposures that did not show up in prefix-only evaluation. This adds a caveat to the view that 'using RAG reduces the risk of memorization leakage.'

Second, researchers are reassessing how far membership inference can be trusted. OLMo-Detect paper is a benchmark built on OLMo 2, whose training data is fully public, and it covers the pre-training, mid-training and post-training stages. It matches the distributions of text used in training (members) and text not used (non-members) to make the comparison fair. It evaluated 15 unsupervised attacks and 3 supervised attacks, and even the best attack reached an AUC of only 0.68. On AUC, 0.5 means a coin flip and 1.0 means perfect separation. Detection worked best at the mid-training stage, but this was due to the type of data, not the training stage (curated math data stood out much more). Scores rose as models grew from 1B to 13B but did not rise further at 32B.

Third, researchers are extending differential privacy to new training methods. The RLVR differential privacy study paper applied problem-level differential privacy to RLVR (reinforcement learning on problems whose answers can be graded automatically). The authors state that, to their knowledge, this is the first guarantee for RLVR training. They trained Qwen2.5-1.5B-Instruct with LoRA under a budget of ε=8. The reward signal added 2.65 points on MATH and 3.24 points on GSM8K. This was 2.3–3.8 points higher than a private SFT method with the same budget, and it retained 85–90% of the gain achieved by GRPO without privacy. On MATH, even with an 8x tighter budget, the loss was at most 1.2 points.

Limitations and open issues

Defenses come at a cost. The GPT-2 training data extraction study paper noted that differentially private training is theoretically sound but lengthens training time and usually reduces performance. It said deduplicating data does help reduce memorization in practice but does not stop every attack. The unit of protection also has to be chosen correctly. If one person's information is spread across several documents, protection at the level of a single document may not be enough. The study added the caveat that it must be applied at an 'appropriate record level.'

Common workarounds are not complete either. Alignment did not remove memorized content paper, and context such as RAG can reduce some exposures while creating new ones paper. Defenses that adjust the output (showing only the top k results, lowering precision) were compared on classification models paper. So the material in this article gives no basis for assuming they work as-is on generative models.

The measurement tools themselves are also at issue. Membership inference is sometimes cited as evidence in audits or copyright disputes over whether 'this data was used for training.' But as the OLMo-Detect paper results show, even the best attack on LLMs reaches an AUC of only 0.68. So 'not detected' should not be taken as evidence of 'not used for training.' Conversely, a weak attack does not mean there is no leakage. Extraction attacks target memorized sentences in the 'worst case,' not on average.

Further reading

If you are new to the topic, start with the GPT-2 training data extraction study paper. It gives the most intuitive explanation of what memorization in LLMs is and why it happens even without overfitting. Its closing discussion of defenses also connects directly to practice. Next, the membership inference attack study paper explains the shadow model technique and its relationship to overfitting, which gives you a foundation for reading later membership inference research.

If you operate a service, read the large-scale extraction study paper, which extended the scope to commercial models, together with the in-context memorization extraction study paper, which covers exposure in RAG settings. If you are designing defenses, get the concepts from the differentially private deep learning study paper, then look at the RLVR differential privacy study paper for concrete numbers on how budget trades off against performance.

If you plan to use membership inference to audit whether data was used, OLMo-Detect paper is strongly recommended. It shows how attacks actually perform and how results get inflated when member and non-member distributions are not matched.