Paper review · October 23, 2026
Exploring the Empirical Privacy Landscape of Synthetic Data
Exploring The Empirical Privacy Landscape of Synthetic Data
Through a controlled comparison of fine-tuning defenses using synthetic medical records and canaries, the authors showed that privacy-aware rewriting strikes a good balance between performance and information leakage.
Problem
Fine-tuning a language model on sensitive data can lead it to memorize and leak training data. Yet few studies have compared data-level defenses (synthesis, rewriting, PII removal) and training-level defenses (such as DP-SGD) under the same conditions, measuring both performance and leakage together.
Method
The authors inserted 100 canaries containing fake drug names into 5,000 synthetic patient records generated with Synthea. Canaries were inserted with 1, 4, 16, 32 or 64 duplicates, for a total of 7,340 records. They compared no training, direct fine-tuning, PII removal, DP-SGD (ε=8), Goldfish, generic rewriting and privacy-aware rewriting (instructed to replace rare details with fictional values). The target models were Qwen3.5-4B-Base and gpt-oss-20b, trained with LoRA. Leakage was measured by the perplexity of canary drug names in sentence completion and question answering, and performance by the perplexity of held-out records and drug names.
Results
- Direct fine-tuning gave the best performance for both models but also leaked heavily. DP-SGD reduced leakage but caused a large performance loss.
- PII removal deletes only identifiers such as names and addresses and leaves non-identifying facts such as fake drug names, so at high duplication it leaked at a level similar to direct fine-tuning.
- For both target models, privacy-aware rewriting ranked higher than generic rewriting on both privacy and performance. DP-SGD and GPT-5.5-based privacy-aware rewriting remained robust to leakage even at 64 duplicates.
- Data privacy as rated by LLM judges correlated strongly with the sentence-completion leakage metric (ρ=0.89). In contrast, data quality as rated by the judges did not predict performance gains (ρ=-0.64).
- Training for 1 epoch on 5 rewritten versions was no better than training for 5 epochs on 1 rewritten version.
Limitations
The rewriting methods have no formal differential privacy guarantee and were evaluated only empirically. Both leakage and performance were measured only with perplexity-based metrics. There are 2 target models, and training used LoRA with a single seed. The data are Synthea synthetic records and may differ from the diversity of real clinical documents. Different canary groups were used for each duplication level, so the difficulty of the canaries themselves is mixed in. Rewriting incurs additional generation cost.
Why it matters
The paper shows that PII removal alone is not enough when fine-tuning models on sensitive data. Because the design of the rewriting prompt strongly affects the performance-privacy balance, the findings can be applied directly when designing data preprocessing pipelines.
Key terms
- Canary
- A unique piece of secret information deliberately planted in training data. If the model reproduces it, that is evidence of memorization and leakage.
- DP-SGD
- A differentially private training method that clips per-example gradients and adds noise to mathematically limit the influence of any individual training example.
- Goldfish
- A training technique that excludes some tokens from the loss calculation according to a fixed rule to reduce verbatim memorization.
Original
Authors: Zihao Zhao, Avi Schwarzschild, Casey Meehan, Katherine Lee, Zhili Feng, Dylan Sam · Posted 2026-10-15 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.