AI glossary · Advanced · AI security · Data engineering
Differential privacy
Differential privacy is a mathematical privacy protection method that adds a calculated amount of 'noise' to data so that no one can tell from the analysis results alone whether a specific individual was included in the data.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Security & safety
- Fields
- AI security, Data engineering
- Added
- 2026-10-09
How it works
The core idea is that 'the result should be almost the same whether or not any one person's data is included'. To achieve this, random noise is added to statistics or to the model training process, hiding the influence any one person has on the result. How much noise to add is controlled by a value called the 'privacy budget'. A small budget means more noise and stronger protection but lower accuracy; a large budget means the opposite.
The privacy budget tug-of-war
- Very small budget
- Balance point suited to the use case
- Very large budget
Why it matters
Simple de-identification, such as removing names or resident registration numbers, can sometimes be undone by matching against other data to re-identify people. Differential privacy is different in that it mathematically guarantees a level of protection regardless of what background knowledge an attacker has. AI models can memorize parts of their training data and reproduce them verbatim, and applying differential privacy to training can reduce this risk of personal data leakage. That is why it draws attention in fields that handle sensitive data, such as healthcare and finance.
Things to know
There is no free protection. Noise lowers the quality of results or model performance, and the cost is larger especially when data is scarce or the analysis concerns small groups. Also, querying the same data repeatedly keeps using up the budget, so total usage must be managed. Saying 'differential privacy was applied' is not enough; you need to see what the budget was set to in order to know the actual level of protection.
Example
Smartphone makers and browser companies use differential privacy when collecting statistics such as frequently used emoji or newly trending words, by adding noise on the device before sending the data. The company can see overall trends but cannot know individual users' input. Government agencies that publish population statistics and companies that train AI models on sensitive data also use the same principle.