Privacy and data protection in AI: GDPR, memorization and DP
AI systems can memorize and expose personal data from training sets, raising privacy risks under GDPR and other frameworks. Differential privacy offers a mathematical safeguard, while regulators like the ICO and EDPB provide guidance on compliance. This article explains the key concepts and practical steps for protecting privacy in AI.
.jpeg){kind=link}
GDPR and AI: core principles
The UK Information Commissioner's Office (ICO) provides guidance on applying data protection principles to AI. The ICO's Guidance on AI and data protection explains that organizations must ensure lawfulness, fairness, and transparency when processing personal data in AI systems. It also covers accountability, governance, and the need for data protection impact assessments (DPIAs).
The European Data Protection Board (EDPB) has issued an EDPB opinion on AI models that addresses legal bases for processing personal data in AI, including the use of web-scraped data and the distinction between personal and non-personal data in models. The opinion emphasizes that AI models may contain personal data even if not directly identifiable, requiring careful analysis under GDPR.
The NIST Privacy Framework offers a voluntary tool for organizations to manage privacy risk. The NIST Privacy Framework helps identify and prioritize actions to protect individuals' privacy while enabling innovation. It is structured around core functions: Identify, Govern, Control, Communicate, and Protect, which can be applied to AI systems.
Memorization and training data extraction
Machine learning models can inadvertently memorize specific training examples, leading to privacy leaks. Research by Shokri et al. demonstrated membership inference attacks where an adversary can determine whether a given record was part of the training dataset by observing the model's predictions. This attack works even with black-box access and was shown on commercial machine learning services.
A more severe risk is training data extraction, as shown by Carlini et al. in their study on large language models. The paper Extracting Training Data from Large Language Models showed that an adversary can recover verbatim text sequences, including personal information like names, phone numbers, and email addresses, by querying a model. The attack succeeded even for sequences that appeared only once in the training data, and larger models were found to be more vulnerable.
These findings highlight that memorization is not just a theoretical concern but a practical threat. The IBM article on AI privacy notes that AI models contain a trove of sensitive data, making them attractive targets for attackers. Data exfiltration and leakage are among the key privacy risks identified, along with collection of sensitive data without consent and use of data beyond disclosed purposes.
Differential privacy as a safeguard
Differential privacy provides a formal framework to limit the information that a model can reveal about its training data. The paper Deep Learning with Differential Privacy by Abadi et al. introduced techniques to train neural networks with differential privacy guarantees. They showed that it is possible to train deep networks with non-convex objectives under a modest privacy budget, with manageable costs in accuracy and efficiency.
The core idea is to add calibrated noise during training, ensuring that the presence or absence of any single individual's data does not significantly affect the output. This provides a quantifiable privacy guarantee, often expressed as epsilon (ε). A smaller ε means stronger privacy but typically more noise, which can reduce model quality.
Differential privacy is not a silver bullet; it requires careful tuning and may not be suitable for all applications. However, it is a recognized technical measure that can complement organizational and legal safeguards. The ICO guidance mentions statistical accuracy and fairness, and differential privacy can help address both by reducing reliance on specific individuals' data.
Practical steps for compliance
Organizations developing AI should conduct a DPIA to identify and mitigate privacy risks, as recommended by the ICO. This includes assessing the necessity and proportionality of data processing, and considering measures like data minimization, anonymization, and encryption. The ICO's guidance also emphasizes transparency: individuals should be informed about how their data is used in AI systems.
The EDPB opinion advises that when personal data is used to train AI models, a valid legal basis must be established under GDPR, such as consent or legitimate interest. For web-scraped data, the opinion stresses the need to assess whether the data is personal and to apply appropriate safeguards. Anonymization is a key goal, but the opinion notes that true anonymization is difficult to achieve in AI models.
The NIST Privacy Framework can help organizations systematically manage privacy risks. By mapping its functions to AI lifecycle stages, from design to deployment, organizations can identify gaps and implement controls. The framework is voluntary but provides a common language for privacy risk management.
Finally, staying informed about evolving guidance is essential. Regulators like the ICO and EDPB continue to update their recommendations, and technical research on attacks and defenses is ongoing. Organizations should monitor these developments and adapt their practices accordingly.
| Risk | Description | Mitigation |
|---|---|---|
| Membership inference | Attacker determines if a record was in training data | Differential privacy, regularization |
| Training data extraction | Attacker recovers verbatim training examples | Differential privacy, output filtering |
| Data collection without consent | Personal data scraped or collected without permission | Consent mechanisms, legal basis |
| Use beyond disclosed purpose | Data used for AI training without notice | Purpose limitation, transparency |
| Data exfiltration | Attackers steal sensitive data from models | Access controls, encryption |
| Bias and discrimination | AI decisions unfairly impact groups | Fairness audits, bias mitigation |
Frequently asked questions
What is a membership inference attack?
A membership inference attack is a technique where an adversary determines whether a specific data record was part of a model's training dataset. This can leak sensitive information about individuals, as shown in the research by Shokri et al.
How does differential privacy work in AI?
Differential privacy adds calibrated noise to the training process, ensuring that the model's outputs do not significantly change whether any single individual's data is included. This provides a formal privacy guarantee, typically measured by an epsilon value.
What does GDPR require for AI systems?
GDPR requires that personal data processing in AI be lawful, fair, and transparent. Organizations must have a valid legal basis, conduct data protection impact assessments, and implement measures to protect individuals' rights, as detailed in ICO and EDPB guidance.
Can AI models memorize personal data?
Yes, AI models can memorize specific training examples, including personal information. Research has shown that large language models can be queried to extract verbatim text sequences, such as names and phone numbers, even if they appear only once in the training data.
What is the NIST Privacy Framework?
The NIST Privacy Framework is a voluntary tool that helps organizations identify and manage privacy risks. It provides a structured approach with core functions like Identify, Govern, Control, Communicate, and Protect, applicable to AI systems.