Privacy in Machine Learning and Differential Privacy

Machine learning feeds on data, and much valuable data is personal: health records, locations, financial information, case files of displaced people. Privacy failures can expose people to discrimination, persecution, fraud or violence. Protecting privacy requires more than removing names — models themselves can leak their training data. This lecture covers the threats and the strongest formal defence we have: differential privacy.

Why "anonymised" data often is not#

• Linkage attacks: Latanya Sweeney showed that ZIP code, birth date and sex alone uniquely identify a large majority of Americans, and re-identified a governor's medical records by linking "anonymised" hospital data with a public voter list.
• The Netflix Prize dataset was de-anonymised by linking movie ratings with public IMDb reviews (Narayanan & Shmatikov, 2008).
• Location traces are highly unique: a few spatio-temporal points can identify most individuals in mobile-phone datasets (de Montjoye et al., 2013).

k-anonymity and related techniques (generalising or suppressing quasi-identifiers) help but remain vulnerable to background knowledge and composition of releases.

Attacks on models#

• Membership inference (Shokri et al., 2017): determine whether a specific person's record was in the training set — itself sensitive (e.g. membership in a dataset of patients with a particular disease). Overfit models are especially vulnerable.
• Model inversion / attribute inference: reconstruct sensitive attributes or representative inputs from a model.
• Training data extraction: large language models have been shown to regurgitate memorised training text verbatim, including personal information (Carlini et al., 2021).
• Leakage through features and embeddings.

Differential privacy (DP)#

Dwork, McSherry, Nissim and Smith (2006) proposed a rigorous definition. A randomised algorithm $M$ is $(\varepsilon, \delta)$-differentially private if for all neighbouring datasets $D$ and $D'$ differing in one individual's data, and all sets of outputs $S$:

Intuition: the output is almost equally likely whether or not any single person participated, so an observer learns very little about any individual. Smaller $\varepsilon$ means stronger privacy; $\delta$ is a small failure probability (much smaller than $1/n$).

Key properties:

• Robust to auxiliary information: guarantees hold whatever the attacker knows.
• Composition: running several DP analyses adds up privacy loss (the "privacy budget").
• Post-processing: anything computed from a DP output remains DP.

The Laplace mechanism#

To release a numeric query $f(D)$ (e.g. a count) with sensitivity $\Delta f$ (the maximum change from one person), add Laplace noise:

import numpy as np

def dp_count(values, predicate, epsilon, rng=np.random.default_rng()):
    true = sum(predicate(v) for v in values)
    return true + rng.laplace(0, 1.0 / epsilon)          # sensitivity of a count is 1

ages = np.random.default_rng(0).integers(0, 90, 10_000)
for eps in [0.1, 1.0, 10.0]:
    print(f"epsilon={eps:>4}: noisy count of children under 5 = {dp_count(ages, lambda a: a < 5, eps):.1f}")
print("true count:", int((ages < 5).sum()))

Large aggregate counts remain accurate while individual contributions are masked. National statistics offices (e.g. the US Census Bureau for the 2020 Census) have adopted DP for official data releases.

DP-SGD: training models with differential privacy#

Abadi et al. (2016) made deep learning differentially private:

1. Compute per-example gradients.
2. Clip each gradient to a maximum norm $C$ (bounding any individual's influence).
3. Add Gaussian noise with standard deviation $\sigma C$ to the sum.
4. Track cumulative privacy loss with an accountant (moments accountant / Rényi DP).

Libraries: Opacus (PyTorch), TensorFlow Privacy. Costs: lower accuracy (especially for small datasets and minority groups), slower training, careful hyperparameter tuning. Pretraining on public data and fine-tuning privately helps considerably.

# Sketch with Opacus
from opacus import PrivacyEngine
# model, optimizer, train_loader defined as usual
privacy_engine = PrivacyEngine()
# model, optimizer, train_loader = privacy_engine.make_private_with_epsilon(
#     module=model, optimizer=optimizer, data_loader=train_loader,
#     target_epsilon=3.0, target_delta=1e-5, epochs=10, max_grad_norm=1.0)
# ... train normally; privacy_engine.get_epsilon(1e-5) reports the budget spent

Privacy by design in practice#

1. Data minimisation: collect only what you need; delete when no longer needed.
2. Purpose limitation and a clear legal basis (e.g. under GDPR or national data-protection laws); informed consent where applicable.
3. Pseudonymisation, access control, encryption at rest and in transit, audit logs.
4. Aggregation and DP for published statistics and dashboards.
5. Federated learning or on-device processing to keep raw data local (next lecture).
6. Privacy impact assessments before new processing.
7. Test models for memorisation (membership inference, canary insertion) before release.