Deepfakes, Synthetic Media and Misinformation

In early 2024, an employee of a multinational company in Hong Kong reportedly transferred about 25 million US dollars after a video call in which the "chief financial officer" and other colleagues were deepfakes. Voice-cloning scams target families with fake calls from "relatives in trouble". Fabricated images of disasters and conflicts circulate within minutes of real events. Generative AI has made realistic synthetic media cheap and fast — creating new threats to trust, security and democracy.

How synthetic media is made#

• Face swapping: autoencoders or GANs trained on footage of two people swap faces in video (the original "deepfakes").
• Face reenactment / lip-sync: drive a target face with another person's expressions, or match lips to new audio.
• Voice cloning: text-to-speech models that imitate a voice from a few seconds of audio.
• Text-to-image and text-to-video: diffusion and transformer models generate photorealistic scenes from prompts.
• LLM-generated text: fluent articles, reviews, comments and personalised messages at scale.

Harms#

• Fraud and scams: impersonating executives, officials or family members.
• Non-consensual intimate imagery: the most common malicious use of deepfakes in several analyses, disproportionately targeting women; devastating for victims.
• Political misinformation: fake speeches, robocalls and images around elections and conflicts.
• Crisis misinformation: fake disaster images or rumours during emergencies can misdirect aid, cause panic or endanger people.
• Harassment and reputational attacks.
• The liar's dividend: when anything could be fake, genuine evidence (of abuses, for instance) can be dismissed as fabricated.
• Erosion of trust in media and institutions.

Detection: an arms race#

Detection methods look for artefacts: inconsistent lighting, unnatural blinking, lip-sync errors, frequency-domain traces of generators, physiological signals (subtle colour changes from pulse), inconsistent reflections, or text statistics.

Provenance and watermarking#

Rather than only detecting fakes, we can authenticate the real and label the synthetic:

• Content provenance (C2PA): the Coalition for Content Provenance and Authenticity standard attaches cryptographically signed "content credentials" recording how media was captured and edited. Some cameras, editing tools and platforms support it.
• Watermarking: invisible signals embedded in AI-generated images, audio or text (e.g. Google DeepMind's SynthID). Text watermarks bias token choices in statistically detectable ways (Kirchenbauer et al., 2023). Limitations: watermarks can sometimes be removed or weakened by editing, paraphrasing or re-generation, and they only work if generators embed them.
• Disclosure policies: platforms and regulations (e.g. transparency obligations for deepfakes under the EU AI Act) require labelling of synthetic content.

# A toy illustration of statistical text watermark detection (green-list idea)
import hashlib, math

def is_green(prev_token, token, gamma=0.5):
    h = int(hashlib.sha256(f"{prev_token}|{token}".encode()).hexdigest(), 16)
    return (h % 1000) / 1000 < gamma            # pseudo-random "green list" seeded by previous token

def watermark_z_score(tokens, gamma=0.5):
    greens = sum(is_green(a, b, gamma) for a, b in zip(tokens, tokens[1:]))
    n = len(tokens) - 1
    return (greens - gamma * n) / math.sqrt(n * gamma * (1 - gamma))

text = "the clinic opens at nine and closes at four on weekdays".split()
print(round(watermark_z_score(text), 2))       # near 0 for unwatermarked text; large for watermarked

A watermarking generator favours "green" tokens; detection counts them — a large z-score indicates watermarked text.

Policy and platform responses#

• Laws against specific harms: non-consensual deepfake imagery, election deepfakes, fraud and impersonation.
• Transparency requirements for synthetic media.
• Platform policies: labelling, removal of deceptive manipulated media, reduced distribution.
• Industry commitments on provenance and watermarking.

Media literacy and organisational practice#

Technology alone will not solve the problem. Individuals and organisations need habits:

• Verify before sharing: check sources, reverse-image search, look for original context, consult fact-checkers.
• Out-of-band verification for unusual requests: call back on a known number; use code words within families and teams for emergency requests.
• Organisational protocols: financial approvals never based on a single call or video; staff training on voice and video impersonation.
• Responsible communication: organisations — especially news and humanitarian agencies — should never present AI-generated images as real documentation, and should label illustrative synthetic content clearly.