Haliassos Alexandros ; Vougioukas Konstantinos ; Petridis Stavros ; Pantic Maja

Abstract

Although current deep learning-based face forgery detectors achieveimpressive performance in constrained scenarios, they are vulnerable to samplescreated by unseen manipulation methods. Some recent works show improvements ingeneralisation but rely on cues that are easily corrupted by commonpost-processing operations such as compression. In this paper, we proposeLipForensics, a detection approach capable of both generalising to novelmanipulations and withstanding various distortions. LipForensics targetshigh-level semantic irregularities in mouth movements, which are common in manygenerated videos. It consists in first pretraining a spatio-temporal network toperform visual speech recognition (lipreading), thus learning rich internalrepresentations related to natural mouth motion. A temporal network issubsequently finetuned on fixed mouth embeddings of real and forged data inorder to detect fake videos based on mouth movements without overfitting tolow-level, manipulation-specific artefacts. Extensive experiments show thatthis simple approach significantly surpasses the state-of-the-art in terms ofgeneralisation to unseen manipulations and robustness to perturbations, as wellas shed light on the factors responsible for its performance. Code is availableon GitHub.