InterSpeech 2021

LiRA: Learning Visual Speech Representations from Audio through Self-supervision
(3 minutes introduction)

Pingchuan Ma (Imperial College London, UK), Rodrigo Mira (Imperial College London, UK), Stavros Petridis (Facebook, UK), Björn W. Schuller (Imperial College London, UK), Maja Pantic (Imperial College London, UK)
The large amount of audiovisual content being shared online today has drawn substantial attention to the prospect of audio-visual self-supervised learning. Recent works have focused on each of these modalities separately, while others have attempted to model both simultaneously in a cross-modal fashion. However, comparatively little attention has been given to leveraging one modality as a training objective to learn from the other. In this work, we propose Learning visual speech Representations from Audio via self-supervision (LiRA). Specifically, we train a ResNet+Conformer model to predict acoustic features from unlabelled visual speech. We find that this pre-trained model can be leveraged towards word-level and sentence-level lip-reading through feature extraction and fine-tuning experiments. We show that our approach significantly outperforms other self-supervised methods on the Lip Reading in the Wild (LRW) dataset and achieves state-of-the-art performance on Lip Reading Sentences 2 (LRS2) using only a fraction of the total labelled data.