Humans involuntarily tend to infer parts of the conversation from lipmovements when the speech is absent or corrupted by external noise. In thiswork, we explore the task of lip to speech synthesis, i.e., learning togenerate natural speech given only the lip movements of a speaker.Acknowledging the importance of contextual and speaker-specific cues foraccurate lip-reading, we take a different path from existing works. We focus onlearning accurate lip sequences to speech mappings for individual speakers inunconstrained, large vocabulary settings. To this end, we collect and release alarge-scale benchmark dataset, the first of its kind, specifically to train andevaluate the single-speaker lip to speech task in natural settings. We proposea novel approach with key design choices to achieve accurate, natural lip tospeech synthesis in such unconstrained scenarios for the first time. Extensiveevaluation using quantitative, qualitative metrics and human evaluation showsthat our method is four times more intelligible than previous works in thisspace. Please check out our demo video for a quick overview of the paper,method, and qualitative results.https://www.youtube.com/watch?v=HziA-jmlk_4&feature=youtu.be