DC Field | Value | Language |
---|---|---|
dc.contributor.author | Kim, Minsu | ko |
dc.contributor.author | Hong, Joanna | ko |
dc.contributor.author | Ro, Yong Man | ko |
dc.date.accessioned | 2021-11-10T06:44:41Z | - |
dc.date.available | 2021-11-10T06:44:41Z | - |
dc.date.created | 2021-09-29 | - |
dc.date.created | 2021-09-29 | - |
dc.date.created | 2021-09-29 | - |
dc.date.issued | 2021-12-06 | - |
dc.identifier.citation | Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021) | - |
dc.identifier.uri | http://hdl.handle.net/10203/289063 | - |
dc.description.abstract | In this paper, we propose a novel lip-to-speech generative adversarial network, Visual Context Attentional GAN (VCA-GAN), which can jointly model local and global lip movements during speech synthesis. Specifically, the proposed VCA-GAN synthesizes the speech from local lip visual features by finding a mapping function of visemes-to-phonemes, while global visual context is embedded into the intermediate speech representation to refine the coarse speech representation in details. To achieve this, a visual context attention module is proposed where it encodes global representations from the local visual features and provides the desired global visual context corresponding to the given coarse speech representation to the generator. In addition to the explicit modelling of local and global visual representations, a synchronization technique is introduced through contrastive learning that guides the generator to synthesize a speech in sync with the given input lip movements. Extensive experiments demonstrate that the proposed VCA-GAN outperforms existing state-of-the-art and is able to effectively synthesize the speech from multi-speaker that has been barely handled in the previous works. | - |
dc.language | English | - |
dc.publisher | Neural Information Processing Systems | - |
dc.title | Lip to Speech Synthesis with Visual Context Attentional GAN | - |
dc.type | Conference | - |
dc.type.rims | CONF | - |
dc.citation.publicationname | Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS 2021) | - |
dc.identifier.conferencecountry | US | - |
dc.identifier.conferencelocation | Virtual | - |
dc.contributor.localauthor | Ro, Yong Man | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.