DC Field | Value | Language |
---|---|---|
dc.contributor.author | Kim, Minsu | ko |
dc.contributor.author | Hong, Joanna | ko |
dc.contributor.author | Park, Se Jin | ko |
dc.contributor.author | Ro, Yong Man | ko |
dc.date.accessioned | 2021-10-15T00:30:30Z | - |
dc.date.available | 2021-10-15T00:30:30Z | - |
dc.date.created | 2021-07-23 | - |
dc.date.created | 2021-07-23 | - |
dc.date.created | 2021-07-23 | - |
dc.date.created | 2021-07-23 | - |
dc.date.issued | 2021-10-15 | - |
dc.identifier.citation | 18th IEEE/CVF International Conference on Computer Vision (ICCV), pp.296 - 306 | - |
dc.identifier.uri | http://hdl.handle.net/10203/288199 | - |
dc.description.abstract | In this paper, we introduce a novel audio-visual multi-modal bridging framework that can utilize both audio and visual information, even with uni-modal inputs. We exploit a memory network that stores source (i.e., visual) and target (i.e., audio) modal representations, where source modal representation is what we are given, and target modal representations are what we want to obtain from the memory network. We then construct an associative bridge between source and target memories that considers the interrelationship between the two memories. By learning the interrelationship through the associative bridge, the proposed bridging framework is able to obtain the target modal representations inside the memory network, even with the source modal input only, and it provides rich information for its downstream tasks. We apply the proposed framework to two tasks: lip reading and speech reconstruction from silent video. Through the proposed associative bridge and modality-specific memories, each task knowledge is enriched with the recalled audio context, achieving state-of-the-art performance. We also verify that the associative bridge properly relates the source and target memories. | - |
dc.language | English | - |
dc.publisher | Computer Vision Foundation, IEEE Computer Society | - |
dc.title | Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video | - |
dc.type | Conference | - |
dc.identifier.wosid | 000797698900030 | - |
dc.type.rims | CONF | - |
dc.citation.beginningpage | 296 | - |
dc.citation.endingpage | 306 | - |
dc.citation.publicationname | 18th IEEE/CVF International Conference on Computer Vision (ICCV) | - |
dc.identifier.conferencecountry | CN | - |
dc.identifier.conferencelocation | Virtual | - |
dc.identifier.doi | 10.1109/ICCV48922.2021.00036 | - |
dc.contributor.localauthor | Ro, Yong Man | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.