Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications

Cited 19 time in webofscience Cited 0 time in scopus
  • Hit : 410
  • Download : 346
Visual events are usually accompanied by sounds in our daily lives. However, can the machines learn to correlate the visual scene and sound, as well as localize the sound source only by observing them like humans? To investigate its empirical learnability, in this work we first present a novel unsupervised algorithm to address the problem of localizing sound sources in visual scenes. In order to achieve this goal, a two-stream network structure which handles each modality, with attention mechanism is developed for sound source localization. The network naturally reveals the localized response in the scene without human annotation. In addition, a new sound source dataset is developed for performance evaluation. Nevertheless, our empirical evaluation shows that the unsupervised method generates false conclusions in some cases. Thereby, we show that this false conclusion cannot be fixed without human prior knowledge due to the well-known correlation and causality mismatch misconception. We show that the false conclusion can be effectively corrected even with a small amount of supervision, i.e., semi-supervised setup. We present the versatility of the learned audio and visual embeddings on the cross-modal content alignment and we incorporate this proposed algorithm into sound saliency based automatic camera view panning in 360 degree videos.
Publisher
IEEE COMPUTER SOC
Issue Date
2021-05
Language
English
Article Type
Article
Citation

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, v.43, no.5, pp.1605 - 1619

ISSN
0162-8828
DOI
10.1109/TPAMI.2019.2952095
URI
http://hdl.handle.net/10203/282411
Appears in Collection
EE-Journal Papers(저널논문)
Files in This Item
111887.pdf(11.95 MB)Download
This item is cited by other documents in WoS
⊙ Detail Information in WoSⓡ Click to see webofscience_button
⊙ Cited 19 items in WoS Click to see citing articles in records_button

qr_code

  • mendeley

    citeulike


rss_1.0 rss_2.0 atom_1.0