Cross-modal alignment and translation for missing modality action recognition

Cited 1 time in webofscience Cited 0 time in scopus
  • Hit : 77
  • Download : 0
DC FieldValueLanguage
dc.contributor.authorPark, Yeonjuko
dc.contributor.authorWoo, Sangminko
dc.contributor.authorLee, Suminko
dc.contributor.authorNugroho, Muhammad Adiko
dc.contributor.authorKim, Changickko
dc.date.accessioned2023-10-17T02:00:35Z-
dc.date.available2023-10-17T02:00:35Z-
dc.date.created2023-10-16-
dc.date.issued2023-11-
dc.identifier.citationCOMPUTER VISION AND IMAGE UNDERSTANDING, v.236-
dc.identifier.issn1077-3142-
dc.identifier.urihttp://hdl.handle.net/10203/313398-
dc.description.abstractMultimodal data provide complementary information on the same context, leading to performance improve-ment in video action recognition. However, in reality, not all modalities are available at test time. To this end, we propose Cross-Modal Alignment and Translation (CMAT) framework for action recognition that is robust to missing modalities. Specifically, our framework first aligns representations of multiple modalities from the same video sample through contrastive learning, effectively alleviating the bias with respect to the type of missing modality. Then, CMAT learns to translate representations of one modality into that of another modality. This allows the representations of the missing modalities to be generated from the remaining modalities during the testing. Accordingly, CMAT fully utilizes multimodal information obtained through abundant interactions across modalities. The proposed CMAT achieves state-of-the-art performances in both complete and missing modality settings on NTU RGB+D, NTU RGB+D 120, and Northwestern-UCLA datasets. Moreover, extensive ablation studies demonstrate the effectiveness of our design.-
dc.languageEnglish-
dc.publisherACADEMIC PRESS INC ELSEVIER SCIENCE-
dc.titleCross-modal alignment and translation for missing modality action recognition-
dc.typeArticle-
dc.identifier.wosid001074796200001-
dc.identifier.scopusid2-s2.0-85170529504-
dc.type.rimsART-
dc.citation.volume236-
dc.citation.publicationnameCOMPUTER VISION AND IMAGE UNDERSTANDING-
dc.identifier.doi10.1016/j.cviu.2023.103805-
dc.contributor.localauthorKim, Changick-
dc.contributor.nonIdAuthorPark, Yeonju-
dc.description.isOpenAccessN-
dc.type.journalArticleArticle-
dc.subject.keywordAuthorVideo action recognition-
dc.subject.keywordAuthorMulti-modal learning-
dc.subject.keywordAuthorMissing modality-
dc.subject.keywordAuthorContrastive learning-
dc.subject.keywordAuthorFeature translation-
Appears in Collection
EE-Journal Papers(저널논문)
Files in This Item
There are no files associated with this item.
This item is cited by other documents in WoS
⊙ Detail Information in WoSⓡ Click to see webofscience_button
⊙ Cited 1 items in WoS Click to see citing articles in records_button

qr_code

  • mendeley

    citeulike


rss_1.0 rss_2.0 atom_1.0