DC Field | Value | Language |
---|---|---|
dc.contributor.author | Kang, Minki | ko |
dc.contributor.author | Han, Moonsu | ko |
dc.contributor.author | Hwang, Sung Ju | ko |
dc.date.accessioned | 2021-01-28T06:07:11Z | - |
dc.date.available | 2021-01-28T06:07:11Z | - |
dc.date.created | 2020-12-03 | - |
dc.date.created | 2020-12-03 | - |
dc.date.created | 2020-12-03 | - |
dc.date.issued | 2020-11-16 | - |
dc.identifier.citation | Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.6102 - 6120 | - |
dc.identifier.uri | http://hdl.handle.net/10203/280128 | - |
dc.description.abstract | We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training, such that we can effectively adapt the language model to a particular target task (e.g. question answering). Specifically, we present a novel reinforcement learning-based framework which learns the masking policy, such that using the generated masks for further pre-training of the target language model helps improve task performance on unseen texts. We use off-policy actor-critic with entropy regularization and experience replay for reinforcement learning, and propose a Transformer-based policy network that can consider the relative importance of words in a given text. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets using BERT and DistilBERT as the language models, on which it outperforms rule-based masking strategies, by automatically learning optimal adaptive maskings. | - |
dc.language | English | - |
dc.publisher | ACL | - |
dc.title | Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation | - |
dc.type | Conference | - |
dc.identifier.wosid | 000855160706026 | - |
dc.identifier.scopusid | 2-s2.0-85101667429 | - |
dc.type.rims | CONF | - |
dc.citation.beginningpage | 6102 | - |
dc.citation.endingpage | 6120 | - |
dc.citation.publicationname | Conference on Empirical Methods in Natural Language Processing (EMNLP) | - |
dc.identifier.conferencecountry | IT | - |
dc.identifier.conferencelocation | Virtual | - |
dc.contributor.localauthor | Hwang, Sung Ju | - |
dc.contributor.nonIdAuthor | Kang, Minki | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.