MASKER: Masked keyword regularization for reliable text classification

Abstract

Pre-trained language models have achieved state-of-the-art performance on text classification. However, it is under-explored that the fine-tuned text classifiers work for out-of-distribution (OOD) samples (drawn far from training distribution) or whether they are generalizable to domain shift. To address the questions, we find that the fine-tuned text classifiers still suffer from the keyword bias: overly relying on a limited number of keywords to make predictions. In particular, we empirically show that keyword bias makes models over-confident for an OOD sample containing keywords and causes a significant accuracy drop under domain shift. Inspired by this finding, we propose a simple yet effective fine-tuning method, named masked keyword regularization (MASKER), that facilitates the prediction based on the contextual information. Specifically, MASKER forces the model to predict the keywords from the rest of the words, while regularizing the model to produce low confident predictions for the masked text simultaneously. For further improvement, we also propose keyword selection schemes that are specific to our training method. We demonstrate that MASKER, applied to various pre-trained language models (e.g., BERT, RoBERTa, and ALBERT) improves OOD detection and cross-domain generalization, without degradation of classification accuracy.