Paraphrase Bidirectional Transformer with Multi-task Learning

Abstract

It is very important to analyze the semantic similarity of two sentences for Natural Language Processing (NLP). This paper proposes a Paraphrase-BERT to perform Paraphrase Identification task. We first fine-tune the pre-trained BERT with MRPC data and add a Whole Word Masking, which is pre-training method recently announced by Google, to the BERT. Finally, we perform Multi-Task Learning (MLT) to improve performance. Specifically, the Question Answering task and the Paraphrase Identification (PI) task are learned sequentially to improve performance of PI task. As a result, it has shown that MLT affect a performance improvement of downstream task (11.11% point absolute accuracy improvement, 7.88% point absolute F1 improvement).