(An) analysis of the roles of semantic answer types for contemporary machine reading question answering models

Abstract

Recently, the performance of Machine Reading Question Answering (MRQA) models has surpassed humans on datasets like SQuAD. For further advances in MRQA techniques, new datasets are being introduced. However, they are rarely based on a deep understanding of the QA capabilities of the existing models tested on the previous datasets. In this study, we analyze the SQuAD and triviaQA dataset quantitatively and qualitatively to show how the MRQA models answer the questions. It turns out that the current MRQA models rely heavily on the use of wh-words and Lexical Answer Types (LAT) in the questions instead of attempting to make use of the meanings of the entire questions and the evidence documents. Based on this analysis, we present the directions for new datasets so that they can help advancing the current QA techniques centered around the MRQA models.