Theory and application of ultra high dimensional sparse representations for efficient and interpretable semantic search

Abstract

Deep learning-based models generally use low-dimensional dense representations to express data samples. Although compact and powerful, it bears several shortcomings that make it unsuitable for tasks requiring processing a large number of samples (e.g., searching documents from web-scale corpus). More specifically, since each dimension of low-dimensional dense representations is highly entangled because of the limited number of dimensions available, it is susceptible to false matches when the number of samples is large. Also, all the dimensions must participate in representing and comparing samples regardless of each sample's characteristics, which is inefficient. Lastly, it is usually hard to interpret the entangled dimensions of dense representations. This thesis shows how high-dimensional sparse representations can cope with such problems in the field of natural language processing (NLP). We first explain the theoretical background and properties of high-dimensional sparse representations. Then we show how high-dimensionality and sparseness allow us to kill two birds, the performance and efficiency when applied to information retrieval (IR) and question answering (QA), the NLP tasks that require accurately finding relevant documents or answers from a vast amount of corpus with low latency. Finally, we introduce a method to interpret the model's outcome in quantitative and qualitative ways.