Utilizing non-local information to large-scale hierarchical text classification

Abstract

Hierarchical text classification to a web taxonomy is challenging because it is a very large-scale problem with hundreds of thousand categories and associated documents. Furthermore, the conceptual levels and training data availabilities of categories vary widely. Compared to the previous work solely relying on machine learning, a narrow-down approach is the state-of-the-art that utilizes a search engine for generating candidates from the taxonomy and builds a classifier for the final category selection. However, we observed the previous work just focusing on local information associated with candidate categories to train a classifier. In this thesis, we take the same approach but address the issue of using non-local information, i.e. global and path information, to improve the effectiveness of classification. To this end, this thesis proposes methods using non-local information based on statistical language modeling framework which is well-developed in information retrieval area by understanding the necessity of non-local information. For evaluation, we constructed a document collection from web pages in the Open Directory Project (ODP). A series of exhaustive experiments and their results show the superiority of our methods and reveal the role of non-local information in hierarchical text classification.