Adaptive information retrieval for automatically indexed queries and documents

Abstract

There are three problems in searching for relevant documents such as noiseness of descriptors, vocabulary gap between documents and a given query, and different importance of query descriptors. The previous probabilistic retrieval models rank documents, considering only the different importance of query descriptors. They ignore the other problems because it is difficult to obtain knowledge appropriate to a particular application, and to use the knowledge correctly in reducing the three problems. At first, this thesis proposes a general ranking function which can correctly handle the three problems. By the way, the function is too complex for a practical information retrieval system to utilize it for effective and efficient document retrieval. The general ranking function is simplified substantially under the assumption of certainty indexing, i.e., binary indexing. The complexity of the simplified ranking function is reduced further by Faithful User Assumption (FUA) that a relevant document has all the concepts represented by a query. A learning method to reduce the three problems is derived formally from FUA. Each time retrieval results are available, it updates the knowledge on importance of query descriptors and relationships between query descriptors and other descriptors. Noise descriptors are also defined in this thesis. The retrieval by the simplified ranking function and the proposed learning method is called Faithful User Retrieval (FUR) under certainty indexing. The effect of the incrementally constructed knowledge and noise query descriptors is investigated through experiments in FUR under certainty indexing and in the previous probabilistic ranking model BIR. When it is not impossible to obtain the distributions of query descriptors in relevant documents for past queries, the retrieval effectiveness of FUR is comparable to that of BIR. If the distributions become available, both of them improve the performance. The degree of improvement of FUR ...