Building word knowledge for information retrieval using statistical information

Abstract

Information Retrieval(IR) is the subfield of computer science that deals with the automated storage and retrieval of documents. In IR systems, a user submits a query to find documents for his/her information need. User query is a representation method to deliver user``s need to IR system. User query, however, has individual varieties according to the users knowledge level and information needs. The property of user query can be characterized as following: subjectivity, incompleteness, and variety. Subjectivity of user query means that user query is generated from the subjective knowledge level of individual information needs. Incompleteness is a property that user``s knowledge cannot be a complete one. The level of incompleteness differs from one user to another. Variety of user query can be described as users generally do not use exact same terms for a single concept. These characteristics of user query yield that IR should use knowledge to process user``s query for finding their information needs. The use of knowledge can reduce the knowledge gap between a user and IR system. There are two main categories of the word knowledge in IR: domain knowledge and lexical knowledge. Domain knowledge represents the knowledge of the domain expert using the similarity of terms. While domain knowledge addresses the meaning coherence among terms, lexical knowledge deals with the individual knowledge on a specific term itself. Namely, lexical knowledge focuses on the variable form diversities in documents on a specific term. In this thesis, it is shown that the two major knowledge, domain knowledge and lexical knowledge, can be built by the statistical anaylsis. For the domain knowledge, Bayesian network is used to encode the statistical behavior of terms. The Collocation map, which is a particular instance of the Bayesian network for encoding term dependency relations, is shown to be useful for the task of automatic domain knowledge construction. The proposed similarity m...