(A) novel term weighting scheme based on discrimination power

Abstract

Term weighting for document ranking and retrieval has been an important research topic in Information Retrieval for decades. We propose a novel term weighting method that utilizes availability of past retrieval results consisting of the queries that contain a particular term, retrieval documents, and their relevance judgments. A term’s evidential weight, DP (Discrimination Power) which we propose in this paper, depends on the degree to which the mean weighting scores for the relevant and non-relevant document distributions are different in the relevance-judged past document collection. It also takes into account the rankings and similarity values of the relevant and non-relevant documents to make a compensation for incorrect positions or scores in the retrieved document list. The experiments were performed using two well-known open-source search engines, Terrier and Indri, and four different ranking models including TFIDF, DFR (Divergence From Randomness) BM25, Hiemstra Language Model, and Indri Language Model. Our experimental result using a standard test collection (TREC-3,4, and 5) shows that a term weighting scheme that incorporates the notion of evidential weights outperforms the four baseline scheme. It is interesting to note that we obtained the performance increase with only a small number of terms found in the relatively small number of past queries. An additional analysis of how the effectiveness changes as the number of terms having DP value increases shows that DP has strong applicability given a large set of queries because the effect of DP is in proportion to the number of DP terms. Further analysis shows the notion of evidential weight, not based on the entire collection but based on the relevance-judged documents, is clearly distinct from IDF. In addition, an experiment was performed and showed significant result on TREC Web Blogs collection to show the proposed method is feasible to apply to general Web search. As a result, we designed a new te...