Efficient signature file declustering methods for parallel processing

Abstract

The signature file is an abstraction of documents, which has been studied as a storage structure for unformatted data. Since the size of the signature file is much smaller than that of a data file, it has been shown that the signature file can effectively work as a filter that immediately discards most non-qualifying documents for a given query. Although sequential organization of a signature file works well for a data file with a small size, its performance becomes a problem when the size of a data file is large. Many organizations of a signature file can improve the performance based on a tree or hashing techniques for single processor systems. There have also been many attempts to make the schemes run for parallel environment. The Hamming Filter shows good declustering performance for some partial match queries. It declusters a signature file by using the Linear Code Decomposition Method(LCDM) that is used for detecting and correcting errors while transmitting data. The LCDM yields practically no execution skew, if the data is not skewed. However, since the LCDM allocates signatures with the same suffix into the same processing node, it can not avoid data skew if many signatures have the same suffix. In addition, it has problems that make the LCDM difficult for parallelism such as non-scalability and non-determinism. In this dissertation we have proposed two signature file declustering methods, called MIN-entropy, Inner-product respectively that overcome the problems in the LCDM. They decluster signature file dynamically based on the current status of signature allocation. Thus, the MIN-entropy and the Inner-product can cope with a variety of workloads and configurations. We have showed through the performance evaluation based on the statistical modeling that the MIN-entropy and the Inner-product give better retrieval performance than the LCDM for data sets with various distributions such as uniform distribution, normal distribution and exponential distribu...