Multidimensional selectivity estimation based on dynamic maintenance of data distribution

Abstract

The Multilevel Grid File(MLGF) is a multidimensional dynamic hashed file organization that gracefully adapts to dynamic environments. In this dissertation we implement the MLGF and analyze the asymptotic growth of its directory size. The asymptotic directory growth is an important factor for evaluating the storage overhead of a multidimensional file organization. We derive that the asymptotic directory growth of the MLGF is linearly dependent on the number of records inserted. To justify this derivation, we perform extensive experiments with various distributions of data: uniform, normal, and exponential distributions. We further perform experiments for more complicated cases where the distributions are highly-skewed or highly-correlated. The results show that the directory size of the MLGF increases linearly in the number of records independently of data distributions, data skew, or correlation. The results also show that the rates of increase are nearly constant in all cases. We also propose a new dynamic method for multidimensional selectivity estimation for range queries that works accurately independent of data distribution. Accurate estimation of selectivity is essential for query optimization and physical database design. Our method employs the MLGF for dynamic estimation of multidimensional distribution of data in a file. We show that each level of the MLGF directory naturally maintains a multidimensional data distribution. We then extend it for further refinement and propose the selectivity estimation method based on the information of the data distribution. A major advantage of the proposed method is that the information is maintained dynamically in the MLGF. In contrast, other static methods such as the histogram method employ static data structures, which require periodic restructuring. Extensive experiments have been performed to test the accuracy of the proposed method for selectivity estimation. We use uniform, normal, exponential distributions,...