Histogram transformation methodology for data distribution in parallel joins

Abstract

As parallel database computers have been more popular, the design of efficient parallel join algorithms has been one of major issues of the database system area. Basically, the conventional parallel join algorithms have two phases: partitioning the joined relations and locally joining the partitioned relations in parallel. In real databases, it is often found that certain values for a given attribute occur more frequently than other values. This phenomenon is referred to as data skew. With skewed data distribution, the join algorithms have encountered many difficulties to achieve good load balancing among processors in the joining phase. This thesis discusses a data distribution problem for load balance on parallel join algorithms to minimize total execution time. We first propose a data distribution framework to resolve load imbalance and bucket overflow in parallel join. Using the histogram transformation technique, the framework transforms a histogram of skewed data to a desired distribution that corresponds to the relative computing power of node processors in the system. Next we propose an efficient parallel join algorithm for handling skewed data based on the proposed data distribution method. The main idea is to use the data distribution framework for allocating relations evenly across all the nodes. The proposed join algorithm works in three phases: the histogram evaluation phase, the partitioning phase, and the joining phase. The histogram evaluation phase first obtains a cumulative histogram of the hashed values of the join attribute for the smaller relation. Then, it determines a histogram equalization transfer function and boundary values for partitioning. In the subsequent partitioning phase, the relations are distributed among the nodes using the boundary values. Finally, the partitioned relations are locally joined on each node processor in parallel. We also present a parallel join algorithm for a heterogeneous system where computing power of eac...