Efficient joins of multiple data streams : fast joins and load shedding

Abstract

There are growing researches for processing of continuous queries over a set of data streams from multiple data sources. These are indispensable to ubiquitous streaming applications, such as IP or sensor network monitoring, moving objects trajectory, and many others. These stream applications require real-time, or near real-time response for very high arrival rates. But, system resources may not be available for the run-time state required by numerous queries. In the first part of this dissertation, we propose a newly improved and practical algorithm for multiple windowed join called AMJoin, which improves the multiple join performance by guaranteeing the detection of join failures in constant time. To achieve this goal, we first design a new data structure called BiHT (Bit-vector Hash Table) and present the overall behavior of AMJoin in detail. In addition, we show various experimental results and their analyses for clarifying its efficiency and practicability. In the next part of this dissertation, we address the problem of load shedding for continuous multi-way join queries over multiple data streams. When the arrival rates of tuples from data streams exceed the system capacity, a load shedding algorithm drops some subset of input tuples to avoid system overloads. To decide which tuples to drop among the input tuples, most existing load shedding algorithms determine the priority of each input tuple based on the frequency or some historical statistics of its join attribute value, and then drop tuples with the lowest priority.However, those v$\It{value-based }$ algorithms cannot determine the priority of tuples properly in environments where join attribute values are unique and each join attribute value occurs at most once in each data stream. In this dissertation, we propose an load shedding algorithm based on $\It{arrival order patterns}$ for multi-way stream joins in such environments. The proposed load shedding algorithm determines the priority of eac...