Performance analysis and caching mechanism study for iterative mapreduce systems

Abstract

MapReduce has become a dominant framework in big data analysis, and thus there have been significant efforts to implement various data analysis algorithms in MapReduce. Many data analysis algorithms are inherently iterative, repeating the same set of tasks until a convergence. To efficiently support iterative algorithms at scale, a few variants of Hadoop and new platforms have been proposed and actively developed in both academia and industry. Representative systems include HaLoop, iMapReduce, Twister, and Spark. In this dissertation, we analyze the performance of iterative MapReduce systems to obtain lessons and study a new caching mechanism through the lessons. To better understand the distributed processing of iterative algorithms, we identify and categorize the limitations of MapReduce in handling iterative algorithms, and then, experimentally compare Hadoop and the aforementioned systems using various workloads and metrics. We thoroughly explore the effectiveness of their new caching, communication, and scheduling mechanisms in support of iterative computation. In general, Spark achieved the best performance. Thus, to learn more about Spark, we experimentally investigate the limitations of Spark in handling iterative algorithms. According to our experiment results, the network I/O overhead was the primary factor that affected system performance the most. The disk I/O overhead also affected system performance, but it was less significant than the network I/O overhead. In addition to these overheads, garbage collection is also heavily affected by the caching mechanism, especially when processing iterative algorithms. Thus, we conducted experiments to analyze the effect of garbage collection according to caching mechanisms in the running of iterative algorithms on Spark. The experimental results show that frequent major garbage collection is a major performance degradation factor, with the garbage collection time accounting for up to 46.51% of the total execution time. In the comparison between the memory cache and the disk cache, their difference becomes larger when the size of the variable data is smaller than that of the eden but the sum of the static and variable data is not. Applying lessons learned from the previous experiments, we study a new Spark memory management system called machine-learning tuning (MLTuning), which automatically optimizes a caching mechanism for Spark. MLTuning comprises two phases: heuristic adjustment phase and regression tuning phase. In the heuristic adjustment phase, the algorithm adaptively tunes the memory by a heuristic rule

in the regression tuning phase, it optimizes the memory by fitting a regression model to the execution logs that are collected during the heuristic adjustment phase. Experimental results indicate that MLTuning can reduce memory overheads and improve the performance of a Spark job by up to 56% compared with existing Spark memory management systems. We believe that our work will contribute toward the efficient distributed processing of iterative algorithms including machine-learning and deep-learning algorithms.