Dynamic virtual mapreduce cluster management with virtual machine and slot reconfiguration

Abstract

With ever increasing demands on large scale data analysis, the importance of distributed data-oriented computing such as MapReduce has been growing precipitously. Meanwhile, cloud computing has emerged to provide flexible computing resources as a utility, replacing privately owned computing clusters. The advent of such cloud computing opens a new opportunity to harness the flexibility of cloud resources for the massive distributed platforms. However, the current distributed data-oriented platforms on the cloud do not fully uti-lize such resource flexibility of the cloud. On the cloud, a MapReduce platform runs on a virtual cluster, con-sisting of many virtual machines (VMs). Even though each user job may exhibit different usages of resources, the current cloud-based platforms allow only a fixed amount of each resource type for a VM, leading to frequent resource imbalances across VMs. In this dissertation, I propose a dynamic resource management system for MapReduce platforms on the cloud, which dynamically reconfigures the resource of each node with VM reconfiguration. The system uses two control modes, depending on the target resource type. Firstly, each node dynamically adjusts the number of slots, resource unit in MapReduce, to mitigate memory and I/O bottlenecks. By controlling the number of outstanding tasks, the slot adjustment reduces the potential memory and I/O bottlenecks in each node. Secondly, the proposed system exploits the unique capability of virtual clusters with the dynamic reconfiguration of CPUs in each VM. With the combination of CPU and slot controls, the second technique can mitigate the computational bottlenecks in addition to memory and I/O delays. By applying these simple policies to local nodes in a distributed manner, the system can achieve improved performance and resource efficiency without the complexity of global resource control. For the evaluation, I constructed a small-scale Hadoop cluster, an open source MapReduce platform,...