Multitenant hadoop with advanced resource management

Abstract

Multitenancy has gained growing importance with the development and evolution of cloud computing technology. In a multitenant environment, multiple tenants with different demands can share a variety of computing resources (e.g., CPU, memory, storage, network, and data) within a single system, while each tenant remains logically isolated. This useful multitenancy concept offers highly efficient and cost-effective systems without wasting computing resources to enterprises requiring similar environments for data processing and management. In this paper, we propose a novel approach supporting multitenancy features for Apache Hadoop, a large scale of distributed system commonly used for processing big data. Hadoop largely consists of HDFS responsible for storing and managing big data, YARN responsible for resource management and job execution management, and MapReduce as a programming model for data processing and analysis. We note that current Hadoop incurs many problems in system utilization due to its file-based management of metadata to which HDFS and YARN frequently refer for their operations, and also due to its limited resource (CPU and memory) management provided by YARN. These problems are also critical obstacles in providing multitenancy in Hadoop. To solve these problems, we propose a RDBMS-based metadata management scheme and an advanced resource management framework for improving the functional aspects of overall Hadoop. For this, we first analyze the Hadoop framework focusing on HDFS and YARN. We next define the problems for supporting multitenancy caused by its metadata and resource management and formally derive the requirements for resolving these problems. Based on the requirements, we then design the details of Multitenant Hadoop that replaces the original HDFS and YARN with the advanced HDFS and Multitenant YARN. Finally, through implementation of the proposed Multitenant Hadoop, we validate its correctness by experimental evaluation and we also show that Multitenant Hadoop satisfies all the derived requirements. The proposed Multitenant Hadoop significantly enhances the metadata and resource management scheme of Hadoop and, as a result, improves its stability, availability, and scalability. This can be much helpful in analyzing and proccessing data with high effectiveness for users who have difficulties with building large scale clusters.