(An) efficient approach to improve space overhead in modern block-based DFS

Abstract

Modern block-oriented distributed storage systems have proliferated in this era of big data and cloud computing. These systems feature block-level data replication in order to guarantee both fault tolerance and data availability. The distributed storage systems typically partition their files into equal-sized blocks and make multiple copies for each block, which are then distributed across machines. However, the data replication strategy severely causes the issue of space overhead as it requires more storage volumes for keeping the copies. Many storage volumes are just wasted only for keeping the block copies of which data may not be accessed in the strategy. We present a novel technique called DynaEC to address the issue in modern block-oriented distribute storage systems. Dynaec substitutes block copies with much fewer parity blocks at runtime by integrating erasure coding techniques into a distributed storage system. In the integration, dynaec provides unique features which have not been well addressed by other studies. First, dynaec provides a unique striping and parity block placement algorithm that encodes data blocks which are randomly distributed across machines to parities and then places the parities to the other machines, guaranteeing data availability for any node failure. Second, parity encoding in dynaec is performed without any change of the original block placement policy in the distributed storage systems. This makes dynaec work seamlessly in the block-oriented distributed storage systems. Third, during the encoding procedure each data node encodes each own data blocks only, not requiring any information about other blocks in other data nodes. As such, the encoding procedure in dynaec can be fully performed in parallel without any synchronization issue. Finally, DynaEC achieves both performance and availability at a similar level to the block-level replication scheme after substitution of replicas with parities. With extensive experiments on our implementation on Hadoop DFS(Distributed File Systems), we show that DynaEC saves storage volumes to the theoretical limit while outperforming the previous approaches in terms of encoding time by orders of magnitude. Solid state drives(SSDs), which have no mechanical moving parts, have received much attention as a new storage media. As SSDs have become more cost effective with the declining price of NAND flash memories, they are widely used from embedded devices to enterprise-level servers. SSDs provide a lot of benefits over HDDs such as low power consumption, shock resilience, light weight, and significantly low random latency. This enables us to reduce the overall operating costs for enterprise data centers although the initial costs of adopting SSDs may still higher than HDDs. However, there remain two serious problems limiting wider deployment of SSDs: limited lifespan and relatively poor random write performance. The main reason of the above problems is the write amplification which is caused by out-of-place update characteristics of NAND flash memories. In parity based DFS, writing a SSD page needs to update not only the data block but also the parity block, both of which are random writes. The more random write operations make more garbage collection triggering, degraded system performance, and shortened SSD lifetime. To solve the parity update issue in SSD-based DFS, we propose a novel technique, called LPUS. LPUS transforms random parity updates to sequential log writes with additional log blocks in the SSD. In contrast to the conventional parity update process, our approach reduces the block fragmentation using sequential parity logs and the number of parity updates by updating parities in a lazy manner. We implement and evaluate the proposed method on SRsim which is an open-source SSD array simulator. The experimental results show that, LPUS reduces write amplification up to 37% and the number of erase operations up to 50% with the reasonable size of log space.