A Design of Kernel-Level Remote Memory Extension System

Abstract

Big data scientists are struggling with a memory capacity of the computer system because of the expansion speed of memory capacity has not kept up with the increasing requirement of large memory applications. Even though large memory application vitally requires big memory system, big memory machine has been too expensive for many researchers and students. By the way, as very high-speed networking technologies such as Infiniband EDR(100 Gbps) have been developed, approaches to utilize remote memory has been considered as a cost effective way to run large memory applications in the HPC cluster environment. For the general users of HPC cluster system who want to run large memory application with administrator's support, we suggest a kernel-level remote memory extension system. We designed a remote memory extension system which mapped remote memory pages to the virtual address space of the large memory application process. The system includes three components such as remote memory consumer, Integrated Memory Manager, and memory provider. We developed a kernel-level remote memory extension device and achieved 4 x improvement of the latency of page fault handling on remote memory.