Design and Implementation of Highly Scalable User-level TCP Stack for Multicore Systems

Abstract

Today, large-scale servers or middleboxes face millions of transactions over TCP every second. However, scaling the performance of short TCP connections (e.g., transactions) on multicore systems is fundamentally challenging. Although many proposals have attempted to address various shortcomings, inefficiency in the kernel implementation still persists. For example, even the state-of-the-art design spends 70 to 80% of CPU cycles in handling TCP connections in the kernel, leaving only small room for innovation in the user-level program. This work presents mTCP, a high-performance user-level TCP stack for multicore systems. mTCP addresses the inefficiency from the ground up-from packet I/O and TCP connection management to the application interface. In addition to adopting well-known techniques, our design (1) translates expensive system call to shared memory access between two threads within the same CPU core, (2) allows efficient flow-level event aggregation, and (3) performs batch processing of RX/TX packets for high I/O efficiency. Our evaluation on an 8-core machine shows that mTCP improves the performance of small message transactions by a factor 25 and 3 than that of latest Linux TCP stack and the best-performing research system known so far. It also improves the performance of various popular applications by 33% to 320% compared with those on the Linux stack.