RDMA as low-latency interconnect for scaled-out software router

Abstract

We evaluate and propose RoCE (RDMA over Converged Ethernet) as an alternative, low-latency back-plane for horizontally scaled software router nodes. By exploring combinations of specific design choices and system parameters in developing internal fabric for software routers, we select a set of pa- rameters and packet I/O APIs that yields the lowest latency and highest throughput. Using the optimal settings derived, we measure and compare end-to-end latency and throughput of a RoCE interconnect against Ethernet using a high-performance userspace network driver (Intel DPDK). Our comparison shows that RoCE keeps lower latency in all packet size ranges while it has throughput penalties for network workloads (e.g., small packet sizes). To mitigate throughput penalties imposed by guaranteeing low latency, we suggest a hardware-assisted, batched forwarding scheme based on scatter/gather prim- itives provided by RoCE NICs. When forwarding ingress network packets, our scheme achieves higher or comparable throughput versus Ethernet in at the cost of several microseconds of latency, effectively reducing internal fabric overheads.