XenLoop: a transparent high performance inter-VM network loopback

Abstract

Advances in virtualization technology have focused mainly on strengthening the isolation barrier between virtual machines (VMs) that are co-resident within a single physical machine. At the same time, a large category of communication intensive distributed applications and software components exist, such as web services, high performance grid applications, transaction processing, and graphics rendering, that often wish to communicate across this isolation barrier with other endpoints on co-resident VMs. State of the art inter-VM communication mechanisms do not adequately address the requirements of such applications. TCP/UDP based network communication tends to perform poorly when used between co-resident VMs, but has the advantage of being transparent to user applications. Other solutions exploit inter-domain shared memory mechanisms to improve communication latency and bandwidth, but require applications or user libraries to be rewritten against customized APIs-something not practical for a large majority of distributed applications. In this paper, we present the design and implementation of a fully transparent and high performance inter-VM network loopback channel, called XenLoop, in the Xen virtual machine environment. XenLoop does not sacrifice user-level transparency and yet achieves high communication performance between coresident guest VMs. XenLoop intercepts outgoing network packets beneath the network layer and shepherds the packets destined to co-resident VMs through a high-speed inter-VM shared memory channel that bypasses the virtualized network interface. Guest VMs using XenLoop can migrate transparently across machines without disrupting ongoing network communications, and seamlessly switch between the standard network path and the XenLoop channel. In our evaluation using a number of unmodified benchmarks, we observe that XenLoop can reduce the inter-VM round trip latency by up to a factor of 5 and increase bandwidth by a up to a factor of 6.

Prioritizing local inter-domain communication in Xen

“…Note that the applications running atop the guest domains are not aware of this para-virtualization, so no modification to them is needed. Another approach for IDC is to use shared memory to exchange data between domains [6]- [9], thus avoiding the involvement of Domain 0 to obtain better performance. However, the shared memory approach requires modifications to the guest domain besides the well supported Xen patch, and may even need to modify the applications as well (a detailed discussion is deferred to Section VII).…”

“…Other research [6]- [9] concentrates on establishing a shared memory data channel between two guest domains via the Xen VMM. As a result, the time spent in the manager domain is avoided, and the performance is improved close to that of native socket communication.…”

“…XenSocket [8] instead maintains another new AF XEN network protocol stack to support IDC. XenLoop [9] utilizes the existing Linux netfilter mechanism to add a XenLoop module between the IP layer and the netfront driver: if the packet is for IDC, the XenLoop module directly communicates with another XenLoop module in the corresponding domain. A discover kernel module in the manager domain is also needed to establish the data channel.…”

Analyzing and Improving MPI Communication Performance in Overcommitted Virtualized Systems

“…The techniques proposed by these studies include VMM-bypassing [12], using XenLoop [13] alike techniques to replace the loopback network with the shared memory [14], and using a dedicated CPU core to handle all communications [15]. However, none of these studies addressed the overcommitted scenarios.…”

Performance Analysis of Large Receive Offload in a Xen Virtualized System

“…al. proposed XenLoop which is a inter-domain communication channel [15]. It bypasses the regular virtual network interfaces in Figure 1, yet provides transparency in the sense that the application programs can still use standard TCP/IP APIs.…”