Device lending in PCI express networks

Abstract

The challenge of scaling IO performance of multimedia systems to demands of their users has attracted much research. A lot of effort has gone into development of distributed systems that add little latency and computing overhead. For machines in PCI Express (PCIe) clusters, we propose Device Lending as a novel solution which works at a system level. Device Lending achieves low latency and extremely low computing overhead without requiring any application-specific distribution mechanisms. For applications, the remote IO resource appears local. In fact, even the drivers of the operating system remain unaware that hardware resources are located in remote machines. By enabling machines in a PCIe cluster to lend a wide variety of hardware, cluster machines can get temporary access to a pool of IO resources. Network cards, FPGAs, SSDs, and even GPUs can easily be shared among computers. Our proposed solution, Device Lending, works transparently without requiring any modifications to drivers, operating systems or software applications. CCS Concepts •Computer systems organization → Distributed architectures; •Software and its engineering → Distributed systems organizing principles;

Flexible device compositions and dynamic resource sharing in PCIe interconnected clusters using Device Lending

“…Device Lending is a mechanism for decoupling devices from the hosts they physically reside in. Using hardware memory mappings, we facilitate the use of remote hardware resources without adding any software overhead [1,2,45]. We have extended our original Device Lending with KVM support for peer-to-peer transfers between multiple devices passed through to a VM.…”

“…SR-IOV-capable devices allow a single physical device to act as multiple virtual devices, allowing a hypervisor to map the same device to several VMs. 1…”

“…In order to make such mappings transparent to both devices and their drivers, we have previously implemented Device Lending [1] for an unmodified Linux kernel using Dolphin's NTB hardware and driver. Our implementation is composed of two parts, namely a ''lender'', allowing a remote unit to use its device, and the ''borrower'' using the device.…”

“…Our original implementation, as described in [1], did not account for peer-to-peer access when borrowing multiple devices from different lenders. As the borrowing system is not aware that the devices reside in different systems, we need a mechanism to resolve I/O addresses to other borrowed devices, in order to fully achieve device-to-device data transfers.…”

“…For clusters of machines interconnected with PCI Express (PCIe), we propose a different strategy to efficient resource sharing called Device Lending [1,2]. In these clusters, I/O devices and interconnection technology are attached to the same PCIe fabric.…”