Early experiences with live migration of SR-IOV enabled InfiniBand

Guay, Wei Lin; Reinemo, Sven-Arne; Johnsen, Bjorn Dag; Yen, C.Y.; Skeie, Tor; Lysne, Olav; Torudbakken, Ola

doi:10.1016/j.jpdc.2015.01.004

Cited by 9 publications

(18 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The vGUID of the SR-IOV VF is migrated together with the VM, but the LID changes. Guay et al 16 In general, when a lossless network is reconfigured, routes have to be recalculated and distributed to all switches while avoiding deadlocks. However, additional signaling and overhead towards the SM is required due to the LID change.…”

Section: Related Workmentioning

confidence: 99%

Efficient routing and reconfiguration in virtualized HPC environments with vSwitch‐enabled lossless networks

Tasoulas

Zahid

Gran

et al. 2018

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

To meet the demands of communication-intensive workloads in the cloud, virtual machines (VMs) should utilize low overhead network communication paradigms. In general, such paradigms enable VMs to directly communicate with the hardware by means of a passthrough technology like Single-Root I/O Virtualization (SR-IOV). However, when passthrough-based virtualization is coupled with lossless interconnection networks, live migrations introduce scalability challenges due to the substantial network reconfiguration overhead. With these challenges in mind, we proposed a virtual switch (vSwitch) SR-IOV architecture for InfiniBand in our previous work titled "Towards the InfiniBand SR-IOV vSwitch Architecture". In this paper, we first suggest solutions to rectify the space-domain scalability issues that are present in vSwitch-enabled subnets as a result of the VMs using dedicated layer-two addresses. Then, we discuss routing strategies for virtualized environments using vSwitches and present a routing algorithm for Fat-Trees. We also present a reconfiguration method that minimizes imposed reconfiguration overhead on Fat-Trees.We perform an extensive evaluation of our prototype algorithms, and as vSwitch-enabled hardware does not yet exist, we deduce from empirical observations by emulating vSwitches with existing hardware, as well as large-scale simulations. Our results show significant reduction in the reconfiguration times as route recalculations can be eliminated, and for certain scenarios, the number of reconfiguration subnet management packets sent to switches is reduced from several hundred thousand down to a single one without degrading the routing quality.

show abstract

Section: Related Workmentioning

confidence: 99%

Efficient routing and reconfiguration in virtualized HPC environments with vSwitch‐enabled lossless networks

Tasoulas

Zahid

Gran

et al. 2018

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…As a consequence, the performance of a fully virtualized cloud is not on par with that of physical, non-virtualized systems. The performance gaps are even larger in the HPC domain where HPC specific technologies such as lossless interconnection networks are common, but not yet ready to be fully virtualized without sacrificing cloud flexibility [21]. Correspondingly, cloud providers have not adopted true HPC clouds yet that can perform on par with traditional HPC systems [22,23], a necessary move in order to accommodate high performance workloads efficiently in the cloud, and therefore bring HPC and efficient Big Data analytics access to an extended audience.…”

Section: Server or Hardware Virtualizationmentioning

confidence: 99%

“…When using software emulated IOV methods, the network interfaces are virtual so their internal state is stored into the memory and gets copied as well, thus, the downtime is in the order of a few milliseconds [60]. In the case of direct device assignment and SR-IOV VFs, the internal state of the network interface cannot be copied because it is stored into the hardware that is not able to move [31,120]. The SR-IOV Virtual Function (VF) assigned to the VM will first need to be detached, the live migration will run, and a new VF will be attached at the destination.…”

Section: Infiniband Sr-iov and Live Migrations In The Cloudmentioning

confidence: 99%

“…Ranadive considered those aspects, and developed tools and methods for monitoring and latency-aware scheduling of the VMs. Nonetheless, the closest related work to ours, that focuses in scalability and reconfiguration of lossless networks while live migrations are part of a cloud data center, is that of Guay [38,120]. In [38], Guay et al present a signaling mechanism that takes into account that all three IB addresses change when a VM is migrating, and notify the peers of the migrated VM to update their cached path information.…”

Section: Infiniband Sr-iov and Live Migrations In The Cloudmentioning

confidence: 99%

“…Then, in Paper II and Paper V we propose the IB SR-IOV vSwitch architecture that allows a VM to be migrated while carrying to the destination all of its associated addresses. In [120], Guay et al also present a prototype that would allow VMs with IB SR-IOV VFs to migrate transparently 11 . This work is concentrating to IB internals, and the presented prototype migrates together with the VM the necessary hardware state of an IB VF so that the migration can happen in an application transparent way.…”

Section: Infiniband Sr-iov and Live Migrations In The Cloudmentioning

confidence: 99%

See 2 more Smart Citations

A Novel Query Caching Scheme for Dynamic InfiniBand Subnets

Tasoulas

Gran

Johnsen³

et al. 2015

2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Self Cite

View full text Add to dashboard Cite

The advent of the Internet of Things, sensor and social networks, to mention just a few examples, all contribute towards the solid establishment of the Big Data era. High Performance Computing (HPC) becomes necessary for the efficient processing of the massive amounts of data our society generates, and cloud computing is a critical component to deliver this processing power to a broader audience that cannot afford to acquire and maintain such complex computing systems themselves. However, HPC specific technology and performance is not yet apt to be delivered efficiently over highly flexible and dynamic environments, as typically are the virtualized cloud infrastructures.In this thesis, we address challenges that arise in high performance dynamic cloud environments, that are equipped with HPC specific technology, in the context of networking and virtualization. We use InfiniBand, a high performance lossless interconnection network as the basis of our research, and first show that lossless networks pose prime challenges when the nature of the infrastructure is very dynamic, i.e. exhibits continuous changes. Then we propose a network I/O virtualization architecture, the InfiniBand vSwitch architecture, that can make lossless network technologies more favorable in the cloud. Moreover, we propose different network reconfiguration methods to enable performance-driven reconfigurations in very large network topologies that are commonly found in data centers. Performance-driven reconfigurations are frequently needed to adapt to unpredictable workload changes resulting from the shared and on-demand nature of a cloud platform, or when cloud providers employ live migration of virtual machines to optimize the resource usage of their infrastructure. Last but not least, we propose a new Quality-of-Service metric, called delay, to capture the directly observable service degradation in consolidated cloud environments. We suggest that the delay can be used as a direct service level agreement metric between cloud providers and cloud tenants. AcknowledgementsFirst, I would like to express my gratitude to my three supervisors; Ernst Gunnar Gran, Tor Skeie and Kyrre Begnum. Without their positiveness, supervision, suggestions and encouragement, my PhD journey would not have been so fruitful and enjoyable during both the good and hard times. Then I would like to thank Bjørn Dag Johnsen from Oracle Norway, our closest collaborate in the ERAC project, the project that mainly funded this thesis, for his enthusiastic attitude and participation in the important discussions that shaped the direction of my work and introduced industrial relevance. My appreciation goes as well to Feroz Zahid for being the most easygoing colleague whom I could imagine sharing office with, and a brilliant and ambitious associate, and Sven-Arne Reinemo for his supervision during my early days.A special thank you must be directed to Hårek Haugerud, Anis Yazidi, Hugo Lewi Hammer, Laurence Marie Anna Habib and the whole NETSYS group at the Oslo and Akershus...

show abstract

Prospects and challenges of virtual machine migration in HPC

Pickartz

Clauss

Breitbart

et al. 2018

Concurrency and Computation

View full text Add to dashboard Cite

Summary The continuous growth of supercomputers is accompanied by increased complexity of the intra‐node level and the interconnection topology. Consequently, the whole software stack ranging from the system software to the applications has to evolve, eg, by means of fault tolerance and support for the rising intra‐node parallelism. Migration techniques are one means to address these challenges. On the one hand, they facilitate the maintenance process by enabling the evacuation of individual nodes during runtime, ie, the implementation of fault avoidance. On the other hand, they enable dynamic load balancing for an improvement of the system's efficiency. However, these prospects come along with certain challenges. On the process level, migration mechanisms have to resolve so‐called residual dependencies to the source node, eg, the communication hardware. On the job level, migrations affect the communication topology, which should be addressed by the communication stack, ie, the optimal communication path between a pair of processes might change after a migration. In this article, we explore migration mechanisms for HPC and discuss their prospects as well as the challenges. Furthermore, we present solutions enabling their efficient usage in this domain. Finally, we evaluate our prototype co‐scheduler leveraging migration for workload optimization.

show abstract

Early experiences with live migration of SR-IOV enabled InfiniBand

Cited by 9 publications

References 21 publications

Efficient routing and reconfiguration in virtualized HPC environments with vSwitch‐enabled lossless networks

Efficient routing and reconfiguration in virtualized HPC environments with vSwitch‐enabled lossless networks

A Novel Query Caching Scheme for Dynamic InfiniBand Subnets

Prospects and challenges of virtual machine migration in HPC

Contact Info

Product

Resources

About