High performance RDMA-based design of HDFS over InfiniBand

Islam, Nusrat Sharmin; Rahman, M. W.; Jose, Jithin; Rajachandrasekar, Raghunath; Wang, H.; Subramoni, Hari; Murthy, Chet; Panda, Dhabaleswar K.

doi:10.1109/sc.2012.65

Cited by 110 publications

(58 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A horizontal data movement request of a block could trigger a vertical data movement because a block may not be resident in memory. Optimizing the performance of horizontal data movement has been the subject of multiple studies [29,17,20], in which hardware acceleration such as RDMA is used to reduce the communication cost. The benefit of these techniques is less profound on HPC systems with network-based storage [26] because the performance is dominated by vertical data movement.…”

Section: Data Movement In Sparkmentioning

confidence: 99%

“…What HPC specific features can we exploit to boost Spark performance? Previous work optimizing data analytics frameworks on HPC systems [20,17] proposes moving away from the client-server distributed paradigm and exploiting the global file name space already available or Remote Direct Memory Access (RDMA) functionality. Upon introduction to HPC systems, we are interesting in evaluating the potential for performance improvement of adopting such techniques into Spark.…”

Section: System Architecture and Data Movementmentioning

confidence: 99%

“…Optimizing data movement in Map-Reduce frameworks has been the subject of numerous recent studies [29,17,20,11]. Hadoop introduced an interface for pluggable custom shu✏e [15,29] for system specific optimizations.…”

Section: Related Workmentioning

confidence: 99%

“…Optimizing the communication between compute nodes (horizontal data movement) has been tackled through RDMAbased mechanisms [29,17,20]. In these studies, optimized RDMA shows its best benefit when the data is resident in memory.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Scaling Spark on HPC Systems

Chaimov

Malony

Canon

et al. 2016

Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

We report our experiences porting Spark to large production HPC systems. While Spark performance in a data center installation (with local disks) is dominated by the network, our results show that file system metadata access latency can dominate in a HPC installation using Lustre: it determines single node performance up to 4⇥ slower than a typical workstation. We evaluate a combination of software techniques and hardware configurations designed to address this problem. For example, on the software side we develop a file pooling layer able to improve per node performance up to 2.8⇥. On the hardware side we evaluate a system with a large NVRAM bu↵er between compute nodes and the backend Lustre file system: this improves scaling at the expense of per-node performance. Overall, our results indicate that scalability is currently limited to O(10 2 ) cores in a HPC installation with Lustre and default Spark. After careful configuration combined with our pooling we can scale up to O(10 4 ). As our analysis indicates, it is feasible to observe much higher scalability in the near future.

show abstract

Section: Data Movement In Sparkmentioning

confidence: 99%

Section: System Architecture and Data Movementmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Scaling Spark on HPC Systems

Chaimov

Malony

Canon

et al. 2016

Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

show abstract

“…However, due to their complexity in hardware implementation and non-transparency to the remote side, send/receive operations do not perform as well as RDMA operations in current InfiniBand platforms. Thus these designs have not achieved the best performance for small data messages and control messages [8,9]. …”

Section: The Architecture Of Infiniband Rdmamentioning

confidence: 99%

Distributed Database Integrated Transaction Processing Technology Research

2017

JWSA

View full text Add to dashboard Cite

Abstract. In this paper, we propose one comprehensive distributed database transaction method for InfiniBand transaction integrity in the distributed heterogeneous environment caused by data and application expansion. The method first integrates the functions of the standard online transaction engine and then uses the optimistic concurrency control technique to deal with the online memory allocation problem. On the one hand, it reduces the delay time of the online transaction and on the other hand, allocates the computing cost of the transaction to the server and the client. In the existing enterprise network environment using YCSB as a benchmark test shows that the different keys under the method of response speed are the fastest. The result shows that this kind of hybrid processing scheme can make full use of the advantages of remote direct memory access and fast network, which can further improve the consistency and reliability of the distributed database system.

show abstract

Twister2: Design of a big data toolkit

Kamburugamuve

Govindarajan

Wickramasinghe

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

Data-driven applications are essential to handle the ever-increasing volume, velocity, and veracity of data generated by sources such as the Web and Internet of Things (IoT) devices. Simultaneously, an event-driven computational paradigm is emerging as the core of modern systems designed for database queries, data analytics, and on-demand applications. Modern big data processing runtimes and asynchronous many task (AMT) systems from high performance computing (HPC) community have adopted dataflow event-driven model. The services are increasingly moving to an event-driven model in the form of Function as a Service (FaaS) to compose services. An event-driven runtime designed for data processing consists of well-understood components such as communication, scheduling, and fault tolerance. Different design choices adopted by these components determine the type of applications a system can support efficiently. We find that modern systems are limited to specific sets of applications because they have been designed with fixed choices that cannot be changed easily. In this paper, we present a loosely coupled component-based design of a big data toolkit where each component can have different implementations to support various applications. Such a polymorphic design would allow services and data analytics to be integrated seamlessly and expand from edge to cloud to HPC environments.

show abstract

High performance RDMA-based design of HDFS over InfiniBand

Cited by 110 publications

References 17 publications

Scaling Spark on HPC Systems

Scaling Spark on HPC Systems

Distributed Database Integrated Transaction Processing Technology Research

Twister2: Design of a big data toolkit

Contact Info

Product

Resources

About