Twister:Net - Communication Library for Big Data Processing in HPC and Cloud Environments

Kamburugamuve, Supun; Wickramasinghe, Pulasthi; Govindarajan, K.; Uyar, Ahmet; Gunduz, Gurhan; Abeykoon, Vibhatha; Fox, Geoffrey

doi:10.1109/cloud.2018.00055

Cited by 12 publications

(18 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Twister2 11,33 is a big data toolkit designed to provide a variety of functionalities to both HPC and dataflow application developers. Twister2: Net 34 is an optimized communication library that contains an MPI-like communication style with TCP-based communication. Application development abstractions are important for creating applications with efficiency.…”

Section: Related Workmentioning

confidence: 99%

“…Considering the dataflow model, specifically for batch data processing, our analysis has been focused on using Apache Spark and Twister2. 11,34,35 Apache Spark is one of the most prominent tools for dataflow frameworks used by many data scientists and big data application developers. 3,10,[38][39][40] In big data application stack, the problem we are trying to optimize comes under the iterative batch applications.…”

Section: Dataflow Model Implementationmentioning

confidence: 99%

See 1 more Smart Citation

Stochastic gradient descent‐based support vector machines training optimization on Big Data and HPC frameworks

Abeykoon

Fox

Kim

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Support vector machines (SVM) is a widely used machine learning algorithm. With the increasing amount of research data nowadays, understanding how to do efficient training is more important than ever. This article discusses the performance optimizations and benchmarks related to providing high-performance support for SVM training. In this research, we have focused on a highly scalable gradient descent-based approach to implementing the core SVM algorithm. In providing a scalable solution, we have designed optimized high-performance computing and dataflow-oriented SVM implementations. A high-performance computing approach means the algorithm is implemented with the bulk synchronous parallel (BSP) model. In addition, we analyzed the language level optimizations and math kernel optimizations on a prominent HPC modeling programming language (C++) and dataflow modeling programming language (Java). In the experiments, we compared the performance of classic HPC models, classic dataflow models, and hybrid models designed on classic HPC and dataflow programming models. Our research illustrates a scientific approach in designing the SVM algorithm at scale in classic HPC, dataflow, and hybrid systems.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Dataflow Model Implementationmentioning

confidence: 99%

Stochastic gradient descent‐based support vector machines training optimization on Big Data and HPC frameworks

Abeykoon

Fox

Kim

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…This is a more general form of an operator, as it can represent SPMD-style operators as well. Twister:Net [40] is one such MPMD-style operator library. Whether it is SPMD or MPMD, we can have eager style operators or dataflow operators.…”

Section: B Spmd and Mpmd Operatorsmentioning

confidence: 99%

HPTMT: Operator-Based Architecture for Scalable High-Performance Data-Intensive Frameworks

Kamburugamuve¹,

Widanage²,

Perera³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Data-intensive applications impact many domains, and their steadily increasing size and complexity demands highperformance, highly usable environments. We integrate a set of ideas developed in various data science and data engineering frameworks. They employ a set of operators on specific data abstractions that include vectors, matrices, tensors, graphs, and tables. Our key concepts are inspired from systems like MPI, HPF (High-Performance Fortran), NumPy, Pandas, Spark, Modin, PyTorch, TensorFlow, RAPIDS(NVIDIA), and OneAPI (Intel). Further, it is crucial to support different languages in everyday use in the Big Data arena, including Python, R, C++, and Java. We note the importance of Apache Arrow and Parquet for enabling language agnostic high performance and interoperability. In this paper, we propose High-Performance Tensors, Matrices and Tables (HPTMT), an operator-based architecture for dataintensive applications, and identify the fundamental principles needed for performance and usability success. We illustrate these principles by a discussion of examples using our software environments, Cylon and Twister2 that embody HPTMT.

show abstract

“…There are diverse big data applications developed on distributed systems. Case studies for face detection, 10 Twister:Net, 11 and Support vector machine (SVM) training optimization 12 are examples. First of all, generally, face detection applications need the processing of a small‐sized and large number of images.…”

Section: Introductionmentioning

confidence: 99%

Metaheuristic task scheduling algorithms for cloud computing environments

Aktan

Bulut

2021

Concurrency and Computation

View full text Add to dashboard Cite

Cloud computing has the advantage of providing flexibility, high-performance, pay-as-you-use, and on-demand service. One of the important research issues in cloud computing is task scheduling. The purpose of scheduling is to assign tasks to available resources while providing optimization on some objectives. Tasks have diversified characteristics, and resources are heterogeneous. These properties make task scheduling an NP-complete problem. In this study, metaheuristic and hybrid metaheuristic algorithms are developed for task scheduling problems in cloud computing environments. We have developed genetic algorithm (GA), differential evolution (DE), and simulated annealing (SA) based metaheuristic algorithms, which are also combined with a greedy approach (GR). In addition to this, we have developed hybrid metaheuristics algorithms, called DE-SA and GA-SA, which are also combined with a greedy approach. The proposed approaches are evaluated in terms of completion time and load balancing of virtual machines. In terms of average completion time, as the number of tasks increases, it has been observed that the DESA algorithm outperforms the solely used DE and SA algorithms. In addition, experiments show that hybrid algorithms improve both the average completion time and the average standard deviation of virtual machine loads for some task groups.

show abstract

Twister:Net - Communication Library for Big Data Processing in HPC and Cloud Environments

Cited by 12 publications

References 18 publications

Stochastic gradient descent‐based support vector machines training optimization on Big Data and HPC frameworks

Stochastic gradient descent‐based support vector machines training optimization on Big Data and HPC frameworks

HPTMT: Operator-Based Architecture for Scalable High-Performance Data-Intensive Frameworks

Metaheuristic task scheduling algorithms for cloud computing environments

Contact Info

Product

Resources

About