Twister2: TSet High-Performance Iterative Dataflow

Wickramasinghe, Pulasthi; Kamburugamuve, Supun; Govindarajan, K.; Abeykoon, Vibhatha; Widanage, Chathura; Perera, Niranda; Uyar, Ahmet; Gunduz, Gurhan; Akkas, Selahattin; Fox, Geoffrey

doi:10.1109/hpbdis.2019.8735495

Cited by 17 publications

(24 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The dataflow model is a trend in Big Data applications [24]. There are several flow-based programming frameworks often defining applications as networks of "black box" processes, which exchange data through predefined inputs and outputs.…”

Section: Related Workmentioning

confidence: 99%

“…There are several flow-based programming frameworks often defining applications as networks of "black box" processes, which exchange data through predefined inputs and outputs. Those frameworks use different ways to define a flow: by using a functional-based language, such as Apache Spark and Twister2 [24]; by using a skeleton-based pipelines, such as Ruffus [25] and Cosmos [26]; or even visually, such as RapidMiner [27], Orange [28] and KNIME [29].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Upgrading a high performance computing environment for massive data processing

Ponce

Santos

Meira

et al. 2019

J Internet Serv Appl

View full text Add to dashboard Cite

High-performance computing (HPC) and massive data processing (Big Data) are two trends that are beginning to converge. In that process, aspects of hardware architectures, systems support and programming paradigms are being revisited from both perspectives. This paper presents our experience on this path of convergence with the proposal of a framework that addresses some of the programming issues derived from such integration. Our contribution is the development of an integrated environment that integretes (i) COMPSs, a programming framework for the development and execution of parallel applications for distributed infrastructures; (ii) Lemonade, a data mining and analysis tool; and (iii) HDFS, the most widely used distributed file system for Big Data systems. To validate our framework, we used Lemonade to create COMPSs applications that access data through HDFS, and compared them with equivalent applications built with Spark, a popular Big Data framework. The results show that the HDFS integration benefits COMPSs by simplifying data access and by rearranging data transfer, reducing execution time. The integration with Lemonade facilitates COMPSs's use and may help its popularization in the Data Science community, by providing efficient algorithm implementations for experts from the data domain that want to develop applications with a higher level abstraction.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Upgrading a high performance computing environment for massive data processing

Ponce

Santos

Meira

et al. 2019

J Internet Serv Appl

View full text Add to dashboard Cite

show abstract

“…While dataflow is a prevalent model in many parallel and distributed programming frameworks [4], functional programming is slowly becoming a commom interface. In addition to Big Data frameworks like Spark, Flink and Swift [9], functional interfaces are also being frequently used in other Data Sciences programming tools (e.g., Scikit-Learn 1 and Pandas 2 use it to express their dataflow models).…”

Section: Related Workmentioning

confidence: 99%

“…Convergence between high-performance computing (HPC) and Big Data has become an important research area, driven in part by the need to incorporate high-level libraries, platforms, and algorithms for machine learning and graph processing, and in part by the idea of using Big Data's fine-grained data awareness to increase the productivity of HPC systems [1], [2]. Several proposals of higher-level abstractions have emerged to address the requirements of these two areas in computer systems [3], [4]. Recent frameworks, like COMPSs [3], Twister2 [4], Spark [5] and Flink [6], share a common dataflow programming model, but are still focused on a single area.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Extension of a Task-Based Model to Functional Programming

Ponce

Lezzi

Badía

et al. 2019

2019 31st International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)

View full text Add to dashboard Cite

Recently, efforts have been made to bring together the areas of high-performance computing (HPC) and massive data processing (Big Data). Traditional HPC frameworks, like COMPSs, are mostly task-based, while popular big-data environments, like Spark, are based on functional programming principles. The earlier are know for their good performance for regular, matrix-based computations; on the other hand, for fine-grained, data-parallel workloads, the later has often been considered more successful. In this paper we present our experience with the integration of some dataflow techniques into COMPSs, a task-based framework, in an effort to bring together the best aspects of both worlds. We present our API, called DDF, which provides a new data abstraction that addresses the challenges of integrating Big Data application scenarios into COMPSs. DDF has a functional-based interface, similar to many Data Science tools, that allows us to use dynamic evaluation to adapt the task execution in runtime. Besides the performance optimization it provides, the API facilitates the development of applications by experts in the application domain. In this paper we evaluate DDF's effectiveness by comparing the resulting programs to their original versions in COMPSs and Spark. The results show that DDF can improve COMPSs execution time and even outperform Spark in many use cases.

show abstract

High‐performance iterative dataflow abstractions in Twister2:TSet

Wickramasinghe

Perera

Kamburugamuve

et al. 2020

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Summary The dataflow model is gradually becoming the de facto standard for big data applications. While many popular frameworks are built around this model, very little research has been done on understanding its inner workings, which in turn has led to inefficiencies in existing frameworks. It is important to note that understanding the relationship between dataflow and high performance computing (HPC) building blocks allows us to address and alleviate many of these fundamental inefficiencies by learning from the extensive research literature in the HPC community. In this article, we present TSets, the dataflow abstraction of Twister2, which is a big data framework designed for high‐performance dataflow and iterative computations. We discuss the dataflow model adopted by TSets and the rationale behind implementing iteration handling at the worker level. Finally, we evaluate TSets to show the performance of the framework and the importance of the worker level iteration model.

show abstract

Twister2: TSet High-Performance Iterative Dataflow

Cited by 17 publications

References 14 publications

Upgrading a high performance computing environment for massive data processing

Upgrading a high performance computing environment for massive data processing

Extension of a Task-Based Model to Functional Programming

High‐performance iterative dataflow abstractions in Twister2:TSet

Contact Info

Product

Resources

About