DALiuGE: A Graph Execution Framework for Harnessing the Astronomical Data Deluge

Wu, Chen; Tobar, Rodrigo; Vinsen, Kevin; Wicenec, A.; Pallot, D.; Lao, Baoqiang; Wang, Ruonan; An, Tao; Boulton, Mark; Cooper, Ian; Dodson, Richard; Dolensky, M.; Mei, Ying; Wang, Feng

doi:10.48550/arxiv.1702.07617

Cited by 1 publication

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The octagon shape in the logical graph (Figure 3) represents an MPIApp, which has a Drop property of num of procs, whose value is set to 5 in this example. During the graph translation stage [7], five MPIApp Drops are generated. One of them represents the Master process, and the other four of them represent the Slave processes.…”

Section: Coarse-grained Porting: Drop Wrapping Of Mpimentioning

confidence: 99%

“…The Data Activated Liu Graph Engine (DALiuGE), a generic, data-driven execution framework, is developed for processing large astronomical datasets [7]. Drop is basic idea of the DALiuGE and is representation of applications and data, AppDrop and DataDrop respectively.…”

Section: Introductionmentioning

confidence: 99%

“…The DALiuGE includes an interface for expressing complex data-reduction pipelines consisting of data sets, algorithmic components and a run-time implementation to execute such pipelines on distributed resources. The full SKA scale testing of the DALiuGE proved that the DALiuGE owns a series of distinctive advantages [7]. However, many system architecture experts urgently demand to test the DALiuGE in more detail in order to have a more reasonable and fair analysis of the applicability for the final SKA data processing.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Empirical Evaluation On the Applicability of the DALiuGE Execution Framework

Mei¹,

Wei²,

Wang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The Square Kilometre Array (SKA) project is an international cooperation project to build the largest radio telescope worldwide. Data processing is one of the biggest challenges of building the SKA telescope. As a distributed execution framework, the Data Activated Liu Graph Engine (DALiuGE) was proposed to be one of the candidates for addressing the massive data of the SKA. DALiuGE has many distinctive features, but its actual ability to handle scientific data is still not evident. In this paper, we perform an objective evaluation of the usability of DALiuGE concerning the execution performance, developer workload, and implementation difficulty of porting the SAGECal to DALiuGE. The evaluation results showed that the DALiuGE enables fast integration of astronomical software, but there are significant differences in the efficiency of different parallel granularities. Even with the deep optimization of the program, there is still a gap between the current DALiuGE and the traditional MPI in execution performance. Therefore, we come to a preliminary conclusion that the DALiuGE has no performance advantage in batch processing of massive data, while it may be more suitable for application scenarios with more customized computational tasks, such as SKA science regional centers.

show abstract

Section: Coarse-grained Porting: Drop Wrapping Of Mpimentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation