Building architectures for data‐intensive science using the ADAGE framework

Abstract

SUMMARYOne of the main activities in data-intensive science is data analysis. Although there are many popular technologies that can assist scientists in various isolated aspects of data analysis, supporting analysis processes in holistic ways that promote system interoperability, integration and automation, as well as scientific reproducibility and efficient data handling, presents many challenges. A common solution to address these challenges is to find efficient ways of integrating various existing technologies together to meet the analysis needs of scientists (which is similar to the idea behind science gateways). We believe that this solution is essentially an exercise in software design; and in many situations, these challenges should be tackled from a software design perspective. Consequently, this paper reviews different architectural design approaches that can be used to address these challenges and proposes a service-oriented framework called the Ad Hoc Data Grid Environment, which consists of an architectural pattern and its associated operational guidelines. The guidelines prescribe a number of activities based on an iterative decomposition approach to produce and evolve software architectures according to constantly changing user needs. The framework is demonstrated on a case study involving analysis processes required for conducting financial event studies.

An open architecture for event-based analytics

“…2 (1 event pattern type involved) -Detecting occurrences of event pattern types No. [1][2][3][4][5][6][7] (Handling data quality issues of dividend events; 7 simple event pattern types involved) -Detecting occurrences of event pattern types No. 8,[10][11][12][13][14] (Calculating earnings; 6 more sophisticated event pattern types involved)…”

“…The open architecture suggested in this paper extends an earlier architecture that was based on the ADAGE framework [6,7]. It provides extended analysis features owing to the ability to integrate complex event processing services to detect patterns of interest from event data.…”

“…As mentioned earlier, the proposed open architecture is an extension of the one presented in [6,7]. In the original architecture, there are three types of services (see Fig.…”

“…Non-technical users struggle to construct workflows that involve complex control-flow operations such as repeating iterations of tasks or defining parallel tasks. The ADAGE Framework [6] gives a set of architectural guidelines based on service-oriented computing principles that restrict the choice of services depending on their role in the analytics pipeline. This idea has been applied in facilitating the definition of event data analysis processes.…”

“…The ideas can be also of value for data scientists, analysts and researchers involved in studying data in particular application domains, such as domain experts involved in financial market analysis as discussed in [6] and [7]. These subject matter experts can work together with computer scientists in defining rules that specify relationships between event occurrences of interest.…”