The BigDawg monitoring framework

Chen, Peinan; Gadepally, Vijay

doi:10.1109/hpec.2016.7761642

Cited by 12 publications

(5 citation statements)

References 9 publications

(2 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[43] divides existing solutions for multi-model data management into four groups: federated systems, polyglot systems, multi-store systems, and polystore systems. A polystore system is a database system with many heterogeneous data stores and various query interfaces, according to [44]. In our solution, we choose to deploy the target data warehouse on the Spark SQL hybrid polystore whose architecture is shown in figure 5, Spark SQL polystore architecture, based on [12], p.29.…”

Section: Polystore Deploymentmentioning

confidence: 99%

Towards a Scalable and Efficient ETL

Gueddoudj,

Chikh

2023

IJCDS

View full text Add to dashboard Cite

Extract, transform, and load (ETL) processes are crucial for building repositories of data from a variety of self-contained sources. Despite their complexity and cost, ETL processes have demonstrated some maturity for traditional, XML, and graph data sources. However the main challenge for ETL processes is double: (1) they do not scale when brought down to managing large and highly varied data sources, involving web-data. (2) the deployment of the target data warehouse in a polystore. The paper reviews various research efforts along this line of research. The paper then proposes a conceptual modeling of these processes using BPMN (Business Process Modeling Notation). These processes are automatically converted to scripts to be implemented within Spark framework. The solution is packaged according a new distributed architecture (Open ETL) that supports both batch and stream processing. To make our new approach more concrete and evaluable, a real case study using the LUBM benchmark, which involves heterogeneous data sources is considered.

show abstract

Section: Polystore Deploymentmentioning

confidence: 99%

Towards a Scalable and Efficient ETL

Gueddoudj,

Chikh

2023

IJCDS

View full text Add to dashboard Cite

show abstract

“…The choice of the minimum instead of the sample mean or other statistics, responds to 1 https://dcor.readthedocs.io/en/latest/auto_examples/index.html 2 https://pypi.org/project/dcor 3 https://anaconda.org/conda-forge/dcor the objective of discarding the effects of other running processes (see, e.g. [37] for further details).…”

Section: Performancementioning

confidence: 99%

dcor: Distance correlation and energy statistics in Python

Ramos-Carreño¹,

Torrecilla²

2023

SoftwareX

View full text Add to dashboard Cite

“…The Monitor [5] records performance information for past queries, which is then used to aid the planner in selecting query plans for future similar queries. It uses a signature-based scheme to determine how similar two queries are.…”

Section: Monitormentioning

confidence: 99%

Database engine integration and performance analysis of the BigDAWG polystore system

Katherine¹,

Gadepally²

2017

2017 IEEE High Performance Extreme Computing Conference (HPEC)

Self Cite

View full text Add to dashboard Cite

The BigDAWG polystore database system aims to address workloads dealing with large, heterogeneous datasets. The need for such a system is motivated by an increase in Big Data applications dealing with disparate types of data, from large scale analytics to realtime data streams to text-based records, each suited for different storage engines. These applications often perform cross-engine queries on correlated data, resulting in complex query planning, data migration, and execution. One such application is a medical application built by the Intel Science and Technology Center (ISTC) on data collected from an intensive care unit (ICU). This thesis presents work done to add support for two commonly used database engines, Vertica and MySQL, to the BigDAWG system, as well as results and analysis from performance evaluation of the system using the TPC-H benchmark.

show abstract

The BigDawg monitoring framework

Cited by 12 publications

References 9 publications

Towards a Scalable and Efficient ETL

Towards a Scalable and Efficient ETL

dcor: Distance correlation and energy statistics in Python

Database engine integration and performance analysis of the BigDAWG polystore system

Contact Info

Product

Resources

About