Extracting deltas from column oriented NoSQL databases for different incremental applications and diverse data targets

Hu, Yong; Deßloch, Stefan

doi:10.1016/j.datak.2014.07.002

Cited by 12 publications

(6 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Of the 8 articles, 7 described various methods for incremental updates, particularly focusing on change data capture (CDC). All describe different categories of CDC, like timestamp-based, audit column-based, trigger-based, log-based, application programming interface-based, and data-based snapshots [16,18,[20][21][22][23][24]: (1) Lynch et al [16] and (2) Lenert et al [18] focused on triggers; (3) Wen [20] focused on timestamps and triggers; (4) Thulasiram and Ramaiah [21] and (5) Sun [22] focused on timestamps; (6) Hu and Dessloch [23] focused on timestamps, audit columns, logs, triggers, and snapshots; and (7) Wei Du and Zou [24] focused on snapshots and MapReduce.…”

Section: Literature Reviewmentioning

confidence: 99%

An Extract-Transform-Load Process Design for the Incremental Loading of German Real-World Data Based on FHIR and OMOP CDM: Algorithm Development and Validation

Henke¹,

Peng²,

Reinecke³

et al. 2023

JMIR Med Inform

View full text Add to dashboard Cite

Background In the Medical Informatics in Research and Care in University Medicine (MIRACUM) consortium, an IT-based clinical trial recruitment support system was developed based on the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM). Currently, OMOP CDM is populated with German Fast Healthcare Interoperability Resources (FHIR) using an Extract-Transform-Load (ETL) process, which was designed as a bulk load. However, the computational effort that comes with an everyday full load is not efficient for daily recruitment. Objective The aim of this study is to extend our existing ETL process with the option of incremental loading to efficiently support daily updated data. Methods Based on our existing bulk ETL process, we performed an analysis to determine the requirements of incremental loading. Furthermore, a literature review was conducted to identify adaptable approaches. Based on this, we implemented three methods to integrate incremental loading into our ETL process. Lastly, a test suite was defined to evaluate the incremental loading for data correctness and performance compared to bulk loading. Results The resulting ETL process supports bulk and incremental loading. Performance tests show that the incremental load took 87.5% less execution time than the bulk load (2.12 min compared to 17.07 min) related to changes of 1 day, while no data differences occurred in OMOP CDM. Conclusions Since incremental loading is more efficient than a daily bulk load and both loading options result in the same amount of data, we recommend using bulk load for an initial load and switching to incremental load for daily updates. The resulting incremental ETL logic can be applied internationally since it is not restricted to German FHIR profiles.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

An Extract-Transform-Load Process Design for the Incremental Loading of German Real-World Data Based on FHIR and OMOP CDM: Algorithm Development and Validation

Henke¹,

Peng²,

Reinecke³

et al. 2023

JMIR Med Inform

View full text Add to dashboard Cite

show abstract

“…23,24 The replication engine scans 2 snapshots to find differential. 23,24 The replication engine scans 2 snapshots to find differential.…”

Section: Data Replication Of Scheme-free Datamentioning

confidence: 99%

“…The second approach is snapshot differential, which is often necessary to infer differential modifications by periodically comparing snapshots of data from the source. Normally, it contains two steps: taking snapshots of data source at different time points and comparing the snapshots to acquire the data changes . The replication engine scans 2 snapshots to find differential.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Stream‐based live data replication approach of in‐memory cache

Yang

2017

Concurrency and Computation

View full text Add to dashboard Cite

Replication is a method to keep the consistency of source data and target data. In our previous work of access-aware in-memory data cache middleware for relational databases, the data are easy to be lost in case that power cuts off. Therefore, we investigate a live data replication approach from in-memory data cache to versioning repository in this paper. This method attempts to recover the in-memory data cache from the versioning repository in failure of access-aware in-memory data cache middleware. Although the replication is not a new problem, the state of art of the replication in the context of document stores is not mature. In our paper, we propose a live data replication approach of in-memory document stores using stream processing framework.First, we introduce cell state model to describe the replication process. To infinitely look back to any revision, we enable our proposed cell state model to support copy-modify-merge model to manage the changed data revisions subsequently. Finally, experimental results show that this approach is more suitable for the replication of continuous in-stream changed data compared with MapReduce-based batch replication.

show abstract

“…Column-oriented databases store columns of data separately, unlike traditional storage where data are stored in the form of complete records. Data reads and retrieval of attributes in such systems is quite fast and comparatively less expensive, as only the relevant column is accessed and concurrent process execution is performed for each column Hu and Dessloch, 2014). Column-oriented databases are highly scalable and eventually consistent.…”

Section: Fig 4 Examples Of Data Models: (A) Key-value; (B) Column-ormentioning

confidence: 99%

Big data storage technologies: a survey

Siddiqa

Karim

Gani

2017

Frontiers Inf Technol Electronic Eng

View full text Add to dashboard Cite

Abstract:There is a great thrust in industry toward the development of more feasible and viable tools for storing fast-growing volume, velocity, and diversity of data, termed 'big data'. The structural shift of the storage mechanism from traditional data management systems to NoSQL technology is due to the intention of fulfilling big data storage requirements. However, the available big data storage technologies are inefficient to provide consistent, scalable, and available solutions for continuously growing heterogeneous data. Storage is the preliminary process of big data analytics for real-world applications such as scientific experiments, healthcare, social networks, and e-business. So far, Amazon, Google, and Apache are some of the industry standards in providing big data storage solutions, yet the literature does not report an in-depth survey of storage technologies available for big data, investigating the performance and magnitude gains of these technologies. The primary objective of this paper is to conduct a comprehensive investigation of state-of-the-art storage technologies available for big data. A well-defined taxonomy of big data storage technologies is presented to assist data analysts and researchers in understanding and selecting a storage mechanism that better fits their needs. To evaluate the performance of different storage architectures, we compare and analyze the existing approaches using Brewer's CAP theorem. The significance and applications of storage technologies and support to other categories are discussed. Several future research challenges are highlighted with the intention to expedite the deployment of a reliable and scalable storage system.

show abstract

Extracting deltas from column oriented NoSQL databases for different incremental applications and diverse data targets

Cited by 12 publications

References 2 publications

An Extract-Transform-Load Process Design for the Incremental Loading of German Real-World Data Based on FHIR and OMOP CDM: Algorithm Development and Validation

An Extract-Transform-Load Process Design for the Incremental Loading of German Real-World Data Based on FHIR and OMOP CDM: Algorithm Development and Validation

Stream‐based live data replication approach of in‐memory cache

Big data storage technologies: a survey

Contact Info

Product

Resources

About