Speeding ETL Processing in Data Warehouses Using High-Performance Joins for Changed Data Capture (CDC)

Tank, Darshan; Ganatra, Amit; Kosta, Y.P.; Bhensdadia, C.K.

doi:10.1109/artcom.2010.63

Cited by 16 publications

(8 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The other two open-source tools, Airbyte and Meltano, were compared in [5], where it was found, after an evaluation of data ingestion tools, that Airbyte is more suitable for the problem of data pipeline design for audit analytics. Furthermore, CDC is a process that identifies and captures only changes to data in a database, thereby increasing its efficiency and then delivering those changes in real time to a downstream process or system [6]. The following section provides more detail about CDC.…”

Section: Functionalitymentioning

confidence: 99%

Building Advanced Web Applications Using Data Ingestion and Data Processing Tools

Šprem,

Tomažin,

Matečić

et al. 2024

Electronics

View full text Add to dashboard Cite

Today, advanced websites serve as robust data repositories that constantly collect various user-centered information and prepare it for subsequent processing. The data collected can include a wide range of important information from email addresses, usernames, and passwords to demographic information such as age, gender, and geographic location. User behavior metrics are also collected, including browsing history, click patterns, and time spent on pages, as well as different preferences like product selection, language preferences, and individual settings. Interactions, device information, transaction history, authentication data, communication logs, and various analytics and metrics contribute to the comprehensive range of user-centric information collected by websites. A method to systematically ingest and transfer such differently structured information to a central message broker is thoroughly described. In this context, a novel tool—Dataphos Publisher—for the creation of ready-to-digest data packages is presented. Data acquired from the message broker are employed for data quality analysis, storage, conversion, and downstream processing. A brief overview of the commonly used and freely available tools for data ingestion and processing is also provided.

show abstract

Section: Functionalitymentioning

confidence: 99%

Building Advanced Web Applications Using Data Ingestion and Data Processing Tools

Šprem,

Tomažin,

Matečić

et al. 2024

Electronics

View full text Add to dashboard Cite

show abstract

“…Ada beberapa metode untuk melakukan real-time CDC, seperti log-scanning (mencatat hasil pengubahan), trigger (menggunakan trigger dalam tabel), dan timestamped (memberi informasi waktu pada rekaman) [4].…”

Section: Real-time Change Data Captureunclassified

“…ETL adalah proses dalam data warehousing yang menangani pengumpulan data dari sistem sumber dan menempatkannya pada sistem tujuan [4].…”

Section: E Extract-transform-loadunclassified

“…Implementasi dari ETL tools bisa bermacam-macam, di dalam RDBMS terdapat tools seperty trigger, query, view, ataupun stored procedure yang bisa digunakan sebagai ETL tools. Dengan adanya ETL tools, maka aplikasi atau pemakai dari luar RDBMS tidak perlu untuk melakukan pemrosesan pada entitas secara langsung [4]. Aplikasi atau pemakai luar cukup menjalankan operasi dari ETL tools tanpa harus mengetahui langkah-langkah proses yang ada di dalamnya.…”

Section: F Etl Toolsunclassified

“…Dalam metode ini, tidak mungkin untuk melakukan pendeteksian rekaman yang dihapus, karena informasi waktu tersebut akan terhapus bersama dengan rekamannya [4].…”

unclassified

See 2 more Smart Citations

Modifikasi Metode Timestamped Change Data Capture pada RDBMS (Relational Database Management System)

Suprayogi¹,

Dachlan²,

Aswin³

2016

Jurnal Nasional Teknik Elektro dan Teknologi Informasi (JNTETI)

View full text Add to dashboard Cite

Abstract-A normalization process must be applied in the data structure in database designing process in RDBMS, in order to eliminate data anomaly and redundancy. This procedure increases access performance as well as reduces unnecessary storage space. It transforms a single entity into multiple entities, normalization process is often so-called decomposition. A problem occurs, however, after a normalized database is being implemented. When a new data recorded there is nothing wrong with its retrieval, while the retrieval process goes into the last data updated in the case of updating activity. Moreover, when a deletion occurred, the data can never be retrieved. Timestamped CDC method offers a way of tracking data change. Basically, timestamped CDC uses historical record to include timestamp everytime change occurs. Tracking updated record is simply done by the previous research of timestamped CDC, while keeping track on deleted data is impossible since the data vanished within the deletion process. This paper modifies the timestamped CDC method so that data deletion tracking can be achieved as well as data updating. Furthermore, this research also makes several testing procedures to find out performance of the modified timestamped CDC structures.Intisari-Proses normalisasi harus dilakukan pada struktur data dalam perancangan basis data pada RDBMS. Proses normalisasi ini akan membersihkan basis data dari anomali dan redundansi, yang pada akhirnya bisa meningkatkan performa dan mengurangi kebutuhan ruang penyimpanan yang tidak diperlukan. Pada proses normalisasi, terjadi pemisahan satu entitas menjadi banyak entitas yang disebut dengan dekomposisi. Tetapi dengan adanya normalisasi, terjadi masalah pada proses pengubahan dan penghapusan data, yang mana proses tersebut tidak bisa dilacak kembali karena tertumpuk data akhir ataupun terhapus. Timestamped CDC merupakan sebuah cara untuk melacak hasil pengubahan data tersebut. Pada dasarnya, timestamped CDC menggunakan rekaman runtutan yang menyertakan jejak waktu ditambahkannya atau diubahnya data tersebut. Pada penelitian sebelumnya, telah berhasil dibuat struktur timestamped CDC yang mampu melacak hasil pengubahan data, tetapi masih belum berhasil melacak penghapusan data pada waktu dilakukan penghapusan karena rekaman yang bersangkutan tersebut ikut terhapus. Makalah ini memodifikasi metode timestamped CDC sehingga mampu melacak penghapusan data. Selain itu makalah ini juga menguji performa struktur modifikasi timestamped CDC yang dibuat.Kata Kunci-Basis Data, Change Data Capture, Timestamp, Normalisasi, Entitas Runtutan, RDBMS.I. PENDAHULUAN Struktur penyimpanan data pada basis data relasional (Relational Database Management System -RDBMS) yang ada saat ini bergantung pada relasi antar entitas, yang seringkali disebut dengan relationship atau join. RDBMS menyimpan data pada entitas-entitas yang terpisah, berbeda dengan sistem penyimpanan yang menaruh semua data pada entitas tunggal, dengna tujuan meningkatkan kecepatan akses dan fleksibilitas.Pada basis data relasi...

show abstract

Real Time Data Warehouse Updates Through Extraction-Transformation-Loading Process Using Change Data Capture Method

Thulasiram¹,

Ramaiah²

2020

Second International Conference on Computer Networks and Communication Technologies

View full text Add to dashboard Cite

Speeding ETL Processing in Data Warehouses Using High-Performance Joins for Changed Data Capture (CDC)

Cited by 16 publications

References 5 publications

Building Advanced Web Applications Using Data Ingestion and Data Processing Tools

Building Advanced Web Applications Using Data Ingestion and Data Processing Tools

Modifikasi Metode Timestamped Change Data Capture pada RDBMS (Relational Database Management System)

Real Time Data Warehouse Updates Through Extraction-Transformation-Loading Process Using Change Data Capture Method

Contact Info

Product

Resources

About