Industrial-strength OLTP using main memory and many cores

Avni, Hillel; Aliev, Alisher; Amor, Oren; Avitzur, Aharon; Бронштейн, И. Е.; Ginot, Eli; Goikhman, Shay; Levy, Eliezer; Levy, Idan; Lu, Fuyang; Mishali, Liran; Mo, Yeqin; Pachter, Nir; Sivov, Dima; Veeraraghavan, Vinoth; Vexler, Vladi; Wang, Lei; Wang, Peng

doi:10.14778/3415478.3415537

Cited by 14 publications

(5 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fig. 1 gives a high-level overview of interactions on a classic database cluster setup of GaussDB [12], [13].…”

Section: A Database Management System (Dbms)mentioning

confidence: 99%

Industry Practice of Coverage-Guided Enterprise-Level DBMS Fuzzing

Wang

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)

View full text Add to dashboard Cite

As an infrastructure for data persistence and analysis, Database Management Systems (DBMSs) are the cornerstones of modern enterprise software. To improve their correctness, the industry has been applying blackbox fuzzing for decades. Recently, the research community achieved impressive fuzzing gains using coverage guidance. However, due to the complexity and distributed nature of enterprise-level DBMSs, seldom are these researches applied to the industry.In this paper, we apply coverage-guided fuzzing to enterpriselevel DBMSs from Huawei and Bloomberg LP. In our practice of testing GaussDB and Comdb2, we found major challenges in all three testing stages. The challenges are collecting precise coverage, optimizing fuzzing performance, and analyzing root causes. In search of a general method to overcome these challenges, we propose RATEL, a coverage-guided fuzzer for enterprise-level DBMSs. With its industry-oriented design, RATEL improves the feedback precision, enhances the robustness of input generation, and performs an on-line investigation on the root cause of bugs. As a result, RATEL outperformed other fuzzers in terms of coverage and bugs. Compared to industrial black box fuzzers SQLsmith and SQLancer, as well as coverage-guided academic fuzzer Squirrel, Ra t e l covered 38.38%, 106.14%, 583.05% more basic blocks than the best results of other three fuzzers in GaussDB, PostgreSQL, and Comdb2, respectively. More importantly, Ra t e l has discovered 32, 42, and 5 unknown bugs in GaussDB, Comdb2, and PostgreSQL.

show abstract

“…Fig. 1 gives a high-level overview of interactions on a classic database cluster setup of GaussDB [12], [13].…”

Section: A Database Management System (Dbms)mentioning

confidence: 99%

Industry Practice of Coverage-Guided Enterprise-Level DBMS Fuzzing

Wang

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)

View full text Add to dashboard Cite

show abstract

“…We have described the idea of multi-engine systems in Section 1. Several production systems already adopted the fast-slow architecture: SQL Server supports memory-optimized tables managed by its Hekaton main-memory engine [21,46]; PostgreSQL supports additional engines through foreign data wrapper [53], which is used by Huawei GaussDB to integrate a main-memory engine [5].…”

Section: Modern Fast-slow Multi-engine Systemsmentioning

confidence: 99%

“…4 This translates into choosing a concurrency control protocol for each engine where a sub-transaction can only commit if its read records are not concurrently modified. A wide range of serializable protocols and systems [5,21,26,38,40,41,43,59,61] exhibit this property: classic two-phase locking avoids clobbered reads by ensuring readers and writers block each other; OCC conducts a commit-time verification to ensure read records are not modified [39]. Coupled with other Skeena techniques, the execution is serializable as long as both engines are set to run at the serializable isolation level that is based on 2PL or OCC.…”

Section: Serializabilitymentioning

confidence: 99%

Skeena: Efficient and Consistent Cross-Engine Transactions

Zhang,

Huang,

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

With the growing DRAM capacity and core count in modern servers, database systems are becoming increasingly multi-engine to feature a heterogeneous set of engines. In particular, a memory-optimized engine and a conventional storage-centric engine may coexist for various application needs. However, handling cross-engine transactions that access more than one engine remains challenging in terms of correctness, performance and programmability.This paper describes Skeena, a holistic approach to cross-engine transactions. We propose a lightweight transaction tracking structure and an atomic commit protocol to ensure correctness and support various isolation levels in multi-engine systems. Evaluation on a 40-core server shows that Skeena (1) does not penalize single-engine transactions and (2) enables the use of cross-engine transactions to improve throughput by up to 30× and/or reduce storage cost by judiciously placing tables in different engines.

show abstract

“…Fig. 1 gives a high-level overview of interactions on a classic database cluster setup of GaussDB [12], [13]. The server listens on a socket to accept queries from clients.…”

Section: Introductionmentioning

confidence: 99%

Industry Practice of Coverage-Guided Enterprise-Level DBMS Fuzzing

Wang

et al. 2021

Preprint

View full text Add to dashboard Cite

As an infrastructure for data persistence and analysis, Database Management Systems (DBMSs) are the cornerstones of modern enterprise software. To improve their correctness, the industry has been applying blackbox fuzzing for decades. Recently, the research community achieved impressive fuzzing gains using coverage guidance. However, due to the complexity and distributed nature of enterprise-level DBMSs, seldom are these researches applied to the industry.In this paper, we apply coverage-guided fuzzing to enterpriselevel DBMSs from Huawei and Bloomberg LP. In our practice of testing GaussDB and Comdb2, we found major challenges in all three testing stages. The challenges are collecting precise coverage, optimizing fuzzing performance, and analyzing root causes. In search of a general method to overcome these challenges, we propose RATEL, a coverage-guided fuzzer for enterprise-level DBMSs. With its industry-oriented design, RATEL improves the feedback precision, enhances the robustness of input generation, and performs an on-line investigation on the root cause of bugs. As a result, RATEL outperformed other fuzzers in terms of coverage and bugs. Compared to industrial black box fuzzers SQLsmith and SQLancer, as well as coverage-guided academic fuzzer Squirrel, RATEL covered 38.38%, 106.14%, 583.05% more basic blocks than the best results of other three fuzzers in GaussDB, PostgreSQL, and Comdb2, respectively. More importantly, RATEL has discovered 32, 42, and 5 unknown bugs in GaussDB, Comdb2, and PostgreSQL.

show abstract

Industrial-strength OLTP using main memory and many cores

Cited by 14 publications

References 26 publications

Industry Practice of Coverage-Guided Enterprise-Level DBMS Fuzzing

Industry Practice of Coverage-Guided Enterprise-Level DBMS Fuzzing

Skeena: Efficient and Consistent Cross-Engine Transactions

Industry Practice of Coverage-Guided Enterprise-Level DBMS Fuzzing

Contact Info

Product

Resources

About