TiDB

Abstract

Hybrid Transactional and Analytical Processing (HTAP) databases require processing transactional and analytical queries in isolation to remove the interference between them. To achieve this, it is necessary to maintain different replicas of data specified for the two types of queries. However, it is challenging to provide a consistent view for distributed replicas within a storage system, where analytical requests can efficiently read consistent and fresh data from transactional workloads at scale and with high availability. To meet this challenge, we propose extending replicated state machine-based consensus algorithms to provide consistent replicas for HTAP workloads. Based on this novel idea, we present a Raft-based HTAP database: TiDB. In the database, we design a multi-Raft storage system which consists of a row store and a column store. The row store is built based on the Raft algorithm. It is scalable to materialize updates from transactional requests with high availability. In particular, it asynchronously replicates Raft logs to learners which transform row format to column format for tuples, forming a real-time updatable column store. This column store allows analytical queries to efficiently read fresh and consistent data with strong isolation from transactions on the row store. Based on this storage system, we build an SQL engine to process large-scale distributed transactions and expensive analytical queries. The SQL engine optimally accesses row-format and column-format replicas of data. We also include a powerful analysis engine, TiSpark, to help TiDB connect to the Hadoop ecosystem. Comprehensive experiments show that TiDB achieves isolated high performance under CH-benCHmark, a benchmark focusing on HTAP workloads.

NHtapDB: Native HTAP Databases

“…We ran the OLxPBench in different configurations by varying the type and sending rate of workload. In the demonstration, we compare NHtapDB with state-of-the-practice HTAP databases -TiDB [3].…”

“…Using a dual-format store [3,5] will introduce data transfer overhead between the row-based store and column-based store as shown in Figure 1. To keep data fresh, row-based data updates must be propagated to column-based storage as soon as possible.…”

HTAP With Reactive Streaming ETL

“…TiDB (Huang et al, 2020) is an approach to HTAP which uses a row-store DBMS and a columnstore DBMS for OLTP and OLAP workloads respectively. In this approach, multiple row-store DBMS replicas, based on the TiKV distributed DBMS, are deployed as a cluster to handle transactional queries.…”

Towards a typology of university technology transfer organizations in China: evidences from Tsinghua University

“…Transaction processing (TP) is a key function of a database, which aims to provide atomicity, consistency, isolation, and durability (ACID) features. Most existing cloud-native databases couple TP either with storage layer [27] or with execution layer [42,43]. Coupling TP with Storage Layer.…”

“…Some two-layer databases couple TP with the storage layer as shown in Figure 1a. For example, TiDB [27] adopts a distributed transactional key-value (KV) store TiKV [13] as the storage layer, where the TiDB server is the stateless SQL layer that receives SQL requests, performs SQL parsing and optimization, and generates a distributed execution plan. TiKV is responsible for storing shareded data and supports distributed transactions.…”