Storage management in AsterixDB

Abstract

Social networks, online communities, mobile devices, and instant messaging applications generate complex, unstructured data at a high rate, resulting in large volumes of data. This poses new challenges for data management systems that aim to ingest, store, index, and analyze such data efficiently. In response, we released the first public version of AsterixDB, an open-source Big Data Management System (BDMS), in June of 2013. This paper describes the storage management layer of AsterixDB, providing a detailed description of its ingestion-oriented approach to local storage and a set of initial measurements of its ingestion-related performance characteristics.In order to support high frequency insertions, AsterixDB has wholly adopted Log-Structured Merge-trees as the storage technology for all of its index structures. We describe how the AsterixDB software framework enables "LSM-ification" (conversion from an in-place update, disk-based data structure to a deferredupdate, append-only data structure) of any kind of index structure that supports certain primitive operations, enabling the index to ingest data efficiently. We also describe how AsterixDB ensures the ACID properties for operations involving multiple heterogeneous LSM-based indexes. Lastly, we highlight the challenges related to managing the resources of a system when many LSM indexes are used concurrently and present AsterixDB's initial solution.

Microblogs data management and analysis

“…This part highlights the current state and the challenges of managing microblogs data through existing big data systems [2], [6], [29], [22], [40], depicted in the last two rows in Figure 2, this part will take 20 minutes in total and will give a briefing on system challenges and motivational case studies. Then, we highlights the data management technologies in the following four systems:…”

“…• AsterixDB [2]: AsterixDB is a generic big data management system that can support various data sources.…”

“…The horizontal axis in Figure 1 represents the year of publication or system release for each technique/system, while the vertical axis represents the research topic. The techniques are then classified into three categories: (1) filled black circle, (2) techniques that deal with historical data, depicted by a blank circle, and (3) techniques that deal with both real-time and historical data, depicted by a blank triangle. Figure 2 gives the detailed tutorial outline and timing that lasts for 1.5 hours.…”

Microblogs Data Management Systems

“…Twitter Birth Industry Academia Taghreed [20] EarlyBird [6] MDMS [22] Topsy Political Index [44] Topsy [41] AsterixDB [2] VoltDB [40] H [17,20,45], or discover users with similar interests [14]. Such users, or group of users, can be used in several scenarios, including posting ads and enhancing their social graph.…”

“…The horizontal axis in Figure 1 represents the year of publication or system release for each technique/system, while the vertical axis represents the research topic. The techniques are then classif ed into three categories: (1) techniques that deal with real-time data, i.e., very recent data, depicted by a f lled black circle, (2) techniques that deal with historical data, depicted by a blank circle, and (3) techniques that deal with both real-time and historical data, depicted by a blank triangle. Figure 2 gives the detailed tutorial outline and timing that lasts for 1.5 hours.…”