“Anti-Caching”-based elastic memory management for Big Data

Abstract

The increase in the capacity of main memory coupled with the decrease in cost has fueled the development of inmemory database systems that manage data entirely in memory, thereby eliminating the disk I/O bottleneck. However, as we shall explain, in the Big Data era, maintaining all data in memory is impossible, and even unnecessary. Ideally we would like to have the high access speed of memory, with the large capacity and low price of disk. This hinges on the ability to effectively utilize both the main memory and disk.In this paper, we analyze state-of-the-art approaches to achieving this goal for in-memory databases, which is called as "Anti-Caching" to distinguish it from traditional caching mechanisms. We conduct extensive experiments to study the effect of each fine-grained component of the entire process of "Anti-Caching" on both performance and prediction accuracy. To avoid the interference from other unrelated components of specific systems, we implement these approaches on a uniform platform to ensure a fair comparison. We also study the usability of each approach, and how intrusive it is to the systems that intend to incorporate it. Based on our findings, we propose some guidelines on designing a good "Anti-Caching" approach, and sketch a general and efficient approach, which can be utilized in most in-memory database systems without much code modification.978-1-4799-7964-6/15/$31.00

On main-memory flushing in microblogs data management systems

“…Our problem can be considered a variation of the anti-caching problem, applied in microblogs platforms rather than relational main-memory databases. How ever, existing techniques [8,15,30] have limitation to solve our problem. First, none of them addresses top-k queries, which is a maj or component of microblogs systems [5,19,28].…”

“…Though, it still uses a traditional policy (LRU) that requires tracking usage of individual data items. This pose a significant overhead to maintain LRU-ordered list for all data items in the system [30] . Unlike this work, kFlushing uses top-k queries as a guide to smartly select flushing victims with minimal overhead that does not limit system scalability.…”

“…Evicting data from main-memory has been studied in both buffer management in database systems [9] and anti-caching in main memory databases [8,15,30] . Our problem can be considered a variation of the anti-caching problem, applied in microblogs platforms rather than relational main-memory databases.…”

“…kFlushing enables existing algorithms for top-k microblog search queries (e.g., [5,6,16,24,19,28]) to reach to their full potential and significantly increasing their memory hit ratio. different terminologies, e.g., buffer management in database management systems (DBMSs) [9], anti-caching in main memory databases [8,15,30], and load shedding in data stream management systems (DSMSs) [1, 12,13]. However, neither buffer management nor anti-caching techniques exploit top-k queries as they decide on flushing an item based on its latest access time regardless of other items.…”

A performance study of big data on small nodes

“…We choose these two sets to investigate three scenarios: (i) both servers have enough memory, (ii) Xeon has enough memory while ARM does not, (iii) both servers do not have enough memory. These scenarios cover memory management issues in modern database sys-tems [34]. The join query is both memory and I/O bounded, as the smaller table is usually used to build an in-memory structure and the other table is scanned once from the storage.…”