In-Memory Big Data Management and Processing: A Survey

Abstract

Growing main memory capacity has fueled the development of in-memory big data management and processing. By eliminating disk I/O bottleneck, it is now possible to support interactive data analytics. However, in-memory systems are much more sensitive to other sources of overhead that do not matter in traditional I/O-bounded disk-based systems. Some issues such as fault-tolerance and consistency are also more challenging to handle in in-memory environment. We are witnessing a revolution in the design of database systems that exploits main memory as its data storage layer. Many of these researches have focused along several dimensions: modern CPU and memory hierarchy utilization, time/space efficiency, parallelism, and concurrency control. In this survey, we aim to provide a thorough review of a wide range of in-memory data management and processing proposals and systems, including both data storage systems and data processing frameworks. We also give a comprehensive presentation of important technology in memory management, and some key factors that need to be considered in order to achieve efficient in-memory data management and processing.

In-Memory Data processing using Redis Database

“…The qualitative comparison of in-memory data management systems [1] are Relational databases, NoSQL databases, Graph databases, Cache systems, Big data analysis systems and real time processing systems on multiple dimensions. The comparison of Relational and NoSQL databases on the basis of the security issues [2], where security is necessary today.…”

“…In present Big data processing, in-memory enumerate has become popular because to increase capacity and high throughput of main memory. Both relational and NoSQL databases are inmemory databases [1] that provides different mechanism for data storage and retrieval. Relational database store data in structure like tabular format, where each relational table consists of rows (tuples) and columns, therefore it depends on the relational model.…”

Map vs. Unordered Map: An Analysis on Large Datasets

“…The unacceptable performance was initially encountered by Internet companies such as Amazon, Google, Facebook and Twitter, [2] but is now also becoming a hindrance for other organizations which desire to provide a reliable real-time service (e.g., real-time auction services, advertising, social gaming). For example, trading companies must detect a sudden change in the stock market prices and react immediately (in milliseconds), which seems unlikely to achieve using traditional disk-based processing systems [3]. In order that we meet the strict requirements for analyzing large amounts of unstructured data in real-time and cater to requests in milliseconds, an IMDB system that loads the entire data into the Random Access Memory (RAM) as and when required, is necessary.…”

“…In recent decades, new breakthroughs are being created due to the emergence of multi-core processors and the availability of large amounts of main memory at tumbling costs. For instance, memory storage capacity and bandwidth have been doubling roughly every three years, while its price has been dropping by a factor of 10 every five years [3]. Evolution of database systems over the last few decades is primarily driven by significant progress in hardware, availability of a large amount of data, emerging applications, collection of data at an unprecedented rate and so on.…”

RkNN Query Processing in Distributed Spatial Infrastructures: A Performance Study

“…Recent developments of spatial big data systems have motivated the emergence of novel technologies for processing large-scale spatial data on clusters of computers in a distributed environment. These Distributed Data Management Systems (DDMSs) can be classified in disk-based [9] and in-memory-based [18]. The disk-based Distributed Spatial Data Management Systems (DSDMSs) are characterized by being Hadoop-based systems and the most representative ones are SpatialHadoop [4] and Hadoop-GIS [1].…”