A Study on Garbage Collection Algorithms for Big Data Environments

Abstract

The need to process and store massive amounts of data—Big Data—is a reality. In areas such as scientific experiments, social networks management, credit card fraud detection, targeted advertisement, and financial analysis, massive amounts of information are generated and processed daily to extract valuable, summarized information. Due to its fast development cycle (i.e., less expensive to develop), mainly because of automatic memory management, and rich community resources, managed object-oriented programming languages (e.g., Java) are the first choice to develop Big Data platforms (e.g., Cassandra, Spark) on which such Big Data applications are executed. However, automatic memory management comes at a cost. This cost is introduced by the garbage collector, which is responsible for collecting objects that are no longer being used. Although current (classic) garbage collection algorithms may be applicable to small-scale applications, these algorithms are not appropriate for large-scale Big Data environments, as they do not scale in terms of throughput and pause times. In this work, current Big Data platforms and their memory profiles are studied to understand why classic algorithms (which are still the most commonly used) are not appropriate, and also to analyze recently proposed and relevant memory management algorithms, targeted to Big Data environments. The scalability of recent memory management algorithms is characterized in terms of throughput (improves the throughput of the application) and pause time (reduces the latency of the application) when compared to classic algorithms. The study is concluded by presenting a taxonomy of the described works and some open problems, with regard to Big Data memory management, that could be addressed in future works.

One-shot Garbage Collection for In-memory OLTP through Temporality-aware Version Storage

“…Programming language and system runtimes (e.g. Java, .NET Framework) provide managed memory management [3,7]. Under the hood, the memory manager tracks the liveness of the objects by tracking the reachability of all dynamically allocated objects, and once the object becomes unreachable, the system cleans and reclaims all the memory of such dead objects [15,23,24].…”

Influencing Factors in the Scalability of Distributed Stream Processing Jobs

“…A few efforts have been done to reduce the GC overhead in Spark, e.g., [70], [71]. Finally, the reader can find a comprehensive overview of GC algorithms for big data systems in [72].…”

Modeling and Management Big Data in Databases—A Systematic Literature Review

“…Furthermore, other recent surveys related to Big Data have been published; for instance, one describing the state-of-the-art about methodologies developed for multimedia Big Data analytics and the challenges, techniques, applications, strategies and future outlook [22]. Another study presents and analyzes in detail the current stage of Big Data environments and platforms and available garbage collection algorithms [23]. These works neither cover the scope of our research questions for Big Data modeling and management, nor achieve the same level of detail and precision.…”