Unleashing real-time analytics: A comparative study of in-memory computing vs. traditional disk-based systems

Abstract

The article presents a comprehensive study evaluating the performance differences between in-memory computing (IMC) and traditional disk-based database systems, specifically focusing on Redis and PostgreSQL. Given the escalating demands for real-time data analytics across various sectors, the research delves into the comparative efficiency of these two data management paradigms in processing large datasets. Utilizing a synthetic dataset of 23.6 million records, we orchestrated a series of data manipulation tasks, including aggregation, table joins, and filtering operations, to simulate real-world data analytics scenarios. The experiment, conducted on a high-performance computing setup, revealed that Redis significantly outperformed PostgreSQL in all tested operations, showcasing the inherent advantages of IMC in terms of speed and efficiency. Data aggregation tasks saw Redis completing the process up to ten times faster than PostgreSQL. Similarly, table joining, and data filtering tasks were executed more swiftly on Redis, emphasizing IMC's potential to facilitate instantaneous data analytics. These findings underscore the pivotal role of IMC technologies like Redis in empowering organizations to harness real-time insights from big data, a critical capability in today's fast-paced business environment. The study further discusses the implications of adopting IMC over traditional systems, considering aspects such as cost, integration challenges, and the importance of skill development for IT teams. Concluding with strategic recommendations, the article advocates for a nuanced approach to incorporating IMC technologies, highlighting their transformative potential while acknowledging the need for balanced investment and operational planning.