The Coming Age of Pervasive Data Processing

Abstract

Emerging Big Data analytics and machine learning applications require a significant amount of computational power. While there exists a plethora of large-scale data processing frameworks which thrive in handling the various complexities of data-intensive workloads, the ever-increasing demand of applications have made us reconsider the traditional ways of scaling (e.g., scale-out) and seek new opportunities for improving the performance. In order to prepare for an era where data collection and processing occur on a wide range of devices, from powerful HPC machines to small embedded devices, it is crucial to investigate and eliminate the potential sources of inefficiency in the current state of the art platforms. In this paper, we address the current and upcoming challenges of pervasive data processing and present directions for designing the next generation of largescale data processing systems.

Addressing the Nested Data Processing Gap: JSONiq Queries on Snowflake Through Snowpark

“…We largely treat each iterator as independent and pass little cross-iterator information throughout the iterator tree. This approach leaves room for introducing translationlevel optimizations that exploit contextual information and potentially synergize with the Snowflake query optimizer [60], [61]. For instance, our logic could identify common expressions in the JSONiq query and materialize them as temporary tables to facilitate the reuse of partial results across the query.…”