AFrame: Extending DataFrames for Large-Scale Modern Data Analysis

Abstract

Analyzing the increasingly large volumes of data that are available today, possibly including the application of custom machine learning models, requires the utilization of distributed frameworks. This can result in serious productivity issues for "normal" data scientists. This paper introduces AFrame, a new scalable data analysis package powered by a Big Data management system that extends the data scientists' familiar DataFrame operations to efficiently operate on managed data at scale. AFrame is implemented as a layer on top of Apache AsterixDB, transparently scaling out the execution of DataFrame operations and machine learning model invocation through a parallel, shared-nothing big data management system. AFrame incrementally constructs SQL++ queries and leverages Aster-ixDB's semistructured data management facilities, user-defined function support, and live data ingestion support. In order to evaluate the proposed approach, this paper also introduces an extensible micro-benchmark for use in evaluating DataFrame performance in both single-node and distributed settings via a collection of representative analytic operations. This paper presents the architecture of AFrame, describes the underlying capabilities of AsterixDB that efficiently support modern data analytic operations, and utilizes the proposed benchmark to evaluate and compare the performance and support for largescale data analyses provided by alternative DataFrame libraries.

A Simple Semantic-Based Data Storage Layout for Querying Point Clouds

“…The two presented approaches, SDBL via Python and SDB via PostgreSQL also differ with respect to their memory model. When using SDBL via Python, the primary data working object is the pandas dataframe, which albeit its strong resemblance to a relational table in terms of its structure, is nevertheless non-persistent [73]. In other words, it is up to the programmer to choose a suitable data format (such as the feather package which we used) in order to save the dataframe onto disk for later use.…”

Enhancing the Interactivity of Dataframe Queries by Leveraging Think Time

“…In the research community, there are multiple notable papers that have tackled dataframe optimization through vastly different approaches. Sinthong et al propose AFrame, a dataframe system implemented on top of AsterixDB by translating dataframe APIs into SQL++ queries that are supported by AsterixDB [15]. Another work by Yan et al aims to accelerate EDA with dataframes by "auto-suggesting" data exploration op-erations [17].…”

PolyFrame: A Retargetable Query-based Approach to Scaling DataFrames (Extended Version)

“…AFrame [24,25] is a library that provides a Pandas DataFrame [19] based syntax to interact with data in Apache AsterixDB. AFrame targets data scientists who are already familiar with Pandas DataFrames.…”

“…One is the total runtime, which includes both the DataFrame creation time and the expression runtime, and the other is the expression-only runtime. This is done to reflect the impact of the schema inferencing process which can be time-consuming for some DataFrame libraries [24,25]. Also, depending on the nature of a given analysis, the DataFrame creation time can dominate the actual expression evaluation time.…”

“…AFrame [24] is a data exploration library that provides a Pandas-like DataFrame experience on top of Apache As-terixDB [2]. AFrame leverages distributed data storage and management in order to accommodate the rapid rate and volume at which modern data arrives.…”