AFrame: Extending DataFrames for Large-Scale Modern Data Analysis (Extended Version)

Sinthong, Phanwadee; Carey, Michael J.

doi:10.48550/arxiv.1908.06719

Cited by 1 publication

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to demonstrate our PolyFrame implementation and empirically validate the generality of our language-rewrite approach working against different database systems, we conducted a set of experiments using the DataFrame benchmark detailed in [25]. That DataFrame benchmark was originally developed to evaluate AFrame and to compare AFrame's performance with that of other DataFrame libraries.…”

Section: Methodsmentioning

confidence: 99%

“…AFrame [24,25] is a library that provides a Pandas DataFrame [19] based syntax to interact with data in Apache AsterixDB. AFrame targets data scientists who are already familiar with Pandas DataFrames.…”

Section: B Aframementioning

confidence: 99%

“…One is the total runtime, which includes both the DataFrame creation time and the expression runtime, and the other is the expression-only runtime. This is done to reflect the impact of the schema inferencing process which can be time-consuming for some DataFrame libraries [24,25]. Also, depending on the nature of a given analysis, the DataFrame creation time can dominate the actual expression evaluation time.…”

Section: A Dataframe Benchmarkmentioning

confidence: 99%

See 2 more Smart Citations

PolyFrame: A Retargetable Query-based Approach to Scaling DataFrames (Extended Version)

Sinthong¹,

Carey²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In the last few years, the field of data science has been growing rapidly as various businesses have adopted statistical and machine learning techniques to empower their decision making and applications. Scaling data analysis, possibly including the application of custom machine learning models, to large volumes of data requires the utilization of distributed frameworks. This can lead to serious technical challenges for data analysts and reduce their productivity. AFrame, a Python data analytics library, is implemented as a layer on top of Apache AsterixDB, addressing these issues by incorporating the data scientists' development environment and transparently scaling out the evaluation of analytical operations through a Big Data management system. While AFrame is able to leverage data management facilities (e.g., indexes and query optimization) and allows users to interact with a very large volume of data, the initial version only generated SQL++ queries and only operated against Apache AsterixDB. In this work, we describe a new design that retargets AFrame's incremental query formation to other query-based database systems as well, making it more flexible for deployment against other data management systems with composable query languages.

show abstract

Section: Methodsmentioning

confidence: 99%