Faster, Higher, Stronger: Redesigning Spreadsheets for Scale

Bendre, Mangesh; Wattanawaroon, Tana; Rahman, Sajjadur; Mack, Kelly; Liu, Yuyang; Zhu, Shichu; Lü, Yu; Yang, Pingjing; Zhou, Xinyan; Chang, Kevin Chen-Chuan; Karahalios, Karrie; Parameswaran, Aditya

doi:10.1109/icde.2019.00217

Cited by 6 publications

(7 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…-MS Excel is a paid software product, therefore it is not available everywhere; -many programming languages have libraries for working with Excel files, but the code for fetching data seems to be more cumbersome and less intuitive in comparison, for example, with SQL queries to relational databases; -Excel file processing speed is slow when there are a significant number of rows [21][22][23].…”

Section: Research Of Existing Solutions Of the Problemmentioning

confidence: 99%

Oil storages and main oil pipelines pumps database development

Buzovskyi¹,

Kologrivov²,

Antonova³

2020

TAPR

View full text Add to dashboard Cite

The object of research is a data model that characterizes the pumping units of main oil pipelines. The paper considers the creation of pumps database and their characteristics as a component of software systems designed to search for sources of energy saving in pipeline transportation of oil and oil products. An overview of existing programs for calculating the joint operation of an oil pumping station and a pipeline is given. It is shown that most programs are inaccessible for analyzing the applied data storage technologies, or involve manual input of entry data without using databases. Comparative analysis of spreadsheets and relational databases is performed. It is shown that relational databases better meet the requirements of data convenience, availability, scalability, and performance. A physical model of a relational database is presented. The parent and child entities have been established, which make it possible to fully reflect all information about the pumping unit, namely:-coefficients of hydraulic characteristics depending on the diameter of the impeller rotor;-characteristics depending only on the pump make;-classification of the pump by design or purpose. Relationship types and referential integrity rules are defined between entities when deleting or updating data. For data management, the SQLite system is proposed, which provides data manipulation in the SQL query language and does not require the development of additional software. The analysis of possible ways of organizing multilingualism using a database is carried out. The text of the SQL query is proposed, which allows to select the pump with its characteristics by the pump make and impeller diameter. A database structure is proposed that allows storing information about equipment for oil pipelines for various purposes, which makes it possible for automated calculations of complex technological processes. The pumping unit database is an open source project that is posted on the public web service GitHub.

show abstract

Section: Research Of Existing Solutions Of the Problemmentioning

confidence: 99%

Oil storages and main oil pipelines pumps database development

Buzovskyi¹,

Kologrivov²,

Antonova³

2020

TAPR

View full text Add to dashboard Cite

show abstract

“…This was the goal of our tool, DATASPREAD. We wanted DATASPREAD to have a spreadsheetlike frontend and a relational database backend [12,15]. This solves dual problems of providing a scalable and interactive backend for spreadsheets, and an intuitive and flexible frontend for databases.…”

Section: Dataspreadmentioning

confidence: 99%

“…To have DATASPREAD function as a frontend for databases, we need to be able to support more expressive queries from the frontend. DATASPREAD can now support relational algebra and SQL queries in conjunction with formulae, all within the frontend [15]. We can also import tables from the backend database and display it on the sheet [12], with the table and the spreadsheet view kept in sync.…”

Section: Next Steps and Selected Related Workmentioning

confidence: 99%

“…Our Maslow's hierarchy for HILDA is analogous to Maslow's original hierarchy of needs, but is instead organized along the increasing sophistication of data analytics needs. At the bottom, we have CATAMARAN, a automatic data extraction tool that makes it possible to touch your data by extracting structure from messy semi-structured log files [23]; then, we have DATA-SPREAD, a scalable spreadsheet tool that allows you to browse and explore one's data, and make modifications [15]; then, we have ZENVISAGE, a visual data exploration tool to view and play with one's data, and see interesting insights [61]; then, we have HELIX, a human-in-the-loop machine learning tool that allows novices to iterate on machine learning models easily to gain a deeper understanding [77]; and finally, ORPHEUSDB, a data versioning tool that manages the data science process and facilitates collaboration [28].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Enabling data science for the majority

Parameswaran

2019

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

Despite great strides in the generation, collection, and processing of data at scale, data science is still extremely inconvenient for the vast majority of the population. The driving goal of our research, over the past half decade, has been to make it easy for individuals and teams-regardless of programming or analysis expertise-manage, analyze, make sense of, and draw insights from large datasets. In this article, we reflect on a comprehensive suite of tools that we've been building to empower everyone to perform data science more efficiently and effortlessly, including DATASPREAD, a scalable spreadsheet tool that combines the benefits of spreadsheets and databases, and ZENVISAGE, a visual exploration tool that accelerates the discovery of trends or patterns. Our tools have been developed in collaboration with experts in various disciplines, including neuroscience, battery science, genomics, astrophysics, and ad analytics. We will discuss some of the key technical challenges underlying the development of these tools, and how we addressed them, drawing from ideas in multiple disciplines. In the process, we will outline a research agenda for tool development to empower everyone to tap into the hidden potential in their datasets at scale.

show abstract

“…DataSpread is our spreadsheet system that utilizes the techniques in this paper [8][9][10]. So far, we have focused on mainmemory resident systems.…”

Section: The Dataspread Systemmentioning

confidence: 99%

Anti-Freeze for Large and Complex Spreadsheets

Bendre

Wattanawaroon

Mack

et al. 2019

Proceedings of the 2019 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

Spreadsheet systems enable users to store and analyze data in an intuitive and flexible interface. Yet the scale of data being analyzed often leads to spreadsheets hanging and freezing on small changes. We propose a new asynchronous formula computation framework: instead of freezing the interface we return control to users quickly to ensure interactivity, while computing the formulae in the background. To ensure consistency, we indicate formulae being computed in the background via visual cues on the spreadsheet. Our asynchronous computation framework introduces two novel challenges: (a) How do we identify dependencies for a given change in a bounded time? (b) How do we schedule computation to maximize the number of spreadsheet cells available to the user over time? We bound the dependency identification time by compressing the formula dependency graph lossily, a problem we show to be NP-Hard. A compressed dependency table enables us to quickly identify the spreadsheet cells that need recomputation and indicate them as such to users. Finding an optimal computation schedule to maximize cell availability is also NP-Hard, and even merely obtaining a schedule can be expensive-we propose an on-the-fly scheduling technique to address this. We have incorporated asynchronous computation in DataSpread, a scalable spreadsheet system targeted at operating on arbitrarily large datasets on a spreadsheet frontend.

show abstract

Faster, Higher, Stronger: Redesigning Spreadsheets for Scale

Cited by 6 publications

References 6 publications

Oil storages and main oil pipelines pumps database development

Oil storages and main oil pipelines pumps database development

Enabling data science for the majority

Anti-Freeze for Large and Complex Spreadsheets

Contact Info

Product

Resources

About