Towards automated prediction of relationships among scientific datasets

Alawini, Abdussalam; Maier, David; Tufte, Kristin; Howe, Bill; Nandikur, Rashmi

doi:10.1145/2791347.2791358

Cited by 4 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the statistics may not be sufficient in determining the relationship, ReConnect asks the user to select a candidate relationship for validation. In order to identify the relationship between all possible pairs in a large collection of datasets, Abdussalam et al [113] further propose a system, entitled ReDiscover, to automate the relationship discovery process without involving user input. ReDiscover first computes column statistics and then feeds them into machine learning models to predict the relationship.…”

Section: Additional Related Workmentioning

confidence: 99%

Effective Data Versioning for Collaborative Data Analytics

Huang

2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

With the massive proliferation of datasets in a variety of sectors, data science teams in these sectors spend vast amounts of time collaboratively constructing, curating, and analyzing these datasets. Versions of datasets are routinely generated during this data science process, via various data processing operations like data transformation and cleaning, feature engineering and normalization, among others. However, no existing systems enable us to effectively store, track, and query these versioned datasets, leading to massive redundancy in versioned data storage and making true collaboration and sharing impossible. In this thesis, we develop solutions for versioned data management for collaborative data analytics. In the first part of this thesis, we extend a relational database to support versioning of structured data. Specifically, we build a system, OrpheusDB, on top of a relational database with a carefully designed data representation and an intelligent partitioning algorithm for fast version control operations. OrpheusDB inherits much of the same benefits of relational databases, while also compactly storing, keeping track of, and recreating versions on demand. However, OrpheusDB implicitly makes a few assumptions, namely that: (a) the SQL assumption: a SQL-like language is the best fit for querying data and versioning information; (b) the structural assumption: the data is in a relational format with a regular structure; (c) the from-scratch assumption: users adopt OrpheusDB from the very beginning of their project and register each data version along with full metadata in the system. In the second part of this thesis, we remove each of these assumptions, one at a time. First, we remove the SQL assumption and propose a generalized query language for querying data along with versioning and provenance information. Second, we remove the structural assumption and develop solutions for compact storage and fast retrieval of arbitrary data representations. Finally, we remove the "from-scratch" assumption, by developing techniques to infer lineage relationships among versions residing in an existing data repository. ii To my parents and my husband, for their love and support. iii ACKNOWLEDGMENTS First of all, I would like to express my sincere appreciation to my awesome advisor Professor Aditya Parameswaran. It is my great fortune to be working with him during my whole Ph.D. study. I learned a lot from him, not only about how to do research but also about how to be a good researcher. He helped me build confidence in myself and makes me believe I can make it. I will always remember and cherish these valuable memories. Thanks a lot. I have also been fortunate to work with some other great professors-Professors Aaron Elmore, Saurabh Sinha, and Amol Deshpande. Thanks, Aaron for his kind support and thought-provoking discussions during our four years' collaboration. I enjoyed working with him a lot. Thanks, Saurabh for providing me much flexibility and many useful suggestions on our project. Thanks, Amol for giving m...

show abstract

Section: Additional Related Workmentioning

confidence: 99%

Effective Data Versioning for Collaborative Data Analytics

Huang

2020

Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

show abstract

“…The EUSES corpus collects spreadsheets used as databases, and for financial, grading, homework, inventory, and modeling purposes. EUSES is frequently used by researchers building spreadsheet tools [Alawini et al 2015;Barowy et al 2014Barowy et al , 2015Cheung et al 2016;Grigoreanu et al 2010;Hermans and Dig 2014;Hermans et al 2012aHermans et al , 2010Hermans et al , 2013Hofer et al 2013;Joharizadeh 2015;Le and Gulwani 2014;Muşlu et al 2015;Singh et al 2017]. All of the categories present in EUSES are represented in the CUSTODES suite.…”

Section: About the Benchmarksmentioning

confidence: 99%

ExceLint: automatically finding spreadsheet formula errors

Barowy

Berger

Zorn

2018

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Spreadsheets are one of the most widely used programming environments, and are widely deployed in domains like finance where errors can have catastrophic consequences. We present a static analysis specifically designed to find spreadsheet formula errors. Our analysis directly leverages the rectangular character of spreadsheets. It uses an information-theoretic approach to identify formulas that are especially surprising disruptions to nearby rectangular regions. We present ExceLint, an implementation of our static analysis for Microsoft Excel. We demonstrate that ExceLint is fast and effective: across a corpus of 70 spreadsheets, ExceLint takes a median of 5 seconds per spreadsheet, and it significantly outperforms the state of the art analysis.

show abstract

“…The second aspect is that these tools focus on single files and they are unaware of the dataflow of a scientific analysis. Data in one file rarely refers to data in other files, and recovering this information without the explicit dataflow is a research question on its own [22]. In our setting we have the explicit dataflow (lineage) in the workflow description and provenance, and this information needs to be augmented with information extracted from data.…”

Section: Introductionmentioning

confidence: 99%

LabelFlow Framework for Annotating Workflow Provenance

et al. 2018

View full text Add to dashboard Cite

Abstract:Scientists routinely analyse and share data for others to use. Successful data (re)use relies on having metadata describing the context of analysis of data. In many disciplines the creation of contextual metadata is referred to as reporting. One method of implementing analyses is with workflows. A stand-out feature of workflows is their ability to record provenance from executions. Provenance is useful when analyses are executed with changing parameters (changing contexts) and results need to be traced to respective parameters. In this paper we investigate whether provenance can be exploited to support reporting. Specifically; we outline a case-study based on a real-world workflow and set of reporting queries. We observe that provenance, as collected from workflow executions, is of limited use for reporting, as it supports queries partially. We identify that this is due to the generic nature of provenance, its lack of domain-specific contextual metadata. We observe that the required information is available in implicit form, embedded in data. We describe LabelFlow, a framework comprised of four Labelling Operators for decorating provenance with domain-specific Labels. LabelFlow can be instantiated for a domain by plugging it with domain-specific metadata extractors. We provide a tool that takes as input a workflow, and produces as output a Labelling Pipeline for that workflow, comprised of Labelling Operators. We revisit the case-study and show how Labels provide a more complete implementation of reporting queries.

show abstract

Towards automated prediction of relationships among scientific datasets

Cited by 4 publications

References 9 publications

Effective Data Versioning for Collaborative Data Analytics

Effective Data Versioning for Collaborative Data Analytics

ExceLint: automatically finding spreadsheet formula errors

LabelFlow Framework for Annotating Workflow Provenance

Contact Info

Product

Resources

About