Collaborative data analytics with DataHub

Abstract

While there have been many solutions proposed for storing and analyzing large volumes of data, all of these solutions have limited support for collaborative data analytics, especially given the many individuals and teams are simultaneously analyzing, modifying and exchanging datasets, employing a number of heterogeneous tools or languages for data analysis, and writing scripts to clean, preprocess, or query data. We demonstrate DataHub, a unified platform with the ability to load, store, query, collaboratively analyze, interactively visualize, interface with external applications, and share datasets. We will demonstrate the following aspects of the DataHub platform: (a) flexible data storage, sharing, and native versioning capabilities: multiple conference attendees can concurrently update the database and browse the different versions and inspect conflicts; (b) an app ecosystem that hosts apps for various dataprocessing activities: conference attendees will be able to effortlessly ingest, query, and visualize data using our existing apps; (c) thrift-based data serialization permits data analysis in any combination of 20+ languages, with DataHub as the common data store: conference attendees will be able to analyze datasets in R, Python, and Matlab, while the inputs and the results are still stored in DataHub. In particular, conference attendees will be able to use the DataHub notebook -an IPython-based notebook for analyzing data and storing the results of data analysis.

The Trials and Tribulations of Working with Structured Data

“…Various scientific communities, including information retrieval (IR), databases, Linked Data, data visualisation and HCI, have looked at such data journeys from different angles. They have proposed new interaction models to engage with a particular species of data, such as graphs [57] or time series [16]; studied information needs and how they are formulated [7]; and developed tools for specific data-related activities, for example statistical analysis [33], visualisation [19], personal information management [31] and teamwork [5].…”

Predicting data science sociotechnical execution challenges by categorizing data science projects

“…Of the articles that do focus on the process organizations that have been used to perform data science projects, most report a low level of process maturity (Das et al, ; Saltz & Shamshurin, ). In addition, it has also been observed that most of these projects are managed in an ad hoc fashion, that is, at a low level of process maturity (Bhardwaj et al, ). Perhaps not surprisingly, it has also been reported that an improved process model would result in higher‐quality outcomes (Mariscal, Marban, & Fernandez, ) and at least some managers are open to improving their process methodology, but might not think of doing it unless prompted (Saltz & Shamshurin, ).…”

Goods

“…DataHub [8,9,10] is a collaborative version-management system for datasets that is similar in spirit to software version control systems like Git and SVN. DataHub enables many users to analyze, collaborate, modify, and share datasets in a centralized repository.…”