From dirt to shovels

Abstract

An ad hoc data source is any semistructured data source for which useful data analysis and transformation tools are not readily available. Such data must be queried, transformed and displayed by systems administrators, computational biologists, financial analysts and hosts of others on a regular basis. In this paper, we demonstrate that it is possible to generate a suite of useful data processing tools, including a semi-structured query engine, several format converters, a statistical analyzer and data visualization routines directly from the ad hoc data itself, without any human intervention. The key technical contribution of the work is a multiphase algorithm that automatically infers the structure of an ad hoc data source and produces a format specification in the PADS data description language. Programmers wishing to implement custom data analysis tools can use such descriptions to generate printing and parsing libraries for the data. Alternatively, our software infrastructure will push these descriptions through the PADS compiler, creating format-dependent modules that, when linked with formatindependent algorithms for analysis and transformation, result in fully functional tools. We evaluate the performance of our inference algorithm, showing it scales linearly in the size of the training data -completing in seconds, as opposed to the hours or days it takes to write a description by hand. We also evaluate the correctness of the algorithm, demonstrating that generating accurate descriptions often requires less than 5% of the available data.

Interactive example-based finding of text items

“…In these contexts there have been many proposals aimed at minimizing the user effort needed to identify and annotate informative examples, e.g., Zhang (2008); Dalvi et al (2016); Budlong et al (2013); Bajaj et al (2015). Some proposals advocated the execution of an initial, fully unsupervised analysis of the data corpus followed by some explicit instructions from the user on how to proceed with the processing of, e.g., logs (Fisher et al, 2008), collections of facts (Etzioni et al, 2005), or relations (Yates et al, 2007).…”