Research directions in data wrangling: Visualizations and transformations for usable and credible data

Abstract

In spite of advances in technologies for working with data, analysts still spend an inordinate amount of time diagnosing data quality issues and manipulating data into a usable form. This process of ‘data wrangling’ often constitutes the most tedious and time-consuming aspect of analysis. Though data cleaning and integration arelongstanding issues in the database community, relatively little research has explored how interactive visualization can advance the state of the art. In this article, we review the challenges and opportunities associated with addressing data quality issues. We argue that analysts might more effectively wrangle data through new interactive systems that integrate data verification, transformation, and visualization. We identify a number of outstanding research questions, including how appropriate visual encodings can facilitate apprehension of missing data, discrepant values, and uncertainty; how interactive visualizations might facilitate data transform specification; and how recorded provenance and social interaction might enable wider reuse, verification, and modification of data transformations

Enterprise Data Analysis and Visualization: An Interview Study

“…Researchers have also advocated the use of visualization across more phases of the analysis life-cycle [22]. Our analysis corroborates this suggestion.…”

“…Such data wrangling, munging, or cleaning [22] involves parsing text files, manipulating data layout and integrating multiple data sources. This process, whether managed by IT staff or by analysts, was often time consuming and tedious.…”

hydroscoper: R interface to the Greek National Data Bank for Hydrological and Meteorological Information

“…The main functions that can be utilized is the family of functions, get_stations, get_timeseries, get_data, etc., to easily download JSON and TXT files as tidy data frames (Wickham 2014). The internal databases of the package can be used to run queries on the available stations and time series, reducing the time needed for downloading and data wrangling (Kandel et al 2011), as these data are rarely modified.…”

Visual cleaning of genotype data

“…Firstly, primarily as a visual data cleaning tool it has commonalities with the recent direction in visualisation research towards data wrangling [3]. This is the visually-aided process of transforming raw or problematic data into a more usable form, which includes data cleaning.…”

“…In their further work, [3] outlines a number of directions in which data wrangling can be applied, and amongst these are visualising raw data, removal of errors, and visualization of missing data, all of which apply to our situation. There is no requirement for any structural data transformation, as in our case the output data necessarily has to be in the same format as the input data.…”