Enhancing (Publications on) Data Quality: Deeper Data Minding and Fuller Data Confession

Abstract

Statistics typically treats data as inputs for analysis, whereas the broader data science enterprise deals with the entire data life cycle, including the phases that output data. This commentary argues that it would benefit statistics and (data) science if we statisticians were also to treat data as products in and of themselves, and accordingly subject them to data minding, a stringent quality inspection process that scrutinizes data conceptualization, data pre-processing, data curation and data provenance, in addition to data collection, the traditional objective of our emphasis before data analysis. A concrete step in promoting deeper data minding is to encourage fuller data confession in (statistical) publications, that is, to entice-or at least not to disincentivize-the authors into providing more details on the genealogy of a given body of data, including an account of its deliberations, especially with respect to sources of adverse influence on data quality. The collection of articles in this special issue (on data science for societies) provides both the inspiration and aspiration for deeper data minding and fuller data confession.

A Not-so-radical Rejoinder: Habituate Systems Thinking and Data (Science) Confession for Quality Enhancement

“…Indeed, Lin reminded us that the quality control should start earlier, that is, at the data collection stage, "addressing the problem before analysis even began." Although this was not a topic for my panel presentation in 2017 and hence it was not included in the article, it has been a central topic of my research since [22], which led to the proposal of "data minding" [26]. The main finding, as reported in [23], supports Lin's emphasis to its core-data quality matters far more than data quantity.…”

“…Recently, I coined the term "data confession" [26] to encourage more disclosures in research publications about defects in data conceptualization, collection or pre-processing, as another component in enhancing the replicability and ultimately the reliability of published scientific studies, since data quality matters far more than data quantity [23,4,27,3]. The retrospective introspection summarized above suggests a more general data science confession (DSC) in our publications, where we can benefit from each others' mistakes and lessons learned, especially how we reason with ourselves, where we can engage in a pure intellectual dialogue without being distracted by suspicions of impure motivations.…”

“…Much of this rejoinder has been devoted to the variancedoubling issue because the discussants' criticisms has encouraged me to explore further the need of systems thinking [25] and of data confession [26]. However, both notions' raison d'être is quality enhancement in everything we do.…”

Smallset Timelines: A Visual Representation of Data Preprocessing Decisions

“…Their influence on estimation outcomes has been demonstrated, quantitatively, in the fields of fair machine learning [17], natural language processing [11], and psychology [46], to name a few. Yet, in general it is less common to encounter meaningful detail about the preprocessing stage in discussions about research outputs, than it is to learn about how the data were collected and modelled [32]. Preprocessing decisions often remain tucked away in code-either inaccessible or difficult to parse, limiting our ability to interpret and replicate results.…”

“…Communicating and documenting data preprocessing is one aspect of data provenance, a broader concept referring to all aspects of dataset production. An influx of interest in data provenance in the machine learning community has led to work exploring how we might better record and utilise information about a dataset's creation [12,19,26,27,32,38,43]. Preprocessing is mentioned in the provenance literature, but because there are many aspects of provenance, it receives limited attention.…”