The Reproducible Data Reuse (ReDaR) Framework to Capture and Assess Multiple Data Streams

Abstract

Much of the literature in knowledge discovery from data (KDD) focuses on algorithms that are faster and more accurate at capturing patterns in a given data set. However, answering a research question is fundamentally connected with how well the data is aligned with the questions being asked. Thus, data selection is one of the most important steps to ensure that models produced from the KDD process are useful in practice. A lack of documentation about the data selection rationale and the transformations needed to semantically align the data streams prevents others from reproducing the research and obfuscates development of best practices in data integration. Our goal in this paper is to provide KDD practitioners with a framework that brings together theories in provenance, information quality, and contextual reasoning, to enable researchers to achieve a semantically aligned dataset with data selection, description, and documentation based on an application-focused assessment. KEYWORDSSemantic alignment; fitness assessment; data selection; reused data; data practices. INTRODUCTIONKnowledge discovery from databases (KDD) is a multi-step process for pattern identification and classification in data that includes data selection, preprocessing, and transformation steps to prepare data for later modeling and interpretation (Fayyad et al., 1996;Mariscal et al., 2010). KDD applications focused on data-and model-based research, however, are dependent on the semantic alignment and selection of data that are relevant to the application goals. There is a lack of documentation and discussion over best practices for the alignment and selection of data from multiple data streams in modeling and knowledge-based research.Researchers in the Earth sciences often analyze and model subsurface geologic systems through the integration of multiple data streams that are conceptually heterogeneous, and syntactically and semantically ambiguous (Gil et al., 2018;Howard et al., 2009;Nickel et al., 2011). Research on global climate change requires integration of regional and global data sets, where semantic ambiguity can mistakenly encourage the comparison of semantically distint data streams (Faghmous and Kumar, 2014;Ford, et al., 2016;Hassani et al., 2019). In health care, there are moves to compile vast distributed datasets to support 'learning health systems' (Etheredge, 2007;Friedman, et al., 2015;Delaney, et al., 2015). In these environments, and others, the fidelity of data mining results and utility of modelbased insights is dependent on the semantic alignment of complex data streams and on the identification and selection of relevant data. And, as recently noted by the National Academies of Sciences, Engineering and Medicine in evaluating the data requirements for reliable, repeatable toxic risk assessments, "Without a clear definition of evidence streams or documented approaches to evidence synthesis, within each evidence stream, it is difficult to assess and reproduce the results of evidence synthesis." (NASEM, 2021).