BayesWipe: A multimodal system for data cleaning and consistent query answering on structured bigdata

Abstract

Recent efforts in data cleaning of structured data have focused exclusively on problems like data deduplication, record matching, and data standardization; none of these focus on fixing incorrect attribute values in tuples. Correcting values in tuples is typically performed by a minimum cost repair of tuples that violate static constraints like CFDs (which have to be provided by domain experts, or learned from a clean sample of the database). In this paper, we provide a method for correcting individual attribute values in a structured database using a Bayesian generative model and a statistical error model learned from the noisy database directly. We thus avoid the necessity for a domain expert or clean master data. We also show how to efficiently perform consistent query answering using this model over a dirty database, in case write permissions to the database are unavailable. We evaluate our methods over both synthetic and real data.

Approximate OLAP of document-oriented databases: A variety-aware approach

“…will evaluate techniques such as [29] that overcome the same problem by issuing a single but more complex query. Another aspect to consider is the introduction of online repairing of AFDs [36,37,38]: the idea is to automatically repair the errors in AFDs at query time, so that the user can directly receive the correct results without the need to modify the original data. Another direction is to broaden the scope of the approach to a scenario with multiple collections, thus extending the support to the whole DOD.…”

Correlation Monitoring Method and model of Science-Technology-Industry in the AI Field: A Case of the Neural Network

“…When the external downloaded data was imported to the local database, some data had conversion errors (Goh VT & Siddiqi, 2008). For example, when ''2019-01-01'' in the text file was converted to the data of Excel or SQL server, it was considered as digital data, so it may become 43,466 (days from 1900-01-01 to 2019-01-01) (De et al, 2014). For the data involving dates, this article designed a comparison data table, which could execute the update command to realize batch modification.…”

Covid Tracker for Medical Front-Liners- A Survey

“…Data cleaning has to be done which includes detection and removal of errors,inconsistencies to improve data quality [4].Certain methods include null value treatment, filling missing values, detection and treatment of outliers in data, data duplication [5].…”