Stream Data Cleaning under Speed and Acceleration Constraints

Abstract

Stream data are often dirty, for example, owing to unreliable sensor reading or erroneous extraction of stock prices. Most stream data cleaning approaches employ a smoothing filter, which may seriously alter the data without preserving the original information. We argue that the cleaning should avoid changing those originally correct/clean data, a.k.a. the minimum modification rule in data cleaning. To capture the knowledge about what is clean , we consider the (widely existing) constraints on the speed and acceleration of data changes, such as fuel consumption per hour, daily limit of stock prices, or the top speed and acceleration of a car. Guided by these semantic constraints, in this article, we propose the constraint-based approach for cleaning stream data. It is notable that existing data repair techniques clean (a sequence of) data as a whole and fail to support stream computation. To this end, we have to relax the global optimum over the entire sequence to the local optimum in a window. Rather than the commonly observed NP-hardness of general data repairing problems, our major contributions include (1) polynomial time algorithm for global optimum, (2) linear time algorithm towards local optimum under an efficient median-based solution , and (3) experiments on real datasets demonstrate that our method can show significantly lower L1 error than the existing approaches such as smoother.

Online Topic-Aware Entity Resolution Over Incomplete Data Streams

“…DEFINITION Note that, instead of setting 𝜖 𝑌 .𝑚𝑖𝑛 to 0 in [19,41], in this paper, we relax this limitation to let the 𝜖 𝑌 .𝑚𝑖𝑛 be any non-negative value less than 𝜖 𝑌 .𝑚𝑎𝑥 (i.e., 0 ≤ 𝜖 𝑌 .𝑚𝑖𝑛 < 𝜖 𝑌 .𝑚𝑎𝑥), such that the CDD rule can have tighter intervals for distance constraints. CDD Rule Detection: We assume that a static data repository 𝑅 is available, which can be collected/inferred by historical stream data [23,37,38,44]. Following the literature [19,41], to infer a CDD rule in the form 𝑋 → 𝐴 𝑗 from 𝑅, we first obtain determinant attributes 𝑋 from (𝑑-1) attributes (other than 𝐴 𝑗 ), where attributes 𝑋 are correlated with 𝐴 𝑗 in 𝑅.…”

“…In this paper, we consider the missing at random (MAR) model [15] for incomplete data. Under the MAR model, we can classify the existing imputation methods of incomplete data into categories such as statistical-based [23], rule-based [12], constraint-based [38,44], and pattern-based [22] imputation methods. Due to textual property and sparseness of ER data sets, these works may fail to impute incomplete data, when there are only a few or even no samples for imputing missing attributes.…”

Exploring Data and Knowledge combined Anomaly Explanation of Multivariate Industrial Data

“…As shown in Table II, the 4-Type constraints embody the dependence on attributes (columns) and entities (rows) for temporal data. T-3: SD, SC [11], T-4: Similarity Constraints T-3: Variance Constraints [12] points in sequence as the simple instance of Type-1 constraints, CFD for relational data and Physical Mechanism for industrial data are concluded as multi-sequence constraints. Constraints, such as SD, SC, and VC, formalizing the dependence of data points along the time in one sequence belongs to Type-3 constraints.…”

“…Ihab F. Ilyas and Xu Chu give an overview of the end-to-end data cleaning process including error detection and repair methods in [10]. Both statistical-based [27], [28] and constraints-based [11], [29] cleaning are widely applied in temporal date quality improvement. [29] extends the idea of constraints from dependencies defined on relational database (e.g., FD, CFD in [30]), and proposes sequential dependencies (SD) to describe the semantics of temporal data.…”

“…[29] extends the idea of constraints from dependencies defined on relational database (e.g., FD, CFD in [30]), and proposes sequential dependencies (SD) to describe the semantics of temporal data. Accordingly, speed constraints are developed in sequential data and applied to time series cleaning solutions [11], [28]. Causality analysis tries to reason about the responsibility of a source in causing errors result.…”