Time Series Data Cleaning under Multi-Speed Constraints

Abstract

As the basis of data management and analysis, data quality issues have increasingly become a research hotspot in related fields, which contributes to optimization of big data and artificial intelligence technology. Generally, physical failures or technical defects in data collectors and recorders cause anomalies in collected data. These anomalies will strongly impact on subsequent data analysis and artificial intelligence processes; thus, data should be processed and cleaned accordingly before application. Existing repairing methods based on smoothing will cause a large number of originally correct data points being over-repaired into wrong values. The constraint-based methods such as sequential dependency and SCREEN cannot accurately repair data under complex conditions since the constraints are relatively simple. A time series data repairing method under multi-speed constraints is further proposed based on the principle of minimum repairing. Then, dynamic programming is used to solve the problem of data anomalies with optimal repairing. Specifically, multiple speed intervals are set to constrain time series data, and a series of candidate repairing points are formed for each data point according to the speed constraints. Next, the optimal repair solution is selected from these candidates based on the dynamic programming method. With regard to the feasibility study of this method, an artificial dataset, two real datasets, and another real dataset with real anomalies are employed for experiments in case of different rates of anomalies and data sizes. Experimental results demonstrate that, compared with the existing methods based on smoothing or constraints, the proposed method has better performance in terms of RMS errors and time cost. In addition, the investigation of clustering and classification accuracy with several datasets reveals the impact of data quality on subsequent data analysis and artificial intelligence. The proposed method can improve the quality of data analysis and artificial intelligence results.

An Expressway ETC Missing Data Restoration Model Considering Multi-Attribute Features

“…Due to the strict equality relationships required by functional dependencies and rule constraints, it is difficult to discover absolute consistency in real time series datasets. To this end, Gao et al [27] proposed a time series restoration method based on multi-interval velocity constraints. Related studies indicate that, for the restoration issues pertaining to time series data, having accurate timestamps ensures satisfactory restoration effects.…”

IoT data cleaning techniques: A survey

“…Since only a single attribute variable is involved, the speed constraint based repairing method on a single sequence can be converted to a linear programming problem for solving, compared to the NP time complexity of conditional function-dependent and denial-constrained restoration algorithms, which is earlier research on rule-based data cleaning. On this basis, in 2018, Yin et al [74] proposed data cleaning method combining variance constraints with speed constraints. In 2021, Gao et al [75] proposed a repair method with multi-interval speed constraints, and Song et al [42] proposed a repair method combining speed constraints and acceleration constraints, which improved the practicality of multi-entity singleattribute rules.…”

“…Repair methods based on velocity constraints [41,73] ; repair methods combining variance constraints with velocity constraints [74] .…”

Improved Vessel Trajectory Prediction Model Based on Stacked-BiGRUs

“…Li et al [7] used an improved A * shortest path algorithm to fully consider road network topology and historical matching points and proposed a new trajectory restoration algorithm. Gao et al [8] used a dynamic programming method to set multiple intervals for sequence data, and searched for candidate repair points in an iterative manner, avoiding excessive repair of sequence data.…”

“…Between January 28 and February 1, 2021, 624,307 AIS data points from 522 vessels were selected as experimental data. If there are at least MinPts points in ϵ field of p i , then// e mercator distance between two points can be found in the M (8) Initialize C temp � ∅, add p to C temp (9) Let N be the points set in the ϵ field of p i (10) For each p i ′ in N (11) If p i ′ is unvisited, then (12) Mark p i ′ as visited (13) If there are at least MinPts points in the ϵ field of p i ′ , then (14) Add points to N (15) End If (16) If p i ′ is not a member of any cluster, then (17) Add p i ′ to C temp (18) End If (19) End If (20) End For (21) Add C temp to C (22) Else mark p i as noise point (23) End If (24) Until all the points are marked, C � C 1 , C 2 , . .…”