Missing data imputation for traffic congestion data based on joint matrix factorization

Jia, Xiaoyi; Dong, Xiaoyu; Chen, Meng; Yu, Xiaohui

doi:10.1016/j.knosys.2021.107114

Cited by 36 publications

(21 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Matrix factorization Sparsity regularized matrix factorization (Zhang et al, 2009) Random missing, row loss, column loss OD LSTM and Graph Laplacian regularized matrix factorization (Yang et al, 2021a) Random missing, fiber mode-1 missing Speed Joint matrix factorization (Jia et al, 2021) Random missing…”

Section: Matrix Basedmentioning

confidence: 99%

Truncated tensor Schatten p-norm based approach for spatiotemporal traffic data imputation with complicated missing patterns

Nie,

Qin,

Sun

2022

Preprint

View full text Add to dashboard Cite

Rapid advances in sensor, wireless communication, cloud computing and data science have brought unprecedented amount of data to assist transportation engineers and researchers in making better decisions. However, traffic data in reality often has corrupted or incomplete values due to detector and communication malfunctions. Data imputation is thus required to ensure the effectiveness of downstream data-driven applications. To this end, numerous tensor-based methods treating the imputation problem as the low-rank tensor completion (LRTC) have been attempted in previous works. To tackle rank minimization, which is at the core of the LRTC, most of aforementioned methods utilize the tensor nuclear norm (NN) as a convex surrogate for the minimization. However, the over-relaxation issue in NN refrains it from desirable performance in practice. In this paper, we define an innovative nonconvex truncated Schatten p-norm for tensors (TSpN) to approximate tensor rank and impute missing spatiotemporal traffic data under the LRTC framework. We model traffic data into a thirdorder tensor structure of time intervals×locations (sensors)×days and introduce four complicated missing patterns, including random missing and three fiber-like missing cases according to the tensor mode-n fibers. Despite nonconvexity of the objective function in our model, we derive the global optimal solutions by integrating the alternating direction method of multipliers (ADMM) with generalized soft-thresholding (GST). In addition, we design a truncation rate decay strategy to deal with varying missing rate scenarios. Comprehensive experiments are finally conducted using four different types of real-world spatiotemporal traffic datasets, which demonstrate that the proposed LRTC-TSpN method performs well under various missing cases even with high rates of data loss, meanwhile outperforming other state-of-the-art tensor-based imputation models in almost all scenarios. In general, the performance of the proposed method achieves up to a 52% improvement in mean absolute error (MAE) compared to baseline methods even when the missing ratio is as high as 90%. Moreover, both theoretical and empirical evidences indicate that our method has robust and efficient convergence performances.

show abstract

Section: Matrix Basedmentioning

confidence: 99%

Truncated tensor Schatten p-norm based approach for spatiotemporal traffic data imputation with complicated missing patterns

Nie,

Qin,

Sun

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…There are many ways to impute missing data. Many methods are based on finding suitable candidates while maintaining the plausibility and semantic consistency in both statistical and machine-learning-based frameworks [ 7 , 8 , 9 , 10 , 11 , 12 ]. These methods may also perform differently under various conditions (e.g., missing pattern, missing rate, variable types, presence of functional dependencies among variables, etc.)…”

Section: Introductionmentioning

confidence: 99%

Empirical Comparison of Imputation Methods for Multivariate Missing Data in Public Health

Pan

Chen

2023

IJERPH

View full text Add to dashboard Cite

Sample estimates derived from data with missing values may be unreliable and may negatively impact the inferences that researchers make about the underlying population due to nonresponse bias. As a result, imputation is often preferred to listwise deletion in handling multivariate missing data. In this study, we compared three popular imputation methods: sequential multiple imputation, fractional hot-deck imputation, and generalized efficient regression-based imputation with latent processes for handling multivariate missingness under different missing patterns by conducting descriptive and regression analyses on the imputed data and seeing how the estimates differ from those generated from the full sample. Limited Monte Carlo simulation results by using the National Health Nutrition and Examination Survey and Behavioral Risk Factor Surveillance System are presented to demonstrate the effect of each imputation method on reducing bias and increasing efficiency for the parameter estimate of interest for that particular incomplete variable. Although these three methods did not always outperform listwise deletion in our simulated missing patterns, they improved many descriptive and regression estimates when used to impute all incomplete variables at once.

show abstract

“…The traffic data matrix is inherently low-rank, reflected in temporal similarity and spatial correlations between adjacent links and non-adjacent links with similar physical, functional and signal attributes. Therefore, many researchers [1], [2], [3], [4], [5], [6] have attempted to recover the traffic data matrix by utilizing its low-rank nature. To better utilize the traffic data similarity, previous studies further decomposed the temporal dimension into "time of day" and "day", and then organized the data into a three-order tensor of size "space × time-of-day × day" as shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

A Parameter-free Nonconvex Low-rank Tensor Completion Model for Spatiotemporal Traffic Data Recovery

He¹,

Jia²,

Hu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Traffic data chronically suffer from missing and corruption, leading to accuracy and utility reduction in subsequent Intelligent Transportation System (ITS) applications. Noticing the inherent low-rank property of traffic data, numerous studies formulated missing traffic data recovery as a low-rank tensor completion (LRTC) problem. Due to the non-convexity and discreteness of the rank minimization in LRTC, existing methods either replaced rank with convex surrogates that are quite far away from the rank function or approximated rank with nonconvex surrogates involving many parameters. In this study, we proposed a Parameter-Free Non-Convex Tensor Completion model (TC-PFNC) for traffic data recovery, in which a log-based relaxation term was designed to approximate tensor algebraic rank. Moreover, previous studies usually assumed the observations are reliable without any outliers. Therefore, we extended the TC-PFNC to a robust version (RTC-PFNC) by modeling potential traffic data outliers, which can recover the missing value from partial and corrupted observations and remove the anomalies in observations. The numerical solutions of TC-PFNC and RTC-PFNC were elaborated based on the alternating direction multiplier method (ADMM). The extensive experimental results conducted on four real-world traffic data sets demonstrated that the proposed methods outperform other state-of-the-art methods in both missing and corrupted data recovery. The code used in this paper is available at: https://github.com/YoungHe49/T-ITS-PFNC.

show abstract

Missing data imputation for traffic congestion data based on joint matrix factorization

Cited by 36 publications

References 25 publications

Truncated tensor Schatten p-norm based approach for spatiotemporal traffic data imputation with complicated missing patterns

Truncated tensor Schatten p-norm based approach for spatiotemporal traffic data imputation with complicated missing patterns

Empirical Comparison of Imputation Methods for Multivariate Missing Data in Public Health

A Parameter-free Nonconvex Low-rank Tensor Completion Model for Spatiotemporal Traffic Data Recovery

Contact Info

Product

Resources

About