A review of missing values handling methods on time-series data

Pratama, Irfan; Permanasari, Adhistya Erna; Ardiyanto, Igi; Indrayani, Rini

doi:10.1109/icitsi.2016.7858189

Cited by 98 publications

(63 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Mean imputation is a method of replacing a missing value with the mean value from the other instances of valid data at that time [ 26 ]. For example, if there is a value missing at 12:30 PM in a certain record, the missing value is replaced with the mean of the values from 12:30 PM in the other records.…”

Section: Methodsmentioning

confidence: 99%

Deep Learning Approach for Imputation of Missing Values in Actigraphy Data: Algorithm Development Study

Jang¹,

Choi²,

Roh³

et al. 2020

JMIR Mhealth Uhealth

View full text Add to dashboard Cite

Background Data collected by an actigraphy device worn on the wrist or waist can provide objective measurements for studies related to physical activity; however, some data may contain intervals where values are missing. In previous studies, statistical methods have been applied to impute missing values on the basis of statistical assumptions. Deep learning algorithms, however, can learn features from the data without any such assumptions and may outperform previous approaches in imputation tasks. Objective The aim of this study was to impute missing values in data using a deep learning approach. Methods To develop an imputation model for missing values in accelerometer-based actigraphy data, a denoising convolutional autoencoder was adopted. We trained and tested our deep learning–based imputation model with the National Health and Nutrition Examination Survey data set and validated it with the external Korea National Health and Nutrition Examination Survey and the Korean Chronic Cerebrovascular Disease Oriented Biobank data sets which consist of daily records measuring activity counts. The partial root mean square error and partial mean absolute error of the imputed intervals (partial RMSE and partial MAE, respectively) were calculated using our deep learning–based imputation model (zero-inflated denoising convolutional autoencoder) as well as using other approaches (mean imputation, zero-inflated Poisson regression, and Bayesian regression). Results The zero-inflated denoising convolutional autoencoder exhibited a partial RMSE of 839.3 counts and partial MAE of 431.1 counts, whereas mean imputation achieved a partial RMSE of 1053.2 counts and partial MAE of 545.4 counts, the zero-inflated Poisson regression model achieved a partial RMSE of 1255.6 counts and partial MAE of 508.6 counts, and Bayesian regression achieved a partial RMSE of 924.5 counts and partial MAE of 605.8 counts. Conclusions Our deep learning–based imputation model performed better than the other methods when imputing missing values in actigraphy data.

show abstract

Section: Methodsmentioning

confidence: 99%

Deep Learning Approach for Imputation of Missing Values in Actigraphy Data: Algorithm Development Study

Jang¹,

Choi²,

Roh³

et al. 2020

JMIR Mhealth Uhealth

View full text Add to dashboard Cite

show abstract

“…In the context of time-series data, researchers in [43] reported that the level of missing values acceptability varies. Some data sets can contain missing values from 5%-50% while others allow up to 80% percent of missing values.…”

Section: Methods For Handling Missing Valuesmentioning

confidence: 99%

An Assessment of Open Data Sets Completeness

Ali¹,

Emran²,

Asmai³

et al. 2019

IJACSA

View full text Add to dashboard Cite

The rapid growth of open data sources is driven by free-of-charge contents and ease of accessibility. While it is convenient for public data consumers to use data sets extracted from open data sources, the decision to use these data sets should be based on data sets' quality. Several data quality dimensions such as completeness, accuracy, and timeliness are common requirements to make data fit for use. More importantly, in many cases, high-quality data sets are desirable in ensuring reliable outcomes of reports and analytics. Even though many open data sources provide data quality guidelines, the responsibility to ensure data of high quality requires commitment from data contributors. In this paper, an initial investigation on the quality of open data sets in terms of completeness dimension was conducted. In particular, the results of the missing values in 20 open data sets measurement were extracted from the open data sources. The analysis covered all the missing values representations which are not limited to nulls or blank spaces. The results exhibited a range of missing values ratios that indicated the level of the data sets completeness. The limited coverage of this analysis does not hinder understanding of the current level of data completeness of open data sets. The findings may motivate open data providers to design initiatives that will empower data quality policy and guidelines for data contributors. In addition, this analysis may assist public data users to decide on the acceptability of open data sets by applying the simple methods proposed in this paper or performing data cleaning actions to improve the completeness of the data sets concerned.

show abstract

“…From the past research, mean or mode substitute imputation methods are the most commonly used because there are simple and straightforward methods [6]. Unfortunately, the mean substitute imputation method can severely distort the distribution of the data.…”

Section: Related Workmentioning

confidence: 99%

Missing Values Imputation using Similarity Matching Method for Brainprint Authentication

Liew¹,

Choo²,

Yin³

2018

ijacsa

View full text Add to dashboard Cite

This paper proposes a similarity matching imputation method to deal with the missing values in electroencephalogram (EEG) signals. EEG signals with rather high amplitude can be considered as noise, normally they will be removed. The occurrence of missing values after this artefact rejection process increases the complexity of computational modelling due to incomplete data input for model training. The fundamental concept of the proposed similarity matching imputation method is founded on the assumption that similar stimulation on a particular subject will acquire comparable EEG signals response over the related EEG channels. Hence, we replaced the missing values using the highest similarity amplitude measure across different trials in this study. Next, wavelet phase stability (WPS) was used to evaluate the performance of the proposed method since WPS portrays better signals information as compared to amplitude measure in this situation. The statistical paired sample t-test was used to validate the performance of the proposed similarity matching imputation method and the preceding mean substitute imputation method. The lower the value of mean difference indicates the better approximation of imputation data towards its original form. The proposed method is able to treat 9.75% more missing value trials, with significantly better imputation value, than the mean substitution method. Continuity of the current study will be focusing on evaluating the robustness of the proposed method in dealing with different rate of missing data.

show abstract

A review of missing values handling methods on time-series data

Cited by 98 publications

References 15 publications

Deep Learning Approach for Imputation of Missing Values in Actigraphy Data: Algorithm Development Study

Deep Learning Approach for Imputation of Missing Values in Actigraphy Data: Algorithm Development Study

An Assessment of Open Data Sets Completeness

Missing Values Imputation using Similarity Matching Method for Brainprint Authentication

Contact Info

Product

Resources

About