A Novel Algorithm for Missing Data Imputation on Machine Learning

Madhu, G.; Bharadwaj, B. Lalith; Nagachandrika, G.; Vardhan, K. Sai

doi:10.1109/icssit46314.2019.8987895

Cited by 17 publications

(13 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach seems to be efficient and effective than using complete feature data set. The experimentation results in these studies (Al-Helali et al , 2020a; Madhu et al , 2019; Sanjar et al , 2020; Sefidian and Daneshpour, 2019; Tran et al , 2017) show this.…”

Section: Discussionmentioning

confidence: 59%

A systematic review of machine learning-based missing value imputation techniques

Thomas

Rajabi

2021

DTA

View full text Add to dashboard Cite

PurposeThe primary aim of this study is to review the studies from different dimensions including type of methods, experimentation setup and evaluation metrics used in the novel approaches proposed for data imputation, particularly in the machine learning (ML) area. This ultimately provides an understanding about how well the proposed framework is evaluated and what type and ratio of missingness are addressed in the proposals. The review questions in this study are (1) what are the ML-based imputation methods studied and proposed during 2010–2020? (2) How the experimentation setup, characteristics of data sets and missingness are employed in these studies? (3) What metrics were used for the evaluation of imputation method?Design/methodology/approachThe review process went through the standard identification, screening and selection process. The initial search on electronic databases for missing value imputation (MVI) based on ML algorithms returned a large number of papers totaling at 2,883. Most of the papers at this stage were not exactly an MVI technique relevant to this study. The literature reviews are first scanned in the title for relevancy, and 306 literature reviews were identified as appropriate. Upon reviewing the abstract text, 151 literature reviews that are not eligible for this study are dropped. This resulted in 155 research papers suitable for full-text review. From this, 117 papers are used in assessment of the review questions.FindingsThis study shows that clustering- and instance-based algorithms are the most proposed MVI methods. Percentage of correct prediction (PCP) and root mean square error (RMSE) are most used evaluation metrics in these studies. For experimentation, majority of the studies sourced the data sets from publicly available data set repositories. A common approach is that the complete data set is set as baseline to evaluate the effectiveness of imputation on the test data sets with artificially induced missingness. The data set size and missingness ratio varied across the experimentations, while missing datatype and mechanism are pertaining to the capability of imputation. Computational expense is a concern, and experimentation using large data sets appears to be a challenge.Originality/valueIt is understood from the review that there is no single universal solution to missing data problem. Variants of ML approaches work well with the missingness based on the characteristics of the data set. Most of the methods reviewed lack generalization with regard to applicability. Another concern related to applicability is the complexity of the formulation and implementation of the algorithm. Imputations based on k-nearest neighbors (kNN) and clustering algorithms which are simple and easy to implement make it popular across various domains.

show abstract

Section: Discussionmentioning

confidence: 59%

A systematic review of machine learning-based missing value imputation techniques

Thomas

Rajabi

2021

DTA

View full text Add to dashboard Cite

show abstract

“…XGBoost, the Missforest can reconstruct the data that is best for a boosting classifier. One previous study [11] showed the Missforest coupled with XGboost can perform relatively well on mild missingness scenario (2.19% to 13.63%). But for our model, it outperforms not only the traditional methods but also the methods that utilizes the neural networks such as MLP and LSTM.…”

Section: Resultsmentioning

confidence: 99%

“…On the other hand, MissForest, a tree-based machine learning method that predicts missing values from related, non-missing data [10], overcomes these limitations but does not make use of time-varying dynamics. In addition, the rate of missing data in traditional studies is relatively low, ranging from 1.98% to 50.65% [6,10,11]. A stronger imputation technique is needed for real-world applications with severe missingness testing.…”

Section: Introductionmentioning

confidence: 99%

Dynimp: Dynamic Imputation for Wearable Sensing Data through Sensory and Temporal Relatedness

Huo

Liang

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In wearable sensing applications, data is inevitable to be irregularly sampled or partially missing, which pose challenges for any downstream application. An unique aspect of wearable data is that it is time-series data and each channel can be correlated to another one, such as x, y, z axis of accelerometer. We argue that traditional methods have rarely made use of both times-series dynamics of the data as well as the relatedness of the features from different sensors. We propose a model, termed as DynImp, to handle different time point's missingness with nearest neighbors along feature axis and then feeding the data into a LSTM-based denoising autoencoder which can reconstruct missingness along the time axis. We experiment the model on the extreme missingness scenario (> 50% missing rate) which has not been widely tested in wearable data. Our experiments on activity recognition show that the method can exploit the multi-modality features from related sensors and also learn from history time-series dynamics to reconstruct the data under extreme missingness.

show abstract

“…However, like many other approaches in literature a missing value mechanism was not considered. Also, in [125], the researchers developed a novel ensemble imputation approach named the missXGBoost imputation technique. The technique has proven to be suitable for continuous attributes of medical applications.…”

Section: Ensemble Methodsmentioning

confidence: 99%

A Survey On Missing Data in Machine Learning

Emmanuel

Maupong

Mpoeleng

et al. 2021

Preprint

View full text Add to dashboard Cite

Machine learning has been the corner stone in analysing and extracting information from data and often a problem of missing values is encountered. Missing values occur as a result of various factors like missing completely at random, missing at random or missing not at random. All these may be as a result of system malfunction during data collection or human error during data pre-processing. Nevertheless, it is important to deal with missing values before analysing data since ignoring or omitting missing values may result in biased or misinformed analysis. In literature there have been several proposals for handling missing values. In this paper we aggregate some of the literature on missing data particularly focusing on machine learning techniques. We also give insight on how the machine learning approaches work by highlighting the key features of the proposed techniques, how they perform, their limitations and the kind of data they are most suitable for. Finally, we experiment on the K nearest neighbor and random forest imputation techniques on novel power plant induced fan data and offer some possible future research direction.

show abstract

A Novel Algorithm for Missing Data Imputation on Machine Learning

Cited by 17 publications

References 18 publications

A systematic review of machine learning-based missing value imputation techniques

A systematic review of machine learning-based missing value imputation techniques

Dynimp: Dynamic Imputation for Wearable Sensing Data through Sensory and Temporal Relatedness

A Survey On Missing Data in Machine Learning

Contact Info

Product

Resources

About