Impacts of Fractional Hot-Deck Imputation on Learning and Prediction of Engineering Data

Song, Ikkyun; Yang, Yicheng; Im, Jongho; Tong, Tong; Ceylan, Hali̇l; Cho, In Ho

doi:10.1109/tkde.2019.2922638

Cited by 17 publications

(8 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The FHDI slightly outperforms the PMM and the FCM imputations. Although this relative performance of the FHDI depends on the adopted incomplete data, this result along with similar prior investigation [2] underpins the positive impact of the P-FHDI on big-data oriented machine learning and statistical learning.…”

Section: Introductionsupporting

confidence: 61%

“…I NCOMPLETE data problem has been pandemic in nearly all scientific and engineering domains. Inadequate handling of missing data may lead to biased or incorrect statistical inference and subsequent machine learning [2]. In the "imputation" methods, the active research areas of missing data-curing, two major questions arose and have been answered for the past decades.…”

Section: Introductionmentioning

confidence: 99%

“…Reliable imputation may play an important role in potential datadriven research. The impacts of the FHDI on the subsequent machine learning have been investigated in [2]. Following the same methodology, we compare the impact of different data curing methods on machine learning and statistical learning in Fig.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Parallel Fractional Hot-Deck Imputation and Variance Estimation for Big Incomplete Data Curing

Yang

Kim

Cho

2022

IEEE Trans. Knowl. Data Eng.

Self Cite

View full text Add to dashboard Cite

The fractional hot-deck imputation (FHDI) is a general-purpose, assumption-free imputation method for handling multivariate missing data by filling each missing item with multiple observed values without resorting to artificially created values. The corresponding R package FHDI [1] holds generality and efficiency, but it is not adequate for tackling big incomplete data due to the requirement of excessive memory and long running time. As a first step to tackle big incomplete data by leveraging the FHDI, we developed a new version of a parallel fractional hot-deck imputation (named as P-FHDI) program suitable for curing large incomplete datasets. Results show a favorable speedup when the P-FHDI is applied to big datasets with up to millions of instances or 10,000 of variables. This paper explains the detailed parallel algorithms of the P-FHDI for large instances (big-n) or high-dimensionality (big-p) datasets and confirms the favorable scalability. The proposed program inherits all the advantages of the serial FHDI and enables a parallel variance estimation, which will benefit a broad audience in science and engineering.

show abstract

Section: Introductionsupporting

confidence: 61%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Parallel Fractional Hot-Deck Imputation and Variance Estimation for Big Incomplete Data Curing

Yang

Kim

Cho

2022

IEEE Trans. Knowl. Data Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The impact of FHDI on the subsequent ML has been extensively discussed in [3] with datasets of moderate size. Following a similar methodology, we investigate the impact of UP-FHDI on deep learning with ultra incomplete datasets.…”

Section: Impact Of Up-fhdi On Deep Learningmentioning

confidence: 99%

“…I NCOMPLETE data commonly occurs in nearly all scientific and engineering domains, which may result in biased estimation of parameters and exacerbate subsequent statistical analyses [2]. Inadequate handling of missing data may lead to incorrect statistical inference and subsequent machine learning (ML) [3]. A popular approach, known as listwise deletion [4], is to omit instances with missing values from analysis, but it may seriously bias sample statistics if the data does not follow the assumption of missing completely at random (MCAR).…”

Section: Introductionmentioning

confidence: 99%

Ultra Data-Oriented Parallel Fractional Hot-Deck Imputation With Efficient Linearized Variance Estimation

Yang

Kwon

Kim

et al. 2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Parallel fractional hot-deck imputation (P-FHDI [1]) is a general-purpose, assumption-free tool for handling item nonresponse in big incomplete data by combining the theory of FHDI and parallel computing. FHDI cures multivariate missing data by filling each missing unit with multiple observed values (thus, hot-deck) without resorting to distributional assumptions. P-FHDI can tackle big incomplete data with millions of instances (big-n) or 10, 000 variables (big-p). However, handling ultra incomplete data (i.e., concurrently big-n and big-p) with tremendous instances and high dimensionality has posed problems to P-FHDI due to excessive memory requirement and execution time. To tackle the aforementioned challenges, we propose the ultra data-oriented P-FHDI (named UP-FHDI) capable of curing ultra incomplete data. In addition to the parallel Jackknife method, this paper enables a computationally efficient ultra data-oriented variance estimation using parallel linearization techniques. Results confirm that UP-FHDI can tackle an ultra dataset with one million instances and 10, 000 variables. This paper illustrates the special parallel algorithms of UP-FHDI and confirms its positive impact on the subsequent deep learning performance.

show abstract

Prediction of Stress-Dependent Soil Water Retention Using Machine Learning

Fazel Mojtahedi,

Akbarpour,

Darzi

et al. 2024

Geotech Geol Eng

View full text Add to dashboard Cite

Impacts of Fractional Hot-Deck Imputation on Learning and Prediction of Engineering Data

Cited by 17 publications

References 31 publications

Parallel Fractional Hot-Deck Imputation and Variance Estimation for Big Incomplete Data Curing

Parallel Fractional Hot-Deck Imputation and Variance Estimation for Big Incomplete Data Curing

Ultra Data-Oriented Parallel Fractional Hot-Deck Imputation With Efficient Linearized Variance Estimation

Prediction of Stress-Dependent Soil Water Retention Using Machine Learning

Contact Info

Product

Resources

About