Analysis and Impact Evaluation of Missing Data Imputation in Day-ahead PV Generation Forecasting

Kim, Tae‐Young; Ko, Woong; Kim, Jin-Ho

doi:10.3390/app9010204

Cited by 69 publications

(30 citation statements)

References 28 publications

(58 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The SOMI algorithm can have low means square error in the weight of each element of the replacement features while maintaining a high convergence speed [13,20]. Table 3 shows SOMI are implemented for imputation with % error classification compared to mean, modus, median and hot desk algorithm [35,36,37], respectively. When the level of the missing lower, the better the accuracy imputation.…”

Section: Resultsmentioning

confidence: 99%

Optimization of Feature Selection Using Genetic Algorithm in Naïve Bayes Classification for Incomplete Data

Khotimah¹,

Miswanto²,

Suprajitno³

2020

IJIES

View full text Add to dashboard Cite

In the case of high dimensional data with missing values, the process of collecting data from various sources may be miss accidentally, which affected the quality of learning outcomes. a large number of machine learning methods can be applied to explore the search area for imputation and selection of features and parameters. ML classification needs preprocessing with self-organizing map imputation (SOMI) before the imputation of missing values is done to improve the accuracy of the model. This study introduces a new approach that combines naïve Bayes classification (NBC) and genetic algorithm (GA) optimization procedures to effectively explore the search space based on a sample of experimental points. GA is a classification model approach based on the selection of features that cause computational problems, such as reduced dimensions, uncertainty and imbalanced data sets with various classes. In the experiment, preprocessing the data using SOMI yielded error results that were up to 10% for various data sets with missing data compared to other methods. In the SOMI-GANB hybrid model, the experimental results show that the proposed method can significantly improve accuracy by up to 90% compared to other imputation methods and without feature selection. SOMI can be used for homogeneous, heterogeneous and mixed data sets. The results from the experiment clearly showed that the proposed method could significantly increase the yield compared to the other imputation methods and without feature selection. The combination of GA and naïve Bayes classification was chosen because they are simple, easy-to-understand methods that are very effective in finding optimal solutions from a set of possible solutions. Naïve Bayes imputation had higher accuracy compared to neural network imputation.

show abstract

Section: Resultsmentioning

confidence: 99%

Optimization of Feature Selection Using Genetic Algorithm in Naïve Bayes Classification for Incomplete Data

Khotimah¹,

Miswanto²,

Suprajitno³

2020

IJIES

View full text Add to dashboard Cite

show abstract

“…Kim et al (2019) estimated that the missing values in the precipitation data of the Korea Meteorological Agency will be up to 16% from year 2015-2016, and about 19% for weather data in 2017 [44]. This estimation drives the Korean government to plan for data imputation strategy as the missing values can affect power generation prediction performance.…”

Section: Methods For Handling Missing Valuesmentioning

confidence: 99%

An Assessment of Open Data Sets Completeness

Ali¹,

Emran²,

Asmai³

et al. 2019

IJACSA

View full text Add to dashboard Cite

The rapid growth of open data sources is driven by free-of-charge contents and ease of accessibility. While it is convenient for public data consumers to use data sets extracted from open data sources, the decision to use these data sets should be based on data sets' quality. Several data quality dimensions such as completeness, accuracy, and timeliness are common requirements to make data fit for use. More importantly, in many cases, high-quality data sets are desirable in ensuring reliable outcomes of reports and analytics. Even though many open data sources provide data quality guidelines, the responsibility to ensure data of high quality requires commitment from data contributors. In this paper, an initial investigation on the quality of open data sets in terms of completeness dimension was conducted. In particular, the results of the missing values in 20 open data sets measurement were extracted from the open data sources. The analysis covered all the missing values representations which are not limited to nulls or blank spaces. The results exhibited a range of missing values ratios that indicated the level of the data sets completeness. The limited coverage of this analysis does not hinder understanding of the current level of data completeness of open data sets. The findings may motivate open data providers to design initiatives that will empower data quality policy and guidelines for data contributors. In addition, this analysis may assist public data users to decide on the acceptability of open data sets by applying the simple methods proposed in this paper or performing data cleaning actions to improve the completeness of the data sets concerned.

show abstract

“…According to [18], we can list two important features of the KNN method: the KNN impute function can effortlessly deal with and predict both quantitative and qualitative attributes, and this method can directly handle a number of missing values. Even though KNN impute has been widely used for dealing with missing data, there are some downsides [19][20][21] when it is executed on a large dataset [15].…”

Section: Related Workmentioning

confidence: 99%

Missing Data Imputation for Geolocation-based Price Prediction Using KNN–MCF Method

Sanjar

Olimov

Kim

et al. 2020

IJGI

View full text Add to dashboard Cite

Accurate house price forecasts are very important for formulating national economic policies. In this paper, we offer an effective method to predict houses’ sale prices. Our algorithm includes one-hot encoding to convert text data into numeric data, feature correlation to select only the most correlated variables, and a technique to overcome the missing data. Our approach is an effective way to handle missing data in large datasets with the K-nearest neighbor algorithm based on the most correlated features (KNN–MCF). As far as we are concerned, there has been no previous research that has focused on important features dealing with missing observations. Compared to the typical machine learning prediction algorithms, the prediction accuracy of the proposed method is 92.01% with the random forest algorithm, which is more efficient than the other methods.

show abstract

Analysis and Impact Evaluation of Missing Data Imputation in Day-ahead PV Generation Forecasting

Cited by 69 publications

References 28 publications

Optimization of Feature Selection Using Genetic Algorithm in Naïve Bayes Classification for Incomplete Data

Optimization of Feature Selection Using Genetic Algorithm in Naïve Bayes Classification for Incomplete Data

An Assessment of Open Data Sets Completeness

Missing Data Imputation for Geolocation-based Price Prediction Using KNN–MCF Method

Contact Info

Product

Resources

About