Study on Missing Values and Outlier Detection in Concurrence with Data Quality Enhancement for Efficient Data Processing

Vinisha, Feby A; Sujihelen, L.

doi:10.1109/icssit53264.2022.9716355

Cited by 8 publications

(4 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The initial stage of data preparation involves reviewing the dataset for missing values, which can compromise the reliability of the model (Vinisha & Helen, 2022). Outlier detection is then crucial, as these abnormal values can significantly impact statistical estimates and data analysis results (Kwak & Kim, 2017).…”

Section: Data Preprocessingmentioning

confidence: 99%

Leveraging Machine Learning for Accurate Groundnut Price Forecasting in Tamil Nadu: An XGboost Approach

Mantaw,

2024

Preprint

View full text Add to dashboard Cite

Accurate forecasting of agricultural commodity prices is vital for enabling effective decision-making within the agricultural ecosystem. This study addresses the implementation of modern machine learning algorithms, including XGBoost, Automated Machine Learning (AutoML) utilising PyCaret, and Auto-ARIMA, for forecasting groundnut prices in Tamil Nadu. The research integrates key temporal information, such as day of the week, month, and lagged data, to boost the prediction performance. The findings illustrate the improved performance of the AutoML technique, with the Light Gradient Boosting Machine (LightGBM) model attaining the lowest Root Mean Squared Error (RMSE) of 516.511 and Mean Absolute Percentage Error (MAPE) of 5.1%. The feature significance analysis indicates the substantial effect of year and lagged factors on the XGBoost model's predictions. The study provides vital information for stakeholders in the agriculture industry, including farmers, traders, and policymakers, by delivering precise price estimates to promote informed decision-making.

show abstract

Section: Data Preprocessingmentioning

confidence: 99%

Leveraging Machine Learning for Accurate Groundnut Price Forecasting in Tamil Nadu: An XGboost Approach

Mantaw,

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, RE data accuracy is highly dependent on weather conditions, which include numerous variables (e.g., dust particles accumulation on sensors) (Zell et al, 2015). Additionally, equipment failure that causes missing data, device misalignment, and sensor cleaning procedure or calibration can introduce data variation (Vinisha & Sujihelen, 2022). This highlighted a vital data issue with real-time energy generation forecasting.…”

Section: Big Datamentioning

confidence: 99%

“…The solar irradiance data are collected using different devices that measure incidents on a sensor surface. However, in a hostile climate such as Saudi Arabia, the devices' performance is affected by various elements, such as dust particles accumulation on sensors (Zell et al, 2015) and missing data due to equipment failure, sensor cleaning procedure, or calibration (Vinisha & Sujihelen, 2022). As a result, this can influence data quality severely due to the different anomaly sources.…”

Section: Introductionmentioning

confidence: 99%

Anomaly Detection in Renewable Energy Big Data Using Deep Learning

Katamoura,

Aksoy

2023

International Journal of Intelligent Information Technologies

View full text Add to dashboard Cite

This work aims to review the literature on anomaly detection (AD) in renewable energy. Due to the significance of the RE data quality and sensor performance, it is crucial to ensure that the measurement device works correctly and maintains data accuracy. The review identifies the relevant studies on big data anomaly detection in the energy field and synthesizes the related techniques. Also, the study shows a need for segmentation annotations for solar system electroluminescence imagery complicating the domain development of anomaly segmentation approaches. Consequently, most processes create machine learning (ML) models using semi-supervised techniques. Still, these approaches need more generalization regarding variation in environmental or systematic conditions. Furthermore, the studies discussed here focus on existing algorithms that used big data and AD to propose an improved analysis framework. Finally, the work presents a framework to solve the problem of identifying sensors' issues that will appear in data anomalies.

show abstract

“…When the absolute value of the heading difference near the missing value is within 5°, it is determined as a straight line trajectory; otherwise, it is a curve trajectory (Gao et al, 2021 ). After calculating the number of missing values according to Table 1 , routes with straight continuous missing values between 0 and 5% of the total route length and curved continuous missing values between 0 and 2% of the route were interpolated, and trajectories exceeding these criteria were rejected (Vinisha and Sujihelen, 2022 ). As the linear interpolation algorithm has high operating efficiency, the route distance designed in this study is relatively short, and the routes are relatively smooth, it is suitable to use linear interpolation for implementation (Huang et al, 2011 ).…”

Section: Production Of Datasetsmentioning

confidence: 99%

Research on autonomous route generation method based on AIS ship trajectory big data and improved LSTM algorithm

Zhuang

Chen

2022

Front. Neurorobot.

View full text Add to dashboard Cite

The autonomous generation of routes is an important part of ship intelligence and it can be realized by deep learning of the big data of automatic identification system (AIS) ship trajectories. In this study, to make the routes generated by long short-term memory (LSTM) artificial neural network more accurate and efficient, a ship route autonomous generation scheme is proposed based on AIS ship trajectory big data and improved multi-task LSTM artificial neural network. By introducing an unsupervised trajectory separation mechanism into LSTM, a fast and accurate separation of trajectories with similar paths is realized. In the process of route generation, first of all, a clustering algorithm is used to cluster the trajectories in massive AIS data according to the density of trajectory points, so as to eliminate the trajectories in the routes that do not belong to the target area. Furthermore, the routes are classified according to the type of ships, and then the classified trajectories are processed and used as datasets. Based on these datasets, an improved LSTM algorithm is used to generate ship routes autonomously. The results show the improved LSTM works better than LSTM when the generated route trajectories are short.

show abstract

Study on Missing Values and Outlier Detection in Concurrence with Data Quality Enhancement for Efficient Data Processing

Cited by 8 publications

References 31 publications

Leveraging Machine Learning for Accurate Groundnut Price Forecasting in Tamil Nadu: An XGboost Approach

Leveraging Machine Learning for Accurate Groundnut Price Forecasting in Tamil Nadu: An XGboost Approach

Anomaly Detection in Renewable Energy Big Data Using Deep Learning

Research on autonomous route generation method based on AIS ship trajectory big data and improved LSTM algorithm

Contact Info

Product

Resources

About