Time series outlier removal and imputing methods based on Colombian weather stations data

Abstract

The time data series of weather stations are a source of information for floods. The study of the previous wintertime series allows knowing the behavior of the variables and the result that will be applied to analysis and simulation models that feed variables such as flow and level of a study area. One of the most common problems is the acquisition and transmission of data from weather stations due to atypical values and lost data; this generates difficulties in the simulation process. Consequently, it is necessary to propose a numerical strategy to solve this problem. The data source for this study is a real database where these problems are presented with different variables of weather. This study is based on comparing three methods of time series analysis to evaluate a multivariable process offline. For the development of the study, we applied a method based on the discrete Fourier transform (DFT), and we contrasted it with methods such as the average and linear regression without uncertainty parameters to complete missing data. The proposed methodology entails statistical values, outlier detection, and the application of the DFT. The application of DFT allows the time series completion, based on its ability to manage various gap sizes and replace missing values. In sum, DFT led to low error percentages for all the time series (1% average). This percentage reflects what would have likely been the shape or pattern of the time series behavior in the absence of misleading outliers and missing data.

Assessment of surface water resource and allocation optimization for diverse demands in Ethiopia's upper Bilate Watershed

“…Extreme values that greatly depart from the median range in a given data collection are known as outliers. For locating these extraordinary data points, the outlier formula is a helpful tool [ [67] , [68] , [69] , [70] ]. In order to analyses the quality of rainfall data in this study and detect outliers, we utilized outliers' mechanisms and calculated outliers using the method below: Calculate the interquartile range (IQR):…”

Performance evaluation of IRI-Plas 2017 model with ionosonde data measurements of ionospheric parameters

“…A reliable indicator of a set of data's degree of dispersion is the median absolute deviation (MAD). MAD is a more widely used outlier removal technique and a known statistical methodology that uses the standard deviation from the mean, which was recommended by different researchers [ [23] , [24] , [25] , [26] , [27] , [28] , [29] , [30] , [31] , [32] ]. The ionosonde measures the electron densities for each layer as well as their estimated altitudes as functions of time [ 33 , 34 ].…”

Grape Yield Prediction Models: Approaching Different Machine Learning Algorithms

“…Climate data is now abundant, relatively cheap and can be found in different resolutions, as weather stations across the world continuously record and monitor various parameters for climate classification, planning, modeling and management purposes [47]. However, measuring instruments are subject to recording errors, malfunctioning, maintenance, network transmission and storage failures among other events that can generate data gaps and result in incomplete datasets [48][49][50]. Data missingness adversely impacts the analysis carried out afterwards and may lead to erroneous findings, false conclusions and inaccurate predictions [51].…”

“…In addition to leveraging univariate patterns, multivariate approaches can be applied to estimate missing data by means of predictive modeling, where non-missing variables are used as predictors. As a result, different statistical and machine learning methods have long been used to address the data missingness issue in different knowledge domains [51,56,57], including climate data [47,49,50,58,59]. In advanced multiple imputation schemes, the process of generating replacement values for missing data is repeated many times, resulting in m complete datasets that are further analyzed.…”