Detection of Outliers and Imputing of Missing Values for Water Quality UV-VIS Absorbance Time Series

Plazas-Nossa, Leonardo; Angulo, Miguel Antonio Ávila; Torres, Andrés

doi:10.14483/udistrital.jour.reving.2017.1.a01

Cited by 7 publications

(4 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This method incorporates classical time series analysis elements such as spectral analysis (15), digital signal processing, dynamic systems, and multivariate statistics. SSA consists of decomposing an original signal into a set of uncorrelated components from which three characteristics can be extracted: trend, oscillation, and noise (16,17).…”

Section: Singular Spectral Analysis (Ssa)mentioning

confidence: 99%

Comparative Analysis between Singular Spectral Analysis and Empirical Mode Decomposition for Structural Damage Detection

González,

Salcedo,

Cano

2023

Ing.

View full text Add to dashboard Cite

Context: In recent years, thanks to technological advances in instrumentation and digital signal processing, noninvasive methods to detect structural damage have become increasingly important. Vibration-based structural health monitoring (SHM) techniques allow detecting the presence and location of damage from permanent changes in the fundamental frequencies of signals. A successfully employed method for damage detection is empirical mode decomposition (EMD). Another method, less used in this field of study, is singular spectral analysis (SSA). This paper describes both methods and presents a simulation study aimed at comparing them and identifying which one is more effective in detecting structural damage. Method: The methods of a reference study known as benchmark SHM were applied to facilitate the comparison. To evaluate the effectiveness of both methods, Monte Carlo simulation was employed. To control the random noise and other factors inherent to the simulation, the procedure was repeated 1.000 times for each type of damage. Results: In the case of severe damage, both methods showed a good performance. However, when the damage was slight, the changes in the fundamental frequency were not apparent. However, a significant change in the amplitude level was observed. In this case, SSA obtained the best results. Conclusions: The EMD and SSA methods, together with high-pass filtering, detected severe damage when the acceleration records had low or no noise. When the acceleration records were contaminated with noise, the likelihood of EMD detecting the damage decreased dramatically. One of the advantages of SSA over EMD is that, for moderate or mild damage patterns, the former does not require filters or the use of the Hilbert-Huang transform to detect damage. In general, it was found that SSA was more effective in detecting damage.

show abstract

Section: Singular Spectral Analysis (Ssa)mentioning

confidence: 99%

Comparative Analysis between Singular Spectral Analysis and Empirical Mode Decomposition for Structural Damage Detection

González,

Salcedo,

Cano

2023

Ing.

View full text Add to dashboard Cite

show abstract

“…Therefore, this step is called "Backward Step to Fill -BSF". But if neither steps of forward and backward filling have the length enough, the total values, including available values and replaced values in previous ranges, from the start of the time series are used to fill the missing values gap; this is called "Total amount of values Step to Fill -TSF" (Plazas et al 2015). Finally, DFT is applied to the resulting time series and 1% of the most important harmonics is used.…”

Section: Emms Functmentioning

confidence: 99%

Time Series Outlier Removal and Imputing Methods Based on Colombian Weather Stations Data

Parra-Plazas

Gaona-García

Plazas-Nossa

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The time data series of weather stations are a source of information for floods. The study of the previous wintertime series allows knowing the behavior of the variables and the result that will be applied to analysis and simulation models that feed variables such as flow and level of a study area. One of the most common problems is the acquisition and transmission of data from weather stations due to atypical values and lost data, this generates difficulties in the simulation process. Consequently, it is necessary to propose a numerical strategy to solve this problem. The data source for this study is a real database where these problems are presented with different variables of weather. This study is based on comparing three methods of time series analysis to evaluate a multivariable process offline. For the development of the study, we applied a method based on the Discrete Fourier Transform (DFT) and we contrasted it with methods such as the average and linear regression without uncertainty parameters to complete missing data. The proposed methodology entails statistical values, outlier detection and the application of the DFT. The application of DFT allows the time series completion, based on its ability to manage various gap sizes and replace missing values. In sum, DFT led to low error percentages for all the time series (1% average). This percentage reflects what would have likely been the shape or pattern of the time series behavior in the absence of misleading outliers and missing data.

show abstract

“…If it is lower, the values from that available range are used to fill the gap. Therefore, this step is called the “backward step to fill (BSF).” But if neither steps of the forward and backward filling have enough length, the total values, including available values and replaced values in previous ranges, from the start of the time series are used to fill the missing values gap; this is called “total amount of values step to fill (TSF)” (Plazas-Nossa et al, 2015 ).…”

Section: Introductionmentioning

confidence: 99%

Time series outlier removal and imputing methods based on Colombian weather stations data

Parra-Plazas

Gaona-García

Plazas-Nossa

2023

Environ Sci Pollut Res

View full text Add to dashboard Cite

The time data series of weather stations are a source of information for floods. The study of the previous wintertime series allows knowing the behavior of the variables and the result that will be applied to analysis and simulation models that feed variables such as flow and level of a study area. One of the most common problems is the acquisition and transmission of data from weather stations due to atypical values and lost data; this generates difficulties in the simulation process. Consequently, it is necessary to propose a numerical strategy to solve this problem. The data source for this study is a real database where these problems are presented with different variables of weather. This study is based on comparing three methods of time series analysis to evaluate a multivariable process offline. For the development of the study, we applied a method based on the discrete Fourier transform (DFT), and we contrasted it with methods such as the average and linear regression without uncertainty parameters to complete missing data. The proposed methodology entails statistical values, outlier detection, and the application of the DFT. The application of DFT allows the time series completion, based on its ability to manage various gap sizes and replace missing values. In sum, DFT led to low error percentages for all the time series (1% average). This percentage reflects what would have likely been the shape or pattern of the time series behavior in the absence of misleading outliers and missing data.

show abstract

Detection of Outliers and Imputing of Missing Values for Water Quality UV-VIS Absorbance Time Series

Cited by 7 publications

References 28 publications

Comparative Analysis between Singular Spectral Analysis and Empirical Mode Decomposition for Structural Damage Detection

Comparative Analysis between Singular Spectral Analysis and Empirical Mode Decomposition for Structural Damage Detection

Time Series Outlier Removal and Imputing Methods Based on Colombian Weather Stations Data

Time series outlier removal and imputing methods based on Colombian weather stations data

Contact Info

Product

Resources

About