Handling Missing Values and Unusual Observations in Statistical Downscaling Using Kalman Filter

Saputra, M Dika; Hadi, Alfian Futuhul; Riski, Abduh; Anggraeni, Dian

doi:10.1088/1742-6596/1863/1/012035

Cited by 11 publications

(9 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, it is required to interpolate and smooth the values of time series to fill the gaps of the missing values. Here, it worth highlighting that previous works have shown that the Kalman filter produces appropriate and efficient values compared to other interpolation methods [23][24][25].…”

Section: Introductionmentioning

confidence: 96%

“…For handling missing values in the produced time series, the Kalman smoothing filter [23,34] was evaluated and compared to regression analysis using the Cook's distance and Hat matrix. Annual density plots helped initial observations on the completed LST and NDVI time series, while the additive seasonal decomposition model and the BFAST algorithm served as tools for analyzing the time series and detecting breakpoints.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Time Series Analysis of Landsat Data for Investigating the Relationship between Land Surface Temperature and Forest Changes in Paphos Forest, Cyprus

et al. 2022

View full text Add to dashboard Cite

This study aims to investigate how alternations of the land surface temperature (LST) affects the normalized difference vegetation index (NDVI) in Paphos forest, Cyprus, using Landsat-5 and Landsat-8 imagery for the time periods 1993–2000 and 2013–2018, respectively. A total of 262 Landsat images were processed to compute the mean monthly NDVI and LST values and create a time series. Using the Cook’s distance, the effect of missing values in the analysis of the time series were examined. Results from the cross-correlation and cross-variograms, decomposition model, and the BFAST algorithm were compared to produce reliable conclusions on forest changes and satellite, meteorological, and environmental data were combined to interpret the changes that occurred inside the forest. The decomposition analysis showed a decrease of 2.7% in the LST for the period 1993–2000 and an increase of 4.6% in the LST during the period 2013–2018. The NDVI trend is negatively correlated to the LST trend for both time periods. An increase in the LST trend was identified in November 1998 as well as in the NDVI trend in October 1994 and May 2014 that was caused by favorable climatic conditions. An increase in the NDVI trend from May 2014 to December 2015 may be related to reduced pityocampa attacks. An abrupt decrease was detected in December 2015 that was probably caused by the locust invasion that occurred in the island earlier that year. A positive correlation appears for LST and NDVI variables for time lags 4, 5, 6, 7, and 8 months. Overall, it was shown that LST and NDVI analysis is very promising for identifying potential forest decline.

show abstract

Section: Introductionmentioning

confidence: 96%

Section: Introductionmentioning

confidence: 99%

Time Series Analysis of Landsat Data for Investigating the Relationship between Land Surface Temperature and Forest Changes in Paphos Forest, Cyprus

et al. 2022

View full text Add to dashboard Cite

show abstract

“…50 Here we replace the missing data in all EPA sites, and PA sensors using the ARIMA model with Kalman filter. [51][52][53][54] The power spectral density of each EPA and PA hourly time series of PM 2.5 data was then calculated using the stats package in R.…”

Section: Spectral Analysis: Methodsmentioning

confidence: 99%

Understanding the source components captured by the Purple Air Network

Kumar

Senarathna

Gurajala

et al. 2023

Preprint

View full text Add to dashboard Cite

PM2.5 has been linked to numerous pollution-mediated adverse health effects and their monitoring is key for taking preventative and mitigative measures. Accurate measurements of PM2.5 concentrations are available at EPA sites, but such data lacks spatial resolution due to a limited number of monitoring locations. In recent years the deployment of low-cost sensor networks has opened up the possibility of acquiring air quality data at a high spatiotemporal resolution. However, the sensitivity, noise, and accuracy of data acquired by low-cost sensors remain a concern. Here, we studied PM2.5 measurements made from EPA and Purple Air (PA) sensor networks in the Chicago area to understand the parameters influencing the performance characteristics of the low-cost sensor network. Using time series decomposition of PM2.5 data into short-term and baseline components using Kolmogorov–Zurbenko (KZ) filter and analysis of the extracted frequency signals, we determine that PA sensor data is more sensitive to meteorological conditions than anthropogenic activities in both short-term, and baseline components. We hypothesize that the low-cost sensor networks may have different sensitivity to aerosol from different sources and hence care must be taken in their calibrations and in their use for evaluating the impact of air quality mitigation policies.

show abstract

“…We cannot impute these values using ARIMA, (e.g. Saputra et al 2021) and NN-techniques (e.g. Li et al 2020;Shu et al 2021), as the SRNN would not then need to learn how to fill the gaps.…”

Section: Parameter Configurationmentioning

confidence: 99%

LSST Cadence Strategy Evaluations for AGN Time-series Data in Wide-Fast-Deep Field

Sheng,

Ross,

Nicholl

2022

Preprint

View full text Add to dashboard Cite

Machine learning is a promising tool to reconstruct time-series phenomena, such as variability of active galactic nuclei (AGN), from sparsely-sampled data. Here we use three Continuous Auto-Regressive Moving Average (CARMA) representations of AGN variability -the Damped Random Walk (DRW) and (over/under-)Damped Harmonic Oscillator (DHO) -to simulate 10-year AGN light curves as they would appear in the upcoming Vera Rubin Observatory Legacy Survey of Space and Time (LSST), and provide a public tool to generate these for any survey cadence. We investigate the impact on AGN science of five proposed cadence strategies for LSST's primary Wide-Fast-Deep (WFD) survey. We apply for the first time in astronomy a novel Stochastic Recurrent Neural Network (SRNN) algorithm to reconstruct input light curves from the simulated LSST data, and provide a metric to evaluate how well SRNN can help recover the underlying CARMA parameters. We find that the light curve reconstruction is most sensitive to the duration of gaps between observing season, and that of the proposed cadences, those that change the balance between filters, or avoid having long gaps in the g-band perform better. Overall, SRNN is a promising means to reconstruct densely sampled AGN light curves and recover the long-term Structure Function of the DRW process (SF ∞ ) reasonably well. However, we find that for all cadences, CARMA/SRNN models struggle to recover the decorrelation timescale (τ ) due to the long gaps in survey observations. This may indicate a major limitation in using LSST WFD data for AGN variability science.

show abstract

Handling Missing Values and Unusual Observations in Statistical Downscaling Using Kalman Filter

Cited by 11 publications

References 10 publications

Time Series Analysis of Landsat Data for Investigating the Relationship between Land Surface Temperature and Forest Changes in Paphos Forest, Cyprus

Time Series Analysis of Landsat Data for Investigating the Relationship between Land Surface Temperature and Forest Changes in Paphos Forest, Cyprus

Understanding the source components captured by the Purple Air Network

LSST Cadence Strategy Evaluations for AGN Time-series Data in Wide-Fast-Deep Field

Contact Info

Product

Resources

About