Toward automating post processing of aquatic sensor data

Machine Learning-Assisted, Process-Based Quality Control for Detecting Compromised Environmental Sensors

“…Classification on phase portraits also outperformed classification on value distributions for the water level and pH data sets (+3% and +2% accuracy, respectively). While the ARIMA time series-based approach has been reported to be the state of the art anomaly detection method for environmental sensor data, 25 it had the lowest validation performance on the data sets used in this study, performing worse than the rule-based test baseline. The Support Vector Machine (SVM) classification-based model was found to have the best overall performance across the three data sets.…”

“…The performance of classification on phase portraits was compared with four other anomaly detection strategies. These include two established time series-based approaches: moving average (MA) and autoregressive integrated moving average (ARIMA) residual analysis (the state of the art). Furthermore, we investigated the value of the phase portrait feature space by trialing our supervised classification approach on a simpler feature space consisting of a 1D histogram of value ( X i = [ x ( t )]).…”

“…Looking toward a future in which the management of environmental systems is driven by vast quantities of real-time data, it is no longer feasible for the QAQC process to be entirely manual. ,, A number of approaches have been proposed to detect environmental sensor data faults. These approaches largely fall into three categories, (1) heuristic rule-based tests, ,− (2) ML and purely statistical methods, ,, and (3) multisensor comparison. − Some studies have applied a combination of these. − These existing methods are often unsuitable for use on large-scale, low-resource sensor networks, however, as they are either expensive, labor-intensive to scale, or anticipate a minimum signal-to-noise ratio to which real-world sensors may not adhere to. In recent years, most QAQC approaches have been built around data that are reliable enough to be cleaned to begin with.…”

Advances in Catchment Science, Hydrochemistry, and Aquatic Ecology Enabled by High-Frequency Water Quality Measurements

“…Because of the complex interactions between hydrochemical parameters measured at high-frequency, machine learning approaches are often used to identify patterns in high-frequency data or to detect any data interdependencies, through unsupervised or supervised learning . Machine learning approaches have been used to automate identification and correction of anomalies in high-frequency water quality data and estimate nutrient concentrations from high-frequency UV–vis absorbance, fluorescence, DO, specific conductivity, and turbidity measurements. , Fewer studies have used machine learning approaches to infer information about patterns and processes from high-frequency water quality data to date. For example, Bieroza and Heathwaite successfully used a fuzzy logic system to determine the direction of the storm event c - q patterns based on a learning data set including volume of flow discharge and mean air temperature during storm events.…”

“…The streamPULSE platform facilitates stream metabolism modeling through providing consistent approaches to sensor data collection and protocols for data quality assurance and control and stream metabolism modeling . Additionally, several freely available toolboxes designed to analyze high-frequency water data have been released in the past years, including the R packages oddwater developed to detect outliers in WQ data from in situ sensors, waterData which calculates and plots anomalies, ensemble hydrograph separation scripts, and EndSplit for end-member splitting analysis and Python packages AbspectroscoPY to analyze UV–vis sensor data and pyhydroqc for automating detection and correction of anomalies in sensor data …”