Advanced monitoring of water systems using in situ measurement stations: data validation and fault detection

Abstract

In situ continuous monitoring at high frequency is used to collect water quality information about water bodies. However, it is crucial that the collected data be evaluated and validated for the appropriate interpretation of the data so as to ensure that the monitoring programme is effective. Software tools for data quality assessment with a practical orientation are proposed. As water quality data often contain redundant information, multivariate methods can be used to detect correlations, pertinent information among variables and to identify multiple sensor faults. While principal component analysis can be used to reduce the dimensionality of the original variable data set, monitoring of some statistical metrics and their violation of confidence limits can be used to detect faulty or abnormal data and can help the user apply corrective action(s). The developed algorithms are illustrated with automated monitoring systems installed in an urban river and at the inlet of a wastewater treatment plant.

Characterizing Long-term Wear and Tear of Ion-Selective pH Sensors

“…By several accounts, the lack of online sensor data quality poses a longstanding challenge for both the advancement of environmental science and engineering practice (Rieger et al, 2005(Rieger et al, , 2006Rosén et al, 2008;Rieger et al, 2010;Haimi et al, 2013;Corominas et al, 2018). It is therefore not surprising that considerable time and energy has been invested in methods for automated quality assessment and quality control of online measurement devices (e.g., Thomann et al, 2002;Thomann, 2008;Corominas et al, 2011;Spindler and Vanrolleghem, 2012;Alferes et al, 2013;Spindler, 2014;Villez and Habermacher, 2016;Le et al, 2018).…”

Flow Forecasting using Deterministic Updating of Water Levels in Distributed Hydrodynamic Urban Drainage Models

“…Since measured water levels are assimilated directly into the model, measurement uncertainties are however transferred to the model. This means that the measurements have to be of good quality, and the use of updating should thus preferably be combined with automated quality control of the measurements, where the raw data from routine monitoring programs are filtered using heuristic or statistical methods e.g., [31] prior to insertion into the Update scheme. Even when the measurements are perfect, direct updating of water levels should be used with care.…”

Validating data quality during wet weather monitoring of wastewater treatment plant influents

“…This requires more complex validation methods using more signals through the application of cross validation or multivariable methods that consider the correlation between different variables (Alferes et al, 2013;Villez et al, 2008). If more sensors are not available, it is also possible to configure software sensors as a real time calculation based on one or more sensors (Lynggaard-Jensen and Lading, 2006;Spindler and Vanrolleghem, 2012).…”

“…Using the online time series information the method developed by Alferes et al (2013) integrates two main steps: outlier handling and fault detection. Since the presence of outliers can seriously affect the results of statistical tests on the data by altering, for example, the variance, the mean and the normality of the data set, they must be first detected and removed to avoid faulty conclusions.…”