Data-Driven Drift Detection in Real Process Tanks: Bridging the Gap between Academia and Practice

Abstract

Sensor drift in Wastewater Treatment Plants (WWTPs) reduces the efficiency of the plants and needs to be handled. Several studies have investigated anomaly detection and fault detection in WWTPs. However, these solutions often remain as academic projects. In this study, the gap between academia and practice is investigated by applying suggested algorithms on real WWTP data. The results show that it is difficult to detect drift in the data to a sufficient level due to missing and imprecise logs, ad hoc changes in control settings, low data quality and the equality in the patterns of some fault types and optimal operation. The challenges related to data quality raise the question of whether the data-driven approach for drift detection is the best solution, as this requires a high-quality data set. Several recommendations are suggested for utilities that wish to bridge the gap between academia and practice regarding drift detection. These include storing data and select data parameters at resolutions which positively contribute to this purpose. Furthermore, the data should be accompanied by sufficient logging of factors affecting the patterns of the data, such as changes in control settings.

Impact of instrumentation reliability on mainstream suspended partial denitrification–anammox (PdNA)

“…Fouling can physically damage the sensor surface and/or promote the formation of an additional layer of diffusion that interferes with the diffusion of target ions and promotes sensor drift (Ryu et al, 2020). Drifting, shown by Figure 3c, is the overestimation or underestimation of actual “grab” measurements that usually increases over time (Cecconi, 2020; Hansen et al, 2022; Papias et al, 2018).…”

“…Yet, it is evident that timely detection of sensor faults is also important. Thus, it would be worthwhile to implement fault detection and isolation methods, to attain and guarantee sensor and data quality reliability (Cecconi, 2020; Corominas et al, 2018; Hansen et al, 2022; Huang et al, 2019; Isermann, 1997; Mali & Laskar, 2020; Russo et al, 2020; Thomann et al, 2002; Thürlimann et al, 2018; Villez et al, 2008, 2011; Wang, Fan, et al, 2021). Alternatively, an investment in wet‐chemistry analyzers known to be more accurate and require less maintenance than ISE sensors (Barker, 2020; Bende‐Michl & Hairsine, 2010; Vanrolleghem & Lee, 2003) should be highly considered for future full‐scale PdNA systems.…”

“…The other option for resolution is anomaly and drifting detection through advance data analytics and machine learning, which presents several methods such as artificial neural networks, principal component analysis, Kalman filter-based methods, and denoising data processing algorithms (Cecconi, 2020;Corominas et al, 2018;Hansen et al, 2022;Huang et al, 2019;Isermann, 1997;Mali & Laskar, 2020;Russo et al, 2020;Thomann et al, 2002;Thürlimann et al, 2018;Villez et al, 2008Villez et al, , 2011Wang, Fan, et al, 2021). In spite of the diverse methods available, there are still barriers that prevent their utilization (Corominas et al, 2018;Hansen et al, 2022;Russo et al, 2020). In this study, ISE probes for ammonium and nitrate were used to measure nitrogen parameters that controlled aeration and carbon dosing in different zones of the pilot.…”

“…Traditionally, sensor issues are resolved by curve fitting, frequent calibration: one point or multipoint, matrix adjustments, cross validation, and routine cleaning and maintenance (Kaelin et al, 2008; Papias et al, 2018; Rieger et al, 2010). The other option for resolution is anomaly and drifting detection through advance data analytics and machine learning, which presents several methods such as artificial neural networks, principal component analysis, Kalman filter‐based methods, and denoising data processing algorithms (Cecconi, 2020; Corominas et al, 2018; Hansen et al, 2022; Huang et al, 2019; Isermann, 1997; Mali & Laskar, 2020; Russo et al, 2020; Thomann et al, 2002; Thürlimann et al, 2018; Villez et al, 2008, 2011; Wang, Fan, et al, 2021). In spite of the diverse methods available, there are still barriers that prevent their utilization (Corominas et al, 2018; Hansen et al, 2022; Russo et al, 2020).…”