Streaming Sensor Data Validation in Networked Infrastructure Systems through Synergic Auto and Cross Similarity Discovery and Analysis

Xing, Lu; Rasekh, Amin; Shafiee, M. Ehsan; Sela, Lina; Preis, Ami

doi:10.1061/9780784482445.044

Cited by 1 publication

(1 citation statement)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Looking toward a future in which the management of environmental systems is driven by vast quantities of real-time data, it is no longer feasible for the QAQC process to be entirely manual. ,, A number of approaches have been proposed to detect environmental sensor data faults. These approaches largely fall into three categories, (1) heuristic rule-based tests, ,− (2) ML and purely statistical methods, ,, and (3) multisensor comparison. − Some studies have applied a combination of these. − These existing methods are often unsuitable for use on large-scale, low-resource sensor networks, however, as they are either expensive, labor-intensive to scale, or anticipate a minimum signal-to-noise ratio to which real-world sensors may not adhere to. In recent years, most QAQC approaches have been built around data that are reliable enough to be cleaned to begin with.…”

Section: Introductionmentioning

confidence: 99%

Machine Learning-Assisted, Process-Based Quality Control for Detecting Compromised Environmental Sensors

Schmidt,

Kerkez

2023

Environ. Sci. Technol.

View full text Add to dashboard Cite

Machine learning (ML) techniques promise to revolutionize environmental research and management, but collecting the necessary volumes of high-quality data remains challenging. Environmental sensors are often deployed under harsh conditions, requiring labor-intensive quality assurance and control (QAQC) processes. The need for manual QAQC is a major impediment to the scalability of these sensor networks. Existing techniques for automated QAQC make strong assumptions about noise profiles in the data they filter that do not necessarily hold for broadly deployed environmental sensors, however. Toward the goal of increasing the volume of high-quality environmental data, we introduce an ML-assisted QAQC methodology that is robust to low signal-to-noise ratio data. Our approach embeds sensor measurements into a dynamical feature space and trains a binary classification algorithm (Support Vector Machine) to detect deviation from expected process dynamics, indicating whether a sensor has become compromised and requires maintenance. This strategy enables the automated detection of a wide variety of nonphysical signals. We apply the methodology to three novel data sets produced by 136 low-cost environmental sensors (stream level, drinking water pH, and drinking water electroconductivity), deployed by our group across 250,000 km 2 in Michigan, USA. The proposed methodology achieved accuracy scores of up to 0.97 and consistently outperformed state-of-the-art anomaly detection techniques.

show abstract