Efficient scalable schemes for monitoring a large number of data streams

Mei, Yajun

doi:10.1093/biomet/asq010

Cited by 236 publications

(227 citation statements)

References 16 publications

Supporting

Mentioning

223

Contrasting

Order By: Relevance

“…Here we assume that the online monitoring is conducted under the unstructured environment in the sense that we do not make any assumptions to relate the occurring event to the local data streams, see Tartakovsky et al [36], Mei [21], and Xie and Siegmund [40]. Also see Lévy-Leduc and Roueff [16] for an application of the unstructured problem to anomaly detection in computer networks.…”

Section: Preliminaries and Backgroundmentioning

confidence: 99%

“…Below we will call the scheme in (4) the "MAX" scheme. The second method is the "SUM" scheme developed in Mei [21] that is defined by the stopping time…”

Section: Preliminaries and Backgroundmentioning

confidence: 99%

“…Existing methods include MAX-scheme (which uses the maximum of local CUSUM statistics as the global statistic, see Tartakovsky et al [36]), SUM-scheme (which uses the sum of local CUSUM statistics as the global statistic, see Mei [21]), the mixtureschemes proposed in Xie and Siegmund [40], and the simultaneous-estimation-based schemes in Wang and…”

Section: Introductionmentioning

confidence: 99%

“…While the parallel local monitoring approach sounds interesting, one allegation often made is that we will lose much information at the global level if we combine local detection procedures, not raw observation themselves, to make a global decision. Indeed, as mentioned earlier, there are two existing methods that combine local detection procedures together: the MAX and SUM schemes that use the maximum or sum of local CUSUM to raise a global alarm, but both methods are known to be inefficient when the number of affected data streams is moderate, see Mei [21] and Xie and Siegmund [40].…”

Section: Introductionmentioning

confidence: 99%

“…Our key idea is to generalize the SUM scheme in Mei [21] by introducing the shrinkage function to local detection statistics in the hope of filtering out those unchanging local data streams. We would like to acknowledge that there might be inherent loss of statistical efficiency in the parallel local monitoring approach as compared to the (non-recursive) global monitoring approach that uses all raw observations, e.g., see Section 5 for the comparison of our proposed schemes with those in Xie and…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Scalable SUM-Shrinkage Schemes for Distributed Monitoring Large-Scale Data Streams

Liu¹,

Zhang²,

Mei

2018

STAT SINICA

View full text Add to dashboard Cite

In this article, we investigate the problem of monitoring independent large-scale data streams where an undesired event may occur at some unknown time and affect only a few unknown data streams. Motivated by parallel and distributed computing, we propose to develop scalable global monitoring schemes by parallel running local detection procedures and by using the sum of the shrinkage transformation of local detection statistics as a global statistic to make a decision. Our approach is illustrated in two concrete examples: one is censoring sensor networks when local sensors can be nonhomogeneous with completely specified local post-change distributions, and the other is the problem of monitoring normally distributed data streams when the local post-change mean shifts are unknown and can be positive or negative.Numerical simulation studies demonstrate the usefulness of the proposed SUM-Shrinkage schemes.

show abstract

Section: Preliminaries and Backgroundmentioning

confidence: 99%

“…Below we will call the scheme in (4) the "MAX" scheme. The second method is the "SUM" scheme developed in Mei [21] that is defined by the stopping time…”

Section: Preliminaries and Backgroundmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Scalable SUM-Shrinkage Schemes for Distributed Monitoring Large-Scale Data Streams

Liu¹,

Zhang²,

Mei

2018

STAT SINICA

View full text Add to dashboard Cite

show abstract

Multisensor change detection on the basis of big time‐series data and Dempster‐Shafer theory

Jafari

Qian

et al. 2016

Concurrency and Computation

View full text Add to dashboard Cite

Summary With the proliferation of the Internet of Things, numerous sensors are deployed to monitor a phenomenon that in many cases can be modeled by an underlying stochastic process. The goal is to detect change in the process with tolerable false alarm rate. In practice, sensors may have different accuracy and sensitivity range, or they decay along time. As a result, the sensed data will contain uncertainties and sometimes they are conflicting. In this study, we propose a novel framework to take advantage of Dempster‐Shafer theory's capability of representation of uncertainty to detect change and effectively deal with complementary hypotheses. Specifically, Kullback‐Leibler divergence is used as the metric to find the distances between the estimated distribution with the before and after change distributions. Mass functions are calculated on the basis of those distance values for each sensor independently, and Dempster‐Shafer combination rule is applied to combine the mass values among all sensors. In the case of high conflict in various sensor readings, Dezert‐Smarandache combination rule is applied, and the belief, plausibility, and pignistic probability are obtained for decision making. Simulation results using both synthetic data and real data demonstrate the effectiveness of the proposed schemes.

show abstract

Fault classification for high‐dimensional data streams: A directional diagnostic framework based on multiple hypothesis testing

Xiang

Tsung

et al. 2021

Naval Research Logistics

View full text Add to dashboard Cite

In various modern statistical process control applications that involve high‐dimensional data streams (HDDS), accurate fault diagnosis of out‐of‐control (OC) streams is becoming crucial. The existing diagnostic approaches either focus on moderate‐dimensional processes or are unable to determine the shift direction accurately, especially when the signal‐to‐noise ratio is low. In this paper, we conduct a bold trial and consider the fault classification problem of the mean vector of HDDS where determining the shift direction of the OC streams is important to perform customized repairs. To this end, under the basic assumptions that the in‐control data streams are normal with mean 0 and variance 1, and that the high‐dimensional observations after the alarm are solely OC, the problem is formulated into a three‐classification multiple testing framework, and an efficient data‐driven diagnostic procedure is developed to minimize the expected number of false positives and to control the missed discovery rate at given level. The procedure is statistically optimal and computationally efficient, and improves the diagnostic effectiveness by considering directional information, which provides insights to guide further decisions. Both theoretical and numerical results reveal the superiority of the new method.

show abstract

Efficient scalable schemes for monitoring a large number of data streams

Cited by 236 publications

References 16 publications

Scalable SUM-Shrinkage Schemes for Distributed Monitoring Large-Scale Data Streams

Scalable SUM-Shrinkage Schemes for Distributed Monitoring Large-Scale Data Streams

Multisensor change detection on the basis of big time‐series data and Dempster‐Shafer theory

Fault classification for high‐dimensional data streams: A directional diagnostic framework based on multiple hypothesis testing

Contact Info

Product

Resources

About