Performance Metrics for Surveillance Schemes

Fraker, Shannon E.; Woodall, William H.; Mousavi, Shabnam

doi:10.1080/08982110701810444

Cited by 46 publications

(43 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Metrics used in the industrial literature to evaluate control charts, such as average run length, are specifically designed for detection of a sustained shift in a parameter [43], which corresponds to the flat outbreak shape simulated in this study, but would be misleading when used to interpret the algorithms' performance for other outbreak scenarios. Therefore, although at times recommended for the evaluation of prospective statistical surveillance [44], performance measures from the industrial literature were not used [43].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Syndromic surveillance using veterinary laboratory data: data pre-processing and algorithm performance evaluation

Dórea

McEwen

McNab

et al. 2013

J. R. Soc. Interface.

View full text Add to dashboard Cite

Diagnostic test orders to an animal laboratory were explored as a data source for monitoring trends in the incidence of clinical syndromes in cattle. Four years of real data and over 200 simulated outbreak signals were used to compare pre-processing methods that could remove temporal effects in the data, as well as temporal aberration detection algorithms that provided high sensitivity and specificity. Weekly differencing demonstrated solid performance in removing day-of-week effects, even in series with low daily counts. For aberration detection, the results indicated that no single algorithm showed performance superior to all others across the range of outbreak scenarios simulated. Exponentially weighted moving average charts and Holt-Winters exponential smoothing demonstrated complementary performance, with the latter offering an automated method to adjust to changes in the time series that will likely occur in the future. Shewhart charts provided lower sensitivity but earlier detection in some scenarios. Cumulative sum charts did not appear to add value to the system; however, the poor performance of this algorithm was attributed to characteristics of the data monitored. These findings indicate that automated monitoring aimed at early detection of temporal aberrations will likely be most effective when a range of algorithms are implemented in parallel.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Therefore, although at times recommended for the evaluation of prospective statistical surveillance [44], performance measures from the industrial literature were not used [43].…”

Section: Discussionmentioning

confidence: 99%

Syndromic surveillance using veterinary laboratory data: data pre-processing and algorithm performance evaluation

Dórea

McEwen

McNab

et al. 2013

J. R. Soc. Interface.

View full text Add to dashboard Cite

show abstract

“…We then define: -CCD = T /C, the proportion of changepoints correctly detected -DN F = T /D, the proportion of detections that are not false detections These intuitive definitions are the same as sensitivity and positive predicted value (PPV ) in the surveillance literature (German et al 2001;Fraker et al 2008), and are equivalent to recall and precision, respectively, in the pattern recognition literature. Similar metrics are discussed in Kifer et al (2004).…”

Section: Detection Ratesmentioning

confidence: 99%

Continuous monitoring for changepoints in data streams using adaptive estimation

Bodenham

Adams

2016

Stat Comput

View full text Add to dashboard Cite

Data streams are characterised by a potentially unending sequence of high frequency observations which are subject to unknown temporal variation. Many modern streaming applications demand the capability to sequentially detect changes as soon as possible after they occur, while continuing to monitor the stream as it evolves. We refer to this problem as continuous monitoring. Sequential algorithms such as CUSUM, EWMA and their more sophisticated variants usually require a pair of parameters to be selected for practical application. However, the choice of parameter values is often based on the anticipated size of the changes and a given choice is unlikely to be optimal for the multiple change sizes which are likely to occur in a streaming data context. To address this critical issue we introduce a changepoint detection framework based on adaptive forgetting factors that, instead of multiple control parameters, only requires a single parameter to be selected. Simulated results demonstrate that this framework has utility in a continuous monitoring setting. In particular, it reduces the burden of selecting parameters in advance. Moreover, the methodology is demonstrated on real data arising from Foreign Exchange markets.

show abstract

“…This metric should be distinguished from the conditional expected delay (CED) metric for which the conditioning is on a process shift occurring at a particular time in Phase II. For more information on the CED, see Fraker, Woodall, and Mousavi (2008) or Kenett and Pollack (2012). In our article we consider only in-control Phase II performance.…”

Section: Introductionmentioning

confidence: 99%

A head-to-head comparative study of the conditional performance of control charts based on estimated parameters

Zwetsloot

Woodall

2016

Quality Engineering

Self Cite

View full text Add to dashboard Cite

General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons). Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: http://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible. ABSTRACTImplementation of the Shewhart, CUSUM, and EWMA charts requires estimates of the in-control process parameters. Many researchers have shown that estimation error strongly influences the performance of these charts. However, a given amount of estimation error may differ in effect across charts. Therefore, we perform a pairwise comparison of the effect of estimation error across these charts. We conclude that the Shewhart chart is more strongly affected by estimation error than the CUSUM and EWMA charts. Furthermore, we show that the general belief that the CUSUM and EWMA charts have similar performance no longer holds under estimated parameters.

show abstract

Performance Metrics for Surveillance Schemes

Cited by 46 publications

References 27 publications

Syndromic surveillance using veterinary laboratory data: data pre-processing and algorithm performance evaluation

Syndromic surveillance using veterinary laboratory data: data pre-processing and algorithm performance evaluation

Continuous monitoring for changepoints in data streams using adaptive estimation

A head-to-head comparative study of the conditional performance of control charts based on estimated parameters

Contact Info

Product

Resources

About