Quantile Sampling for Practical Delay Monitoring in Internet Backbone Networks

Abstract

Point-to-point delay is an important network performance measure as it captures service degradations caused by various events. We study how to measure and report delay in a concise and meaningful way for an ISP, and how to monitor it efficiently. We analyze various measurement intervals and potential metric definitions. We find that reporting high quantiles (between 0.95 and 0.99) every 10-30 minutes as the most effective way to summarize the delay in an ISP. We then propose an active probing scheme to estimate a high quantile with bounded error. We show that only a small number of probes are sufficient to provide an accurate estimate. We validate the proposed delay monitoring technique on real data collected on the Sprint IP backbone network. To make our work complete, we lastly compare the overhead of our active probing technique with a passive sampling scheme and show that for delay measurement, active probing is more practical.

Tracking of multiple quantiles in dynamically varying data streams

“…MDUMIQE is required to satisfy the monotone property which sets a limitation on how far MDUMIQE can update the estimates in each iteration. More specifically, for MDUMIQE, the estimate Q n (q k ) must always be between Q n (q k−1 ) and Q n (q k+1 ), recall (6). Whenever the difference between Q n (q k−1 ) and Q n (q k+1 ) is small, we can only do small updates of Q n (q k ).…”

“…Next, we turn to assumption 5 which requires that w Q n (q k ) has a Lipschitz derivative with respect to Q n (q k ). Unfortunately it is not obvious that this is satisfied since H has a discontinuous derivative with respect to Q n (q k ) due to the min-function in (6). To show that both assumptions 2 and 5 are satisfied, we need to perform a subtle modification of (12) as follows.…”

“…In this paper we consider the problem of estimating quantiles when data arrive sequentially (data stream). The problem has been considered for many applications like portfolio risk measurement in the stock market [10,1], fraud detection [19], signal processing and filtering [16], climate change monitoring [20], SLA violation monitoring [14,15] and back-bone network monitoring [6].…”

A Novel Incremental Quantile Estimator Using the Magnitude of the Observations

“…In this paper we consider the problem of estimating quantiles when data arrive sequentially (data stream). The problem has been considered for many applications like portfolio risk measurement in the stock market [10], [1], fraud detection [29], signal processing and filtering [25], climate change monitoring [30], SLA violation monitoring [23], [24] and back-bone network monitoring [8].…”

A new quantile tracking algorithm using a generalized exponentially weighted average of observations

“…In this paper we consider the problem of estimating quantiles of streaming data. Streaming quantile estimation has been considered for a wide range of applications like portfolio risk measurement in the stock market [15,1], fraud detection [40], signal processing and filtering [32], climate change monitoring [41], SLA violation monitoring [30,31], network monitoring [7,22], Monte Carlo simulation [36], structural health monitoring [16] and non-parametric statistical testing [21],…”