Modified repeated median filters

Abstract

We discuss moving window techniques for fast extraction of a signal comprising monotonic trends and abrupt shifts from a noisy time series with irrelevant spikes. Running medians remove spikes and preserve shifts, but they deteriorate in trend periods. Modified trimmed mean filters use a robust scale estimate such as the median absolute deviation about the median (MAD) to select an adaptive amount of trimming. Application of robust regression, particularly of the repeated median, has been suggested for improving upon the median in trend periods. We combine these ideas and construct modified filters based on the repeated median offering better shift preservation. All these filters are compared w.r.t. fundamental analytical properties and in basic data situations. An algorithm for the update of the MAD running in time O(log n) for window width n is presented as well.

Robust Detail‐Preserving Signal Extraction

“…Instabilities have not been observed, although RM-filters are not Lipschitz-continuous because of the hard thresholding [8]. A TRM-filter can be substantially more efficient than the RM with the same width n, depending on the amount of trimming.…”

“…Neither can they preserve arbitrary shifts during trends, nor can they remove spikes completely, not even under idealized conditions. Only DWMTMs can keep their good properties during trends if the inner window is sufficiently short [18,8,29].…”

“…Trimmed repeated median (TRM)-filters suggested in [8] use the RM in the initial step and apply least squares to the trimmed observations in a second step. A suitable trimming constant q t can be obtained by estimating the variability about the RM regression line e.g.…”

Methods and Algorithms for Robust Filtering

“…Fast algorithms for the update of the filter output are needed for online signal extraction. Denoting the length of the time window by n, the median of the proceeding window can be updated in logarithmic time (O(log n)) using linear space if the data in the window are stored in sorted order using Bernholt and Fried (2003), and another update algorithm needing only linear space running in O(n log n) average time is presented by Fried, Bernholt and Gather (2004).…”

“…For the MRM, however, O(n log n) time is needed at least for the second repeated median. Detailed descriptions of the update algorithms can be found in Bernholt et al (2004) and Fried, Bernholt and Gather (2004). Table 1 summarizes the time and the space needed for the updates of the filtering procedures.…”