A Two-Dimensionally Coincident Second Difference Cosmic Ray Spike Removal Method for the Fully Automated Processing of Raman Spectra

Abstract

Charge-coupled device detectors are vulnerable to cosmic rays that can contaminate Raman spectra with positive going spikes. Because spikes can adversely affect spectral processing and data analyses, they must be removed. Although both hardware-based and software-based spike removal methods exist, they typically require parameter and threshold specification dependent on well-considered user input. Here, we present a fully automated spike removal algorithm that proceeds without requiring user input. It is minimally dependent on sample attributes, and those that are required (e.g., standard deviation of spectral noise) can be determined with other fully automated procedures. At the core of the method is the identification and location of spikes with coincident second derivatives along both the spectral and spatiotemporal dimensions of two-dimensional datasets. The method can be applied to spectra that are relatively inhomogeneous because it provides fairly effective and selective targeting of spikes resulting in minimal distortion of spectra. Relatively effective spike removal obtained with full automation could provide substantial benefits to users where large numbers of spectra must be processed.

Development and Integration of Block Operations for Data Invariant Automation of Digital Preprocessing and Analysis of Biological and Biomedical Raman Spectra

“…4B) fully automated cosmic ray spike removal. 61 A major problem encountered with CSR is the unique identification of spikes to prevent the unintended modification of other spectral features.…”

“…A threshold is therefore established based on the average frequency of occurrence of cosmic ray spikes in spectra and the use of Poissonian statistics. 61 Although we have established such a threshold based on the spike frequency observed in our own experience, 61 it is detector dependent and also site dependent because the frequency of cosmic ray strikes increases with altitude. 123 Regardless, note that the threshold is independent of the analyte(s) being investigated, and hence no user input is required to determine it.…”

“…Automation Aspects. In a fully automated CSR method that we developed, 61 spikes are identified and located based on coincident second differences (discrete analog of the second derivative) along both the spectral and spatiotemporal dimensions of two-dimensional spectral datasets. A spike is considered present if the (negativegoing) second difference ''peak'' is more negative, along both dimensions, than 2.45 times the standard deviation of the (zero-mean) noise in the second difference spectrum.…”

“…This threshold is equivalent to a threshold of three times the standard deviation of the noise in the original spectrum because the noise in the second difference spectrum increases by a factor of ffiffiffi 6 p , but the absolute value of the spike increases by a factor of 2. 61 Thus, it is a nonarbitrary threshold in the sense of being based on the limit-of-detection (LOD) convention. 122 Thereafter, it is removed by adding one-half the (negative) second difference spike's intensity to the initially small as peak apices are being stripped, increase when peak centers are being stripped (red area), and decrease again as peak bases are being stripped.…”

“…61 (F) In contrast, after despiking with the commonly used 5 3 5 median filter, undesirable effects include the selective (glycogen, dark gray arrows) or widespread (phenylalanine,~1000 cm À1 ) modification of some peaks, as well as the removal of artifacts (light gray arrow).…”