Standardized Measurement Error: A Universal Measure of Data Quality for Averaged Event-Related Potentials (v_r1)

Abstract

Event-related potentials (ERPs) can be very noisy, and yet there is no widely accepted metric of ERP data quality. Here we propose a universal measure of data quality for ERP research: the standardized measurement error (SME). Whereas some potential measures of data quality provide a generic quantification of the noise level, the SME quantifies the expected error in the specific amplitude or latency value being measured in a given study (e.g., the peak latency of the P3 wave). It can be applied to virtually any value that is derived from averaged ERP waveforms, making it a universal measure of data quality. In addition, the SME quantifies the data quality for each individual participant, making it possible to identify participants with low- quality data and “bad” channels. When appropriately aggregated across individuals, SME values can be used to quantify the impact of the single-trial EEG variability and the number of trials being averaged together on the effect size and statistical power in a given experiment. If SME values were regularly included in published papers, researchers could identify the recording and analysis procedures that produce the highest data quality, which could ultimately lead to increased effect sizes and greater replicability across the field. Thus, the SME is a both a universal and useful metric of ERP data quality.

Data Quality and Reliability Metrics for Event-Related Potentials (ERPs): The Utility of Subject-Level Reliability

“…As acknowledged by Luck et al (2020), it is difficult to know what constitutes a "small enough" SME score (and between-trial standard deviation, for our purposes). Comparing SME estimates among participants can shed light on which participants might have poor data quality relative to other participants within a group.…”

“…The SME was recently proposed as an estimate of data quality (Luck et al, 2020). The SME is computed for a mean ERP amplitude scoring approach (e.g., average ERP activity between 0 and 100 ms locked to a response) by using the formula for the standard error of the mean; however, as recommended by Luck et al (2020), this estimate is to be referred to as the SME when quantifying data quality (SME is an umbrella term for the data quality estimates recommended in Luck et al, 2020). We provide the formula for the standard error of the mean below, which is referred to as SME (notation was changed to match the current manuscript's notations; Luck et al, 2020).…”

“…An advantage of SME is that it provides an estimate of data quality specific to the ERP measurement approach used (e.g., the mean amplitude from a specified time window for the scores later used in statistical analysis). The present manuscript focuses solely on the estimation of SME for mean amplitude approaches for the sake of simplicity (for application of SME to other measurement approaches see Luck et al, 2020).…”

“…Recently, Luck, Stewart, Simmons, and Rhemtulla (2020) proposed a metric referred to as the standardized measurement error (SME) that can capture how noisy a single subject's ERP score is. Unlike conventional classical test theory measures, the SME can be applied at both the subject-and group-level to determine the impact of noise on ERP scores.…”

“…Unlike conventional classical test theory measures, the SME can be applied at both the subject-and group-level to determine the impact of noise on ERP scores. However, despite the advantages of the SME as an objective index of data quality (see Luck et al, 2020), the SME provides no inherent information about whether measurement precision is "high enough" for a particular purpose (e.g., comparison of ERP scores across conditions, persons, or groups).…”