Error measurement in craniometrics: The comparative performance of four popular assessment methods using 2000 simulated cranial length datasets (g-op)

Abstract

For measurements to be accurate and precise, measurement errors should be small. In the anthropometry and craniofacial identification literature, four methods are commonly used for assessing measurement error: Pearson's product moment correlation coefficient (r), intra-class correlation coefficients (ICC), statistical significance tests (often reported by P-values) and the technical error of measurement (TEM; also known as Dalberg's error/ratio). In this paper, the performance of all four of these statistics were evaluated using maximum cranial lengths (g-op) from Howells (n=2524), by duplicating the dataset and mathematically adding known degrees of error to the second set. This was repeated under a broad array of trials (2000 total) each with slightly different amounts of error simulation to comprehensively assess the four error metrics in terms of descriptive power and utility, using the same data for each of the four error assessment methods. Data simulations included the addition of random and systematic errors of different sizes with absolute differences ranging from 1 to 50mm (or in relative terms, 28% of the original measurement). Two sample sizes (n=25 and 2524 individuals) were explored and all analyses were conducted in R. P-values from Student's t-tests only showed significant differences (P<0.05) for the larger sample size when the error was systematic. Small samples, and/or any with random error, did not yield low or significant P-values (P<0.05). When raw differences were <4mm for 95% of the sample (n=2524), the ICC and r were high (>0.97) and remained so even after tripling the error, such that 95% of the sample possessed raw differences up to 12mm (r=0.8). In contrast, the TEM was low initially (<2mm or r-TEM<1%), and then increased (<4.5mm and 2.5%, TEM and r-TEM respectively). These data show that P-values, ICC and r values hold substantial limits for error description as they do not always flag error well. In contrast, TEM appears to covary with error more saliently and holds the advantage that changes are reported in the units of the original measurement. For these reasons, TEM is recommended in favour to P-values, ICC and r.

Facial soft tissue thicknesses in craniofacial identification: Data collection protocols and associated measurement errors

“…Not surprisingly then, studies employing t-tests (including Hodson's first use for FSTT measurement errors in 1985) rarely find statistically significant differences between repeated measurement sessions [29,32,33,39,40,49,73,91,122]. Counter to what prior studies claim, the lack of statistical significance following t-testing is not a reliable indicator of little or negligible error because the test is insensitive to error normally distributed about a mean of zero [168].…”

“…<Table 2 about here> Only 5% of studies use two separate data acquisition sessions to assess the full measurement protocol and utilise the technical error of measurement (TEM) for error calculation [168]. These studies pertain only to needle puncture [65,67] and B-mode ultrasound [87,88].…”

“…It is important to note that in some cases, entirely inappropriate statistical methods have been used to assess the measurement error, e.g., one way ANOVA for independent (not dependant) samples [53] and a surprisingly large number of cephalogram studies attempt to report measurement errors using ttests [29,32,33,39,40,49,73,91,122]. Student's t-tests should be avoided when examining measurement errors since these tests will only identify systematic error, not normally distributed errors about a mean of zero [168], which most of the measurement error is likely to be. Not surprisingly then, studies employing t-tests (including Hodson's first use for FSTT measurement errors in 1985) rarely find statistically significant differences between repeated measurement sessions [29,32,33,39,40,49,73,91,122].…”

The utility of elliptical Fourier analysis for estimating ancestry and sex from lateral skull photographs

“…Consequently, standard protocols typically recommend employment of these methods in the laboratory (see e.g., Defense POW/MIA Accounting Agency (DPAA) Laboratory Manual, SOP 3.4 [23]). This is not problematic if the laboratory is relatively close to the field site, and indeed in many instances it is favorable since laboratory conditions often favor higher quality assurance with well-lit conditions for examinations [24]. However, laboratory examinations may not be ideal in all contexts, e.g., where transport of remains occurs across large distances and where early intervention could avoid unnecessary and large expenses of returning skeletal material at a later date when determined not to be of interest for identification.…”

“…Inter-observer reproducibility was also assessed by an anthropologist (CS), independent of the primary analyst (JC). The technical error of measurement (TEM) and its relative equivalent (rTEM) were calculated for these error tests [14,24,48,49]. To reduce the number of variables assessed for this part of the study,…”