Reliability and validity of expert assessment based on airborne and urinary measures of nickel and chromium exposure in the electroplating industry

Abstract

The reliability and validity of six experts’ exposure ratings were evaluated for 64 nickel-exposed and 72 chromium-exposed workers from six Shanghai electroplating plants based on airborne and urinary nickel and chromium measurements. Three industrial hygienists and three occupational physicians independently ranked the exposure intensity of each metal on an ordinal scale (1–4) for each worker's job in two rounds: the first round was based on responses to an occupational history questionnaire and the second round also included responses to an electroplating industry-specific questionnaire. Spearman correlation (rs) was used to compare each rating's validity to its corresponding subject-specific arithmetic mean of four airborne or four urinary measurements. Reliability was moderately-high (weighted kappa range=0.60–0.64). Validity was poor to moderate (rs= -0.37–0.46) for both airborne and urinary concentrations of both metals. For airborne nickel concentrations, validity differed by plant. For dichotomized metrics, sensitivity and specificity were higher based on urinary measurements (47–78%) than airborne measurements (16–50%). Few patterns were observed by metal, assessment round, or expert type. These results suggest that, for electroplating exposures, experts can achieve moderately-high agreement and (reasonably) distinguish between low and high exposures when reviewing responses to in-depth questionnaires used in population-based case-control studies.

Agreement of Experts and Non-Experts in a Desktop Exercise Evaluating Exposure to Asthmagens in the Cotton and Textile, and Other Industries

“…Overall, agreement (ICC1 and ICC3) between expert assessors was fair to good and was generally better than the agreement of non-experts, although CIs largely overlapped. This is, in general, comparable with that found in other studies (Goldberg et al, 1986;Kromhout et al, 1987;Macaluso et al, 1993;de Cock et al, 1996;Stewart et al, 2000;Mannetje et al, 2003;Walker et al, 2003;Steinsvag et al, 2007Steinsvag et al, , 2008Chen et al, 2014), although these also showed that agreement differs between agents, industries, and individual experts, which is something we could take into account to a small degree (e.g. cotton industry versus others).…”

“…These results further indicate that agreement between assessors does not increase with the amount of information available to the rater. This has been observed previously in a number of industries (Post et al, 1991;de Cock et al, 1996;Stewart et al, 2000;Friesen et al, 2011;Chen et al, 2014), but is not universally true as for general population studies, small improvements have been observed when more information became available to the raters (Tielemans et al, 1999;Mannetje et al, 2003). What is of specific interest in this study, however, is the clear dip in agreement for medium information (compared with little and much detail) that can be observed especially for the experts but also for the non-experts, which has not been observed before.…”

“…However, simulated sensitivity and specificity in our study are likely inflated because of the high estimated prevalence of exposure in our study. Other occupational studies have reported lower sensitivity but varying specificity (Steinsvag et al, 2008;Chen et al, 2014), but community-based studies, with a much lower prevalence of specific exposures, indicated similar specificity but much lower (~0.3) sensitivity than reported in our study (Benke et al, 1997;Tielemans et al, 1999).…”

“…Estimated sensitivity of this desktop exercise was very good (>90%) and better than that shown previously (Steinsvag et al, 2008;Chen et al, 2014). However, this can be ascribed to the fact that sensitivity is a function of agreement (ICC) and the prevalence of exposure, with high prevalence of exposure in these selected job descriptions.…”

“…Furthermore, the methodology is based on the assumption that the validity of the assessments is better than chance, which has been shown to be the case for relative ranking in our study and others (Post et al, 1991;Friesen et al, 2011). However, measurement data were not available so accuracy could not be directly assessed, whereas previous studies have indicated that the assumption of 'better than chance' may not necessarily be correct (Logan et al, 2009;Vadali et al, 2012;Chen et al, 2014).…”