Establishing Validity and Cross-Context Equivalence of Measures and Indicators

Abstract

Quantitative research depends on using measures to collect data that are valid (ie, reflect well the phenomena of interest) and perform equivalently across contexts. Demonstrating validity and cross-context equivalence requires specifically designed studies, but many such studies have problems that have limited their usefulness. This article explains validity and cross-context equivalence of measures (and important related concepts) and clarifies how to establish them. Validation is the process of determining whether a measure or indicator is suitable for providing useful analytical measurement for a given purpose and context. Cross-context equivalence means that a measure performs comparably across contexts. Four types of equivalence are construct, item, measurement, and scalar. Establishing validity and cross-context equivalence requires representing mathematically the errors (ie, imprecision, undependability, and inaccuracy) of a measure and using appropriate statistical methods to quantify these errors. Studies aiming to provide evidence about the validity of a measure need to clarify the purpose and context for use of that measure. Choose one of the two conceptual systems for validation; obtain data to establish the extent to which the measure is well constructed, reliable, and accurate; and use analytic methods beyond simple correlations to provide a basis for making reasoned judgment about whether the measure provides useful analytic measurement for the particular purpose(s) and context. Establishing accuracy of a measure requires having available other measures known to be accurate as comparators; in the case that no other measure understood to be more accurate is available, then the study will be able to establish agreement rather than validity.

Best Practices for Conducting and Interpreting Studies to Validate Self-Report Dietary Assessment Methods

“…78 Random error occurs in both directions from true intake 79 and leads to unreliable estimates. 29,78 A main source of random error in estimating usual dietary intake is day-to-day variation, 25 which primarily affects data collected using short-term measures such as 24HR and FR. 50,62 Random errors may also be due to variation in data collection parameters (eg, time of day or day of week of data collection) or biological or environmental factors (eg, variation in vitamin C content in relation to storage conditions).…”

“…A measure or indicator is valid if each of six criteria are met: 1) its construction is well-grounded in theory; 2) its performance is consistent with that theory; it is 3) precise, 4) dependable, and 5) accurate within specified performance standards; and 6) its accuracy is attributable to the well-grounded theory for that purpose and context ." 29 This holistic conceptualization indicates that distinctions among different types of validity are somewhat arbitrary and that data from a method are only useful when they represent the construct of interest. 31 Two conceptual systems for validation within the field of nutrition sciences include the biometric, in which truth is typically observable, 32 and the psychometric, [33][34][35] in which measures of an underlying construct, for example, depression or self-esteem, typically depend on self-reported subjective assessments and truth is not observable.…”

“…As a result, psychometric methods have often been adopted in the validation of dietary assessment tools. 33 In the psychometric system, the six concepts noted above and explored in detail elsewhere by Frongillo and colleagues 29 are often described as face or content validity (1), construct validity (2), testeretest reliability (3 and 4), and criterion validity (5 and 6). Validity typically takes precedence 31,[35][36][37] over reliability because even in the event that a method provides consistent measurements, those measurements are useful only when they are sufficiently accurate for a specified purpose.…”

“…Frongillo and colleagues 29 described limitations in validation studies in the field of nutrition more generally. An issue specific to dietary assessment is that claims of validity often rely on suboptimal methods.…”

“…27,28 True usual intake is not observable in most studies, 27 underscoring the need for robust validation to understand whether a given method yields estimates of intake that are reasonable for providing data for a given purpose, context, and population. 29 Understanding the properties of an assessment method can also help to identify analytic methods and study design considerations to mitigate error in the measurement of dietary exposures. 30 We identified complexities and recommended best practices related to the validation of self-report dietary assessment methods.…”