External Validity and Model Validity: A Conceptual Approach for Systematic Review Methodology

Abstract

Background. Evidence rankings do not consider equally internal (IV), external (EV), and model validity (MV) for clinical studies including complementary and alternative medicine/integrative medicine (CAM/IM) research. This paper describe this model and offers an EV assessment tool (EVAT©) for weighing studies according to EV and MV in addition to IV. Methods. An abbreviated systematic review methodology was employed to search, assemble, and evaluate the literature that has been published on EV/MV criteria. Standard databases were searched for keywords relating to EV, MV, and bias-scoring from inception to Jan 2013. Tools identified and concepts described were pooled to assemble a robust tool for evaluating these quality criteria. Results. This study assembled a streamlined, objective tool to incorporate for the evaluation of quality of EV/MV research that is more sensitive to CAM/IM research. Conclusion. Improved reporting on EV can help produce and provide information that will help guide policy makers, public health researchers, and other scientists in their selection, development, and improvement in their research-tested intervention. Overall, clinical studies with high EV have the potential to provide the most useful information about “real-world” consequences of health interventions. It is hoped that this novel tool which considers IV, EV, and MV on equal footing will better guide clinical decision making.

Scale development: ten main limitations and recommendations to improve future research practices

“…It is verified either by calculating the average variance extracted for each factor when the shared variance accounted for 0.50 or more of the total variance or by correlating their scales with a measure of overall quality (Ladhari 2010). In the sequence of convergent validity, the following methods were identified as favorites in the assessment of construct validity: discriminant validity (the extent to which the scale's score does not correlate with unrelated constructs) (e.g., Coker et al 2011), predictive/nomological validity (the extent to which the scores of one construct are empirically related to the scores of other conceptually related constructs) (e.g., Sharma 2010), criterion validity (the empirical association that the new scale has with a gold standard criterion concerned with the prediction of a certain behavior) (e.g., Tanimura et al 2011), internal (signifies whether the study results and conclusions are valid for the study population), and external validity (generalizability of study) (e.g., Bolton and Lane 2012;Khorsan and Crawford 2014). Considering the importance of validity to ensure the quality of the collected data and the generalized potential of the new instrument, future studies should allow different ways to assess the validity of the new scale, thus increasing the psychometric rigor of the analysis.…”

Long-Term Improvements After Multimodal Rehabilitation in Late Phase After Stroke

“…Clinical trials often use excessively strict enrolment criteria, thereby excluding many individuals who could potentially benefit from new interventions, thus, limiting generalizability to clinical practice. 24 Many of the excluded individuals were not randomized into the current study because of their geographic location, because that would have made it practically difficult for the individual to attend the program. In addition, an inclusion of the wellrecovered participants, such as in the largest exclusion group modified Rankin Scale 0 to 1, would probably not have been cost-effective because of their mild deficits, and individuals with severe disabilities were not considered for these therapies because of problems to participate.…”

The Impact of Massage Therapy on Function in Pain Populations—A Systematic Review and Meta-Analysis of Randomized Controlled Trials: Part III, Surgical Pain Populations

“…The External Validity Assessment Tool (EVAT ©) [ 33 ] was used to measure the generalizability of research to other individuals (i.e., external validity) and other settings (i.e., model validity) outside the confines of a study. …”