Consequences Validity Evidence: Evaluating the Impact of Educational Assessments

Cook, David A.; Lineberry, Matthew

doi:10.1097/acm.0000000000001114

Cited by 104 publications

(58 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The framework comprises 5 sources of evidence: content, response process, internal structure, relationship to other variables, and consequences [24]. Table 1 presents these sources, how they were accommodated, and a statistical analysis.…”

Section: Methodsmentioning

confidence: 99%

Developing and Gathering Validity Evidence for a Simulation-Based Test of Competencies in Lung Ultrasound

et al. 2018

View full text Add to dashboard Cite

Background: Clinical lung ultrasound (LUS) is a fast bedside diagnostic tool which can assist clinicians in decisions regarding the treatment and monitoring of patients with respiratory symptoms. LUS training and education differ widely, and is often done in a clinical setting, with potential risks for patients if decisions are made based on the wrong interpretations. No clear guidelines or recommendations for objective and standardized assessment of LUS skills exist, and those that do are often based on a fixed time-frame or an arbitrary number of examinations performed; this does not ensure adequate competencies. Objectives: The study aimed to develop and gather validity evidence for a practical, simulation-based test in LUS. Methods: Nine cases were developed in collaboration with 3D Systems Healthcare, Littleton, CO, USA, representing the most common diagnosis and sonographic findings in patients with respiratory symptoms. Thirty-six participants with different levels of competence in LUS, completed the test. The participants were divided into groups, i.e., novices, intermediates, and experienced, according to their experience with LUS, the number of examinations they had performed, and any research they had conducted. Their answers were used for item analyses. Results: The intraclass correlation coefficient, Cronbachs’ α, was 0.69 summarized, and there was a statistically significant difference (p < 0.001) between the novices and the trained participants (intermediates and experienced). A pass/fail score of 16 points was calculated according to the contrasting-groups method. Conclusion: We developed a test for the assessment of clinical competencies in LUS. The test proved solid validity evidence, and a pass/fail standard without any false-negatives, and only 2 explained false-positives.

show abstract

Section: Methodsmentioning

confidence: 99%

Developing and Gathering Validity Evidence for a Simulation-Based Test of Competencies in Lung Ultrasound

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Intended uses of the assessment include evaluating of mixed methods trainings, understanding learners baseline skills, guiding and tailoring educational strategies, and assessing mixed methods learning progress. Regarding consequences validity 24,25 of the impact of assessing, interpreting, and applying scores of the Self-Rated Mixed Methods Skills Assessment, we speculate on consequences for learners and faculty. For the learner, assessment may lead to enhanced motivation to learn and training focused on self-identified needs.…”

Section: Discussionmentioning

confidence: 97%

“…Finally, we examined consequential validity, which is concerned with the impact of the activity of assessment and subsequent use of scores. 24,25 …”

mentioning

confidence: 99%

Development of a Self-Rated Mixed Methods Skills Assessment: The National Institutes of Health Mixed Methods Research Training Program for the Health Sciences

Guetterman

Creswell

Wittink

et al. 2017

J Contin Educ Health Prof

View full text Add to dashboard Cite

Introduction Demand for training in mixed methods is high, with little research on faculty development or assessment in mixed methods. We describe the development of a Self-Rated Mixed Methods Skills Assessment and provide validity evidence. The instrument taps six research domains: “Research question,” “Design/approach,” “Sampling,” “Data collection,” “Analysis,” and “Dissemination.” Respondents are asked to rate their ability to define or explain concepts of mixed methods under each domain, their ability to apply the concepts to problems, and the extent to which they need to improve. Methods We administered the questionnaire to 145 faculty and students using an internet survey. We analyzed descriptive statistics and performance characteristics of the questionnaire using Cronbach’s alpha to assess reliability and an ANOVA that compared a mixed methods experience index with assessment scores to assess criterion-relatedness. Results Internal consistency reliability was high for the total set of items (.95) and adequate (>=.71) for all but one subscale. Consistent with establishing criterion validity, respondents who had more professional experiences with mixed methods (e.g., published a mixed methods paper) rated themselves as more skilled, which was statistically significant across the research domains. Discussion This Self-Rated Mixed Methods Assessment instrument may be a useful tool to assess skills in mixed methods for training programs. It can be applied widely at the graduate and faculty level. For the learner, assessment may lead to enhanced motivation to learn and training focused on self-identified needs. For faculty, the assessment may improve curriculum and course content planning.

show abstract

“…This reorientation of focus is particularly suited to the assessment of NLEs given that the different types of exam operate within different systems and in different contexts. Furthermore, with its five domains of validity evidence, the APA framework allows us to examine not only the evidence that NLEs measure what they intend to measure, but also, with its inclusion of the consequences domain, we can focus on the “impact, beneficial or harmful and intended or unintended, of the [national licensing] assessment” [26]. …”

Section: Methodsmentioning

confidence: 99%

The impact of large scale licensing examinations in highly developed countries: a systematic review

et al. 2016

View full text Add to dashboard Cite

BackgroundTo investigate the existing evidence base for the validity of large-scale licensing examinations including their impact.MethodsSystematic review against a validity framework exploring: Embase (Ovid Medline); Medline (EBSCO); PubMed; Wiley Online; ScienceDirect; and PsychINFO from 2005 to April 2015. All papers were included when they discussed national or large regional (State level) examinations for clinical professionals, linked to examinations in early careers or near the point of graduation, and where success was required to subsequently be able to practice. Using a standardized data extraction form, two independent reviewers extracted study characteristics, with the rest of the team resolving any disagreement. A validity framework was used as developed by the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education to evaluate each paper’s evidence to support or refute the validity of national licensing examinations.Results24 published articles provided evidence of validity across the five domains of the validity framework. Most papers (n = 22) provided evidence of national licensing examinations relationships to other variables and their consequential validity. Overall there was evidence that those who do well on earlier or on subsequent examinations also do well on national testing. There is a correlation between NLE performance and some patient outcomes and rates of complaints, but no causal evidence has been established.ConclusionsThe debate around licensure examinations is strong on opinion but weak on validity evidence. This is especially true of the wider claims that licensure examinations improve patient safety and practitioner competence.

show abstract

Consequences Validity Evidence: Evaluating the Impact of Educational Assessments

Cited by 104 publications

References 44 publications

Developing and Gathering Validity Evidence for a Simulation-Based Test of Competencies in Lung Ultrasound

Developing and Gathering Validity Evidence for a Simulation-Based Test of Competencies in Lung Ultrasound

Development of a Self-Rated Mixed Methods Skills Assessment: The National Institutes of Health Mixed Methods Research Training Program for the Health Sciences

The impact of large scale licensing examinations in highly developed countries: a systematic review

Contact Info

Product

Resources

About