Ensuring content validity of psychological and educational tests – the role of experts

Abstract

Many test developers try to ensure the content validity of their tests by having external experts review the items, e.g. in terms of relevance, difficulty, or clarity. Although this approach is widely accepted, a closer look reveals several pitfalls need to be avoided if experts' advice is to be truly helpful. The purpose of this paper is to exemplarily describe and critically analyse widespread practices of involving experts to ensure the content validity of tests. First, I offer a classification of tasks that experts are given by test developers, as reported on in the respective literature. Second, taking an exploratory approach by means of a qualitative meta-analysis, I review a sample of reports on test development (N = 72) to identify the common current procedures for selecting and consulting experts. Results indicate that often the choice of experts seems to be somewhat arbitrary, the questions posed to experts lack precision, and the methods used to evaluate experts' feedback are questionable. Third, given these findings I explore in more depth what prerequisites are necessary for their contributions to be useful in ensuring the content validity of tests. Main results are (i) that test developers, contrary to some practice, should not ask for information that they can reliably ascertain themselves ("truth-apt statements"). (ii) Average values from the answers of the experts which are often calculated rarely provide reliable information about the quality of test items or a test. (iii) Making judgements about some aspects of the quality of test items (e.g. comprehensibility, plausibility of distractors) could lead experts to unreliable speculations. (iv)When several experts respond by giving incompatible or even contradictory answers, there is almost always no criterion enabling a decision between them. In conclusion, explicit guidelines on this matter need to be elaborated and standardised (above all, by the AERA, APA, and NCME "Standards").

Comparative analysis of Student's live online learning readiness during the coronavirus (COVID-19) pandemic in the higher education sector

“…Among the 35 items, 8 items were used to determine the motivation for learning and self-directed learning, 7 items were used to determine learner control and technology readiness, and 5 items were used to determine online communication self-efficacy. To ensure content validity, we referred to the approach adopted in Beck (2020) . The questionnaire was assessed by reviewing the scale with five experts with professional expertise in online teaching and learning from various regions, including Hong Kong, Macau, China, and Australia.…”

Item Difficulty in Item Analysis of Intelligence Test Items

“…Given these findings, I delve more into the conditions that must be met for their contributions to be beneficial in assuring test content validity. Finally, it is necessary to develop and standardize precise recommendations on this subject (Beck, 2020).…”

Mixed methods instrument validation: Evaluation procedures for practitioners developed from the validation of the Swiss Instrument for Evaluating Interprofessional Collaboration

“…These include a literature review and conducting expert interviews or focus groups to extract relevant dimensions and develop items that capture them [ 1 , 2 ]. In a further qualitative process, developed items are judged by experts whether they capture all aspects of a dimension and are well understood by prospective respondents [ 3 ]. This is sometimes supplemented by a quantitative assessment of whether the items are relevant and understandable, such as in the example of the Content Validity Index [ 4 – 6 ].…”