Best practice versus actual practice: an audit of survey pretesting practices reported in a sample of medical education journals

Abstract

Background: Despite recommendations from survey scientists, surveys appear to be utilized in medical education without the critical step of pretesting prior to survey launch. Pretesting helps ensure respondents understand questions as survey developers intended and that items and response options are relevant to respondents and adequately address constructs, topics, issues or problems. While psychometric testing is important in assessing aspects of question quality and item performance, it cannot discern how respondents, based upon their lived experiences, interpret the questions we pose.Aim: This audit study explored whether authors of medical education journal articles within audited journals reported pretesting survey instruments during survey development, as recommended by survey scientists and established guidelines/standards for survey instrument development.Methods: Five national and international medical education journals publishing survey articles from Jan. 2014 – Dec. 2015 were audited to determine whether authors reported pretesting during survey development. All abstracts within all issues of these journals were initially reviewed. Two hundred fifty-one articles met inclusion criteria using a protocol piloted and revised prior to use.Results: The number of survey articles published per journal ranged from 11 to 106. Of 251 audited articles, 181 (72.11%) described using a new instrument without pretesting, while 17 (6.77%) described using a new instrument where items were pretested. Fifty-three (21.12%) articles described using pre-existing instruments; of these, no articles (0%) reported pretesting existing survey instruments prior to use.Conclusions: Findings from this audit study indicate that reported survey pretesting appears to be lower than that reported in healthcare journals. This is concerning, as results of survey studies and evaluation projects are used to inform educational practices, guide future research, and influence policy and program development. Findings apply to both survey developers and faculty across a range of fields, including evaluation and medical education research.

Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030

“…The best practices for survey design are well established and have been documented [1]. However, research literature documenting the development of new survey instruments remains limited [2], and reports about the pre-testing of survey tools focus predominantly on psychometric validity [3].…”

An Examination of Resident Perspectives on Survey Participation and Methodology: Implications for Educational Practice and Research

“…In medical education, self-administered questionnaires are routinely used to gather information for educational needs assessments, innovation and quality improvement/patient safety projects, program evaluations, and research studies. [1][2][3][4] Survey results are used to revise curricula, influence program development, develop or challenge policies, aid in funding, 1,[4][5][6] and meet accreditation requirements. 7,8 Despite the importance of survey methodology across numerous fields, response rates have been declining for years, especially for physicians [9][10][11][12] and other healthcare providers.…”

“…19 In addition to declining response rates, the overall quality of data derived from self-administered questionnaires in health professions education is often low, due to survey design issues and response processes, 20 all of which contribute to measurement error. 1,3,[21][22][23] Importantly, the validity of inferences based on survey results is closely tied to factors such as survey design, 5,24,25 assessment of survey item quality, 1,5,[21][22][23][26][27][28][29][30] respondents' cognitive and sociocultural response processes, 5,22,28,[30][31][32] response rates and sampling, 16 and whether respondents actually resemble the population of interest (ie, sample representativeness). 12 Breakoffs, where respondents accept survey invitations, start surveys, and then break off prior to finishing, also affect data quality and ultimately survey inferences.…”

“…For practical steps to creating surveys (which is beyond the scope of this article) we direct readers to the survey literature 21,23,26,29,31,32 and articles designed for health professions education audiences. [1][2][3]22,49,50 Given the confidential nature of data collection and anonymous nature of data analysis, we were unable to engage in member checking. Like other qualitative studies with small sample sizes, results may not generalize to other contexts and residency cultures.…”

Are sexual health survey items understood as intended by African and Asian migrants to Australia? Methods, results and recommendations for qualitative pretesting

“…According to Colbert and colleagues, ‘[w]hile numerous factors affect the quality of data derived from survey instruments … if different respondents do not understand questions in the same way and as researchers intended, then the other issues are moot’. 8 Pretesting of survey instruments is ‘the only way to evaluate in advance whether a questionnaire causes problems’. 9 However, there is a dearth of publicly available pretesting data in relation to SHBBV surveys in migrant contexts.…”