Using Focus Groups, Expert Advice, and Cognitive Interviews to Establish the Validity of a College Student Survey

Abstract

Surveys are the preferred approach for collecting data from large numbers of students about their college experiences. Many instruments are locally developed, used a few times and discarded. Others, such as the College Student Experiences Questionnaire (CSEQ) (Pace & Kuh, 1998) and the Cooperative Institutional Research Program (CIRP) instruments (Astin & Sax, 1999), are nationally normed and administered annually. It is not unusual for items used on nationally normed instruments to be adapted for use on local surveys because researchers perceive them to be "good" and "reliable," especially if psychometric analyses demonstrate that the items are sound.

Evaluating the Rationale for Affirmative Action in College Admissions: Direct and Indirect Relationships between Campus Diversity and Gains in Understanding Diverse Groups

“…The NSSE survey asks students to indicate the frequency with which they engage in activities that represent good educational practice and to report how much they have learned or gained in a variety of areas (Kuh et al,200). Selfreport data are widely used in research on college effects, and the reliability and validity of these data have been studied extensively (see Baird,976;Berdie,97;Pace,985;Pike,995;Pohlmann & Beggs, The College Student Report meets these criteria and yields accurate, meaningful information about students' college experi ences (Kuh, 200;Kuh et al, 200;Ouimet, Bunnage, Carini, Kuh, & Kennedy, 2004).…”

Validation Study of the Abbreviated Version of the Lubben Social Network Scale Spanish Translation among Mexican and Mexican-American Older Adults

“…Focus groups were preferred as a data collection mechanism as it was expected that interaction within each group would bring a more dynamic process through reactions or extended opinions from older adults (Barbour 2007). In addition, survey and measurement experts commonly prescribe the use of focus groups as a pretesting step for new or modified survey or measurement instruments (Ouimet et al 2004).…”

Development of the Quantitative Reasoning Items on the National Survey of Student Engagement

“…These data collection procedures helped us address the broadest definition of validity, whether the items produce results consistent with their intent. The three main goals of the cognitive interviews and focus groups were (1) to explore the thought processes that students use to answer the survey items, (2) to obtain information about the meaning students make of the survey items, and (3) to identify item or terms that were not well understood or that could lead to survey response error (Collins 2003;Drennan 2003;Ouimet et al 2004;Willis 2005). Over the course of the three-year NSSE update process, we conducted cognitive interviews and focus groups with around 180 students at 12 different four-year colleges and universities in the Midwest.…”