Findings derived from self-reported, structured survey questionnaires are commonly used in evaluation and applied research to inform policy-making and program decisions. Although there are a variety of issues related to the quality of survey evidence (e.g., sampling precision), the validity of response processes-how respondents process thoughts and perceptions when answering questionnaires-is crucial. We assess the extent to which cognitive interviews, used to test survey questions as part of the process of questionnaire design and refinement, can strengthen the quality of survey evidence for small-and medium-scale evaluations. To illustrate how cognitive interviews can reveal respondents' response processes and improved questionnaire interpretations, we present excerpts from two types of evaluations (e.g., needs assessment) in two domains (e.g., education). The paper concludes with a brief summary about how the use of cognitive interviews can be augmented in survey development, refinement, and adaptation to improve survey questionnaire interpretations in evaluations with limited resources.

No abstract

The authors examine student ratings within a new framework that emphasizes six distinct aspects of validity : content, substantive, structural, generalizability, external, and consequential. They It is fitting that as the new millennium arrives, a major revision of the Standards for Educational and Psychological Testing (American Psychological Association, 1985) has been completed and is the subject of much debate and dialogue. It reflects some changes in how the assessment community thinks about the concept of validity. The literature responsible for this shift began with Messick' s benchmark chapter on validity in the third edition of Educational Measurement (1989) and continued with Shepard' s article in the Review of Educational Research (1993) and articles on the consequences of assessment by Lane, Parke, and Stone, 1998;Linn, 1998;Reckase, 1998;Yen, 1998;Cronbach, 1989; and Moss, 1992, 1996. How validation efforts should be conducted and how assessment results are used is shifting as a consequence of this body of work.What are the implications of this work for student ratings of instruction? We take up this fundamental question in this chapter. We begin this effort by defining the concepts that are the basis for a new unified concept of validity. We briefly summarize the past validity research on student ratings using the traditional validity framework. Based on the concepts presented, we then do a critical analysis of student ratings research using the new validation framework. Synthesizing the results of the analysis, we propose an initial agenda for future student ratings' validation efforts.

No abstract

In evaluation and applied social research, focus groups may be used to gather different kinds of evidence (e.g., opinion, tacit knowledge). In this article, we argue that making focus group design choices explicitly in relation to the type of evidence required would enhance the empirical value and rigor associated with focus group utilization. We offer a descriptive framework to highlight contrasting design characteristics and the type of evidence they generate. We present examples of focus groups from education and healthcare evaluations to illustrate the relationship between focus group evidence, design, and how focus groups are conducted. To enhance the credibility of focus group evidence and maximize potential learning from this popular qualitative data collection method, we offer a set of questions to guide evaluators reflection and decision making about focus group design and implementation.

The recent No Child Left Behind legislation has defined a vital role for large-scale assessment in determining whether students are learning. Given this increased role of standardized testing as a means of accountability, the purpose of this article is to consider how individual differences in motivational and psychological processes may contribute to performance on high-stakes math assessments. The authors consider individual differences in processes that prior research has found to be important to achievement: achievement goals, value, self-concept, self-efficacy, test anxiety, and cognitive processes. The authors present excerpts from interviews with eighth-grade test takers to illustrate these different achievement-related motivational beliefs, affect, and cognitive processing. Implications for future research studying the situational pressures involved in high-stakes assessments are discussed.

No abstract

No abstract