Findings derived from self-reported, structured survey questionnaires are commonly used in evaluation and applied research to inform policy-making and program decisions. Although there are a variety of issues related to the quality of survey evidence (e.g., sampling precision), the validity of response processes-how respondents process thoughts and perceptions when answering questionnaires-is crucial. We assess the extent to which cognitive interviews, used to test survey questions as part of the process of questionnaire design and refinement, can strengthen the quality of survey evidence for small-and medium-scale evaluations. To illustrate how cognitive interviews can reveal respondents' response processes and improved questionnaire interpretations, we present excerpts from two types of evaluations (e.g., needs assessment) in two domains (e.g., education). The paper concludes with a brief summary about how the use of cognitive interviews can be augmented in survey development, refinement, and adaptation to improve survey questionnaire interpretations in evaluations with limited resources.
Purpose
Data use cultures in schools determine data use practices. Such cultures can be muted by powerful macro accountability and organizational learning cultures. Further, strong equity-oriented data use cultures are challenging to establish. The purpose of this paper is to engage these cultural tensions.
Design/methodology/approach
The data discourse and decisions of four grade-level teams in two elementary schools in one district were studied through observation of 62 grade-level meetings over the course of a year. The observations focused on “data talk,” defined as the structure and content of team conversations about interim student performance data.
Findings
Distinct macro cultures of accountability and organizational learning existed in the two schools. The teams’ own data use cultures partly explained the absence of a focus on equity, and none of the teams used student performance data to make instructional decisions in support of the district’s equity aims. Leadership missed opportunities to cultivate an equity-focused data use culture.
Practical implications
School leaders who advocate that equity importantly guides data use routines, and can anticipate how cultures of accountability or organizational learning “show up” in data use conversations, will be better prepared to redirect teachers’ interpretations of data and clarify expectations of equity reform initiatives.
Originality/value
This study is novel in its concept of “data talk,” which provided a holistic but nuanced account of data use practices in grade-level meetings.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.