Usability and social consequences of the early identification system as a universal screener for social, emotional, and behavioral risks.

Abstract

The Early Identification System (EIS) was developed to overcome many of the usability challenges of school-based behavior screeners. Several prior studies have documented the technical adequacy of the EIS. The present study expanded this work by examining the use, relevance, values implications, and social consequence of EIS implementation in a sample of 54 K-12 schools and 23,104 students in the Midwestern United States. We found that nearly all schools, teachers, and students completed the EIS as planned. Schools used the data to provide universal, selective, and indicated services to a high percentage of students identified with risks as well as to plan professional development for educators based on the screening data. Seventy-nine percent of schools implemented the EIS system with high fidelity, and fidelity was unrelated to student demographic composition within schools. These findings suggest that the EIS may overcome many of the usability barriers that plague common behavior screeners. Limitations and implications for advancing the science of social consequence evaluation are discussed. Impact and ImplicationsThe present findings suggest the EIS is a cost-effective, efficient, and useful tool for screening youth mental health concerns that schools implement with high fidelity. The EIS may overcome many barriers that prevent most schools from conducting ongoing universal screening of youth social, emotional, and behavioral concerns.

Social consequences of educational measures: A commentary.

“…Ideally, authors should report multiple forms of validity evidence with an argument to support the generalization of this evidence from one use, sample, or score to another—especially for less-researched measures. Herman et al (2023) present an extensive list of validity results in their description of the EIS . This example illustrates the detail often necessary to ensure the intended interpretation and avoid erroneous inferences.…”

“…Herman and Bonifay (2023), Splett et al (2023), and Herman et al (2023) discuss evidence of consequential validity in reference to assessment-based decisions. Splett and colleagues suggest that a school’s limited capacity to provide intervention may represent poor consequential validity, that is, “consequential validity claims of the utility of universal SEB screening results to decrease unaddressed SEB risk remain untested” (p. 144).…”

“…Herman and Bonifay (2023) and Herman et al (2023) argue that the use of national norms may impair the social consequences of assessment, often overidentifying students (see also Shinn, 2007). The criticism of national norms extends to criterion-referenced measures, as both ignore the local context.…”

“…Splett et al (2023) investigate the use of the Behavioral and Emotional Screening System ( BESS ; Kamphaus & Reynolds, 2007) to identify students at risk for social–emotional behavior (SEB) difficulties and highlight the challenges schools face when many students meet the criteria for intervention supports. Herman et al (2023) examine the consequences of using the Early Identification System ( EIS ) to identify, prevent, and address students’ social, emotional, and behavioral issues. They show that schools used the data to plan services for at-risk students as well as professional development for staff members.…”

“…Although the articles address critical topics and debates related to assessment, decision-making, and intervention in schools, not all of their analyses tightly fit the focus of this special issue, the social consequences of assessment. Some conclusions—such as the “disconnect between screener results and universal and professional development supports” (Herman et al, 2023, p. 155) or that “consequential validity of SEB screening should … establish utility and impact on access to intervention resources” (Splett et al, 2023, p. 146)—offer salient and seldom-recognized implications for serving students at risk for SEB difficulties. Yet, these ideas take a step beyond the consequential validity evidence for the screeners as defined by Messick (e.g., 1995, 1998), the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014; hereafter, AERA, 2014), or other sources.…”