Improving Content Assessment for English Language Learners: Studies of the Linguistic Modification of Test Items

Abstract

Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions expressed in the ETS Research Report series and other published accounts of ETS research are those of the authors and not necessarily those of the Officers and Trustees of Educational Testing Service.The Daniel Eignor Editorship is named in honor of Dr. Daniel R. Eignor, who from 2001 until 2011 served the Research and Development division as Editor for the ETS Research Report series. The Eignor Editorship has been created to recognize the pivotal leadership role that Dr. Eignor played in the research publication process at ETS. ETS Research Report Series ISSN 2330-8516 R E S E A R C H R E P O R T Improving Content Assessment for English Language Learners: Studies of the Linguistic Modification of Test ItemsJohn W. Young, Teresa C. King, Maurice Cogan Hauck, Mitchell Ginsburgh, Lauren Kotloff, Julio Cabrera, & Carlos CavalieEducational Testing Service, Princeton, NJ This article describes two research studies conducted on the linguistic modification of test items from K-12 content assessments. In the first study, 120 linguistically modified test items in mathematics and science taken by fourth and sixth graders were found to have a wide range of outcomes for English language learners (ELLs) and non-ELLs, with regard to performance on the original and modified versions of the same items. The original items were disclosed items from two states and were used in their standards-based assessments. The modified versions of these items were developed by a team of Educational Testing Service (ETS) assessment developers and researchers. The ETS research team identified several kinds of modifications that appeared promising and applied the modifications systematically. Ideally, modifying items leads to improved performance by ELLs while having little to no impact on the performance of non-ELLs. However, both groups of students performed better on some items, about the same on some other items, and worse on yet some other items. Cognitive interviews were conducted as a follow-up study to investigate which features of the linguistically modified items produced the observed outcomes for ELLs and non-ELLs. The results from this study added little clarity with regards to which of the linguistic modifications were effective and how they improved the understanding and performance of ELLs on these content items.Keywords En...

Designing Bilingual Test Items: Teacher’s Strategies

“…The main consideration is how the item can be understood by the students since she realizes that her classroom is categorized as a lower achievement with LEP, she tries to simplify the language of the test items. This is in line with Young et al, (2014) that the teacher might develop a linguistic modification to ensure the test is accessible for the students. One of the modifications is simplified vocabulary, that is, the identification of challenging words which are not related to the content and replace the words with accessible ones.…”

Considerations for Designing Accessible Educational Scenario-Based Assessments for Multiple Populations: A Focus on Linguistic Complexity

“…Previous research has been conducted to investigate linguistic complexity as a possible source of CIV in assessments using various approaches including textual analysis and think-aloud protocols among others (Martiniello, 2008;Sato et al, 2010;Young et al, 2014). The studies have been conducted to investigate possible ways to modify test items without modifying the assessed construct and to identify possible consequences associated with the presence of CIV due to text complexity on cognition and test-taking behavior (Abedi, 2006;Martiniello, 2008;Turmo and Elstad, 2009).…”

“…Examples of vocabulary-related linguistic complexity included items that had more than one unknown word in each sentence, the inclusion of words that were long, had multiple meanings, or were morphologically complex. Young et al (2014) used data from K-12 content assessments to investigate linguistic modification of 120 test items in mathematics and science administered to 4th and 6th grade students that were identified as having a wide range of outcomes for ELs and non-ELs on the item performance between the original and modified versions of the items. No systematic differences were found in relation to item performance for either group.…”

“…The study conducted by Young et al (2014) was an expansion of the work conducted by Sato et al (2010), which only contained five modification categories (context, graphics, vocabulary/wording, sentence structure, and format/style). Abedi (2006) and Martiniello (2008) suggest that the use of complex construct-irrelevant language in items may disadvantage some populations.…”

Advancing Human Assessment: A Synthesis Over Seven Decades

“…For those with disabilities, topics included accessibility (Hansen et al 2012;Stone et al 2016), accommodations (Cook et al 2010), instrument and item functioning (Buzick and Stone 2011;Steinberg et al 2011), computer-adaptive testing (Stone et al 2013;Stone and Davey 2011), automated versus human essay scoring , and the measurement of growth (Buzick and Laitusis 2010a, b). For English learners, topics covered accessibility Young et al 2014), accommodations (Wolf et al 2012a, b), instrument functioning (Gu et al 2015;Young et al 2010), test use (Lopez et al 2016;Wolf and Farnsworth 2014;Wolf and FaulknerBond 2016), and the conceptualization of English learner proficiency assessment systems ).…”