Re-fitting for a different purpose: A case study of item writer practices in adapting source texts for a test of academic reading

Abstract

The important yet under-researched role of item writers in the selection and adaptation of texts for high-stakes reading tests is investigated through a case study involving a group of trained item writers working on the International English Language Testing System (IELTS). In the first phase of the study, participants were invited to reflect in writing, and then audio-recorded in a semantic-differential-based joint discussion, on the processes they employed to generate test material. The group were next observed at a simulated item writers' editing meeting to refine their texts and items for an IELTS reading test module. The participants' written descriptions and recorded discussions provided rich data on how source texts were perceived, selected and adapted for the Test. The study reports findings from textual analyses using indices of readability and lexical density from the original material sourced by the item writers and their adapted versions for the test. Results from qualitative and quantitive analyses are discussed in terms of the implications for the IELTS reading module of editing actions such as: reducing redundancy and technical language, changing styles, deciding on potentially sensitive issues and relationships between texts and test items. The important issue of text authenticity in tests such as IELTS is also broached.Item writers make an essential contribution to test validity, but have an ambiguous status in the practice of educational testing. On the one hand, item writing is seen to be a creative art (Ebel, 1951;Wesman, 1971) requiring extensive training and mentoring of practitioners and the flexible, expert interpretation of guidelines. On the other hand, test

Vocabulary Coverage of Reading Tests: Gaps Between Teaching and Testing

“…However, for testing reading, it might not be necessary to reach the minimal reading level for all students; it might be sufficient to assess their reading proficiency and to allow for shortfalls in some situations. Perhaps due to this concern, doubt has been cast on the use of 95% vocabulary coverage as the minimal target in reading tests (see Green & Hawkey, ; Webb & Paribakht, ). For example, in Green and Hawkey (), the BNC 2,000 word families and the Academic Word List (Coxhead, ) were not expected to reach 90% vocabulary coverage of the four reading texts for IELTS (Nation & Waring, ).…”

“…Perhaps due to this concern, doubt has been cast on the use of 95% vocabulary coverage as the minimal target in reading tests (see Green & Hawkey, ; Webb & Paribakht, ). For example, in Green and Hawkey (), the BNC 2,000 word families and the Academic Word List (Coxhead, ) were not expected to reach 90% vocabulary coverage of the four reading texts for IELTS (Nation & Waring, ). Likewise, in examining the vocabulary coverage of 38 reading texts from CanTEST, Webb and Paribakht () found that 3,000 word families were sufficient to reach 95% vocabulary coverage of a text, but 14,000 word families were required to reach 95% vocabulary coverage of another text.…”

Syntactic Complexity in Adapted Teaching Materials: Differences Among Grade Levels and Implications for Benchmarking

“…As mentioned earlier, these measures reflect five discrete subconstructs, including clausal coordination, clausal subordination, nonfinite elements, phrasal coordination, and noun phrase complexity, and three composite features including overall sentence complexity, overall T‐unit complexity, and elaboration at the clause level. These results suggest that expert teachers and material developers attended to different subconstructs or dimensions of syntactic complexity in selecting and adapting teaching materials (Green & Hawkey, ). All eight syntactic complexity measures exhibited a clear upward trend across the 12 grade levels, indicating that overall, texts across the 12 grade levels become increasingly more complex in all dimensions of syntactic complexity.…”

“…The importance of including syntactic complexity as an explicit and distinct component in text complexity models has been well argued for (Frantz et al., ), and tools for evaluating text complexity do indeed all incorporate measures for this construct (Graesser et al., ; Sheehan et al., ). However, to date, little, if any, systematic effort has been made to establish benchmarks for syntactic complexity for text adaptation purposes and to make feedback regarding syntactic complexity accessible to teachers during the text adaptation process, despite the importance of this construct in this process (Crossley et al., ; Green & Hawkey, ). There is thus a clear need for more research in this area.…”

“…The Modern Language Journal, 104, 1, (2020) DOI: 10.1111/modl.12622 0026-7902/20/192-208 $1.50/0 C National Federation of Modern Language Teachers Associations text adaptation decisions (Crossley et al, 2007). The text adaptation literature, however, has regarded it problematic to solely rely on the intuitive approach to text adaptation, as it leads to considerable variation and inconsistency in the decisions made (Crossley, Allen, & McNamara, 2012;Green & Hawkey, 2012). With recent advances in corpus linguistics and natural language processing, computational tools for automating text readability or complexity analysis are becoming increasingly available.…”