Empiricism versus connoisseurship: Establishing the appropriacy of texts in tests of academic reading

Abstract

Providers of tests of languages for academic purposes generally claim to provide evidence on the extent to which students are likely to be able to cope with the future demands of reading in specified real-life contexts. Such claims need to be supported by evidence that the texts employed in the test reflect salient features of the texts the test takers will encounter in the target situation as well as demonstrating the comparability of the cognitive processing demands of accompanying test tasks with target reading activities. This paper will focus on the issue of text comparability. For reasons of practicality, evidence relating to text characteristics is generally based on the expert judgement of individual test writers, arrived at through a holistic interpretation of test specifications. However, advances in automated textual analysis and a better understanding of the value of pooled qualitative judgement have now made it feasible to provide more quantitative approaches focusing analytically on a wide range of individual characteristics. This paper will employ these techniques to explore the comparability of texts used in a test of academic reading comprehension and key texts used by first-year undergraduates at a British university. It offers a principled means for test providers and test users to evaluate this aspect of test validity.

The relationship between academic vocabulary coverage and scores on a standardized English proficiency test

“…However, little research has been conducted on the coverage of different frequency lists in existing standardized language proficiency tests and its relationship to test takers' performance. Green, Ünaldı, and Weir (2010) examined the features in IELTS tests and reported a significantly lower coverage of academic words in the test than in the undergraduate texts, and attributed this to the use of texts from newspapers and magazines in the test.…”

Reconstructing Readability: Recent Developments and Recommendations in the Analysis of Text Difficulty

“…Though many adults still enjoy the novels and essays that are often required reading for school children, adults also have to navigate their way through news, technical writing, and informational texts intended for members of special fields as well as the general public. Not many of the methods utilized in these areas, though, are available for general use (for research in specialty texts and special populations, see Kirsch and Mosenthal 1990;Meyer et al 1993;Kim et al 2007;Feng 2009;Green et al 2010;Heilman et al 2007;Peterson and Ostendorf 2009).…”

A Corpus Study of Structural Types of Lexical Bundles in MUET Reading Texts

“…A specialised corpus of MUET test papers made up of only the reading passages categorised into two main disciplines namely arts and science was built. Building such a small corpus has been proven significant by many (Green, Unaldi & Weir 2010, Romer 2004) because it can be used to systematically analyse the kind of language adopted in a particular context, in this case, an examination setting. It can also quantitatively study frequently occurring language patterns in reading texts which appeared to be insufficient as highlighted by Biber et.al (2004) as cited in Green et al (2010).…”