Development of a Scale for Assessing the Level of Computer Familiarity of Toefl Examinees

Abstract

EDUCATIONAL TESTING SERVICE, ETS, the ETS logo, GRE, TOEFL, and the TOEFL logo are registered trademarks of Educational Testing Service. The modernized ETS logo is a trademark of Educational Testing Service. AbstractThe increasing use of computer-based testing raises concerns about equity and bias. Specifically, many in the field of language testing are concerned that the introduction of a computer-based TOEFL test in 1998 may introduce bias by confounding the measurement of English-language proficiency with level of computer familiarity. As the first step in a series of research efforts to address this question, a 23-item questionnaire focusing on individuals' access to, attitude toward, and experience with using computers as well as related technology was developed and administered to TOEFL test takers in April and May of 1996 as part of a TOEFL test administration. Results of this survey administration are documented in a paper by Kirsch, Jamieson, Taylor, and Eignor (1998). This paper has two purposes. The first is to explain decisions as to which questionnaire items would subsequently be used together as a score to separate TOEFL examinees into levels of computer familiarity. The second purpose is to provide details on the procedures used to assess the underlying factor structure of the complete questionnaire. Factor analysis procedures (specifically, a principal factors approach with both varimax orthogonal and promax oblique rotations) were used for both purposes.Eleven items, which loaded heavily on the first factor of a two-factor common factor solution with promax oblique rotation, were selected to create a computer familiarity score. This score was then used to classify TOEFL examinees into one of three familiarity groups: low, moderate, and high. A four-factor common factor solution with promax oblique rotation was seen as providing the best accounting of the underlying factor structure of the complete questionnaire.The Test of English as a Foreign Language (TOEFL ®) was developed in 1963 by the National Council on the Testing of English as a Foreign Language. The Council was formed through the cooperative effort of more than 30 public and private organizations concerned with testing the English proficiency of nonnative speakers of the language applying for admission to institutions in the United States. In 1965, Educational Testing Service (ETS®) and the College Board assumed joint responsibility for the program. In 1973, a cooperative arrangement for the operation of the program was entered into by ETS, the College Board, and the Graduate Record Examinations (GRE®) Board. The membership of the College Board is composed of schools, colleges, school systems, and educational associations; GRE Board members are associated with graduate education.ETS administers the TOEFL program under the general direction of a Policy Council that was established by, and is affiliated with, the sponsoring organizations. Members of the Policy Council represent the College Board, the GRE Board, and such institutions and ...

Skills Diagnosis Using IRT‐Based Latent Class Models

“…In a more large‐scale testing application, the RUM has been applied to data from the TOEFL (e.g., see Eignor, Taylor, Kirsch, & Jamieson, 1998). In a comprehensive study of the skills underlying the test, Jang (2005) described the process of development of a Q ‐matrix and validation of the skills following the application.…”

The TOEFL CBT (Computer-based test)

“…These demonstrations show how the items are to be answered and give test-takers an opportunity to practise using the computer. The inclusion of these tutorials -together with tutorials on how to use a mouse, how to scroll and how to use the screen icons/'testing tools' -follows on from research carried out (see Eignor et al, 1998;Kirsch et al, 1998;Taylor et al, 1998). It is also possible to buy two sample tests that are available on a compact disk called POWERPREP (see ETS, 2000b;2001c).…”

“…On the basis of research done (see Eignor et al, 1998;Kirsch et al, 1998;Taylor et al, 1998), computer familiarity tutorials have been designed to precede each administration, and there are currently three tutorials addressing the computer skills needed to take TOEFL CBT: how to use a mouse, how to scroll and how to use the screen icons/'testing tools'. There is no evidence to suggest that students who are not accustomed to computers are penalized by taking the test in this way, but more research is needed on this.…”

A Proposal on the Validation Model of Equivalence between PBLT and CBLT

“…In this case study, we investigated the computer familiarity of all 296 subjects with the adapted version of Computer Familiarity Questionnaire (Eignor, Taylor, Kirsch, & Jamieson, 1998). Since the adapted questionnaire contains 11 questions with 4 choices for each, the scores from the adapted questionnaire range from 11 to 44, the higher the score the higher the computer familiarity.…”