Prototyping new item types

Nissan, Susan; Schedl, Mary

doi:10.4324/9780203181287.ch19

Cited by 3 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They can also result in improvements in the instructions in future test forms. This study adds to the research that has already been conducted at the previously described phases of test development (e.g., McKinley, Mills, Reese, Schaeffer, & Steffen, 1993;Nissan & Schedl, 2012;Parshall, Harmes, Davey, & Pashley, 2010).…”

Section: Cognitive Interview Methodsmentioning

confidence: 88%

Usability of Interactive Item Types and Tools Introduced in the New GRE^® revised General Test

Swiggett

Kotloff

Ezzo

et al. 2014

ETS Research Report Series

View full text Add to dashboard Cite

The computer‐based Graduate Record Examinations® (GRE®) revised General Test includes interactive item types and testing environment tools (e.g., test navigation, on‐screen calculator, and help). How well do test takers understand these innovations? If test takers do not understand the new item types, these innovations may introduce construct‐irrelevant variance, with test takers performing differently than they would with more familiar item types. Similarly, the navigational and other test environment tools are another potential source of variance, if some test takers understand how to use them and others do not. In this study, we examined the reactions, engagement, and difficulties encountered as 20 potential test takers completed Verbal and Quantitative Reasoning sections of a practice GRE test. Participants were sophomores and juniors from colleges and universities in the local area. Their reactions were captured through the use of cognitive laboratory sessions that incorporated interviews that required test takers to think aloud, as well as researcher observations as test takers worked quietly. Results of the analysis of this data revealed that some participants needed time to figure out what was being asked of them when they encountered the new item types, although most were able to answer each item eventually. On the other hand, most participants stated that they did not even notice the test environment tools, and few were observed actually using the tools. Several participants provided suggestions about improving the usability of the new item types and test environment tools.

show abstract

Section: Cognitive Interview Methodsmentioning

confidence: 88%

Usability of Interactive Item Types and Tools Introduced in the New GRE^® revised General Test

Swiggett

Kotloff

Ezzo

et al. 2014

ETS Research Report Series

View full text Add to dashboard Cite

show abstract

“…This report describes a variety of investigations carried out on the EIT that were done during the development of the TOEFL Essentials test. Not including initial small‐scale design and usability studies, development of the EIT proceeded in the three major phases of prototyping, piloting, and field testing (Kenyon & MacGregor, 2012; Nissan & Schedl, 2012). The prototyping phase focused on usability issues related to the EIT and the impact of specific design features.…”

Section: Introductionmentioning

confidence: 99%

Developing an Innovative Elicited Imitation Task for Efficient English Proficiency Assessment

Davis¹,

Norris²

2021

ETS Research Report Series

View full text Add to dashboard Cite

The elicited imitation task (EIT), in which language learners listen to a series of spoken sentences and repeat each one verbatim, is a commonly used measure of language proficiency in second language acquisition research. The TOEFL® Essentials™ test includes an EIT as a holistic measure of speaking proficiency, referred to as the “Listen and Repeat” task type. In this report, we describe the design considerations that informed the development of the EIT for TOEFL Essentials. We also report the results of a series of investigations conducted during the prototyping and pilot phases of test development, which were undertaken with the goal of confirming task design specifications, evaluating scoring performance, and obtaining initial validity evidence to support score interpretation and use of the EIT in the TOEFL Essentials test. We found that task design variables generally performed as expected. The length of input sentence was strongly associated with performance (Pearson r = .88), consistent with the construct measured by the EIT, while other task variables not directly related to the EIT construct did not impact performance (e.g., graphics, speaker accent, and response time). Scorers drawn from TOEFL iBT test raters were able to score responses consistently with over 98% exact or adjacent interrater agreement on a 6‐point scale, and scores on the pilot version of the EIT were highly reliable (Cronbach's α = .93 on the 15‐item pilot version). Correlations between EIT scores and other measures were generally as expected: Correlations with other speaking tasks were high (.78–.84) and slightly to somewhat lower for other language measures (.73 for writing, .68 for listening, and .57 for reading). Correlation with an independent measure of holistic language proficiency (C‐test) was moderately high (.69), as expected. We discuss the study findings in terms of the TOEFL Essentials test validity argument and point out limitations to the current results along with future research needs. Overall, we believe that the findings provide initial support to warrant the use of the EIT as operationalized in the TOEFL Essentials test.

show abstract

Awareness Of Word Analysability And Its Role In Expanding Learners’ Vocabulary

Abdulrazzaq

2022

SILC

View full text Add to dashboard Cite

Some linguists have claimed that foreign language learners are unaware that English words containing Greek and Latin roots are analyzable. One purpose of the present study was to assess this claim. A second purpose was to see whether making these learners aware of the analyzability of these words would help them to expand their vocabulary. The final purpose was to find out whether students find the meanings of certain types of such words easier to guess than others. The 30 subjects in this study were pretested, given instruction in analyzing words into their component roots and guessing their meanings, and then post-tested. Descriptive and inferential statistics were used to evaluate the hypotheses, and these yielded the following results: (1) advanced Iraqi EFL learners are not aware of the analyzability of this type of words, and (2) making these learners aware of this analyzability will enable them to expand their vocabulary considerably, (3) learners find guessing the meanings of words containing Greek roots easier than those of words containing Latin roots, and (4) learners find guessing the meanings of words whose roots have not undergone any linguistic changes easier than those of words whose roots have undergone such changes.

show abstract

Prototyping new item types

Cited by 3 publications

References 0 publications

Usability of Interactive Item Types and Tools Introduced in the New GRE^® revised General Test

Usability of Interactive Item Types and Tools Introduced in the New GRE^® revised General Test

Developing an Innovative Elicited Imitation Task for Efficient English Proficiency Assessment

Awareness Of Word Analysability And Its Role In Expanding Learners’ Vocabulary

Contact Info

Product

Resources

About

Prototyping new item types

Cited by 3 publications

References 0 publications

Usability of Interactive Item Types and Tools Introduced in the New GRE® revised General Test

Usability of Interactive Item Types and Tools Introduced in the New GRE® revised General Test

Developing an Innovative Elicited Imitation Task for Efficient English Proficiency Assessment

Awareness Of Word Analysability And Its Role In Expanding Learners’ Vocabulary

Contact Info

Product

Resources

About

Usability of Interactive Item Types and Tools Introduced in the New GRE^® revised General Test

Usability of Interactive Item Types and Tools Introduced in the New GRE^® revised General Test