Building and Supporting a Case for Test Use

Bachman, Lyle F.

doi:10.1207/s15434311laq0201_1

Cited by 294 publications

(278 citation statements)

References 29 publications

Supporting

Mentioning

211

Contrasting

Unclassified

Order By: Relevance

“…Chapelle, Enright, and Jamieson (2008) proposed an argument-based approach to building interpretive and validity arguments for the TOEFL test. Chapelle et al's approach built primarily on Kane and his associates' (Kane, 2004;Kane, Crooks, & Cohen, 1999) and Bachman's (2005) frameworks for developing interpretive and validity arguments for test score interpretation and use based on Toulmin's (2003) theory of practical reasoning. On the one hand, developing an interpretive argument or a concept map of how one might go about building an argument for an intended test score interpretation and use is part of a test design process (Kane, 2004).…”

Section: Standards For Educational and Psychological Testing (Americamentioning

confidence: 99%

INVESTIGATING THE VALUE OF SECTION SCORES FOR THE TOEFL iBT® TEST

Sawaki

Sinharay

2013

ETS Research Report Series

View full text Add to dashboard Cite

This study investigates the value of reporting the reading, listening, speaking, and writing section scores for the TOEFL iBT ® test, focusing on 4 related aspects of the psychometric quality of the TOEFL iBT section scores: reliability of the section scores, dimensionality of the test, presence of distinct score profiles, and the section scores' generalizability for norm-referenced decisions as well as the dependability of criterion-referenced decisions for international student admission.Four operational TOEFL iBT test forms were analyzed for all examinees as well as for 3 native language (L1) groups (Arabic, Korean, and Spanish).Haberman's (2008) subscore analysis suggested that the speaking section score had added value due to its relative distinctness from the other modalities. Consistent with the subscore analysis results, a series of exploratory factor analyses (EFAs) indicated the possibility of the presence of 2 correlated factors-a reading/listening/writing factor and a speaking factor.In contrast, the CFAs conducted separately for the 3 L1 groups as well as a multiple-group confirmatory factor analyses (CFAs) identified a correlated 4-factor model with reading, listening, speaking, and writing factors as the best representation of the structure of the entire test for all examinees as well as for the 3 L1 groups. Reliability of the observed section scores for norm-referenced score interpretations and the dependability of classification decisions made based on different cut scores were generally satisfactory while they were also found to be relatively low in some circumstances. Based on the mixed results concerning the value-added information the TOEFL iBT section scores provide, recommendations for future research directions and some key issues of consideration for high-stakes decision making based on the section scores were summarized.Key words: dependability, dimensionality, factor analysis, generalizability theory, reliability, score profile, subscore analysis ii TOEFL ® was developed in 1963 by the National Council on the Testing of English as a Foreign Language. The Council was formed through the cooperative effort of more than 30 public and private organizations concerned with testing the English proficiency of nonnative speakers of the language applying for admission to institutions in the United States. In 1965, Educational Testing Service (ETS) and the College Board ® assumed joint responsibility for the program. In 1973, a cooperative arrangement for the operation of the program was entered into by ETS, the College Board, and the Graduate Record Examinations ® (GRE ® ) Board. The membership of the College Board is composed of schools, colleges, school systems, and educational associations; GRE Board members are associated with graduate education. The test is now wholly owned and operated by ETS.ETS administers the TOEFL program under the general direction of a policy board that was established by, and is affiliated with, the sponsoring organizations. Members of the TOEFL Board (previo...

show abstract

Section: Standards For Educational and Psychological Testing (Americamentioning

confidence: 99%

INVESTIGATING THE VALUE OF SECTION SCORES FOR THE TOEFL iBT® TEST

Sawaki

Sinharay

2013

ETS Research Report Series

View full text Add to dashboard Cite

show abstract

“…The assumptions are that the test scores and other related information provided to users are relevant, useful, and sufficient for making intended decisions and promote positive effects on teaching and learning (Bachman, 2005).…”

Section: The Validity Argument For Scoring Tpo With Speechrater V10mentioning

confidence: 99%

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATER^SM V1.0

Xi¹,

Higgins²,

Zechner³

et al. 2008

ETS Research Report Series

View full text Add to dashboard Cite

This report presents the results of a research and development effort for SpeechRater SM Version 1.0 (v1.0), an automated scoring system for the spontaneous speech of English language learners used operationally in the Test of English as a Foreign Language™ (TOEFL ® ) Practice Online assessment (TPO). The report includes a summary of the validity considerations and analyses that drive both the development and the evaluation of the quality of automated scoring. These considerations include perspectives on the construct of interest, the context of use, and the empirical performance of the SpeechRater in relation to both the human scores and the intended use of the scores. The outcomes of this work have implications for short-and long-term goals for iterative improvements to SpeechRater scoring. which is used by prospective test takers to prepare for the official TOEFL iBT test. This study reports the development and validation of the system for low-stakes practice purposes. The process we followed to build this system represented a principled approach to maximizing 2 essential qualities: substantively meaningful and technically sound. In developing and evaluating the features and the scoring models to predict human assigned scores, we engaged both content and technical experts actively to ensure the construct representation and technical soundness of the system. We compared primarily two alternative methodologies of building scoring modelsmultiple regression and classification trees-in terms of their construct representation and empirical performance in predicting human scores. Based on the evaluation results, we concluded that a multiple regression model with feature weights determined by content experts was superior to the other competing models evaluated.We then used an argument-based approach to integrate and evaluate the existing evidence to support the use of SpeechRater SM v1.0 in a low-stakes practice environment. The argumentbased approach provided a mechanism for us to articulate the strengths and weaknesses in the validity argument for using SpeechRater v1.0 and put forward a transparent argument for using it for a low-stakes practice environment. In particular, the construct representation of the multiple regression model with expert weights was sufficiently broad to justify its use in a low-stakes application. While some higher-order aspects of the speaking construct (such as content and organization) are missing, more basic aspects of the construct (such as pronunciation and fluency) are richly represented. In addition, these different parts of the speaking construct tend to be highly correlated, so that the absence of higher order factors is not as detrimental to the model's agreement with human raters as it otherwise might be. The model's agreement with human raters was not sufficiently high to support high-stakes decisions but was still suitable for use in low-stakes applications. The correlation of the 6-item aggregate score with human raters was .57 and was deemed acceptable given the lo...

show abstract

“…According to Shohamy (2001a), all stakeholders such as policy makers, test-writers, students, parents and teachers should be involved in test development and administration. The effects of tests on teachers, students, institutions, and society are accordingly considered one type of validity evidence as many researchers (Bachman, 2005;Cronbach, 1988;Kane, 1992;McNamara & Roever, 2006;Shohamy, 2001;Pan, 2009) have also stressed the importance of justifying test use and investigating its consequences.…”

Section: Discussionmentioning

confidence: 99%

Teachers’ Interpretations and Power in a High-Stakes Test: A CLA Perspective

Tahmasbi¹,

Yamini²

2012

ELR

View full text Add to dashboard Cite

As one of the high-stakes tests known world-wide, Iranian University Entrance Exam (IUEE) influences the lives of a range of people including EFL teachers. This study, conducted within the framework of Critical Language Assessment (CLA), examined the power of EFL teachers in IUEE and their interpretations of language scores therein.The results of factor analysis demonstrated that high-school teachers have no power in IUEE development and administration processes. Teachers also interpreted tests not as indicators of language ability or knowledge but as test-taking skills. Finally, the study reserves the rights for teachers to be involved in the process of IUEE and suggests interpretive approaches to assessment that allow for different meanings and interpretations rather than a single score.

show abstract

Building and Supporting a Case for Test Use

Cited by 294 publications

References 29 publications

INVESTIGATING THE VALUE OF SECTION SCORES FOR THE TOEFL iBT® TEST

INVESTIGATING THE VALUE OF SECTION SCORES FOR THE TOEFL iBT® TEST

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATER^SM V1.0

Teachers’ Interpretations and Power in a High-Stakes Test: A CLA Perspective

Contact Info

Product

Resources

About

Building and Supporting a Case for Test Use

Cited by 294 publications

References 29 publications

INVESTIGATING THE VALUE OF SECTION SCORES FOR THE TOEFL iBT® TEST

INVESTIGATING THE VALUE OF SECTION SCORES FOR THE TOEFL iBT® TEST

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATERSM V1.0

Teachers’ Interpretations and Power in a High-Stakes Test: A CLA Perspective

Contact Info

Product

Resources

About

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATER^SM V1.0