Automated scoring and feedback systems: Where are we and where are we heading?

Xi, Xiaoming

doi:10.1177/0265532210364643

Cited by 107 publications

(74 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It would be more conclusive if approached through systematic evidence relevant to the interpretations, uses, and consequences of AWE, compiled under an argument-driven validity framework Xi, 2010). For example, propose a conceptual model for validation of AWE use in local settings, intended as a heuristic for writing program administrators.…”

Section: Need For Empirical Evidencementioning

confidence: 99%

Automated Writing Analysis for Writing Pedagogy

Cotos

2015

WAP

View full text Add to dashboard Cite

This article aims to engage specialists in writing pedagogy, assessment, genre study, and educational technologies in a constructive dialog and joint exploration of automated writing analysis as a potent instantiation of computer-enhanced assessment for learning. It recounts the values of writing pedagogy and, from this perspective, examines legitimate concerns with automated writing analysis. Emphasis is placed on the need to substantiate the construct-driven debate with systematic empirical evidence that would corroborate or refute interpretations, uses, and consequences of automated scoring and feedback tools intended for specific contexts. Such evidence can be obtained by adopting a validity argument framework. To demonstrate an application of this framework, the article presents a novel genre-based approach to automated analysis configured to support research writing and provides examples of validity evidence for using it with novice scholarly writers. Automated Writing Analysis for writing pedagogy: From healthy tension to tangible prospects Elena Cotos AbstractThis article aims to engage specialists in writing pedagogy, assessment, genre study, and educational technologies in a constructive dialog and joint exploration of automated writing analysis as a potent instantiation of computer-enhanced assessment for learning. It recounts the values of writing pedagogy and, from this perspective, examines legitimate concerns with automated writing analysis. Emphasis is placed on the need to substantiate the construct-driven debate with systematic empirical evidence that would corroborate or refute interpretations, uses, and consequences of automated scoring and feedback tools intended for specific contexts. Such evidence can be obtained by adopting a validity argument framework. To demonstrate an application of this framework, the article presents a novel genre-based approach to automated analysis configured to support research writing and provides examples of validity evidence for using it with novice scholarly writers.

show abstract

Section: Need For Empirical Evidencementioning

confidence: 99%

Automated Writing Analysis for Writing Pedagogy

Cotos

2015

WAP

View full text Add to dashboard Cite

show abstract

“…As Galaczi (2010) reminds us, a key concept in language testing is "fitness for purpose": "Tests are not just valid, they are valid for a specific purpose, and as such, different test formats have different applicability for different contexts, age groups, proficiency levels, and score-user requirements" (p. 26). In the same vein, numerous researchers highlight the importance of establishing construct validity with reference to the inferences and decisions made on the basis of test scores (Bernstein et al, 2010;Galaczi, 2010;Xi, 2010). As Bernstein et al (2010) point out, computer test scores provide one piece of evidence about a candidate's performance but should not be necessarily used as the only basis of decisionmaking.…”

Section: Resultsmentioning

confidence: 99%

Analysing Test-Takers’ Views on a Computer-Based Speaking Test

Amengual-Pizarro

García-Laborda

2017

Profile: Issues Teach. Prof. Dev.

View full text Add to dashboard Cite

Key words:Computer-based language testing, oral assessment, second language testing.Este estudio analiza la opinión de los candidatos sobre un examen oral con ordenador para averiguar los aspectos que consideran más relevantes en la evaluación oral a través de las nuevas tecnologías y explorar las principales ventajas y desventajas de este tipo de pruebas comparadas con pruebas orales con evaluadores humanos. Se distribuyó un pequeño cuestionario a 80 candidatos que realizaron el examen oral aptis en la Universidad de Alcalá en abril de 2016. Los resultados revelan que los candidatos consideran que las pruebas orales con ordenador son válidas y adecuadas para la evaluación de la competencia oral. Curiosamente, los datos demuestran que las características personales de los candidatos juegan un papel primordial en la elección del método de evaluación más idóneo.Palabras clave: evaluación de lenguas a través del ordenador, evaluación de la destreza oral, evaluación de segundas lenguas.

show abstract

“…Unlike automated writing assessment, ASE involves an additional layer of complexity in that the test takers' oral output must first be recognized before it can be evaluated (Xi, 2010a). Despite ongoing research and recent advancements in automated speech recognition (ASR), these technologies are not robust at recognizing non-native accented speech because most ASR-based systems have been designed for a narrow range of native speech patterns.…”

Section: Research On Automated Assessmentmentioning

confidence: 99%

“…Other ASE systems (e.g., SpeechRater developed by ETS) compensate for this limitation with free speech recognition by expanding the speaking construct to include pronunciation, vocabulary, and grammar, in addition to fluency (Xi, Higgins, Zechner, & Williamson, 2008). According to Xi (2010a), currently neither of these approaches "has successfully tackled the problem of under-or misrepresentation of the construct of speaking proficiency in either the test tasks used or the automated scoring methodologies, or both" (p. 294).…”

Section: Research On Automated Assessmentmentioning

confidence: 99%

Computer‐Assisted Language Testing

Suvorov

Hegelheimer

2013

The Companion to Language Assessment

View full text Add to dashboard Cite

Technological advancements that occurred since the late 1980s have revolutionized many aspects of human life including second language (L2) education. Language learners and teachers are now presented with a plethora of technology‐enhanced instructional materials and resources varying from simple electronic dictionaries to full‐fledged online language programs. The ubiquity of technology has also affected language assessment and engendered the field of computer‐assisted language testing (CALT), dedicated to exploring how computers and technology can be utilized for evaluating different language skills of non‐native speakers. Computers have had a significant impact on language testing. First, Web‐based L2 tests have created new opportunities for measuring the L2 proficiency of multiple students anywhere in the world without a need for them to be physically present at a language test center. In addition, progress in natural language processing (NLP) techniques have made it possible for test takers to receive immediate assessment of their L2 abilities. Technological advances have also resulted in the development of more secure and reliable language tests. Nowadays one can either choose from a variety of computer‐based language tests designed for different groups of users and different purposes or develop customized language assessment using available tools and applications. However, even though technology appears to have promising potential for language testing, its use is not without challenges and problems. This chapter begins with a brief introduction of the field and presents a framework for the description of computer‐assisted language tests. Next, it includes a description of major existing language tests and platforms for test development and delivery. The following discussion focuses on the principal recent developments in CALT and reports on the main trends in research. The last section of the chapter concludes by outlining the key challenges and future directions for CALT.

show abstract

Automated scoring and feedback systems: Where are we and where are we heading?

Cited by 107 publications

References 20 publications

Automated Writing Analysis for Writing Pedagogy

Automated Writing Analysis for Writing Pedagogy

Analysing Test-Takers’ Views on a Computer-Based Speaking Test

Computer‐Assisted Language Testing

Contact Info

Product

Resources

About