Ihmisarvioijien näkemyksiä suullisen kielitaidon automaattisesta arvioinnista, digitaalisesta arviointiprosessista sekä puhesuorituksista arvioitavista ulottuvuuksista

Abstract

This study investigated human raters’ perceptions of automated assessment of oral language skills. The raters (n = 37) participated in three assessment rounds organized by the DigiTala research project using Moodle and Zoom. The raters assessed Finnish and Swedish learners’ speech samples using one holistic and five analytical rating scales created in the project. After the assessment, the raters responded to a questionnaire that included Likert-scale and open-ended questions. Numerical responses were analyzed with descriptive statistics, open responses with content analysis. The raters think that automated scoring could support human rating. The assessment rounds were carried out successfully. The selected dimensions proved to be essential parts of the speaking performances. The results will benefit those working on automated assessment and oral language assessment.

New data, benchmark and baseline for L2 speaking assessment for low-resoure languages

“…The tasks are described in detail in [19], rating scales in [20]. For more details on data collection see [21,22].…”

“…Each assessor participated in our rater training to improve the coherence of the ratings. To be able to analyze the coherence and quality of the ratings, the set of recordings selected for each assessor overlapped with the sets of at least two other assessors [21,22].…”