Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing

East, Martin

doi:10.1016/j.asw.2009.04.001

Cited by 71 publications

(45 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Assessing writing ability and the reliability of ratings have been a challenging concern for decades and there is always variation in the elements of writing preferred by raters and there are extraneous factors causing variation (Blok, 1985;Chase, 1968;Chase, 1983;Darus, 2006;East, 2009;Engelhard, 1994;Gyagenda & Engelhard, 1998a;Gyagenda & Engelhard, 1998b;Hughes, Keeling & Tuck, 1980;Hughes, Keeling & Tuck, 1983;Hughes & Keeling, 1984;Kan, 2005;Klein & Hart, 1968;Klein & Taub, 2005;Marshall & Powers, 1969;Murphy & Balzer, 1989;Schaefer, 2008;Slomp, 2012;Sulsky & Balzer, 1988;Wexley & Youtz, 1985;Woehr & Huffcutt, 1994). Fisher, Brooks, and Lewis (2002) state fitness for purpose requirement is the core of all testing work, and direct writing assessments are subjective and thereby more prone to reliability issues.…”

Section: Conclusion and Recommendationsmentioning

confidence: 99%

Measuring Essay Assessment: Intra-rater and Inter-rater Reliability

Kayapinar¹

2014

EJER

View full text Add to dashboard Cite

Problem Statement: There have been many attempts to research the effective assessment of writing ability, and many proposals for how this might be done. In this sense, rater reliability plays a crucial role for making vital decisions about testees in different turning points of both educational and professional life. Intra-rater and inter-rater reliability of essay assessments made by using different assessing tools should also be discussed with the assessment processes. Purpose of Study:The purpose of the study is to reveal possible variation or consistency in grading essay writing ability of EFL writers by the same/different raters using general impression marking (GIM), essay criteria checklist (ECC), and essay assessment scale (ESAS), and discuss rater reliability.Methods: Quantitative and qualitative data were used to present the discussion and implications for the reliability of ratings and the consistency of the measurement results. The assessing tools were applied to 44 EFL university students and 10 graders assessed the essay writing ability of the students by using GIM, ECC, and ESAS in different occasions. Findings and Results:The findings and results of the analyses indicated that using general impression marking is evidently not reliable for assessing essays. The coefficients obtained from checklist and scale assessments, considering the correlation coefficients, estimated variance components, and generalizability coefficients present valuable information, clearly show that there is always variation among the results.

show abstract

Section: Conclusion and Recommendationsmentioning

confidence: 99%

Measuring Essay Assessment: Intra-rater and Inter-rater Reliability

Kayapinar¹

2014

EJER

View full text Add to dashboard Cite

show abstract

“…Araştırmada DPA kullanan öğrencilerle kullanmayan öğrencilerin başarıları karşılaştırılmıştır. Bütünsel DPA'ların nesnel ve güvenilir değerlendirme için uygun olmadığı gerekçesiyle (Elbow, 2000;Gunning, 2006;East, 2009;Koutsoftas ve Gray, 2012) bu çalışmada göreve özel çözümleyici puanlama anahtarının kullanılması yeğlenmiştir.…”

Section: Yöntemunclassified

Dereceli Puanlama Anahtarı Kullanan Öğrencilerin Başarıları Üzerine Bir İnceleme

Bayraktar¹,

Aslan²

2016

Ana Dili Eğitimi Dergisi

View full text Add to dashboard Cite

ÖzBu çalışmanın genel amacı, performans ödevlerini hazırlarken Derecelendirilmiş Puanlama Anahtarı (DPA) kullanan ortaokul öğrencileriyle kullanmayan öğrencilerin başarıları arasında fark olup olmadığını belirlemektir. Çalışma grubu, aynı Türkçe öğretmeninden ders alan ve Türkçe dersindeki akademik başarıları birbirine denk iki 6. sınıf öğrencisinden (n=44) oluşmaktadır. Bu gruplardan birisi rastlantısal atama yoluyla deney grubu (n=22), diğeri ise kontrol grubu (n=22) olarak belirlenmiştir. Araştırmanın verileri öğrencilerin performans ödevlerinden aldıkları puanlardan toplanmıştır. Çalışmanın ilk aşamasında öğrencilerin düzeyine ve Türkçe Öğretimi Programı'nın 6. sınıf kazanımlarına uygun bir performans ödevi hazırlanmıştır. Daha sonra bu ödevin nasıl yapılacağını açıklayan ve her iki gruptaki öğrencilere de verilen ayrıntılı bir yönerge ile alanyazın taraması, uzmanların görüş ve öneriler i doğrultusunda dokuz boyutu olan çözümleyici (analitik) bir DPA hazırlanmıştır. Ayrıntılı yönerge her iki gruba, DPA ise yalnızca deney grubuna verilmiştir. Öğrencilerin hazırladıkları ödevler bu DPA'nın belirlenen dokuz boyutuna göre deneyimli iki Türkçe öğretmeni tarafından değerlendirilmiştir. Shapiro-Wilk testine göre dağılımların normal olduğu belirlenmiş, bu nedenle Pearson korelasyon katsayısı hesaplanmış ve değerlendi riciler arasındaki tutarlılık .77 olarak belirlenmiştir. Bu değer .05 düzeyinde anlamlı olup, aynı zamanda iki puanlayıcı uyumu için yeterli düzeydedir. İki grubun performans ödevlerindeki başarı puanları arasında fark olup olmadığına Mann-Whitney U testi ile bakılmış; sonuçta yalnızca dil ve a nlatım boyutlarında deney grubu öğrencilerinin lehine anlaml ı bir fark olduğu görülmüştür.Anahtar sözcükler: Dereceli puanlama anahtarı, Türkçe dersi, performans ödevi, değerlendirme. A Study on the Achievement of Students who Use Scoring Rubrics AbstractThe general aim of this study is to determine achievement differences between students who use rubrics while preparing their performance homework assignments and those who don't. The study group consisted of forty-four 6 th grade students who were taking the Turkish course from

show abstract

“…This pattern is evident in work on rating scale validation for writing, where researchers have most commonly used statistical techniques to show that a scale can be used reliably (e.g. East, 2009;Harsch & Martin, 2012), how well different scale categories (levels) perform (e.g. Knoch, 2007Knoch, , 2009, and how many different dimensions subcategories of an analytic scale measure (e.g.…”

Section: Rating Scale Validationmentioning

confidence: 99%

Second Language Pronunciation Assessment

Fulcher¹,

Browne²

2016

View full text Add to dashboard Cite

General rightsThis document is made available in accordance with publisher policies. Please cite only the published version using the reference above. Full terms of use are available: http://www.bristol.ac.uk/pure/about/ebr-terms This series brings together titles dealing with a variety of aspects of language acquisition and processing in situations where a language or languages other than the native language is involved. Second language is thus interpreted in its broadest possible sense. The volumes included in the series all offer in their different ways, on the one hand, exposition and discussion of empirical findings and, on the other, some degree of theoretical reflection. In this latter connection, no particular theoretical stance is privileged in the series; nor is any relevant perspective -sociolinguistic, psycholinguistic, neurolinguistic, etc. -deemed out of place. The intended readership of the series includes final-year undergraduates working on second language acquisition projects, postgraduate students involved in second language acquisition research, and researchers, teachers and policy-makers in general whose interests include a second language acquisition component. Second Language Pronunciation Assessment SECOND LANGUAGE ACQUISITIONFull details of all the books in this series and of all our other publications can be found on http://www.multilingual-matters.com, or by writing to Multilingual Matters, St Nicholas House, 31-34 High Street, Bristol BS1 2AW, UK. SECOND LANGUAGE ACQUISITION: 107 Second Language Pronunciation AssessmentInterdisciplinary Perspectives All rights reserved. No part of this work may be reproduced in any form or by any means without permission in writing from the publisher. Edited by Talia Isaacs and Pavel Trofi movich MULTILINGUAL MATTERSThe policy of Multilingual Matters/Channel View Publications is to use papers that are natural, renewable and recyclable products, made from wood grown in sustainable forests. In the manufacturing process of our books, and to further support our policy, preference is given to printers that have FSC and PEFC Chain of Custody certification. The FSC and/or PEFC logos will appear on those books where full certification has been granted to the printer concerned.Typeset by Nova Techset Private Limited, Bengaluru and Chennai, India. Printed and bound in the UK by the CPI Books Group Ltd. Printed and bound in the US by Edwards Brothers Malloy, Inc. In Memory of Alan Davies and Danielle Guénette AcknowledgementsThis edited volume, which brings together different but complementary research perspectives to establish a common platform in which to discuss issues relevant to assessing second language (L2) pronunciation, would not have been possible without the contributions and commitment of the authors, who explore key issues through different disciplinary lenses in the chapters that make up this volume. The vision for the book arose during a cold Canadian winter at the beginning of the second decade of the 21st century, when a sense of momentum for i...

show abstract

Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing

Cited by 71 publications

References 19 publications

Measuring Essay Assessment: Intra-rater and Inter-rater Reliability

Measuring Essay Assessment: Intra-rater and Inter-rater Reliability

Dereceli Puanlama Anahtarı Kullanan Öğrencilerin Başarıları Üzerine Bir İnceleme

Second Language Pronunciation Assessment

Contact Info

Product

Resources

About