Using generalizability theory to investigate the variability and reliability of EFL composition scores by human raters and e-rater
PORTA LINGUARUM, cilt.2022, sa.38, ss.27-45, 2022 (AHCI, SSCI, Scopus)
- Yayın Türü: Makale / Tam Makale
- Cilt numarası: 2022 Sayı: 38
- Basım Tarihi: 2022
- Doi Numarası: 10.30827/portalin.vi38.18056
- Dergi Adı: PORTA LINGUARUM
- Derginin Tarandığı İndeksler: Arts and Humanities Citation Index (AHCI), Social Sciences Citation Index (SSCI), Scopus, MLA - Modern Language Association Database, DIALNET
- Sayfa Sayıları: ss.27-45
- Anahtar Kelimeler: EFL writing assessment, generalizability theory, scoring variability, scoring reliability, automated writing evaluation (AWE), AUTOMATED WRITING EVALUATION
- Karadeniz Teknik Üniversitesi Adresli: Evet
Özet
Using the generalizability theory (G-theory) as a theoretical framework, this study aimed at investigating the variability and reliability of holistic scores assigned by human raters and e-rater to the same EFL essays. Eighty argumentative essays written on two different topics by tertiary level Turkish EFL students were scored holistically by e-rater and eight human raters who received a detailed rater training. The results showed that e-rater and human raters assigned significantly different holistic scores to the same EFL essays. G-theory analyses revealed that human raters assigned considerably inconsistent scores to the same EFL essays although they were given a detailed rater training and more reliable ratings were attained when e-rater was integrated in the scoring procedure. Some implications are given for EFL writing assessment practices.