Using generalizability theory to investigate the variability and reliability of EFL composition scores by human raters and e-rater

SARI E., Han T.

PORTA LINGUARUM, cilt.2022, sa.38, ss.27-45, 2022 (AHCI) identifier identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 2022 Sayı: 38
  • Basım Tarihi: 2022
  • Doi Numarası: 10.30827/portalin.vi38.18056
  • Dergi Adı: PORTA LINGUARUM
  • Derginin Tarandığı İndeksler: Arts and Humanities Citation Index (AHCI), Social Sciences Citation Index (SSCI), Scopus, MLA - Modern Language Association Database, DIALNET
  • Sayfa Sayıları: ss.27-45
  • Anahtar Kelimeler: EFL writing assessment, generalizability theory, scoring variability, scoring reliability, automated writing evaluation (AWE), AUTOMATED WRITING EVALUATION
  • Karadeniz Teknik Üniversitesi Adresli: Evet

Özet

Using the generalizability theory (G-theory) as a theoretical framework, this study aimed at investigating the variability and reliability of holistic scores assigned by human raters and e-rater to the same EFL essays. Eighty argumentative essays written on two different topics by tertiary level Turkish EFL students were scored holistically by e-rater and eight human raters who received a detailed rater training. The results showed that e-rater and human raters assigned significantly different holistic scores to the same EFL essays. G-theory analyses revealed that human raters assigned considerably inconsistent scores to the same EFL essays although they were given a detailed rater training and more reliable ratings were attained when e-rater was integrated in the scoring procedure. Some implications are given for EFL writing assessment practices.