ASSESSING EFL STUDENTS’ ESSAY WRITING WITH AUTOMATED WRITING EVALUATION: VALIDITY AND RELIABILITY EVIDENCE

Abstract

This study aimed to examine the validity and reliability of Automated Writing Evaluation (AWE) in assessing EFL students’ essay writing. A quantitative validation design was employed involving 60 undergraduate students and 60 argumentative essays evaluated by two human raters and an AWE system using an analytic rubric covering content and task achievement, organization, vocabulary, grammatical accuracy, and mechanics. Human and automated scoring reliability was examined using the Intraclass Correlation Coefficient (ICC), while validity was investigated through Pearson correlation, agreement analysis, paired-samples t-test, and Bland–Altman analysis. The findings showed that the AWE system demonstrated excellent test–retest reliability (ICC = .95). The automated scores also showed a strong correlation with human ratings (r = .87) and good overall agreement (ICC = .86). No significant difference was found between the AWE and human scores (p = .749). The strongest agreement was observed for mechanics and grammatical accuracy, whereas the weakest agreement was found for content and task achievement. These findings indicate that AWE has considerable potential as a consistent and valid assessment tool for EFL essay writing; however, human judgment remains necessary, particularly for evaluating complex and interpretive dimensions of writing. The findings are in line with recent research indicating that automated essay scoring can achieve strong reliability while still presenting limitations in construct representation and dimension-level scoring.