AUTOMATED ASSESSMENT OF SPOKEN ENGLISH AT CONSUMER SCALE: AN ASR + LLM PIPELINE EVALUATED AGAINST HUMAN EXAMINERS
Abstract
The rapid growth of online English-learning platforms has created demand for spoken-language assessment that is instant, low-cost, and consistent, yet human examining remains the accepted standard. This study evaluates an automated pipeline that couples automatic speech recognition (ASR) with a large language model (LLM) scorer against certified human examiners. A private corpus of 1,240 monologue responses from 310 learners, collected on the basis of the Speaklish.uz platform, was scored on a CEFR-aligned analytic rubric by two trained raters and by the pipeline. Agreement between the automated overall score and the human consensus reached a Pearson correlation of r = 0.87 and a quadratic weighted kappa of 0.79, approaching the inter-rater ceiling of r = 0.91. The mean absolute error was 0.42 band on a nine-band scale, and 84.6 % of automated scores fell within half a band of the human consensus. Word error rate of the ASR stage was 11.3 % overall but rose to 19.8 % for CEFR A2 speakers, and this degradation propagated into pronunciation scoring. The findings indicate that an ASR + LLM pipeline can deliver examiner-comparable reliability for formative, consumer-scale assessment, provided that low-proficiency speech and high-stakes decisions retain human oversight.
References
1. HolonIQ. Global EdTech Market Sizing: Language Learning. 2024. URL: https://www.holoniq.com/notes/global-language-learning-market
2. Radford A., Kim J.W., Xu T. et al. Robust Speech Recognition via Large-Scale Weak Supervision // Proceedings of the 40th International Conference on Machine Learning (ICML). 2023. Vol. 202. P. 28492-28518.
3. Naismith B., Mulcaire P., Burstein J. Automated Evaluation of Written Discourse Coherence Using Large Language Models // Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications. 2023. P. 394-403.
4. Zechner K., Higgins D., Xi X., Williamson D.M. Automatic Scoring of Non-Native Spontaneous Speech in Tests of Spoken English // Speech Communication. 2009. Vol. 51, No. 10. P. 883-895.
5. Council of Europe. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Companion Volume. Strasbourg: Council of Europe Publishing, 2020. 278 p.
6. Litman D., Strik H., Lim G.S. Speech Technologies and the Assessment of Second Language Speaking // Language Assessment Quarterly. 2018. Vol. 15, No. 3. P. 294-309.
7. O‘zbekiston Respublikasi Prezidentining “Chet tillarni o‘rganishni ommalashtirish tizimini tubdan takomillashtirish chora-tadbirlari to‘g‘risida”gi PQ-4406-son qarori. 2019-yil 19-may.
8. Xi X. Automated Scoring and Feedback Systems: Where Are We and Where Are We Heading? // Language Testing. 2010. Vol. 27, No. 3. P. 291-300.
9. Bannò S., Matassoni M. Proficiency Assessment of L2 Spoken English Using Wav2vec 2.0 // Proceedings of the IEEE Spoken Language Technology Workshop (SLT). 2022. P. 1088-1095.
10. Wang X., Evanini K., Zechner K., Mulholland M. Modeling Discourse Coherence for the Automated Scoring of Spontaneous Spoken Responses // Proceedings of the 8th ISCA Workshop on Speech and Language Technology in Education (SLaTE). 2019. P. 132-136.
11. OpenAI. GPT-4 Technical Report. 2023. URL: https://arxiv.org/abs/2303.08774
12. Fu J., Ng S.-K., Liu P. GPTScore: Evaluate as You Desire // Proceedings of the 2024 Conference of the North American Chapter of the ACL. 2024. P. 6556-6576.
13. Isaacs T., Trofimovich P. Pronunciation and Second Language Assessment // Language Assessment Quarterly. 2017. Vol. 14, No. 2. P. 89-95.
14. Ramanarayanan V., Suendermann-Oeft D., Lange P. et al. Assembling the Jigsaw: How Multiple Open Standards Are Synergistically Combined in the HALEF Multimodal Dialog System // Multimodal Interaction with W3C Standards. Cham: Springer, 2017. P. 295-310.
15. British Council, IDP: IELTS Australia, Cambridge Assessment English. IELTS Speaking Band Descriptors (Public Version). 2023. URL: https://www.ielts.org/-/media/pdfs/speaking-band-descriptors.ashx
16. Speaklish. ORPO-dataset. Hugging Face Datasets. URL: https://huggingface.co/datasets/Speaklish/ORPO-dataset
