AI-Integrated Computational Framework for Pronunciation Error Analysis Through Native–Learner Speech Comparison
DOI:
https://doi.org/10.63158/journalisi.v8i5.1812Keywords:
Automatic Pronunciation Assessment, Pronunciation Accuracy Score, Whisper ASR, Native–Learner Speech Comparison, K-Means ClusteringAbstract
This study proposes an AI-integrated computational framework for pronunciation error analysis through native–learner speech comparison. The framework integrates Whisper-based Automatic Speech Recognition (ASR), acoustic feature extraction, statistical analysis, and machine learning to analyze learner pronunciation. Unlike previous studies that primarily rely on individual transcription-based or acoustic measures, the proposed framework integrates multiple pronunciation indicators into a composite Pronunciation Accuracy Score (PAS) formulated in this study. The framework was evaluated using a dataset comprising 45 non-native English learners and one native English speaker. PAS combines Pronunciation Error Rate (PER), Word Error Rate (WER), Pitch Similarity, Rhythm Similarity, and Mel-Frequency Cepstral Coefficient (MFCC) Similarity. Pearson correlation analysis showed that Rhythm Similarity (r = 0.544) and Pitch Similarity (r = 0.510) exhibited the strongest positive correlations with PAS, whereas Word Error Rate showed the strongest negative correlation (r = −0.495). K-Means clustering identified three learner groups with distinct pronunciation characteristics within the tested dataset. Although evaluated on a relatively small dataset without validation against expert human ratings, the proposed framework provides a promising foundation for Computer-Assisted Pronunciation Training (CAPT) and future multilingual pronunciation assessment.
Downloads
References
[1] M. Farrús, "Automatic speech recognition in L2 learning: A review based on PRISMA methodology," Languages, vol. 8, no. 4, art. no. 242, 2023, doi: 10.3390/languages8040242.
[2] H. Ahlawat, N. Aggarwal, and D. Gupta, "Automatic speech recognition: A survey of deep learning techniques and approaches," Int. J. Cogn. Comput. Eng., vol. 6, pp. 201–237, 2025, doi: 10.1016/j.ijcce.2024.12.007.
[3] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, "HuBERT: Self-supervised speech representation learning by masked prediction of hidden units," IEEE/ACM Trans. Audio Speech Lang. Process., vol. 29, pp. 3451–3460, 2021, doi: 10.1109/TASLP.2021.3122291.
[4] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," in Proc. 34th Conf. Neural Inf. Process. Syst., vol. 33, 2020. [Online]. Available: http://arxiv.org/abs/2006.11477
[5] V. Roger, J. Farinas, and J. Pinquier, "Deep neural networks for automatic speech processing: A survey from large corpora to limited data," EURASIP J. Audio Speech Music Process., vol. 2022, no. 1, pp. 1–15, 2022, doi: 10.1186/s13636-022-00251-w.
[6] M. Amrate and P. H. Tsai, "Computer-assisted pronunciation training: A systematic review," ReCALL, vol. 37, no. 1, pp. 22–42, 2025, doi: 10.1017/S0958344024000181.
[7] M. Y. C. Jiang et al., "Integrating automatic speech recognition technology into vocabulary learning in a flipped English class for Chinese college students," Front. Psychol., vol. 13, art. no. 902429, 2022, doi: 10.3389/fpsyg.2022.902429.
[8] Y. El Kheir, A. Ali, and S. A. Chowdhury, "Automatic pronunciation assessment — A review," in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, doi: 10.18653/v1/2023.findings-emnlp.557.
[9] Y. Gong, Z. Chen, I.-H. Chu, P. Chang, and J. Glass, "Transformer-based multi-aspect multi-granularity non-native English speaker pronunciation assessment," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2022, pp. 7262–7266, doi: 10.1109/ICASSP43922.2022.9746743.
[10] B.-C. Yan et al., "An effective pronunciation assessment approach leveraging hierarchical transformers and pre-training strategies," in Proc. 62nd Annu. Meeting Assoc. Comput. Linguistics, Stroudsburg, PA, USA, 2024, pp. 1737–1747, doi: 10.18653/v1/2024.acl-long.95.
[11] W. Dong, C. Cucchiarini, R. van Hout, and H. Strik, "Predicting accentedness and comprehensibility through ASR scores and acoustic features," Comput. Speech Lang., vol. 95, art. no. 101858, 2026, doi: 10.1016/j.csl.2025.101858.
[12] E. J. Yeo, K. Choi, S. Kim, and M. Chung, "Speech intelligibility assessment of dysarthric speech by using goodness of pronunciation with uncertainty quantification," in Proc. INTERSPEECH, Dublin, Ireland, 2023, pp. 166–170, doi: 10.21437/Interspeech.2023-173.
[13] M. Kim, "Digital enhancement of pronunciation assessment: Automated speech recognition and human raters," Phonet. Speech Sci., vol. 15, no. 2, pp. 13–20, 2023, doi: 10.13064/ksss.2023.15.2.013.
[14] Y. Gao, B. M. L. Srivastava, and J. Salsman, "Spoken English intelligibility remediation with PocketSphinx alignment and feature extraction improves substantially over the state of the art," in Proc. IMCEC, 2018, doi: 10.1109/IMCEC.2018.8469649.
[15] J. Wang and Y. Zheng, "Assessing intelligibility as conceptualised within the CEFR-companion volume framework using adaptive comparative judgement," Lang. Test., vol. 43, no. 1, pp. 9–36, 2025, doi: 10.1177/02655322251338605.
[16] S. Ali, S. Tanweer, S. Khalid, and N. Rao, "Mel frequency cepstral coefficient: A review," in Proc. 2nd Int. Conf. ICT Digit., Smart, Sustain. Develop., 2021, doi: 10.4108/eai.27-2-2020.2303173.
[17] F. W. S. dos Santos and H. K. de Souza, "Examining vowel intelligibility in Brazilian Portuguese EFL learners: A study using automatic speech recognition," Ilha do Desterro, vol. 77, pp. 243–274, 2024, doi: 10.5007/2175-8026.2024.e98870.
[18] L. R. Rabiner and B.-H. Juang, Fundamentals of Speech Recognition, 1st ed. Englewood Cliffs, NJ, USA: PTR Prentice Hall, 1993.
[19] L. R. Rabiner and M. R. Sambur, "An algorithm for determining the endpoints of isolated utterances," Bell Syst. Tech. J., vol. 54, no. 2, pp. 297–315, 1975, doi: 10.1002/j.1538-7305.1975.tb02840.x.
[20] S. Graf, T. Herbig, M. Buck, and G. Schmidt, "Features for voice activity detection: A comparative analysis," EURASIP J. Adv. Signal Process., vol. 2015, no. 1, Dec. 2015, doi: 10.1186/s13634-015-0277-z.
[21] J. Ramírez, J. C. Segura, C. Benítez, A. De La Torre, and A. Rubio, "Improved voice activity detection combining noise reduction and subband divergence measures," in Proc. INTERSPEECH, Jeju Island, Korea, 2004, pp. 961–964, doi: 10.21437/Interspeech.2004-347.
[22] Z. Zaland et al., "Multichannel speech enhancement for automatic speech recognition: A literature review," PeerJ Comput. Sci., vol. 11, art. no. e2772, 2025, doi: 10.7717/peerj-cs.2772.
[23] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, "Robust speech recognition via large-scale weak supervision," in Proc. 40th Int. Conf. Mach. Learn., 2023, pp. 28492–28518. [Online]. Available: http://arxiv.org/abs/2212.04356
[24] V. K. Singh, "Transcripto: Fine-tuning multilingual ASR for Indian grievance feedback calls," J. Inf. Syst. Eng. Manag., vol. 10, no. 42s, 2025, doi: 10.52783/jisem.v10i42s.7880.
[25] E. M. N. Saeed, B. Al-sellami, M. Tawfik, and A. A. M. Abushariah, "KANWhisper: Leveraging learnable activation functions for interpretable and efficient Arabic automatic speech recognition," Sci. Rep., vol. 16, art. no. 55863, 2026, doi: 10.1038/s41598-026-55863-5.
[26] A. Babaeian, "Speaking to machine: Validity of pronunciation assessment by technology-driven systems," Integr. J. Res. Arts Humanit., vol. 5, no. 3, pp. 276–279, 2025, doi: 10.55544/ijrah.5.3.31.
[27] J. Gao, "An investigation of high-proficiency L2 English speakers' oral test performance: A profiling approach," Front. Commun., vol. 7, art. no. 926409, 2022, doi: 10.3389/fcomm.2022.926409.
[28] S. Inceoglu, W. H. Chen, and H. Lim, "Monitoring student behavior in autonomous automatic speech recognition-based pronunciation practice," System, vol. 124, art. no. 103387, 2024, doi: 10.1016/j.system.2024.103387.
[29] V. Berisha and J. M. Liss, "Responsible development of clinical speech AI: Bridging the gap between clinical research and technology," NPJ Digit. Med., vol. 7, no. 1, Dec. 2024, doi: 10.1038/s41746-024-01199-1.
[30] M. Bashori, R. van Hout, H. Strik, and C. Cucchiarini, "'Look, I can speak correctly': Learning vocabulary and pronunciation through websites equipped with automatic speech recognition technology," Comput. Assist. Lang. Learn., vol. 37, no. 4, pp. 1335–1363, 2024, doi: 10.1080/09588221.2022.2080230.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Information Systems and Informatics
This work is licensed under a Creative Commons Attribution 4.0 International License.
Author's Declaration
- The Authors certify that they have read, understood, and agreed to the Journal of Information Systems and Informatics (JournalISI) submission guidelines, policies, and submission declaration. The submission has been prepared using the provided template.
- The Authors certify that all authors have approved the publication of this manuscript and that there is no conflict of interest.
- The Authors confirm that the manuscript is their original work, has not received prior publication, is not under consideration for publication elsewhere, and has not been previously published.
- The Authors confirm that all authors listed on the title page have contributed significantly to the work, have read the manuscript, attest to the validity and legitimacy of the data and its interpretation, and agree to its submission.
- The Authors confirm that the manuscript is not copied from or plagiarized from any other published work.
- The Authors declare that the manuscript will not be submitted for publication in any other journal or magazine until a decision is made by the journal editors.
- If the manuscript is finally accepted for publication, the Authors confirm that they will either proceed with publication immediately or withdraw the manuscript in accordance with the journal’s withdrawal policies.
- The Authors agree that, upon publication of the manuscript in this journal, they transfer copyright or assign exclusive rights to the publisher, including commercial rights
