Hostname: page-component-5b777bbd6c-5mwv9 Total loading time: 0 Render date: 2025-06-21T12:41:03.953Z Has data issue: false hasContentIssue false
  • English
  • Français

An investigation of machine translation output quality and the influencing factors of source texts

Published online by Cambridge University Press:  01 July 2021

Sangmin-Michelle Lee Open the ORCID record for Sangmin-Michelle Lee [Opens in a new window]
Sangmin-Michelle Lee*
Affiliation:
Kyung Hee University, Republic of Korea (sangminlee@khu.ac.kr)
Get access Rights & Permissions [Opens in a new window]

Abstract

The use of machine translation (MT) in the academic context has increased in recent years. Hence, language teachers have found it difficult to ignore MT, which has led to some concerns. Among the concerns, its accuracy has become a major factor that shapes language teachers’ pedagogical decision to use MT in their language classrooms. Despite the urgency of the issue, studies on MT output quality in foreign language education remain scarce. Moreover, as MT is advancing every year, updated studies are imperative. Therefore, the present study investigated the quality of MT outputs (Google Translate) from Korean to English by comparing it with the English-translated texts of intermediate English as a foreign language students. The study also examined the factors within the source texts that affect the quality of MT outputs. Five trained evaluators examined multiple aspects of MT output samples (N = 104) and students’ English texts (N = 104), including mechanics, vocabulary, grammar, and context. The results showed that both texts were equally comprehensible, but MT outperformed the students in most aspects under investigation. The study further found that only two factors in the source texts – punctuation and sentence complexity – influenced MT output quality, whereas lexical and grammatical accuracy, lexical diversity, and contextual understanding did not affect it. Based on the results, the study presents classroom implications for using MT for educational purposes.

Keywords

machine translationqualitysource textinfluencing factors
Type
Research Article
Information
ReCALL , Volume 34 , Issue 1 , January 2022 , pp. 81 - 94
Check for updates
Copyright
© The Author(s), 2021. Published by Cambridge University Press on behalf of European Association for Computer Assisted Language Learning

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Abraham, L. B. (2009) Web-based translation for promoting language awareness: Evidence from Spanish. In Abraham, L. B. & Williams, L. (eds.), Electronic discourse in language learning and language teaching. Amsterdam: John Benjamins, 65–83. https://doi.org/10.1075/lllt.25.06abrl CrossRefGoogle Scholar
Aiken, M. & Balan, S. (2011) An analysis of Google Translate accuracy. Translation Journal, 16(2). http://translationjournal.net/journal//51pondering.htm Google Scholar
Alhaisoni, E. & Alhaysony, M. (2017) An investigation of Saudi EFL university students’ attitudes towards the use of Google Translate. International Journal of English Language Education, 5(1): 72–82. https://doi.org/10.5296/ijele.v5i1.10696 CrossRefGoogle Scholar
Amaral, L. & Meurers, D. (2011) On using intelligent computer-assisted language learning in real-life foreign language teaching and learning. ReCALL, 23(1): 4–24. https://doi.org/10.1017/S0958344010000261 CrossRefGoogle Scholar
Bahri, H. & Mahadi, T. S. T. (2016) Google Translate as a supplementary tool for learning Malay: A case study at Universiti Sains Malaysia. Advances in Language and Literary Studies, 7(3): 161–167. https://doi.org/10.7575/aiac.alls.v.7n.3p.161 Google Scholar
Barr, D. (2013) Embedding technology in translation teaching: Evaluative considerations for courseware integration. Computer Assisted Language Learning, 26(4): 295–310. https://doi.org/10.1080/09588221.2012.658406 CrossRefGoogle Scholar
Bowker, L. & Ciro, J. B. (2019) Machine translation and global research: Towards improved machine translation literacy in the scholarly community . Bingly: Emerald Publishing. https://doi.org/10.1108/9781787567214 Google Scholar
Briggs, N. (2018) Neural machine translation tools in the language learning classroom: Students’ use, perceptions, and analyses. The JALT CALL Journal, 14(1): 3–24. https://doi.org/10.29140/jaltcall.v14n1.221 CrossRefGoogle Scholar
Choi, Y. H. & Lee, J. (2006) L1 use in L2 writing process of Korean EFL students. English Teaching, 61(1): 205–225.Google Scholar
Clifford, J., Merschel, L. & Munné, J. (2013) Surveying the landscape: What is the role of machine translation in language learning? The Acquisition of Second Languages and Innovative Pedagogies, 10: 108–121. https://doi.org/10.7203/attic.10.2228 Google Scholar
Cook, G. (2012) Translation in language teaching: An argument for reassessment. Oxford: Oxford University Press.Google Scholar
Correa, M. (2014) Leaving the “peer” out of peer-editing: Online translators as a pedagogical tool in the Spanish as a second language classroom. Latin American Journal of Content and Language Integrated Learning, 7(1): 1–20. https://doi.org/10.5294/laclil.2014.7.1.1 CrossRefGoogle Scholar
Ducar, C. & Schocket, D. H. (2018) Machine translation and the L2 classroom: Pedagogical solutions for making peace with Google Translate. Foreign Language Annals, 51(4): 779–795. https://doi.org/10.1111/flan.12366 CrossRefGoogle Scholar
Fredholm, K. (2015) Online translation use in Spanish as a foreign language essay writing: Effects on fluency, complexity and accuracy. Revista Nebrija de Lingüística Aplicada a la Enseñanza de las Lenguas, 18: 7–24.Google Scholar
Fredholm, K. (2019) Effects of Google Translate on lexical diversity: Vocabulary development among learners of Spanish as a foreign language. Revista Nebrija de Lingüística Aplicada a la Enseñanza de las Lenguas, 13(26): 98–117. https://doi.org/10.26378/rnlael1326300 Google Scholar
Garcia, I. (2016) Can machine translation help the language learner? International conference ICT for language learning. https://conference.pixel-online.net/conferences/ICT4LL2010/common/download/Proceedings_pdf/TRAD02-Garcia.pdf Google Scholar
Garcia, I. & Pena, M. I. (2011) Machine translation-assisted language learning: Writing for beginners. Computer Assisted Language Learning, 24(5): 471–487. https://doi.org/10.1080/09588221.2011.582687 CrossRefGoogle Scholar
Giannetti, T. R. (2016) Google Translate as a resource for writing: A study of error production in seventh grade Spanish. St. John Fisher College, master’s thesis.Google Scholar
Godwin-Jones, R. (2019) In a world of SMART technology, why learn another language? Educational Technology & Society, 22(2): 4–13.Google Scholar
Goulet, M.-J., Simard, M., Parra Escartín, C. & O’Brien, S. (2017) Using machine translation for academic writing in English as a second language: Results of an exploratory study on linguistic quality. ASp: la revue du GERAS, 72: 5–28. https://doi.org/10.4000/asp.5045 CrossRefGoogle Scholar
Groves, M. & Mundt, K. (2015) Friend or foe? Google Translate in language for academic purposes. English for Specific Purposes, 37: 112–121. https://doi.org/10.1016/j.esp.2014.09.001 CrossRefGoogle Scholar
Im, H.-J. (2017) The university students’ perceptions or attitudes on the use of the English automatic translation in a general English class: Based on English writing lessons. Korean Journal of General Education, 11(6): 727–751.Google Scholar
Jolley, J. R. & Maimone, L. (2015) Free online machine translation: Use and perceptions by Spanish students and instructors. In Moeller, A. J. (ed.), Learn language, explore cultures, transform lives: Selected papers from the 2015 Central States Conference on the Teaching of Foreign Languages. Richmond: Robert M. Terry, 181–200.Google Scholar
Kazemzadeh, A. & Kashani, A. (2014) The effect of computer-assisted translation on L2 learners’ mastery of writing. International Journal of Research Studies in Language Learning, 3: 29–44.Google Scholar
Kim, E.-Y. (2011) Using translation exercises in the communicative EFL writing classroom. ELT Journal, 65(2): 154–160. https://doi.org/10.1093/elt/ccq039 CrossRefGoogle Scholar
Kim, S. (2019) Playing with machine translation in language classroom: Affordances and constraints. Multimedia-Assisted Language Learning, 22(2): 9–28.Google Scholar
Kim, Y. & Yoon, H. (2014) The use of L1 as a writing strategy in L2 writing tasks. GEMA Online Journal of Language Studies, 14(3): 33–50. https://doi.org/10.17576/GEMA-2014-1403-03 CrossRefGoogle Scholar
Kliffer, M. D. (2008) Post-editing machine translation as an FSL exercise. Porta Linguarum, 9: 53–67. https://doi.org/10.30827/Digibug.31745 Google Scholar
Kol, S., Schcolnik, M. & Spector-Cohen, E. (2018) Google Translate in academic writing courses? The EUROCALL Review, 26(2): 50–57. https://doi.org/10.4995/eurocall.2018.10140 CrossRefGoogle Scholar
Lee, S. (2019) Korean college students’ perceptions toward the effectiveness of machine translation on L2 revision. Multimedia-Assisted Language Learning, 22(4): 206–225. https://doi.org/10.15702/mall.2019.22.4.206 Google Scholar
Lee, S.-M. (2020) The impact of using machine translation on EFL students’ writing. Computer Assisted Language Learning, 33(3): 157–175. https://doi.org/10.1080/09588221.2018.1553186 CrossRefGoogle Scholar
Marinac, M. & Barić, I. (2018) Teachers’ attitudes toward and use of translation in the foreign language classroom at institutions of higher education in Croatia. Theory and Practice in Language Studies, 8(8): 906–915. https://doi.org/10.17507/tpls.0808.02 CrossRefGoogle Scholar
Maruf, S. & Haffari, G. (2018) Document context neural machine translation with memory networks. In Gurevych, I. & Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long papers). Stroudsburg: Association for Computational Linguistics, 1275–1284.CrossRefGoogle Scholar
Murtisari, E. T., Widiningrum, R., Branata, J. & Susanto, R. D. (2019) Google Translate in language learning: Indonesian EFL students’ attitudes. The Journal of AsiaTEFL, 16(3): 978–986. https://doi.org/10.18823/asiatefl.2019.16.3.14.978 CrossRefGoogle Scholar
Niño, A. (2008) Evaluating the use of machine translation post-editing in the foreign language class. Computer Assisted Language Learning, 21(1): 29–49. https://doi.org/10.1080/09588220701865482 CrossRefGoogle Scholar
Niño, A. (2009) Machine translation in foreign language learning: Language learners’ and tutors’ perceptions of its advantages and disadvantages. ReCALL, 21(2): 241–258. https://doi.org/10.1017/S0958344009000172 CrossRefGoogle Scholar
O’Brien, S., Simard, M., & Goulet, M.-J. (2018) Machine translation and self-post-editing for academic writing support: Quality explorations. In Moorkens, J., Castilho, S., Gaspari, F. & Doherty, S. (eds.), Translation quality assessment: From principles to practice. Cham: Springer, 237–262. https://doi.org/10.1007/978-3-319-91241-7_11 CrossRefGoogle Scholar
O’Neill, E. (2013) Online translator usage in foreign language writing. In Swanson, P. B. & Hoyt, K. (eds.), Dimension 2013. World language learning: Setting the global standard. Decatur: Southern Conference on Language Teaching, 74–88.Google Scholar
O’Neill, E. M. (2014) Real-life technology and the L2 French classroom: Online translation usage among intermediate French students. Proceedings of the 2014 American Association of Teachers of French Convention. New Orleans, LA, 19–22 July.Google Scholar
O’Neill, E. M. (2016) Measuring the impact of online translation on FL writing scores. The IALLT Journal, 46(2): 1–39. https://doi.org/10.17161/iallt.v46i2.8560 CrossRefGoogle Scholar
Park, J. (2017) Analysis of the types of translation errors by the Google Translator. Studies on English Language & Literature, 59(4): 279–299.Google Scholar
Ruiz, N. & Federico, M. (2014) Complexity of spoken versus written language for machine translation. In Cettolo, M., Federico, M., Specia, L. & Way, A. (eds.), Proceedings of the 17th Annual Conference of the European Association for Machine Translation. Allschwil: European Association for Machine Translation, 173–180.Google Scholar
Shadiev, R., Sun, A. & Huang, Y.-M. (2019) A study of the facilitation of cross-cultural understanding and intercultural sensitivity using speech-enabled language translation technology. British Journal of Educational Technology, 50(3): 1415–1433. https://doi.org/10.1111/bjet.12648 CrossRefGoogle Scholar
Stapleton, P. & Kin, B. L. K. (2019) Assessing the accuracy and teachers’ impressions of Google Translate: A study of primary L2 writers in Hong Kong. English for Specific Purposes, 56: 18–34. https://doi.org/10.1016/j.esp.2019.07.001 CrossRefGoogle Scholar
Steding, S. (2009) Machine translation in the German classroom: Detection, reaction, prevention. Die Unterrichtspraxis/Teaching German, 42(2): 178–189. https://doi.org/10.1111/j.1756-1221.2009.00052.x CrossRefGoogle Scholar
Sun, D. (2017) Application of post-editing in foreign language teaching: Problems and challenges. Canadian Social Science, 13(7): 1–5. https://doi.org/10.3968/9698 Google Scholar
Thue Vold, E. (2018) Using machine-translated texts to generate L3 learners’ metalinguistic talk. In Haukås, Å., Bjørke, C. & Dypedahl, M. (eds.), Metacognition in language learning and teaching. New York: Routledge, 67–97. https://doi.org/10.4324/9781351049146-5 CrossRefGoogle Scholar
Tsai, S.-C. (2019) Using Google Translate in EFL drafts: A preliminary investigation. Computer Assisted Language Learning, 32(5–6): 510–526. https://doi.org/10.1080/09588221.2018.1527361 CrossRefGoogle Scholar
White, K. D. & Heidrich, E. (2013) Our policies, their text: German language students’ strategies with and beliefs about web-based machine translation. Die Unterrichtspraxis/Teaching German, 46(2): 230–250. https://doi.org/10.1111/tger.10143 CrossRefGoogle Scholar
Yang, Z., Chen, W., Wang, F. & Xu, B. (2018) Unsupervised neural machine translation with weight sharing. In Gurevych, I. & Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long papers). Stroudsburg: Association for Computational Linguistics, 46–55.CrossRefGoogle Scholar
Supplementary material: File

Lee supplementary material

Lee supplementary material 1

Download Lee supplementary material(File)
File 490 Bytes
Supplementary material: File

Lee supplementary material

Lee supplementary material 2

Download Lee supplementary material(File)
File 20.8 KB