Detection of Machine-Translated Croatian-English Phrases: Do Translators Agree in their Assessment?
DOI:
https://doi.org/10.15291/csi.5069Keywords:
Croatian–English, human evaluation, inter-rater agreement, machine translationAbstract
This study examines the extent to which machine-translated and human-translated Croatian–English phrases can be distinguished by expert evaluators by analysing inter-rater agreement. For this purpose, a parallel corpus was compiled from four Croatian research articles in the fields of Education and Psychology, together with their human translations and machine translations produced using the online tool onlinedoctranslator. com. Two professional translators, native speakers of Croatian with extensive translation experience, were asked to classify selected English translations of Croatian phrases as either machine-translated or human-translated. The analysis shows a raw inter-rater agreement of 75 % for both machine-translated and human-translated phrase sets, indicating a moderate level of agreement between the raters. These results suggest that even expert translators do not consistently reach the same conclusions when judging the origin of short, decontextualised translated phrases. The findings support the view that, at the phrase level, machine-translated output is increasingly difficult to distinguish from human translation on the basis of surface linguistic features alone.
Downloads
References
Borucinsky, Mirjana, Jana Kegalj and Dario Zagorec. 2025. “Translation Quality Evaluation of Croatian-to-English Machine-Translated Administrative Texts”. Journal of Language and Literary Studies 51: 193–211.
Brezina, Vaclav. 2021. Statistics in Corpus Linguistics. Cambridge: Cambridge University Press.
Castilho, Sheila, Stephen Doherty, Federico Gaspari and Joss Moorkens. 2018. “Approaches to Human and Machine Translation Quality Assessment”. In Joss Moorkens, Sheila Castilho, Federico Gaspari and Stephen Doherty (Eds.). Translation Quality Assessment: From Principles to Practice. Heidelberg: Springer: 9–38.
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Panayota Georgakopoulou, Pintu Lohar, Andy Way, Antonio Valerio Miceli-Barone and Maria Gialama. 2017. “A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators”. In Proceedings of Machine Translation Summit XVI: Research Track. Nagoya: Machine Translation Summit: 116–131.
Čanžar, Margareta. 2024. Human Evaluation of Machine-Translated Texts. Diplomski rad (MA thesis). Rijeka: University of Rijeka, Faculty of Humanities and Social Sciences.
Chatzikoumi, Eirini. 2020. “How to evaluate machine translation: A review of automated and human metrics”. Natural Language Engineering 26 (2): 137–161.
Chichirau, Malina, Rik van Noord and Antonio Toral. 2023. “Automatic Discrimination of Human and Neural Machine Translation in Multilingual Scenarios”. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation. Tampere: European Association for Machine Translation: 217–226.
Dunđer, Ivan. 2020. “Machine Translation System for the Industry Domain and Croatian Language”. Journal of Information and Organizational Sciences 44 (1): 33–50.
Dunđer, Ivan. 2021. “Pregled razvoja tehnologije automatskog strojnog prevođenja”. Polytechnic & Design 9 (2): 90–100.
Dunđer, Ivan, Sanja Seljan and Marko Pavlovski. 2020. “Automatic Machine Translation of Poetry and a Low-Resource Language Pair”. In Proceedings of the 43rd
International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO 2020). Rijeka: MIPRO: 1034–1039.
Gwet, Kilem Li. 2014. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. Gaithersburg, MD: Advanced Analytics.
Online Doc Translator. n. d. “Online Doc Translator”. (1 June 2024). (online source) Seljan, Sanja and Ivan Dunđer. 2014. “Combined Automatic Speech Recognition and Machine Translation in Business Correspondence Domain for English–Croatian”. International Journal of Computer, Information, Systems and Control Engineering 8: 1069–1075.
Seljan, Sanja and Ivan Dunđer. 2015a. “Automatic Quality Evaluation of Machine-Translated Output in Sociological-Philosophical-Spiritual Domain”. In Proceedings of the 10th Iberian Conference on Information Systems and Technologies (CISTI 2015) 2: 128–131.
Seljan, Sanja and Ivan Dunđer. 2015b. “Machine Translation and Automatic Evaluation of English/Russian–Croatian”. In Proceedings of the International Conference “Corpus Linguistics – 2015” (CORPORA 2015): 72–79.
Seljan, Sanja, Marija Brkić and Vlasta Kučiš. 2011. “Evaluation of Free Online Machine Translations for Croatian-English and English-Croatian Language Pairs”. In Clive Billenness, Annette Hemera, Vladimir Mateljan, Mihaela Stančić, Hrvoje Stančić and Sanja Seljan (Eds.). The Future of Information Sciences—Information Sciences and e-Society. Zagreb: Department of Information Sciences: 331–344.
Seljan, Sanja, Marko Tucaković and Ivan Dunđer. 2015. “Human Evaluation of Online Machine Translation Services for English/Russian–Croatian”. In WorldCIST’15 – 3rd World Conference on Information Systems and Technologies. Advances in Intelligent Systems and Computing: 1089–1098.
Seljan, Sanja, Ivan Dunđer and Angelina Gašpar. 2013. “From Digitisation Process to Terminological Digital Resources”. In Proceedings of the 36th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO 2013): 1329–1334.
Seljan, Sanja, Nikolina Škof Erdelja, Vlasta Kučiš, Ivan Dunđer and Mirjana Pejić Bach. 2020. “Quality Assurance in Computer-Assisted Translation in Business Environments”. In Natural Language Processing for Global and Local Business. Hershey, PA: IGI Global: 247-270.
Seljan, Sanja, Ivan Dunđer and Marko Pavlovski. 2020. “Human Quality Evaluation of Machine-Translated Poetry”. In: Proceedings of the 43rd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO 2020). Rijeka: MIPRO: 1040–1045.
Toral, Antonio. 2020. “Reassessing claims of human parity and super-human performance in machine translation at WMT 2019”. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation: 185–194.
