Automatic Language Identification for Indonesian-Malaysian Language Using Machine Learning

• Hlavní autoři: Abdiansah, Abdiansah, Rizqie, Muhammad Qurhanul
• Médium: UMS Journal (OJS)
• Jazyk: eng
• Vydáno: Department of Informatics, Universitas Muhammadiyah Surakarta, Indonesia 2023
• Témata: Language Identification; Indonesian; Malaysian; Support Vector Machine
• On-line přístup: https://journals.ums.ac.id/index.php/khif/article/view/21669
• Shrnutí: Language Identification (LID) aims to guess or identify which language the text or sound is coming from. Language identification tends to be easier in languages with different characteristics (e.g., Indonesian and English), but not for languages with similar characteristics (e.g., Indonesian and Malaysian). Similar languages can cause ambiguity that will be a bias for machine learning. Using Support Vector Machine (SVM) technique, this research tried to identify the Indonesian or Malaysian language. The training and testing data are taken from Leipzig Corpora Collection and Twitter dataset. The feature representation technique uses TF-IDF, and the baseline testing uses Naive Bayes Multinomial. We used two training techniques: split (20:80) and 10-cross validation. The experimental results show that the accuracy between the baseline and SVM is not too far. Both provide accuracy of around 90% and above. The results indicate that Indonesian and Malaysian language identification accuracy is relatively high even though using simple techniques.