Fine Tuning Whisper Small for Low-Resource Makassar Automatic Speech Recognition
Abstract
Automatic speech recognition (ASR) for regional languages is often constrained by limited paired speech–text data and by evaluation settings that can hide important differences between familiar and previously unseen transcript content. This study examines full-model fine-tuning of Whisper Small for Makassar ASR using a processed corpus of 6,119 utterances (6.3882 h). The corpus was partitioned into 4,894 training utterances, 612 validation utterances, and a locked 613-utterance test set. Because the same three speakers occur in all partitions, the evaluation is explicitly speaker-dependent. For a fair reviewer-driven baseline, the original zero-shot Whisper Small model and the fine-tuned checkpoint were re-evaluated on the identical test items with the same processor, 16-kHz audio preparation, Indonesian transcription setting, deterministic one-beam decoding, and a shared 225-token generation safety cap. Under this controlled protocol, overall WER decreased from 120.84% to 51.78%, while CER decreased from 62.00% to 16.19%. A post-hoc transcript-content audit identified 136 seen-content test utterances (22.19%) and 477 unseen-content utterances (77.81%). On unseen content, WER decreased from 125.88% to 60.91%, an absolute reduction of 64.97 percentage points. Error decomposition showed that substitutions remained the dominant error class after fine-tuning. The zero-shot system also showed repetitive long-generation behavior in targeted ceiling-hit cases. These findings support language- and corpus-specific adaptation of Whisper Small for Makassar, while also showing why aggregate error rates should be interpreted together with content-overlap, speaker-design, decoding, and reproducibility audits.
References
Alharbi, S., Alrazgan, M., Alrashed, A., Alnomasi, T., Almojel, R., Alharbi, R., Alharbi, S., Alturki, S., Alshehri, F., & Almojil, M. (2021). Automatic speech recognition: Systematic literature review. IEEE Access, 9, 131858–131876. https://doi.org/10.1109/ACCESS.2021.3112535
Slam, W., Li, Y., & Urouvas, N. (2023). Frontier research on low-resource speech recognition technology. Sensors, 23(22), 9096. https://doi.org/10.3390/s23229096
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning (Vol. 202, pp. 28492–28518). PMLR.
Liu, Y., Yang, X., & Qu, D. (2024). Exploration of Whisper fine-tuning strategies for low-resource ASR. EURASIP Journal on Audio, Speech, and Music Processing, 2024, 29. https://doi.org/10.1186/s13636-024-00349-3
Pratama, R. S. A., & Amrullah, A. (2024). Analysis of Whisper automatic speech recognition performance on low resource language. Jurnal Pilar Nusa Mandiri, 20(1), 1–8. https://doi.org/10.33480/pilar.v20i1.4633
Jafar, J., Tb, M. A. W., Aziz, F., Iriany, R., & Nasir, N. (2025). Fine-tuning Whisper model for Mandar speech recognition: Approach and performance evaluation. Journal of Applied Engineering and Technological Science, 7(1), 261–273. https://doi.org/10.37385/jaets.v7i1.7170
Polat, H., Turan, A. K., Koçak, C., & Ulaş, H. B. (2024). Implementation of a Whisper architecture-based Turkish automatic speech recognition (ASR) system and evaluation of the effect of fine-tuning with a Low-Rank Adaptation (LoRA) adapter on its performance. Electronics, 13(21), 4227. https://doi.org/10.3390/electronics13214227
Ghimire, R. R., Poudyal, P., & Bal, B. K. (2024). Improving on the limitations of the ASR model in low-resourced environments using parameter-efficient fine-tuning. In Proceedings of the 21st International Conference on Natural Language Processing (ICON) (pp. 408–415).
Chaparala, K. D., Zarrella, G., Fischer, B. T., Kimura, L., & Jones, O. P. (2024). Mai Ho‘omāuna i ka ‘Ai: Language models improve automatic speech recognition in Hawaiian. In Proceedings of the 4th NeurIPS Efficient Natural Language and Speech Processing Workshop (Vol. 262, pp. 576–583). PMLR.
Pratap, V., Tjandra, A., Shi, B., Tomasello, P., Babu, A., Kundu, S., Elkahky, A., Ni, Z., Vyas, A., Fazel-Zarandi, M., Baevski, A., Adi, Y., Zhang, X., Hsu, W.-N., Conneau, A., & Auli, M. (2024). Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97), 1–52.
Ardila, R., Branson, M., Davis, K., Kohler, M., Meyer, J., Henretty, M., Morais, R., Saunders, L., Tyers, F., & Weber, G. (2020). Common Voice: A massively-multilingual speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference (pp. 4218–4222). European Language Resources Association.
Conneau, A., Ma, M., Khanuja, S., Zhang, Y., Axelrod, V., Dalmia, S., Riesa, J., Rivera, C., & Bapna, A. (2023). FLEURS: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT) (pp. 798–805). IEEE. https://doi.org/10.1109/SLT54892.2023.10023141
Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449–12460.
Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J., Baevski, A., Conneau, A., & Auli, M. (2022). XLS-R: Self-supervised cross-lingual speech representation learning at scale. In Interspeech 2022 (pp. 2278–2282). https://doi.org/10.21437/Interspeech.2022-143
Sakti, S., & Titalim, B. A. (2023). Leveraging the multilingual Indonesian ethnic languages dataset in self-supervised models for low-resource ASR task. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 1–8). IEEE. https://doi.org/10.1109/ASRU57964.2023.10389730
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (Vol. 97, pp. 2790–2799). PMLR.
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019). SpecAugment: A simple augmentation method for automatic speech recognition. In Interspeech 2019 (pp. 2613–2617). https://doi.org/10.21437/Interspeech.2019-2680
Wang, H., Bin, J., Gou, C., Yang, L., Hou, B., & Qin, M. (2025). Low-resource speech recognition by fine-tuning Whisper with Optuna-LoRA. Applied Sciences, 15(24), 13090. https://doi.org/10.3390/app152413090
Azis, H., Fajriansyah M, M. F., Darwis, H., Purnawansyah, Hasanuddin, T., & Sugiarti. (2026). Development of a low-resource automatic speech recognition system for the Makassar dialect. In 20th International Conference on Ubiquitous Information Management and Communication (IMCOM) (pp. 1–8). IEEE. https://doi.org/10.1109/IMCOM69009.2026.11360893
