Amharic at the margins
An audit of what large multilingual corpora collect under the name of Amharic
This event is part of a demonstration of what Lera offers for running knowledge events. Its content is written with care and draws on real sources, but the conference has not taken place, and apart from the convener its participants are fictional.
Amharic is the working language of the Ethiopian state and has a script of its own, a syllabary inherited from Ge'ez. Even so, the corpora on which multilingual language models are trained count Amharic as a "low-resource" language. This poster reports a manual audit of a sample of the text that three widely used web corpora publish under the Amharic label. Our question was simple: is what appears under this label actually Amharic? And if it is, where does it come from and what does it talk about?
We took our method from the audit by Kreutzer and colleagues (2022) and adapted it to the Ge'ez script and to Amharic as it is written on social media. Two native speakers judged each sentence independently: fluent Amharic, machine-made or broken Amharic, another language in the same script, or a string that is not language at all. Wherever their judgements differed, we settled the case by discussion rather than by counting votes.
The picture that emerged is of a language seen through the eyes of an automatic labeller: anything written in this script has been assumed to be Amharic. The fluent text, too, comes mostly from a handful of sources: religious writing, state media and machine-translated commercial pages. Everyday speech, poetry, local knowledge of farming and healing, and writing by women are almost nowhere to be seen. A model that learns Amharic from this corpus learns the language from the mouth of the state and the pulpit.
Findings
- Language labels in these corpora mistake the script for the language; Tigrinya and liturgical Ge'ez are silently counted as Amharic.
- Fluent, natural text made up less than half of our sample, and most of it came from a few internet domains.
- The problem for Amharic is the quality and range of the data, not its volume; ten times more data from the same sources will not teach a model the language people speak.
References
Kreutzer, Julia, et al. 2022. "Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets." Transactions of the Association for Computational Linguistics 10: 50–72.
Joshi, Pratik, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. "The State and Fate of Linguistic Diversity and Inclusion in the NLP World." Proceedings of ACL 2020, 6282–6293.