“Low-Resource” Is a Policy, Not a Property
Urdu and the economics of language technology
This event is part of a demonstration of what Lera offers for running knowledge events. Its content is written with care and draws on real sources, but the conference has not taken place, and apart from the convener its participants are fictional.
In the literature of language processing, a language with little labelled digital data is called “low-resource”. The word sounds neutral, as though it reported a natural scarcity, like a dry country. Consider Urdu: more than two hundred million people speak or read it, it has centuries of poetry, prose and journalism behind it, and yet in many inventories of language technology it is still “low-resource”. The resources are not scarce. What is scarce is investment and decision.
In 2020 Pratik Joshi and his colleagues sorted the world’s languages into six classes according to available data and showed that most of the world’s languages sit in the lowest class while a handful of languages hold almost all the resources. The gap does not reflect the number of speakers; it reflects the geography of money and research institutions. The audit by Kreutzer and colleagues in 2022 showed something bleaker: in large web-crawled corpora, a significant share of what was recorded under the names of low-resource languages was in fact text in another language or not language at all. The statistics said the data existed, and it did not.
Urdu shows how this scarcity is manufactured. For decades Urdu newspapers and books were typeset in proprietary layout software whose output was closer to an image than to text. As a result, a large part of printed Urdu sits in archives but cannot be read by a machine. Nastaliq, the script Urdu readers regard as their language’s natural hand, was for years poorly supported in fonts and operating systems. None of this is a property of the language. All of it consists of decisions that companies, governments and universities made or failed to make.
So the answer must be political as well as technical. A language corpus should be a public good, with a clear licence and governance by the language community. Optical character recognition for Nastaliq, digitising newspaper archives and building open dictionaries are tasks the market will not take on by itself. The experience of the Masakhane network in Africa, described by Nekoto and colleagues in 2020, shows that participatory research, carried out by speakers and researchers of the language themselves, can achieve in a few years what had not been done in decades.
For a system that identifies knowledge independently of any one language, this argument has a concrete consequence: the quality of a language must not be confused with the quality of its machine translation. If a concept is written in Urdu, that text is the original, and its English translation is another form of the same knowledge object. A language in which original knowledge is written will not stay “low-resource” for ever.
Questions for discussion
- Which institution should pay to digitise the printed archives of languages such as Urdu?
- How is a language corpus held as a public good protected from unauthorised harvesting by companies?
- Should the label “low-resource” be dropped, or can its political meaning be made explicit?
References
Joshi, Pratik, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. “The State and Fate of Linguistic Diversity and Inclusion in the NLP World.” In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6282–6293.
Kreutzer, Julia, et al. 2022. “Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets.” Transactions of the Association for Computational Linguistics 10: 50–72.
Nekoto, Wilhelmina, et al. 2020. “Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages.” In Findings of the Association for Computational Linguistics: EMNLP 2020, 2144–2160.