Beyond Data Hunger: Cultivating Language Technology for All

by

Author: Denis Avetisyan

A new approach to building inclusive language tools prioritizes ethical data creation and community involvement for languages often left behind.

This review proposes a ‘Data Care’ framework – grounded in CARE principles – to address the challenges of developing sustainable language technologies for low-resource languages like Serbian, focusing on bias mitigation, corpus development, and digital sovereignty.

While large language models increasingly dominate the field of natural language processing, their development often replicates existing linguistic and cultural biases, particularly for underrepresented languages. This study, ‘From Data Scarcity to Data Care: Reimagining Language Technologies for Serbian and other Low-Resource Languages’, examines these challenges through the lens of Serbian, revealing how historical data loss and contemporary engineering-driven approaches exacerbate inequities in language technology. It proposes a ‘Data Care’ framework-grounded in principles of Collective Benefit, Authority to Control, Responsibility, and Ethics-to shift bias mitigation from a reactive fix to an integral component of corpus design and governance. Can this model offer a pathway toward more inclusive and sustainable language technologies that genuinely reflect the nuances of diverse linguistic communities?

Decoding the Digital Echo: Linguistic Loss in the Age of AI

The rapid advancement of Large Language Models (LLMs) presents a paradoxical challenge for linguistic diversity; while promising to unlock new possibilities in natural language processing, these models inherently favor languages with abundant digital representation. Languages lacking extensive online resources – encompassing everything from digitized texts and websites to transcribed speech – are effectively marginalized in this new technological landscape. This isn’t simply a matter of technological access, but a form of digital exclusion, where the very tools designed to connect and empower inadvertently reinforce existing power imbalances. The performance of LLMs is directly correlated with the size and quality of the data used for training; consequently, languages with limited digital footprints struggle to achieve comparable results, hindering their integration into essential applications like machine translation, voice assistance, and content creation. This creates a cycle of disadvantage, where under-represented languages receive even less attention, further exacerbating the data scarcity and limiting their participation in the digital age.

The Serbian language vividly illustrates the challenges faced by many languages in the age of artificial intelligence, stemming from a history of textual heritage loss and the resulting data scarcity. While English language models benefit from training datasets containing 3 to 14 trillion words, the combined digital corpus of all South Slavic languages totals just 23 billion words, with Serbian accounting for approximately 7 to 9 billion. This dramatic disparity isn’t merely a quantitative difference; it represents a significant impediment to linguistic preservation and the development of language technologies capable of accurately representing and processing Serbian. The limited data available restricts the efficacy of Large Language Models, hindering their ability to understand nuance, context, and the full richness of the language, and ultimately contributing to a digital divide that marginalizes less-represented linguistic communities.

The erosion of linguistic heritage directly impedes the creation of robust Digital Corpora, essential resources for modern language technology. A comprehensive Digital Corpus – a large, structured set of texts – serves as the foundation for training Artificial Intelligence and Natural Language Processing models. When historical destruction and present-day data scarcity limit the available textual material, the resulting corpus becomes unrepresentative of the language’s full range of expression and cultural nuance. This deficiency doesn’t merely affect academic linguistic research; it actively hinders technological advancement, producing AI systems that perform poorly with, or even misrepresent, the nuances of the language. Consequently, the ability to accurately translate, generate text, or provide information in these under-resourced languages is severely compromised, perpetuating a digital divide and diminishing access to information and technology for its speakers.

The creation of truly representative language technologies demands a dedicated commitment to data-driven approaches that prioritize cultural heritage and inclusivity. This extends beyond simply amassing large datasets; it necessitates careful consideration of the historical and societal contexts that shape a language, particularly those facing digital scarcity. Successfully bridging the digital divide requires actively curating and preserving linguistic resources, ensuring that data collection respects the nuances and diversity within a language community. By focusing on responsible data practices and prioritizing inclusivity, developers can move beyond merely replicating existing biases in large language models and instead foster technologies that genuinely reflect and support all languages – safeguarding linguistic diversity for future generations and promoting equitable access to the benefits of artificial intelligence.

Reclaiming the Code: A Framework for Responsible AI Development

Historically, data collection for natural language processing has relied on readily available, large-scale text sources, typically prioritizing majority languages and digital content. This approach systematically disadvantages minority languages due to their limited digital presence and the resulting scarcity of training data. Consequently, models trained on these biased datasets exhibit lower performance and higher error rates when applied to minority languages, often misinterpreting linguistic structures or failing to recognize culturally specific terminology. This disparity isn’t merely a technical issue; it directly impacts the utility of AI applications – such as machine translation, speech recognition, and content moderation – for speakers of under-represented languages, potentially exacerbating existing societal inequalities and hindering access to information and services.

The Data Care framework represents a shift in data handling practices, moving beyond conventional approaches to emphasize equitable and respectful data lifecycle management. It is built upon the CARE Principles – Collective Benefit, Authority to Control, Responsibility, and Ethics – which dictate that data collection and use should prioritize benefits for the communities from which the data originates, ensure those communities retain control over their data, establish clear lines of accountability for data handling, and adhere to ethical standards throughout the entire process. This means data governance must actively involve the communities affected, data usage agreements should be transparent and mutually beneficial, and processes for addressing harms resulting from data misuse must be in place. Implementation necessitates a focus on data sovereignty and the recognition of data as a cultural heritage, rather than simply a resource for analysis.

The Data Care framework moves beyond identifying and correcting biases after data collection, instead proactively influencing corpus construction to achieve equitable representation. This involves deliberate strategies during data sourcing, annotation, and validation to ensure sufficient and nuanced inclusion of minority languages and diverse cultural perspectives. Specifically, it advocates for diversifying data sources beyond readily available online content, prioritizing community-sourced data, and employing culturally-aware annotation schemes. This approach seeks to avoid the amplification of existing societal imbalances inherent in datasets built on historically biased information, ultimately fostering more inclusive and accurate AI models.

Effective implementation of the Data Care framework necessitates close collaboration between linguistic experts and computational method specialists. Linguistic expertise is critical for identifying and addressing nuanced cultural contexts, ensuring appropriate data annotation, and validating the representation of diverse languages within a corpus. Simultaneously, computational methods provide the tools for large-scale data processing, automated analysis of bias, and the development of algorithms that can operationalize the CARE Principles. This integration allows for a feedback loop where linguistic insights inform computational model design, and computational analysis reveals potential biases or areas requiring further linguistic review, ultimately leading to more responsible and equitable AI systems.

Unlocking the Linguistic Vault: Technological Pathways to Inclusion

Optical Character Recognition (OCR) is the process of converting images of text – such as scanned documents or photographs of historical materials – into machine-readable text data. This conversion is a prerequisite for creating Digital Corpora, which are large, structured sets of texts used for linguistic analysis, natural language processing, and historical research. The accuracy of OCR directly impacts the quality of the resulting Digital Corpus; errors in OCR necessitate manual correction, a time-consuming and resource-intensive process. Modern OCR systems employ a combination of image processing techniques and machine learning algorithms to achieve high accuracy rates, even with degraded or handwritten documents. Without effective OCR, the vast majority of historical texts remain inaccessible to computational analysis, hindering efforts to preserve and understand cultural heritage and linguistic evolution.

Cross-lingual transfer learning addresses the challenge of limited data availability in low-resource languages like Serbian by leveraging knowledge acquired from languages with abundant resources, such as English or French. This is typically achieved through pre-training models on high-resource languages and then fine-tuning them with the smaller datasets available for Serbian. Techniques include parameter sharing, where model weights learned from the high-resource language are transferred and adapted, and multilingual embeddings, which map words from different languages into a shared vector space. By transferring learned representations, cross-lingual methods reduce the need for extensive Serbian-specific data, improving performance on tasks like machine translation, named entity recognition, and sentiment analysis, while requiring fewer computational resources than training models from scratch.

Native Language Models (NLMs) for Serbian offer improved performance over generalized models due to their specific training on Serbian datasets. Generalized models, trained on a broad range of languages, often lack the capacity to accurately represent the morphological complexity and syntactic structures unique to Serbian. This includes handling inflectional richness, case markings, and word order variations that are prevalent in the language. By focusing solely on Serbian data, NLMs can learn these nuances, resulting in higher accuracy in tasks such as machine translation, sentiment analysis, and text summarization. The increased precision stems from the model’s ability to better predict word sequences and contextualize meaning within the specific linguistic framework of Serbian, ultimately delivering more relevant and reliable outputs.

Data sovereignty is a critical consideration in the development of Serbian language technologies, necessitating adherence to the cultural and legal frameworks governing data collection and usage within Serbian-speaking communities. With an estimated 10 to 12 million speakers, the Serbian language represents a significant population requiring dedicated data resources; however, simply amassing data is insufficient. Compliance with relevant data protection regulations, respecting intellectual property rights, and ensuring community consent are paramount. Targeted data collection initiatives must prioritize ethical considerations and acknowledge the specific cultural context of the Serbian language and its users to foster trust and facilitate sustainable language technology development.

A Future Forged in Plurilingualism: Linguistic Equity and AI’s Potential

The creation of truly inclusive artificial intelligence necessitates acknowledging the pluricentric nature of languages like Serbian. Unlike languages with a single, codified standard, Serbian exhibits multiple accepted varieties – notably, Ekavian and Ijekavian – each with its own established norms and usage. Failing to recognize this linguistic reality results in language models that inherently privilege one variety over others, effectively marginalizing speakers of the non-dominant forms. A model trained solely on Ekavian Serbian, for example, will struggle to accurately process or generate Ijekavian, creating a digital experience that feels alienating or even inaccurate to a significant portion of the Serbian-speaking population. Therefore, robust and representative datasets incorporating all standard varieties are crucial, allowing AI to understand and respond to the full spectrum of linguistic expression within the Serbian language community, and fostering a more equitable and genuinely inclusive technological landscape.

A national artificial intelligence strategy deliberately incorporating linguistic diversity isn’t merely about inclusivity; it’s a catalyst for broader technological advancement. Prioritizing multiple languages, including those historically underrepresented in digital spaces, unlocks innovation by fostering unique approaches to problem-solving and data analysis. This approach expands the potential talent pool contributing to AI development, moving beyond monolingual datasets and algorithms. Furthermore, equitable access to technology – where AI systems understand and respond effectively to all languages – ensures that the benefits of this powerful technology are shared by all communities, rather than exacerbating existing inequalities. By viewing linguistic diversity as a valuable resource, a nation can position itself at the forefront of a more creative, inclusive, and globally relevant AI landscape.

The revitalization of underrepresented languages in the digital sphere hinges on a conscientious approach to data and the implementation of cutting-edge technology. Principles of Data Care – encompassing responsible collection, curation, and usage – are paramount, ensuring linguistic communities maintain control over their digital heritage and benefit from its application. Coupled with this, Native Language Models – AI systems trained specifically on the nuances of a particular language – offer a powerful alternative to relying on generalized, often less accurate, multilingual models. These models, built with careful attention to data provenance and community input, not only improve the performance of language technologies for Serbian and similar languages, but also foster a sense of ownership and agency, ultimately empowering these linguistic communities to actively participate in and shape the future of AI.

The future of linguistic equity isn’t about merely archiving languages, but actively cultivating their presence in the digital world. Recognizing this, current initiatives prioritize the digitization of Serbian, addressing historical data scarcity that has hindered its representation in artificial intelligence. These efforts aren’t simply about preservation; they are about building a robust ecosystem where Serbian can thrive technologically, fostering innovation and ensuring equitable access to AI for its speakers. By proactively creating and curating datasets, and developing native language models, the focus shifts from mitigating loss to empowering a vibrant, technologically-engaged linguistic community and shaping a future where Serbian, and other underrepresented languages, are not just maintained, but actively celebrated within the digital landscape.

The pursuit of language technologies for low-resource languages, as detailed in this paper, inherently demands a challenging of existing norms. It necessitates dismantling the assumption that readily available datasets are sufficient, and instead, constructing new methodologies. This echoes Alan Turing’s sentiment: “Sometimes people who are unhappy tend to look for a person to blame.” The article proposes a proactive ‘Data Care’ framework-a deliberate effort to avoid inheriting biases from existing data-rather than retrospectively attempting to correct flawed systems. Just as Turing explored the limits of computation by questioning fundamental assumptions, this work advocates for an active, critical approach to data creation and corpus development, prioritizing ethical considerations and community involvement to build truly inclusive language models.

Beyond the Toolbox

The call for ‘Data Care’ represents more than simply adding another layer to the technical stack. It’s an admission that the usual methods – scaling models, clever algorithms – hit a wall when the very foundation is shaky. Serbian, and languages like it, aren’t merely lacking data; they’re contending with the ghosts of linguistic marginalization, biases embedded in existing digital infrastructure. The real challenge isn’t building a better machine translation engine, it’s interrogating why certain voices were silenced in the first place, and whether technology can ever truly redress that imbalance.

Future work must, therefore, embrace a degree of constructive demolition. The pursuit of ‘digital sovereignty’ isn’t about building walled gardens; it’s about understanding the inherent power dynamics woven into the fabric of these systems. Corpus development, framed solely as a data-gathering exercise, misses the point. It demands a critical examination of what constitutes ‘representative’ data, and who gets to define it.

The field now faces a peculiar paradox. Having spent decades reverse-engineering language, it must now begin to unmake its assumptions. Native language models are a start, but they’re only useful if the underlying data isn’t simply a reflection of existing inequalities. The ultimate test won’t be achieving a benchmark score, but whether these tools genuinely amplify, rather than merely echo, the voices they claim to serve.

Original article: https://arxiv.org/pdf/2512.10630.pdf

Contact the author: https://www.linkedin.com/in/avetisyan/

See also:

2025-12-14 11:33