End-to-End Adaptation of LLMs for Low-Resource Languages

This paper presents a full end-to-end pipeline—corpus creation, custom tokenization, continued pre-training, fine-tuning, and quantization—for adapting a LLaMA 3.1 base model into a functional Sinhala LLM under constrained hardware, offering a reproducible template for low-resource language adaptation.

Problem Statement

LLM capabilities are heavily concentrated in high-resource languages, leaving languages like Sinhala with limited access to modern NLP tools despite having millions of speakers. Building a Sinhala LLM is hard because usable digital text is scarce (much literature exists only in print), the script's complex orthography breaks standard tokenizers, and most research teams lack the large-scale compute typically assumed for LLM adaptation work.

Key Novelty

A custom tokenization strategy engineered specifically for Sinhala's complex orthography while remaining compatible with LLaMA 3.1's pre-trained architecture
An end-to-end low-resource corpus pipeline combining web-scale collection, translation, digitization of printed sources, and noise reduction to overcome data scarcity
A hardware-constrained adaptation recipe (continued pre-training + fine-tuning + quantization) demonstrated to work within limited memory/compute budgets, packaged as a reproducible framework for other low-resource languages

Evaluation Highlights

Qualitative gains in Sinhala text generation quality, contextual understanding, and response coherence relative to the base (non-adapted) LLaMA model
Feasibility demonstration: successful continued pre-training, fine-tuning, and quantization of a modern LLM entirely under limited hardware/memory resources

Signal Assessment

3/10 The work applies established techniques (continued pre-training, fine-tuning, tokenizer extension, quantization) to a new low-resource language rather than introducing new methodology, but it delivers real practical value and a reusable framework for an underserved NLP community.

Methodology

Aggregate and digitize Sinhala text from scarce online sources and printed literature, supplemented with translated content to expand corpus size
Apply noise reduction and cleaning to construct a high-quality training corpus suitable for continued pre-training
Design and integrate a custom tokenizer tailored to Sinhala's complex orthography while preserving compatibility with the LLaMA 3.1 architecture
Perform continued pre-training of the LLaMA 3.1 base model on the curated Sinhala corpus
Fine-tune the pre-trained model for improved instruction-following and response quality
Quantize the model to enable training and inference under limited hardware/memory constraints
Evaluate text generation quality, contextual understanding, and response coherence of the resulting Sinhala LLM

System Components

Sinhala Corpus Pipeline

Collects, translates, digitizes, and cleans Sinhala text from limited online and printed sources to build a training-ready dataset

Custom Sinhala Tokenizer

A tokenization scheme designed for Sinhala's complex orthography, aligned with LLaMA 3.1's existing vocabulary/architecture to minimize retraining overhead

Continued Pre-training Module

Adapts LLaMA 3.1 base model weights to Sinhala linguistic patterns using the curated corpus

Fine-tuning Stage

Refines the pre-trained model for coherent, context-aware Sinhala response generation

Quantization Module

Compresses the model to reduce memory/compute footprint, enabling training and deployment on limited hardware

Results

Metric/Benchmark	Base LLaMA 3.1 (Sinhala)	Adapted Sinhala LLM (This Paper)	Delta
Text generation quality	Fragmented/limited fluency in Sinhala	Improved fluency and grammaticality	Qualitative improvement
Contextual understanding	Weak grasp of Sinhala context	Better contextual comprehension	Qualitative improvement
Response coherence	Inconsistent, incoherent outputs	More consistent, coherent responses	Qualitative improvement

Key Takeaways

Continued pre-training plus fine-tuning plus quantization can produce a usable low-resource-language LLM without frontier-scale compute
Corpus construction (translation, digitization, cleaning) is often the largest practical bottleneck in low-resource LLM projects, not model architecture
Custom tokenization tailored to a language's script/orthography while staying compatible with the base model's architecture is a key enabler of successful adaptation
The described pipeline can serve as a reusable blueprint for adapting LLMs to other underrepresented languages with similar data and hardware constraints

Abstract

While Large Language Models (LLMs) have revolutionized information processing, their benefits are disproportionately skewed toward high-resource languages, leaving languages like Sinhala behind. Building on our earlier work, “An Approach to Training and Fine-Tuning Large Language Models for Low-Resource Languages,” this extended version presents the complete development and evaluation of a Sinhala Large Language Model (LLM) obtained by training a LLaMA 3.1 base model for this underrepresented language. Developing the Sinhala LLM required addressing challenges inherent to low-resource language modelling. The scarcity of online Sinhala text and the prevalence of printed-only literature required extensive efforts in corpus creation, translation, and noise reduction. Sinhala’s complex orthography further necessitated the design of a custom tokenization strategy aligned with existing pre-trained architectures. Additionally, the process of continued pre-training, fine-tuning, and quantization was constrained by limited hardware and memory resources. Despite these limitations, the trained Sinhala LLM demonstrates significant progress in adapting large-scale architectures to a low-resource context. Experimental results highlight consistent improvements in text generation quality, contextual understanding, and response coherence. This study demonstrates that effective LLMs can be built for low-resource languages even with limited hardware, providing a reproducible framework for researchers facing similar constraints.