SLlama: A Small Language Model for Extremely Low Resource Domains

ACL ARR 2025 February Submission4795 Authors

Abstract: Efficient language modeling is critical for low-resource languages and computationally constrained environments. In this work, we introduce SLlama, a parameter-efficient variant of the LlamA architecture, designed through a combination of Repeated Reduced Hidden Size and Projection (RRHP), Permutated Weight Attention (PWA), Shared Projection Multi-Layer Perceptron (SPMLP), and Layer Weight Sharing across key components. These modifications substantially reduce parameter count while preserving linguistic competence. Compared to BabyLlama, the baseline model trained on the BabyLM Challenge dataset, SLlama achieves a 31.72\% improvement in linguistic knowledge acquisition without distillation, while maintaining a comparable GLUE score with lower computational costs. The model’s ability to excel in linguistic tasks with minimal data suggests its potential for low-resource language modeling. Through extensive evaluations, we demonstrate that SLlama remains robust under intricate linguistic assessments, offering a practical alternative for NLP applications where efficiency is paramount. Our findings highlight the feasibility of extreme parameter reduction without sacrificing core linguistic capabilities, paving the way for more accessible and resource-efficient language models.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: parameter-efficient-training; data-efficient training;NLP in resource-constrained settings

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 4795