Reading Between the Lines: Commonsense Reasoning in Small Language Models

Wasif Feroze, Hong Qu, Muhammad Shahid, Shaohuan Cheng

Abstract: Recently, large language models have been scaled down from large to smaller parameters. Large language models have generalized to many tasks with pre-training, and these also excelled in commonsense reasoning with targeted fine-tuning. Commonsense reasoning is the capability to make judgments and draw conclusions based on everyday knowledge that humans typically acquire through life experiences. Reasoning ability in language models involves understanding implicit relationships, contextual cues, and causal connections in various scenarios. Despite the progress of large models in many tasks, commonsense reasoning has proved challenging in few-shot settings. In this paper, we propose the evaluation of small language models for commonsense reasoning using the instruction tuning method. We performed experiments on two datasets for commonsense reasoning and evaluated the performance of the models with different quantization processes in one-shot settings. Our results show that the model demonstrates promising results; however, further fine-tuning is required to enhance their commonsense reasoning abilities. Our study contributes to understanding the potential and limitations of small language models.