Upaya at the FinLLM Challenge Task 1 and 2: DistFin: Distillation based Fine-Tuning for Financial Tasks

Ashvini Kumar Jindal, Pawan Kumar Rajpoot, Ankur Parikh

Abstract: With the advent of Large Language Models (LLM) in finance, financial text analysis and generation tasks have received growing attention. Financial text classification and financial text summarization are some of the very important text analysis and generation tasks, respectively. Adapting LLMs to these tasks is very crucial for domain adaptation. This paper presents a method to fine-tune LLMs to Financial Argument Classification and Financial Abstractive Summarization. The argument classification task focuses on argument unit classification to test the capabilities of LLMs to identify and categorize texts as premises or claims. The summarization task aims to abstract financial texts into concise summaries. The dataset was released along with shared tasks as a part of the 8th Financial Technology and Natural Language Processing (FinNLP), co-located with IJCAI 2024. In this paper, we employed a distillation-based fine-tuning of Llama-3 (8B parameters) to learn the rationale/step generated by Llama-3 (70B parameters) along with labels. In the argument classification task, we achieved an F1-score (evaluation metric) of 0.4166. In the summarization task, we got the 2nd rank with the Rouge-1 score (evaluation metric) of 0.5294.