A Comparative Study on the Development of a Thai Legal QA Framework Using Large Language Models and Mixed Legal Datasets

Supachoke Hanwiboonwat, Chaichana Thavornthaveekul, Prachya Boonkwan, Apivadee Piyatumrong, Peerapon Vateekul

Abstract: In the present day, large language models (LLMs) such as GPT-4o play a significant role in answering legal questions in the Thai language. However, creating a system for answering legal questions for the general public is a highly challenging and complex task. This is due to the fact that Thai legal code documents use complex language and are often lengthy. This research focuses on developing a framework for legal question answering (QA) targeted at the general public, with the aim of establishing best practices for creating effective legal QA systems. To enhance the system’s performance, we constructed our own dataset and integrated data from a variety of sources. Intensive experiments were conducted to identify the most suitable LLM for legal applications in the Thai context. We proposed a method to improve QA performance in the legal domain with LLM by fine-tuning multiple task datasets. The entire process is thoroughly detailed, covering aspects such as fine-tuning and the use of retrieval-augmented generation (RAG), including techniques like keyword search and contextual search, as well as reranking processes. Additionally, this study compares various prompt formats to find the most effective one for answering legal questions for the general public. The results from our model are comparable to larger models with more parameters, such as GPT-4o, in legal examination tasks and perform better in answering legal questions tasks.

External IDs:dblp:conf/nldb/HanwiboonwatTBPV25