Rule-Guided Language Model Alignment for Text Generation Management in Industrial Use Cases

Shunichi Akatsuka, Aman Kumar, Xian Yeow Lee, Lasitha Vidyaratne, Dipanjan Dipak Ghosh, Ahmed K. Farahat

Keywords: Large Language Models, finetuning, model alignment, dataset generation, automotive

TL;DR: Make LLMs follow safety rules by efficiently revising existing response and finetuning the revised response

Abstract: Recent advances in Large Language Models (LLMs) have shown significant success in various natural language tasks. However, when implementing LLMs in industry applications, they often need to follow domain-specific rules. Since these rules can be complex and numerous, it is often difficult to precisely identify which rule should be applied to the response. In this paper, we propose a simple yet effective method to address this issue, by taking the following two steps: (1) generate a dataset of rule-applied responses using simplified rule selection, and (2) train an LLM on this dataset. Since the rule selection is not designed to be perfect, the responses in the dataset do not always follow all the necessary rules. However, by training an LLM on this dataset, we expect the LLM to generalize over the rules and correctly identify the task-to-rule dependency. We demonstrated our method in the automotive repair domain, to make a repair recommendation LLM to follow safety rules. Our experimental results show that our approach improves LLM performance compared to solely applying the rules using the simplified rule selection. This suggests that our method could enhance the utility of LLMs in industry applications.

Submission Number: 104