Towards an approach combining Knowledge Graphs and Prompt Engineering for Merging Large Language Models

Furel De Consol TEGUIMENE YENDJI, Azanzi Jiomekong, Sanju Tiwari, Carick Appolinaire ATEZONG YMELE, Janice Anta Zebaze

Keywords: Large Language Models, knowledge graphs, prompt engineering, LLM merging, AI

TL;DR: This paper proposes a novel approach of LLM merging based on the association of MergeKit method with Prompt engineering and knowledge graphs

Abstract:

Large Language Models (LLMs) have emerged as transformative tools in areas
ranging from education to software development, but their high computing power
and energy costs limit their accessibility. Deploying these powerful LLMs in
developing countries presents high costs and infrastructural challenges, such as
limited access to high-performance computing resources. We address this problem
by proposing an approach that relies on lightweight, open-source LLMsm each
optimized in sub-tasks of larger complex problems. Our methodology combines
knowledge graphs (KGs), prompt engineering and LLM merging to build a model that
works efficiently on global tasks. Our results show an increase in the performance
of the fused LLM of 0.32, reaching a final score of 0.63, which is a significant
improvement over the baseline. This work demonstrates that resource-efficient
and open-source LLMs, when combined strategically, can provide accessible and
effective AI solutions.

Submission Number: 9