Skill based framework for harnessing emergent abilities of LLMs for knowledge management
Abstract: This paper introduces a skill-based framework for enhancing the emergent abilities of Large Language Models (LLMs) within knowledge management applications, leveraging Retrieval-Augmented Generation (RAG). LLMs exhibit emergent abilities that can significantly impact their performance in complex tasks. Our approach explores and harnesses these abilities by defining skills, optimizing model performance through the DSPy framework, and assessing impact using a combination of discrete and continuous metrics. We conducted experiments on LLMs of varying scales, focusing on models like GPT-3.5 and Mistral 7B, across skill associated datasets (Emotion-based, fact-based persona, persona emotional state, crisp answers). Our results indicate that the DSPy optimization enhances LLM performance, particularly in generating contextually rich responses while reducing operational costs. This study not only sheds light on the mechanisms through which emergent abilities develop in LLMs but also illustrates how skill-based frameworks can systematically leverage these properties to improve efficiency and effectiveness in real-world applications.
Paper Type: Long
Research Area: Generation
Research Area Keywords: retrieval-augmented generation, analysis, optimization methods, evaluation and metrics
Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Publicly available software and/or pre-trained models, Theory
Languages Studied: English
Submission Number: 5334
Loading