Find Before you Fine-Tune (FiT): How to Identify Small-Scale LLM Suitable for Cybersecurity Question-Answering Tasks?
Abstract: This research presents an innovative automated framework that dynamically pairs Retrieval-Augmented Generation (RAG) with various pre-trained and fine-tuned Large Language Models (LLMs) to enhance their effectiveness in cybersecurity applications. RAG, initially introduced to leverage external knowledge for Languade Models and LLMs, proves insufficient on its own in domains requiring acute precision, like cybersecurity, where it's crucial to distinguish between relevant and irrelevant information. Our framework addresses this gap by evaluating and matching the most appropriate LLM to RAG based on specific cybersecurity tasks. This not only facilitates the provision of contextually accurate and pertinent information but also streamlines the analytical process, significantly saving time for cybersecurity analysts and improving their capability to identify and respond to security threats efficiently. Our findings suggest that instruction tuning causes a knowledge drop, fine-tuning may worsen hallucination in the cybersecurity domain, and the evaluation tasks in our framework are able to predict the post fine-tuning behavior of LLMs.
Paper Type: Long
Research Area: NLP Applications
Research Area Keywords: Large Language Models, Cybersecurity, Contextualization, Fine-tuning
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models
Languages Studied: English
Submission Number: 885
Loading