Leveraging Pre-Trained Large Language Models (LLMs) for On-Premises Comprehensive Automated Test Case Generation: An Empirical Study

Hamza Mohammed

Abstract:

The rapidly evolving field of Artificial Intelligence (AI)-
assisted software testing has predominantly focused on au-
tomated test code generation, with limited research explor-
ing the realm of automated test case generation from user
stories requirements. This paper presents a comprehensive
empirical study on harnessing pre-trained Large Language
Models (LLMs) for generating concrete test cases from nat-
ural language requirements given in user stories. We inves-
tigate the efficacy of various prompting and alignment tech-
niques, including prompt chaining, few-shot instructions, and
agency-based approaches, to facilitate secure on-premises de-
ployment. By integrating our learnings with an on-premises
model setup, wherein we deploy a RoPE scaled 4-bit quan-
tized LLaMA 3 70B Instruct model, optionally augmented
with LoRA adapters trained on QA datasets, we demonstrate
that this approach yields more accurate and consistent test
cases despite VRAM constraints, thereby maintaining the se-
curity benefits of an on-premises deployment.