Leveraging LLMs for Formal Grammar Generation in Programming Contest Testing
Abstract: Test cases are crucial for ensuring the program's correctness and evaluating performance in programming. The high diversity of test cases within constraints is necessary to distinguish between correct and incorrect answers. Automated source code generation is currently a popular area due to the inefficiency of manually generating test cases. Recent attempts involve generating conditional cases from problem descriptions using deep-learning models that learn from source code. However, this task requires a combination of complex skills such as extracting syntactic and logical constraints for a given test case from a problem, and generating test cases that satisfy the constraints. In this work, we introduce a modified context-free grammar that explicitly represents the syntactical and logical constraints embedded within programming problems. Our innovative framework for automated test case generation separates restriction extraction from test case generation, simplifying the task for the model. Our experimental results show that, compared to current methods, our framework produces test cases that are more precise and effective. All the codes in this paper are available in https://anonymous.4open.science/r/neural_translation_for_test_case_generation.
Paper Type: Long
Research Area: NLP Applications
Research Area Keywords: software engineering, automated program testing, formal grammar, test case generation, large language model
Contribution Types: NLP engineering experiment, Theory
Languages Studied: formal language, English
Submission Number: 1213
Loading