Optimizing Class-Level Code Generation: Enhancing In-Context Learning in Large Language Models with Pruning Techniques
Abstract: Recently, many large language models (LLMs) have been proposed, showing advanced proficiency in code generation. Such generation focuses on generating independent and often method-level code, thus leaving it unclear how LLMs perform in generating more complicated tasks. To fill this research gap, researchers have studied to construct prompt to achieve good performance in complicated code generation, i.e., class-level code generation, and they have launched a corresponding benchmark, ClassEval, on generating and evaluating class-level code. However, the obvious difficulty of class-level code generation prompt construction lies in that class-level prompt has longer texts than those of method-level code generation, and at the same time the input length of the released models always has a limitation, such as: GPT-3.5 and GPT-4 with 4096 tokens and 8192 tokens limitations respectively.\\
Therefore, it is important to research how to construct the class-level prompt by pruning some code tokens. Through the pruning strategy, we add more code examples into prompt to deliver as many semantic information as possible to LLMs. We introduce a new pruning strategy, namely attention-guided strategy, to this research point. By this pruning strategy, we conduct experiments on code generation by GPT-3.5, a kind of LLM proved to achieve excellent performance on generation tasks. In our work, we adopt ClassEval benchmark dataset specialized for class-level code generation to conduct our experiments.
Additionally, we evaluate the strategy both in method-level and class-level metrics, finding that this pruning strategy is effective to prune appropriate tokens for LLM to generate class-level code. Above all, attention-guided strategy outperforms the randomly pruning strategy with 4.2\%, 10.2\% and 13\% higher class-level code generation accuracy by LLM. We also analyze the impact of the quantity of code reduction on the quality of code generation in LLM, concluding that pruning under 40\% of code snippets with extra 4 examples included can take great advantage of the intelligence of LLM to contribute to perfect class-level code generation.
Paper Type: Long
Research Area: Machine Learning for NLP
Research Area Keywords: NLP, NLP Applications
Languages Studied: Python
Submission Number: 2871
Loading