Fuzzy-Assisted Contrastive Decoding Improving Code Generation of Large Language Models

Shuai Wang, Liang Ding, Yibing Zhan, Yong Luo, Shuai Liu, Weiping Ding

Abstract: Large language models (LLMs) play a crucial role in intelligent code generation tasks. Most existing work focuses on pretraining or fine-tuning specialized code LLMs, e.g., CodeLlama. However, pretraining or fine-tuning a code LLM requires a vast corpus of data, significant computational resources, and considerable human effort. Compared to pretraining or fine-tuning LLMs, a simple and flexible method of contrastive decoding has garnered widespread attention to improve the text generation quality of LLMs. While contrastive decoding can indeed improve the text generation quality of LLMs, our research has found that directly using contrastive decoding: 1) introduces erroneous information into the logit distribution generated from normal prompts (i.e., user’s input), particularly in the code generation of LLMs; 2) significantly impedes the inference and decoding time of LLMs. In this work, the limitations of using contrastive decoding directly are systematically highlighted, and a novel real-time fuzzy-assisted contrastive decoding (FCD) mechanism is proposed to improve the code generation quality of LLMs. The proposed FCD mechanism initially categorizes prompts into high-quality and low-quality groups based on the results of the evaluator (i.e., unit test) before integrating the LLM. Next, feature values (e.g., standard deviation, peak value, etc.) related to the logit distribution of predicted tokens during the LLM’s inference process for both high-quality and low-quality prompts are extracted. Finally, the extracted feature values are used to train the fuzzy neural network (i.e, fuzzy min–max neural network) offline, allowing for the prejudgement of the reliability of the logit distribution for normal prompt outputs. This prevents the direct use of erroneous information from contrastive decoding and improves the code generation quality of LLMs. Through extensive experiments, it has been demonstrated that the proposed FCD mechanism can significantly improve the code generation quality of LLMs through FCD. Moreover, the FCD mechanism can also reduce the time required for inference and contrastive decoding.

External IDs:dblp:journals/tfs/WangDZLLD25