GLID$^2$E: A Gradient-Free Lightweight Fine-tune Approach for Discrete Sequence Design

Hanqun Cao, Haosen Shi, Chenyu Wang, Sinno Jialin Pan, Pheng-Ann Heng

Track: Machine learning: computational method and/or computational results

Nature Biotechnology: No

Keywords: Diffusion model, Biological sequence design, RL-based fine-tuning, AI4Science

Abstract:

The design of biological sequences is essential for engineering functional biomolecules that contribute to advancements in human health and biotechnology. Recent advances in diffusion models, with their generative power and efficient conditional sampling, have made them a promising approach for sequence generation. To enhance model performance on limited data and enable multi-objective design and optimization, reinforcement learning (RL)-based fine-tuning has shown great potential. However, existing fine-tuning methods are often unstable in discrete optimization when not using gradients or become computationally inefficient when relying on gradient-based approaches, creating significant challenges for achieving both control and stability in the tuning process.
To address these issues, we propose GLID$^2$E, a gradient-free RL-based tuning approach for discrete diffusion models. Our method introduces a clipped likelihood constraint to regulate the exploration space and reward shaping to better align the generative process with design objectives, ensuring a more stable and efficient tuning process. By integrating these techniques, GLID$^2$E mitigates training instabilities commonly encountered in RL and diffusion-based frameworks, enabling robust optimization even in challenging biological design tasks. In two biological systems, GLID$^2$E achieves competitive performance in function-based design while ensuring lightweight and efficient tuning.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Presenter: ~Hanqun_Cao2

Format: Yes, the presenting author will attend in person if this work is accepted to the workshop.

Funding: Yes, the presenting author of this submission falls under ICLR’s funding aims, and funding would significantly impact their ability to attend the workshop in person.

Submission Number: 26