Generative Proto-Sequence: Sequence-Level Decision Making for Long-Horizon Reinforcement Learning

Netanel Fried, Liad Giladi, Gilad Katz

Abstract: Deep reinforcement learning (DRL) methods often face challenges in environments characterized by large state spaces, long action horizons, and sparse rewards, where effective exploration and credit assignment are critical. We introduce Generative Proto-Sequence (GPS), a novel generative DRL approach that produces variable-length discrete action sequences. By generating entire action sequences in a single decision rather than selecting individual actions at each timestep, GPS reduces the temporal decision bottleneck that impedes learning in long-horizon tasks. This sequence-level abstraction provides three key advantages: (1) it facilitates more effective credit assignment by directly connecting state observations with the outcomes of complete behavioral patterns; (2) by committing to coherent multi-step strategies, our approach facilitates better exploration of the state space; and (3) it promotes better generalization by learning macro-behaviors that transfer across similar situations rather than memorizing state-specific responses. Evaluations across diverse maze navigation tasks of varying sizes and complexities demonstrate that GPS outperforms leading action repetition and temporal methods in the large majority of tested configurations, where it converges faster and achieves higher success rates.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We have addressed all the concerns and questions raised by the Action Editor. Main changes include:
1. The use of additional experimental seeds.
2. We updated the related work section as requested by the Action Editor.
3. We reviewed all claims made throughout the paper and moderated them as needed.

We would like to thank the Action Editor for the constructive suggestions, and for granting us a little more time to complete our experiments.

Best regards,
The Authors

Video: https://drive.google.com/file/d/1ABRFrOwI8g_UMIzkk2ou6psNRstponHT/view?usp=drive_link

Code: https://github.com/liadgiladi/Generative-Proto-Sequence

Assigned Action Editor: ~Mirco_Mutti1

Submission Number: 5634