Event Tables for Efficient Experience Replay
Event Certifications: lifelong-ml.cc/CoLLAs/2023/Journal_Track
Abstract: Experience replay (ER) is a crucial component of many deep reinforcement learning (RL) systems.
However, uniform sampling from an ER buffer can lead to slow convergence and unstable asymptotic
behaviors. This paper introduces Stratified Sampling from Event Tables (SSET), which partitions
an ER buffer into Event Tables, each capturing important subsequences of optimal behavior. We
prove a theoretical advantage over the traditional monolithic buffer approach and combine SSET with
an existing prioritized sampling strategy to further improve learning speed and stability. Empirical
results in challenging MiniGrid domains, benchmark RL environments, and a high-fidelity car racing
simulator demonstrate the advantages and versatility of SSET over existing ER buffer sampling
Submission Length: Long submission (more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=EPvGaTGR0J&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DTMLR%2FAuthors%23your-submissions)
Changes Since Last Submission: The paper was desk rejected due to font inconsistency. After careful analysis it was the "times.sty" latex package that was messing up the fonts set by tmlr.sty. I removed it and verified it comparing the fonts used with an approved tmlr submission.
Assigned Action Editor: ~Martha_White1
License: Creative Commons Attribution 4.0 International (CC BY 4.0)
Submission Number: 690
Loading