Adaptive Depth Tsetlin Automaton

Ali Nikhalat-Jahromi, Ali Mohammad Saghiri, Mohammad Reza Meybodi

Keywords: Reinforcement Learning, Tsetlin Automaton, Automated Machine Learning, Single-State Reinforcement Learning

Abstract: The Tsetlin Automaton (TA) is a foundational single-state reinforcement learning model, but its fixed depth parameter ($N$) poses a significant limitation for navigating the exploration and exploitation dilemma. Despite remarkable advancements, existing TA models lack adaptability in real-world scenarios where dynamic depth adjustments are essential. In this paper, we introduce the Adaptive Depth Tsetlin Automaton (ADTA), a novel solution addressing this challenge. ADTA integrates TA with a reinforcement agent capable of dynamically modifying $N$. We analyze ADTA using Lyapunov stability theorem and Markov chain analysis within a dual-environment framework: the outer environment, where TA operates to maximize rewards, and the inner environment, where a reinforcement learning agent evaluates TA's performance based on $N$. Through actions like 'Grow,' 'Shrink,' and 'Stop,' the inner agent configures $N$ dynamically. Unlike conventional TA configurations with fixed $N$, our approach demonstrates improved reward maximization and regret minimization. Furthermore, we present numerical simulations that corroborate our theoretical results.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3207