On Low Frequencies Fourier Features in Reinforcement Learning

TMLR Paper5200 Authors

Abstract: Recent reinforcement learning (RL) methods have adopted ideas from image processing tasks by employing Fourier Features (FFs) encoding. This approach enables a typical multilayer perceptron (MLP) to learn different frequency features. However, a disparity exists between the scale of frequencies used for image and RL problems. Previous works employed significant lower frequencies to successfully train RL agents and defer to the Neural Tangent Kernels (NTK) theory for justification. However, we observed that NTK cannot provide satisfactory explanations. We present a novel perspective empirically to show why lower frequencies are essential for the successful training of RL agents. Our empirical investigation is based on the cross-correlation among state dimensions and their overall cross energy spectral density (CSD). Based on our empirical observation, we propose a simple enhancement to the current FFs formulation and achieve performance improvements over current FFs formulation and baseline methods.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Oleg_Arenz1

Submission Number: 5200