Learning What Matters: Endogenous Representation through Advantage Estimation and Sparse Attention

RLC 2025 Workshop CRLW Submission4 Authors

Keywords: exogenous mdp, representation learning, advantage estimation, sparse attention

Abstract:

Recent works hypothesize that directly learning the advantage function can induce \textit{endogenous} representations that only depend on variables that are causally related to the agent's action.
However, previous empirical evidence is limited to small linear environments.
We investigate this hypothesis in a more complex image-based environment, showing that advantage function learning leads to improved generalization and robustness. In addition, by using a sparsity-regularized Transformer as the function approximator, we qualitatively demonstrate that the learned advantage function attends only to the decision-relevant part of the observations, leading to improved interpretability.

Submission Number: 4