WaveCorr: Deep Reinforcement Learning with Permutation Invariant Policy Networks for Portfolio Management

Saeed Marzban, Erick Delage, Jonathan Li

Keywords: permutation invariance, portfolio management, deep reinforcement learning, policy network

Abstract: The problem of portfolio management represents an important and challenging class of dynamic decision making problems, where rebalancing decisions need to be made over time with the consideration of many factors such as investors’ preferences, trading environment, and market conditions. In this paper, we present a new portfolio policy network architecture for deep reinforcement learning (DRL) that can exploit more effectively cross-asset dependency information and achieve better performance than state-of-the-art architectures. In doing so, we introduce a new form of permutation invariance property for policy networks and derive general theory for verifying its applicability. Our portfolio policy network, named WaveCorr, is the first convolutional neural network architecture that preserves this invariance property when treating asset correlation information. Finally, in a set of experiments conducted using data from both Canadian (TSX) and American stock markets (S\&P 500), WaveCorr consistently outperforms other architectures with an impressive 3\%-25\% absolute improvement in terms of average annual return, and up to more than 200\% relative improvement in average Sharpe ratio. We also measured an improvement of a factor of up to 5 in the stability of performance under random choices of initial asset ordering and weights. The stability of the network has been found as particularly valuable by our industrial partner.

One-sentence Summary: This paper develops a new permutation invariance property for deep neural network architectures and designs a new convolutional portfolio policy network architecture that better exploits cross-asset dependencies present in stock prices time series.