DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer

Joseph Suarez, Justin Johnson, L. Fei-Fei

Abstract: We present a generic dynamic architecture that employs a problem specific differentiable forking mechanism to leverage discrete logical information about the problem data structure. We adapt and apply our model to CLEVR Visual Question Answering, giving rise to the DDRprog architecture; compared to previous approaches, our model achieves higher accuracy in half as many epochs with five times fewer learnable parameters. Our model directly models underlying question logic using a recurrent controller that jointly predicts and executes functional neural modules; it explicitly forks subprocesses to handle logical branching. While FiLM and other competitive models are static architectures with less supervision, we argue that inclusion of program labels enables learning of higher level logical operations -- our architecture achieves particularly high performance on questions requiring counting and integer comparison. We further demonstrate the generality of our approach though DDRstack -- an application of our method to reverse Polish notation expression evaluation in which the inclusion of a stack assumption allows our approach to generalize to long expressions, significantly outperforming an LSTM with ten times as many learnable parameters.

TL;DR: A generic dynamic architecture that employs a problem specific differentiable forking mechanism to encode hard data structure assumptions. Applied to CLEVR VQA and expression evaluation.

Keywords: CLEVR, VQA, Visual Question Answering, Neural Programmer

Data: [CLEVR](https://paperswithcode.com/dataset/clevr)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1803.11361/code)