# Variable Binding Mechanisms in Recurrent Neural Networks

Tl;dr - check out the demo in `demo_rcopy.ipynb`

The repository contains files for exploring variable binding mechanisms in recurrent architectures. This repo serves as a companion to the paper

## Installation

The full software is written in Python 3.6 and require the following packages:

- `numpy`
- `scipy`
- `matplotlib`
- `pytorch`
- `pytorch-lightning`
- `tensorboard`  (for logging)

to install run the following in the root directory of the repository:

```pip install .```

The variable memory analysis presented in the paper also requires ```emt-tools``` (provided in the supplementary).

## Usage

Prior to use, create ```EXPERIMENT_OUTPUT_DIR``` environment variable with the appropriate
path to the directory where you want to store/read the results of the experiments.

Experiments are in the form of python scripts in the ```experiments_public``` directory. 
The scripts can be run from the root directory of the repository as follows:

```python experiments/<experiment_dir>/<experiment_name>.py```

The py files are files that are used for training the models on various tasks. 
Currently, these are the tasks that are available:

- repeat copy: ```1_0_repeat_copy.py```
- all_linearVB: ```1_0_binary_linearVB.py``` (also contains a slurm cluster script to train multiple models in cluster)

Python notebooks of experiments can also be found in the respective experiment directory in the
```experiment_notebooks``` directory. These notebooks will typically have interpretability experiments 
on all the variable binding tasks.

## Reproducing the results in the paper

The results in the paper can be reproduced by running the following:

- First train the models using the cluster script ```experiments_public/2_all_linearVB/1_0_binary_linearVB.slurm```. Alternatively,
a restricted set of trained models (due to space constraints) can be found in the folder called ```learned_models.tar.gz```. Extract it to the appropriate directory
and set the environment variable ```EXPERIMENT_OUTPUT_DIR```.
- Run the associated notebooks in ```experiments_public/2_all_linearVB/experiment_notebooks/1_0_binary_linearVB.ipynb``` to generate the 
  results in the paper.
- If you want to play with training and interpreting individual variable binding tasks, run the ```demo.rcopy.ipynb``` notebook 
in the root directory of the repository with various task ids.
