# Mixed-Memory RNNs for Learning Long-term Dependencies in Irregularly Sampled Time Series

## Requirements

**Packages**
- Python 3.5 or newer
- TensorFlow 2.0 or newer
- PyTorch 1.8 or newer
- pytorch-lightning 1.3 or newer
- JAX (only for the gradient norm experiments)

Tested with python3.8 and TensorFlow 2.6/PyTorch 1.8 on Ubuntu 20.04

## Module description

- ```pt_trainer.py```: Training script for the models implemented in PyTorch
- ```irregular_sampled_data.py```: Framework agnostic dataset module (can be used by PyTorch and Tensorflow)
- ```torch_node_cell.py```: Implementation of the mmRNN in PyTorch
- ```cornn.py``` PyTorch implementation of the coRNN (adapted for irregularly sampled sequences)
- ```lipschitz_rnn.py``` PyTorch implementation of the Lipschitz RNN (adapted for irregularly sampled sequences)
- ```node_cell.py```: Implementation in Tensorflow of all other continuous-time RNNs (as well as mmRNN) used in the experimental evaluation in the paper
- ```xor_task.py```: Executable to run the synthetic XOR experiment in Tensorflow (both the dense and the event-based modes)
- ```person_activity.py```: Executable to run the Person activity experiment in Tensorflow
- ```et_mnist.py```: Executable to run the Event-based sequential MNIST experiment in Tensorflow
- ```walker_kinematic.py```: Executable to run the Walker2d kinematic simulation experiment in Tensorflow

Each of the four executable python scripts contain the code for loading and pre-processing the data, as well the code to train and evaluate the models.

## Example usage

The four executable python scripts use some command line argument parsing to specify the RNN type and hyperparameters.
The RNN type can be specified by ```--model RNN```, where ```RNN``` is one of the following

| ```--model RNN``` | Description                           |
| ----------------- | ------------------------------------- |
| ```lstm```        | Augmented LSTM                        |
| ```ctrnn```       | CT-RNN                                |
| ```node```        | ODE-RNN                               |
| ```ctgru```       | CT-GRU                                |
| ```grud```        | GRU-D                                 |
| ```gruode```      | GRU-ODE                               |
| ```vanilla```     | Vanilla RNN with time-dependent decay |
| ```bidirect```    | Bidirectional RNN (LSTM with ODE-RNN) |
| ```phased```      | PhasedLSTM                            |
| ```hawk```        | Hawkes process LSTM                   |
| ```cornn```       | coRNN                                 |
| ```irnn```        | iRNN                                  |
| ```lipschitz```   | Lipschitz RNN                         |
| ```mmrnn```       | mmRNN (ours)                          |




For instance

```bash
python3 xor_task.py --model lstm --epochs 500 --dense
```

runs the XOR sequence classification experiment with the dense encoding (=regularly sampled time-series).
By omitting the ```--dense``` flag one can run the same experiment but with the event-based encoding (=irregularly sampled time-series)

## Datsets

Data for the XOR experiment are generated *on the fly*, i.e., no manual downloading necessary.
The MNIST data are loaded through the ```tf.keras.datasets``` API, i.e., no manual downloading necessary.
Data for the Walker kinematic experiments are included in this repository, i.e., no manual downloading necessary.
Data for the person activity task however, must be downloaded first. This can be done by ```source download_datasets.sh```.

## Logging

Each executable python script stores the result of the experiment in the directory ```results``` (which will be created if it does not exists).
The ```results``` directory will have the following structure:

- ```results/xor_event``` Results of the event-based XOR task
- ```results/xor_dense``` Results of the dense encoded XOR task
- ```results/smnist``` Results of the event-based sequential MNIST experiment
- ```results/person_activity``` Results of the Person activity dataset
- ```results/walker``` Results of the Walker2d kinematic simulation dataset

The results for different RNN types will be logged in separate files.
For instance, ```results/xor_event/lstm_64.csv``` will contain the results of the augmented LSTM with 64 hidden units on the event-based XOR task. The naming of the RNN models is the same as for the ```--model``` argument as described above.

## Gradient norms

The vector norms of the state-previous state Jacobian are computed using JAX's ```jacrev``` functionality with the script ```grad_norms.py```.