# Lipschitz regularized gradient flows and generative particles
This repository is the official implementation of *Lipschitz regularized gradient flows and latent generative particles* in Tensorflow 2

## Requirements
To install requirements:

-tensorflow-gpu=2.7.0 or 2.8.0
```pip install tensorflow==2.7.0```

## Dependent python libraries
```
pip install numpy, tensorflow, matplotlib, scipy, 
pip install umap-learn  # gene_expression_example
```

## Main function
* ```f-Gamma-gpa.py```: Lipschitz regularized GPA for general purpose
* ```f-Gamma-gpa-ae.py```: Latent Lipschitz regularized GPA which imports an autoencoder in ```util/saved_model```

call main function with parameters in ```util/input_args.py```. Generic parameters are
* ```--f KL/alpha```: choose f-divergence
* ```--formulation DV/LT```: Specify the Variational formula. For KL divergence, use DV for Donsker-Varadhan. For alpha divergence, use LT for Legendre-transform
* ```-L (float)/(None)```: Specify Lipschitz constant for the discriminator phi
* ```--dataset```: Choose dataset in the list. See ```util/input_args.py```. Different dataset entails different dataset property parameters.
* ```-NN fnn/cnn/fnn-cnn```, ```-N_fnn_layers (list of positive integers)```, ```--epochs_nn (int)```, ```--lr_NN (float)```: Specify parameters for neural network (discriminator phi) structure and training 
* ```--epochs (int)```, ```--lr_P` (float)``: Specify parameters for particles update
* ```--plot_result True/False```: Control plotting the result after the calculation


## Experiments 

### Conditional Image generation
* Conditional GPA conditioned by MNIST digit labels, CIFAR10 object labels
* Generate images from logistic distribution
* FID calculated with autoencoder_mnist model stored in ```util/save_model/FID_MNIST_deep_64_encoder``` and ```util/save_model/FID_MNIST_deep_64_decoder```
* FID calculated with autoencoder_mnist model stored in ```util/save_model/FID_MNIST_deep_256_encoder``` and ```util/save_model/FID_MNIST_deep_256_decoder```
* MNIST latent gpa autoencoder stored in ```util/save_model/MNIST_deep_128_encoder''' and  ```util/save_model/MNIST_deep_128_decoder'''
* CIFAR10 latent gpa autoencoder stored in ```util/save_model/CIFAR10_deep_128_encoder''' and  ```util/save_model/CIFAR10_deep_128_decoder'''

```
# GPA - 200 samples MNIST
python3 f-Gamma-gpa.py --f KL --formulation DV -L 1 -N_Q 200 -N_P 200 --dataset MNIST --N_conditions 10 -NN cnn-fnn -N_cnn_layers 128 128 128 -N_fnn_layers 784 --epochs_nn 5 --epochs 10000 --lr_NN 0.001 --lr_P 0.5 --random_seed 0 --save_iter 100 --plot_result True --exp_no 1

# GPA - 2000 samples MNIST
python3 f-Gamma-gpa.py --f KL --formulation DV -L 1 -N_Q 2000 --mb_size_Q 200 -N_P 200 --dataset MNIST --N_conditions 10 -NN cnn-fnn -N_cnn_layers 128 128 128 -N_fnn_layers 784 --epochs_nn 5 --epochs 20000 --lr_NN 0.001 --lr_P 0.5 --random_seed 0 --save_iter 100 --plot_result True

# Latent GPA - 200 samples MNIST
python3 f-Gamma-gpa-ae.py --f KL --formulation DV -L 1 -N_Q 200 --dataset MNIST_ae -NN fnn -N_latent_dim 128 -N_fnn_layers 256 512 256 --epochs_nn 10 --epochs 10000 --lr_NN 0.001 --lr_P 0.5 --random_seed 0 --exp_no latent-128_1 --save_iter 100 --plot_result True --N_conditions 10

# Latent GPA - 2000 samples MNIST
python3 f-Gamma-gpa-ae.py --f KL --formulation DV -L 1 -N_Q 2000 --mb_size_Q 200 --dataset MNIST_ae -NN fnn -N_latent_dim 128 -N_fnn_layers 256 512 256 --epochs_nn 10 --epochs 20000 --lr_NN 0.001 --lr_P 0.5 --random_seed 0 --exp_no latent-128_1 --save_iter 100 --plot_result True --N_conditions 10

# Latent GPA - 200 samples MNIST
python3 f-Gamma-gpa-ae.py --f KL --formulation DV -L 1 -N_Q 200 --dataset CIFAR10_ae -NN fnn -N_latent_dim 128 -N_fnn_layers 256 512 256 --epochs_nn 10 --epochs 50000 --lr_NN 0.001 --lr_P 0.1 --random_seed 0 --exp_no latent-128_2 --save_iter 100 --plot_result True --N_conditions 10 

# Latent GPA - 2000 samples MNIST
python3 f-Gamma-gpa-ae.py --f KL --formulation DV -L 1 -N_Q 2000 -mb_size_Q 200 --dataset CIFAR10_ae -NN fnn -N_latent_dim 128 -N_fnn_layers 256 512 256 --epochs_nn 10 --epochs 50000 --lr_NN 0.001 --lr_P 0.1 --random_seed 0 --exp_no latent-128_2 --save_iter 100 --plot_result True --N_conditions 10 
```



### Gene expression data merging
* Influence of choosing latent dimensions
* N_dim=2,5,10,20,50,100,200
* N_fnn_layers=32 32 32		# for N_dim=2,5,10
* N_fnn_layers=64 64 64		# for N_dim=20,50,100
* N_fnn_layers=128 128 128	# for N_dim=200
* Step 1: Download geo data using ```gene_expression_example/create_dataset.R``` (already have data!)
* Step 2: Store PCA result (source_norm_dataset_dim_##.csv and target_norm_dataset_dim_##.csv) using ```gene_expression_example/pca_dim_reduction_normalized.py``` (already have data!)
* Step 3: Run ```f-Gamma-gpa.py``` as below
* Step 4: Decode the output using ```gene_expression_example/back_mapping_normalized.py```

```
python3 f-Gamma-gpa.py --f KL --formulation DV -L 1 --dataset BreastCancer -NN fnn -N_fnn_layers $N_fnn_layers --activation_ftn relu --epochs_nn 10 --epochs 5000 --lr_NN 0.1 --lr_P 5.0 --exp_no 0 --save_iter 100 --plot_result True -N_dim $N_dim
```

### Comparison with Score based model
* Mixture of 2D Student-t: Compare alpha=2-divergence Lip=1 GPA to NCSN for heavy tailed data from t(0.5) distribution in 2D
* Mixture of 2D Gaussians2: Compare alpha=2-divergence Lip=1 GPA to NCSN for heavy tailed data from t(0.5) distribution in 2D

```
# Mixture of 2D Student-t
python3 f-Gamma-gpa.py --f alpha -alpha 2 -L 1.0 --dataset Mixture_of_student_t -nu 0.5 -NN fnn -N_fnn_layers 32 32 32 --epochs_nn 5 --epochs 10000 --lr_NN 0.005 --lr_P 1.0 --random_seed 0 --plot_result True --exp_no 0 -N_P 500 -N_Q 200

# Mixture of 2D Gaussians2
python3 f-Gamma-gpa.py --f KL --formulation DV -L 1.0 --dataset Mixture_of_gaussians2 -sigma_Q 1.0 -NN fnn -N_fnn_layers 32 32 32 --epochs_nn 5 --epochs 10000 --lr_NN 0.005 --lr_P 1.0 --random_seed 0 --plot_result True --exp_no 0 -N_P 500 -N_Q 200

python3 ncsn.py # change the dataset variable inside the script
```

### 2D Mixture of Gaussians 1
* Fit data to Multi-wells
* Stable with L=1,10,100, but different converging speed
* It diverges without Lipschitz bound 
* random_seed=0 for trajectory plot
* random_seed=0,1,2 for Divergence, Lipschitz regularized FI loss plots

```
ython3 f-Gamma-gpa.py --f KL --formulation DV -L 1.0 --dataset Mixture_of_gaussians -sigma_Q 0.5 -NN fnn -N_fnn_layers 32 32 32 --epochs_nn 5 --epochs 10000 --lr_NN 0.005 --lr_P 1.0 --random_seed 0 --plot_result True --exp_no 0 --optimizer SGD

python3 f-Gamma-gpa.py --f KL --formulation DV -L 10.0 --dataset Mixture_of_gaussians -sigma_Q 0.5 -NN fnn -N_fnn_layers 32 32 32 --epochs_nn 5 --epochs 10000 --lr_NN 0.005 --lr_P 1.0 --exp_no 0 --random_seed 0 --plot_result True --optimizer SGD

python3 f-Gamma-gpa.py --f KL --formulation DV -L 100.0 --dataset Mixture_of_gaussians -sigma_Q 0.5 -NN fnn -N_fnn_layers 32 32 32 --epochs_nn 5 --epochs 10000 --lr_NN 0.005 --lr_P 1.0 --exp_no 0 --random_seed 0 --plot_result True --optimizer SGD

python3 f-Gamma-gpa.py --f KL --formulation DV --dataset Mixture_of_gaussians -sigma_Q 0.5 -NN fnn -N_fnn_layers 32 32 32 --epochs_nn 5 --epochs 10000 --lr_NN 0.005 --lr_P 1.0 --exp_no 0 --random_seed 0 --plot_result True --optimizer SGD
```

