# GeNIe: Generative Hard Negative Images Through Diffusion

## Overview


GeNIe is a novel data augmentation technique employing Generative text-based latent Diffusion models. GeNIe merges contrasting data points (an image from the source category and a text prompt from the target category) using a latent diffusion model conditioned on the text prompt. By limiting diffusion iterations, we preserve low-level and background features from the source image while representing the target category, creating challenging samples. Additionally, our method further enhances its effectiveness by dynamically adjusting the noise level for each image, known as GeNIe-Ada. Rigorous experimentation across both few-shot and long-tail distribution scenarios substantiates the superior performance of our approach over existing techniques.

![genie_teaser4](https://anonymous.4open.science/r/GeNIe-F6C6/assets/62820830/33aea37e-cfaa-4f5e-824a-cd7d729b451c)

## Visualizing Generated Samples

GeNIe effectively generates hard negatives for the source image class by preserving low-level features and transforming its main target class according to the prompt. 
![genie_teaser6](https://anonymous.4open.science/r/GeNIe-F6C6/blob/main/assets/genie_vis_supp.jpg)

## Noise Adaptive Sampling

A small noise ratio generates an image that closely resembles the source. Conversely, when the noise ratio is large, it resembles the target category's semantics. Thus, to choose the optimal ratio adaptively for each source image to generate images that preserve the low-level semantics of the source image while effectively representing the semantics of the target category, we propose GeNIE-Ada. Genie-Ada traces the semantic trajectory from source image embeddings to target class embedding through the lens of the classifier `f(.)`, to then adaptively select the sample right after the largest semantic shift.  
![genie_teaser5](https://anonymous.4open.science/r/GeNIe-F6C6/blob/main/assets/geniepp_teaser3.jpg)


## Requirements

All our experiments use the PyTorch library. Install PyTorch and ImageNet datasets following the [official PyTorch ImageNet training code](https://github.com/pytorch/examples/tree/master/imagenet). We used Python 3.7 for our experiments.

## Getting Started 
Please install dependencies in a virtual environment: 
 
 ```
 pip install transformers
 pip install diffusers==0.19.0
 ```


## Few-Shot on tiered-ImageNet and mini-ImageNet


1. Use the imagenet_sampler.ipynb notebook to generate episodes for mini-ImageNet and tiered-ImageNet

2. To generate augmentations for all baselines (txt2img, img2img, and GeNIe), utilize the generate_data.sh script as follows:
```
generate_data.sh 0 20 30
```
This script employs GPU 0 to generate augmentations for episodes 20 to 30.

3. To generate Noise Adaptive dataset for few-shot learning - run `/few_shot/noise_adaptive.ipynb`

3. Train on the augmented dataset:
```
CUDA_VISIBLE_DEVICES=0 python ./train.py --data_path /home/datadrive/mini_imagenet_fs --backbone resnet18 --eval_path /home/datadrive/mini_imagenet_fs/models/mini_r18_v2.pth --transform weak --caching_epochs 5 --n_shots 1 --clf LR --augs_name train_negative_noise_noad_r18_v3 &> PATH_TO_LOG.txt & 
```
Change all paths and GPU/CUDA device IDs as per your settings. 

Our code for generating samples is based on [BOOMERANG](https://colab.research.google.com/drive/1PV5Z6b14HYZNx1lHCaEVhId-Y4baKXwt) and for few-shot testing is based on [UNISIAM](https://github.com/bbbdylan/unisiam). 

## License

This project is under the MIT license.
