# Shuffled Transformers for Blind Training

Official implementation of paper: 

Shuffled Transformers for Blind Training

Anonymous authors
Paper under double-blind review
## Overview
Conventional split learning faces the challenge of preserving training data and model privacy as a part of the training is beyond the data owner's control. We tackle this problem by introducing blind training, i.e., training without being aware of the data or the model, realized by shuffled Transformers. This is attributed to our intriguing findings that the inputs and the model weights of the Transformer encoder blocks, the backbone of Transformer, can be shuffled without degrading the model performance. We not only have proven the shuffling invariance property in theory, but also design a privacy-preserving split learning framework following the property, with little modification to the original Transformer architecture. 

![](firgures/framework.png)

## Core Codes

The implementation of our method is located in `./ViT_Base/timm_pretrain.py`, from line 79 to line 96. 

## Environments
Main packages: Python >= 3.8, PyTorch >= 1.10.0

## Run 
### Vit
In folder `ViT_Base`
```
python transfer_learning.py --b=32 --model='timm_pretrain' --data='../data' --R
```
#### Parameters

`--b` sets batch_size

`--model` sets the model used. 'timm_pretrain' is the original ViT pre-trained model, which is not suitable for column shuffle. And 'vit_base_square' is the modified model, whose pre-trained checkpoint is not provided yet. 

Use `--R` or `--RC` to include row shuffle or row column shuffle. If `--R` or `--RC` is used, please use --pe to include position embedding, otherwise it is removed. 
### Bert
In folder `Bert_d2l`

```
python main.py
```



## Dir：
```
├─ViT_Base
│      compute_efficiency.py
│      decryption.py
│      encryption.py
│      timm_pretrain.py
│      test.py
│      transfer_learning.py
│  ├─key
│  │		generate_key.py
│  │		key_768.pt
│  │		unkey_768.pt
│  ├─ViT_base_square
│  │		vit_timm.py
│  ├─pretrain_on_imagenet
│  │	    pretrain_on_imagenet.py
│  │	    pretrain_on_imagenet_no_pe.py
│  │	    pretrain_on_imagenet_normal.py
│  │	    vit_timm.py
│  │	    vit_timm_no_pe.py
│  │	    vit_timm_normal.py
├─Bert_d2l
│      bert.py
│      main.py
│      train.py
│      utils.py
```

Copyright is kept to pay respect to the source code provided by others.
 




