## Big Learning: A Universal Machine Learning Paradigm? A PyTorch Implementation

[//]: # (<p align="center">)

[//]: # (  <img src="" width="480">)

[//]: # (</p>)


This is a PyTorch/GPU implementation of the paper [Big Learning: A Universal Machine Learning Paradigm?](https://arxiv.org/pdf/2207.03899.pdf):

* On the image experiments associated with MNIST and CelebA.

* This repo is a modification on the [MAE repo](https://github.com/facebookresearch/mae.git). Installation and preparation follow that repo.

* This repo is based on [`timm==0.3.2`](https://github.com/rwightman/pytorch-image-models), for which a [fix](https://github.com/rwightman/pytorch-image-models/issues/420#issuecomment-776459842) is needed to work with PyTorch 1.8.1+.

### Catalog

- [x] Big learning on the MNIST dataset
- [x] Big learning on the CelebA dataset


###  Big learning on the MNIST dataset:
```
python main_MNIST_GAN_DT_small.py \
    --method generate \
    --GNoise GIt \
    --DST DIn \
    --Sratio=-1.
    --Tratio=-1.
    --communicate
    --CommuSratio=-1.
    --D_no_padST
    --GLastNorm LR \
    --DLastNorm LN \
    --lambdaGP 10. \
    --batch_size 256 \
    --input_size 64 \
    --patch_size 8 \
    --embed_dimG 256 \
    --depthG 6 \
    --num_headsG 8 \
    --decoder_embed_dimG 512 \
    --decoder_depthG 6 \
    --decoder_num_headsG 8 \
    --embed_dimD 256 \
    --depthD 6 \
    --num_headsD 8 \
    --toImg 3conv \
    --data_path ${IMAGE_DIR} \
    --output_dir ${RESULTS_DIR} \
```


### Big learning on the CelebA dataset:
```
python main_CelebA_GAN_DT_small.py \
    --method generate \
    --GNoise GIt \
    --DST DIn \
    --Sratio=-1.
    --Tratio=-1.
    --communicate
    --CommuSratio=-1.
    --D_no_padST
    --GLastNorm LR \
    --DLastNorm LN \
    --lambdaGP 10. \
    --batch_size 256 \
    --input_size 120 \
    --patch_size 10 \
    --embed_dimG 256 \
    --depthG 6 \
    --num_headsG 8 \
    --decoder_embed_dimG 512 \
    --decoder_depthG 6 \
    --decoder_num_headsG 8 \
    --embed_dimD 256 \
    --depthD 6 \
    --num_headsD 8 \
    --toImg 3conv \
    --set_scheduler warmupC \
    --warmup_epochs 10 \
    --data_path ${IMAGE_DIR} \
    --output_dir ${RESULTS_DIR} \
```

