# Description of source code

---

# Setup

---

- Python = 3.8
- CUDA version = 11.4
- Install dependencies:
    
    ```python
    pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
    ```
    
    ```python
    pip install -r requirements.txt
    ```
    

# Multilayer perceptron (MLP)

---

To reproduce the results in Table1, use following commands in source_code/CIFAR10 directory: 

- Original activation function: ReLU
    - Baseline
        - python 7LayerMLP_R1.py
    - Dropout (q = 0.3)
        - python 7LayerMLP_R2.py
    - ZeroLiers (k = 3)
        - python 7LayerMLP_R3.py
    - ZeroLiers-L-k ($k_0$  = 3)
        - python 7LayerMLP_R4.py

- Original activation function: LeakyReLU
    - Baseline
        - python 7LayerMLP_LR1.py
    - Dropout (q = 0.2)
        - python 7LayerMLP_LR2.py
    - ZeroLiers (k = 3)
        - python 7LayerMLP_LR3.py
    - ZeroLiers-L-k ($k_0$  = 3)
        - python 7LayerMLP_LR4.py

- Original activation function: ELU
    - Baseline
        - python 7LayerMLP_E1.py
    - Dropout (q = 0.3)
        - python 7LayerMLP_E2.py
    - ZeroLiers (k = 3)
        - python 7LayerMLP_E3.py
    - ZeroLiers-L-k ($k_0$  = 3)
        - python 7LayerMLP_E4.py

- Original activation function: GELU
    - Baseline
        - python 7LayerMLP_G1.py
    - Dropout (q = 0.2)
        - python 7LayerMLP_G2.py
    - ZeroLiers (k = 3)
        - python 7LayerMLP_G3.py
    - ZeroLiers-L-k ($k_0$  = 3)
        - python 7LayerMLP_G4.py

- Original activation function: SiLU
    - Baseline
        - python 7LayerMLP_S1.py
    - Dropout (q = 0.3)
        - python 7LayerMLP_S2.py
    - ZeroLiers (k = 3)
        - python 7LayerMLP_S3.py
    - ZeroLiers-L-k ($k_0$  = 3)
        - python 7LayerMLP_S4.py

- Original activation function: Mish
    - Baseline
        - python 7LayerMLP_M1.py
    - Dropout (q = 0.3)
        - python 7LayerMLP_M2.py
    - ZeroLiers (k = 3)
        - python 7LayerMLP_M3.py
    - ZeroLiers-L-k ($k_0$  = 3)
        - python 7LayerMLP_M4.py

To reproduce the results in Figure 11-(a), use following commands in source_code/CIFAR10 directory: 

- SGD
    - python 7LayerMLP_R4_SGD.py

- Adam
    - python 7LayerMLP_R4.py
    
- RMSprop
    - python 7LayerMLP_R4_RMS.py
    
- SGD with momentum
    - python 7LayerMLP_R4_SGDM.py
    

# Autoencoder (AE)

---

To reproduce the results in Table 2, use following commands in source_code/CIFAR10 directory:

- Original activation function: ReLU
    - Baseline
        - python 7LayerAE_R1.py
    - Dropout (q = 0.05)
        - python 7LayerAE_R2.py
    - ZeroLiers-L-k ($k_0$  = 3)
        - python 7LayerAE_R3.py

- Original activation function: LeakyReLU
    - Baseline
        - python 7LayerAE_LR1.py
    - Dropout (q = 0.05)
        - python 7LayerAE_LR2.py
    - ZeroLiers-L-k ($k_0$  = 3)
        - python 7LayerAE_LR3.py

To reproduce the results in Figure 11-(b), use following commands in source_code/CIFAR10 directory: 

- SGD
    - python 7LayerAE_R3_SGD.py

- Adam
    - python 7LayerAE_R3.py

- RMSprop
    - python 7LayerAE_R3_RMS.py
    
- SGD with momentum
    - python 7LayerAE_R3_SGDM.py

# Denoising Autoencoder (DAE)

---

To reproduce the results in Table 3, use following commands in source_code/CIFAR10 directory:

- Original activation function: ReLU
    - Baseline
        - python 7LayerDAE_R1.py
    - Dropout (q = 0.05)
        - python 7LayerDAE_R2.py
    - ZeroLiers-L-k ($k_0$  = 3)
        - python 7LayerDAE_R3.py

- Original activation function: LeakyReLU
    - Baseline
        - python 7LayerDAE_LR1.py
    - Dropout (q = 0.05)
        - python 7LayerDAE_LR2.py
    - ZeroLiers-L-k ($k_0$  = 3)
        - python 7LayerDAE_LR3.py

# BERT

---

We use 2 NVIDIA V100 GPUs to train the three-layer BERT with a batch size of 128. 

To reproduce the results in Figure 8, use following commands in each directory:

- Original:
    - Directory:
        - source_code/BERT/Original
    - Command:
        - CUDA_VISIBLE_DEVICES=0,1 python bert_wik2.py
        
- ZeroLiers (k = 4):
    - Directory:
        - source_code/BERT/ZeroLiers
    - Command:
        - CUDA_VISIBLE_DEVICES=0,1 python bert_wik2.py
        
- ZeroLiers-L-k ($k_0$  = 4):
    - Directory:
        - source_code/BERT/ZeroLiers-L-k
    - Command:
        - CUDA_VISIBLE_DEVICES=0,1 python bert_wik2.py
        
- Baseline:
    - Directory:
        - source_code/BERT/Baseline
    - Command:
        - CUDA_VISIBLE_DEVICES=0,1 python bert_wik2.py
        

# GPT-2

---

We use 2 NVIDIA V100 GPUs to train the one-layer GPT-2 with a batch size of 24. 

To reproduce the ZeroLiers-L-k ($k_0$  = 3) result in Figure 12, use following command in source_code/GPT2 directory:

- CUDA_VISIBLE_DEVICES=0,1 python gpt2_wiki103_v2.py