# A Half-Space Projected Stochastic Gradient Method for Group-Sparsity Regularized Problem

PyTorch Implementation of non-convex experiments in "A Half-Space Projected Stochastic Gradient Method for Group-Sparsity Regularized Problem". 

## Abstract

Optimizing with group sparsity is significant in enhancing model interpretability in machining learning applications, e.g., feature selection, compressed sensing and model compression. However, for large-scale stochastic training problems, effective group-sparsity exploration are typically hard to achieve. Particularly, the state-of-the-art stochastic optimization algorithms usually generate merely dense solutions. To overcome this shortage, we propose a stochastic method—Half-space Stochastic Projected Gradient method (HSPG) to search solutions of high group sparsity while maintain the convergence. Initialized by a simple Prox-SG Step, the HSPG method relies on a novel Half-Space Step to substantially boost the
sparsity level. Numerically, HSPG demonstrates its superiority in deep neural networks, e.g., VGG16, ResNet18 and MobileNetV1, by computing solutions of higher group sparsity, competitive objective values and generalization accuracy.

## Requirements

+ torch: 1.4.0
+ torchvision: 0.5.0

Please follow the instructions [here](<https://pytorch.org/get-started/locally/>) to install PyTorch.


## Set up Dataset

In `datasets.py`, `DATA_DIR`represents the path to dataset. Please replace this constant variable with the path to the dataset.

## Training

Common command line arguments are listed as follows:

+ lmbda: Weighting factor lambda
+ dataset_name: We support `fashion_mnist` and `cifar10`.
+ backend: We support `vgg16` and `resnet18` and `mobilenetv1`
+ epsilon: Control parameter epsilon in Algorithm 3. The default value is `0.00`.

**For Prox-SG**, run this command:

```bash
python proxsg.py --lmbda 0.001 \
                 --dataset_name <dataset_name> \
                 --backend <backend>
```

An example of ResNet18 on CIFAR10:

```bash
python proxsg.py --lmbda 0.001 \
                 --dataset_name cifar10 \
                 --backend resnet18
```



**For Prox-SVRG**, run this command:

```bash
python proxsvrg.py --lmbda 0.001 \
                   --dataset_name <dataset_name> \
                   --backend <backend>
```

An example of ResNet18 on CIFAR10:

```bash
python proxsvrg.py --lmbda 0.001 \
                   --dataset_name cifar10 \
                   --backend resnet18
```



**For HSPG**, run the following command:

```bash
python hspg.py --lmbda 0.001 \
                   --dataset_name <dataset_name> \
                   --backend <backend> \
                   --n_p 150 \
                   --epsilon <epsilon>
```

An example of ResNet18 on CIFAR10:

```bash
python hspg.py --lmbda 0.001 \
                   --dataset_name cifar10 \
                   --backend resnet18 \
                   --n_p 150 \
                   --epsilon 0.05
```

The log file and model are saved in the folder of `results` and  `checkpoints` respectively.



Alternatively, we can also run `bash run.sh` to test all experiments.

## Evaluation

To evaluate our model, see the log files and models which are saved in `results` and `checkpoints` respectively.