# Dynamic Memory Sparsification
This directory provides sample training code for Dynamic Memory Sparsification (DMS) as a part of a NeurIPS submission.

The code is based on the public implementation of DMC available at https://github.com/NVIDIA/Megatron-LM/tree/dmc . We modify the main file `dmc.py` and save the modifications as `dms.py`, so the two versions could be compared.

We assume the existence of additional input args:
 * `dms_enabled` which turns on DMS training and disables DMC
 * `dms_window_size` which controlls the size of the sliding window
 * `dms_decision_neuron_scaling_factor` which re-scales the decision neuron to speed up its training
 * `dms_immediate_eviction` which enables the (inferior) immediate eviction mode

# License
We attach the license for the original code.
