The main experiment is set up based on [pythia](https://github.com/EleutherAI/pythia).

Training:

1. Install the environment in requirements.txt

2. Download dataset following Pythia instructions

3. Modify config in file `configs`

4. Modify the `job.sh` file

5. Run:
**bash job.sh**

Please refer to Pythia code for evaluation.


Our analysis experiments are conducted based on tools and datasets from [sparse probing](https://github.com/wesg52/sparse-probing-paper)


If you want to use the L2E block directly, its location is `megatron/model/L2E.py`