This zip contains code for training minimal transformer architectures on sequences drawn from Graphs.

## Files

- `model.py` - Core transformer implementation
- `generate_sequences.py` - Sequence dataset generation
- `prepare_sequences.py` - Data preprocessing
- `train.py` - Training script
- `test.py` - Evaluation script




## Original Code Attribution

The original codebase was adapted from:
https://openreview.net/forum?id=WFbZusv14E&referrer=%5Bthe%20profile%20of%20Yifei%20Shen%5D(%2Fprofile%3Fid%3D~Yifei_Shen1)

Additional inspiration from nanoGPT: https://github.com/karpathy/nanoGPT

## Quick Start

### 1. Generate Data
```bash
python generate_sequences.py --sequence_length 6 --num_sequences 10000 --permutation_type reversal
```

### 2. Process Data
```bash
python prepare_sequences.py --min_value 0 --max_value 100 --permutation_type reversal
```

### 3. Train Model
```bash
python train.py --n_layer 1 --n_head 1 --n_embd 120 --max_iters 10000
```

### 4. Test Model
```bash
python test.py --ckpt_iter 10000 --config 1_1_120
```

