# Metagen Supplemental Material

## Accessing MetaDB and MetaBench

We have uploaded all relevant data to a publicly accessible S3 account. To pull down this data, please run the following command: 

`aws s3 sync s3://metagen-datasets/anonymized-release/ <destination folder>`

In case there are any issues with e.g. permissions for the s3 sync, we also provide an alternative method. First, run the following command to retrieve a precomputed list of all files in the `anonymized-release` folder:
`wget https://metagen-datasets.s3.amazonaws.com/anonymized-release/all_files_fullpaths.txt`

Then, `cd` into the folder where you wish to download our data, and run the following command:
`wget --input-file=<path to previously downloaded file> --cut-dirs=1 --no-host-directories --force-directories`

In the documentation and throughout our code, root paths (beginning with `/`) indicate a path relative to this top-level directory.

## Evaluating Benchmarks

The files /benchmark/{train, validate, test}.txt contain the model names used as part of the training/validation/testing stages, respectively. Each collection is mutually exclusive; no model appears in more than one of these files. The split listed in these files is consistent across all tasks and query types.

To evaluate a MetaBench benchmark, you need

1. A test.jsonl file from /benchmark
2. A predicted_code directory containing .py and .json predictions, named i.py where i is the index of the test example in test.jsonl
3. A predicted_successs directory containing the results of running all \*.py programs from predicted_code that create valid metamaterials


Run metagen/scripts/collate_results.py. Modify the dictionaries in the scripts to point to your data paths for the model results you wish to evaluate.
This will create a pandas dataframe with all results, as well as the latex tables that appear in our paper and supplemental.


## Running MetaDSL Programs: Processing data

MetaDSL programs are located in the /models/ directory.

The easiest way to run a MetaDSL program is using our docker image metagendb/metagen:

`docker run -t metegendb/metagen process_material path/to/program.py path/to/write/success path/to/write/failure`

If you load the metagen subfolder in a devcontainer, you can also run programs standalone in ipython notebooks to get interactive visualizations as a CAD server.

## Reproducing Training and Evaluation: LlaVA
See llava_train_and_test/README.md

## Reproducing Training and Evaluation: Nova
Unfortunately, due to AWS restrictions, the Nova models are not currently shareable. Amazon Bedrock formatted training and testing splits are available in the same training_data.zip file reference in llava_train_and_test/README.md if you would like to retrain from scratch.

