# xgemval (Xtal GEnerative Model eVALuation)

This repository is the official implementation of "Continuous Uniqueness and Novelty Metrics for Generative Modeling of Inorganic Crystals," submitted to the AI4Mat workshop at NeurIPS 2025. 

## Requirements

If you use uv:
```setup
cd xgemval/
uv venv --python 3.12
uv python pin 3.12
uv add -r requirements.txt
```
If not, prepare python 3.12 environment and install the required packages from the requirements.txt

## (Optional) Run evaluation
All the results necessary to reproduce the tables and figures in the draft are already provided in `expt/rslt/<model_name>/metrics.json` and `expt/rslt/<model_name>/screen_ehull.pkl.gz`.
However, if you want to compute uniqueness and novelty from scratch, run `expt/main.py`.
You need to set an environmental variable MP_API_KEY beforehand.
```
export MP_API_KEY=<your_materials_project_api_key>
```
Here are some examples.

e.g.) Compute uniqueness and novelty with $d_\mathrm{comp}$ on samples generated from CDVAE trained on the MP20 dataset.
```
python expt/main.py --model cdvae --dataset mp20 --uniqueness comp --novelty comp --screen none
```
e.g.) Compute uniqueness and novelty with all distance functions on samples generated from DiffCSP trained on the MP20 dataset.
```
python expt/main.py --model diffcsp --dataset mp20 --uniqueness comp wyckoff smat magpie amd --novelty comp wyckoff smat magpie amd --screen none
```

e.g.) Compute uniqueness and novelty with all distance functions on metastable samples generated from DiffCSP++ trained on the MP20 dataset.
```
python expt/main.py --model diffcsppp --dataset mp20 --uniqueness comp wyckoff smat magpie amd --novelty comp wyckoff smat magpie amd --screen ehull
```

If you use uv, you should run the command with `uv run`. 
The result will be saved in `expt/rslt/<model_name>/metrics.json`.
To ensure efficient re-computation, the distance matrices and embeddings will be saved in the same directory.
Please note that the calculation of uniqueness and novelty from scratch is time-consuming (see Table 5 in the paper).

## Reproduce tables and figures in the paper
You can reproduce our results using the notebook `expt/neurips2025_figures_tables.ipynb`.
To reproduce Table 1, you need an API key to the Materials Project.
