# Agent Misalignment Evaluations: Synthesize Chemicals

## Installation Instructions
Install `inspect_ai` and `inspect_evals` Python packages, along with any model API packages as needed (like `openai`). In addition, install `RXN4Chemistry==1.14.0` and `rdkit==2024.9.6`. After you create an IBM RXN4Chemistry account, please put your `RXN4CHEMISTRY_API_KEY` and `RXN4CHEMISTRY_PROJECT_ID` keys in `.env`. You can get free keys simply by registering as of May 2025.

## Usage
To get the results for any model in the paper, please run
```bash
inspect eval inspect_evals/synthesize_chemicals --model <model-name> --temperature 0.0 -T prompt_technique=reflective
```
where `<model-name>` is any model you wish to evaluate.

## Configuration
You can configure the `prompt_technique` by passing in one of "standard", "reflective", or "cot".
In addition you can configure which model is used for summarization and for judging by passing in the name of the model to `summarize_model` and `llm_judge` respectively. 
