
# AutoToS


## Installation

Use the package manager pip to install the required dependencies.

```bash
pip install -r requirements.txt
```

## Environment Variables
Create a .env file in the root directory and add the following information, replacing "your key" with your key:

```bash
API_KEY="your key"
API_BASE_URL="http://0.0.0.0:4000"
```
Our code works with any [LiteLLM Proxy Server](https://github.com/BerriAI/litellm?tab=readme-ov-file#step-1-start-litellm-proxy). You may adapt the [DomainTestBase](./src/domain_base_class.py) class to work with LM of your choice.

## Export Utils File
To ensure the utils file is correctly exported, run the following command in the root directory path:
```bash
export PYTHONPATH=$PYTHONPATH:./src
```



## Experiments
To run experiments you can use the following command line argument in the src/ folder:
```bash
python experiments.py  --model name_of_model --domain name_of_domain  
```
and if you want to test with a complex validator during the successor soundness test you can write
```bash
python experiments.py  --model name_of_model --domain name_of_domain --complex-validation
```

Currently available domains are  
 * `24game` - the 24Game is a mathematical puzzle, where the task is to get from 4 numbers to a single number 24 through a series of arithmetical operations. We use the dataset from [ToT repository](https://github.com/princeton-nlp/tree-of-thought-llm/tree/master/src/tot/data/24)
 * `blocks` - the BlocksWorld domain is one of the classical planning domains, where the goal is to transform from one configuration of blocks to another
 * `cw` - 5x5 mini crossword puzzles
 * `sokoban` - a famous game where an agent needs to push boxes in a maze to their target positions, and
 * `prontoqa` - a logical inference [dataset](https://github.com/asaparov/prontoqa) 


You also can simply run all experiments consecutively with the `--domain all` option

