# dishonesty-bench

1. Run `git submodule update --init --recursive` in the root folder
2. Run `uv sync` in the root folder
4. For running experiments with the (Together API)[https://docs.together.ai/] you need to create an API key and run `export TOGETHER_API_KEY=xxxxx`.
5. Install as local package: `uv pip install -e .`

# Example usage
## Extract activations

```
uv run python src/probing/extract_activations.py --model meta-llama/Llama-3.1-8B --layers -1 --datasets data/raw/geometry_of_truth/cities_cities_conj.csv --output_dir data/processed/activations 
``` 

## Train probe

```
uv run python src/probing/train.py --activations_dir data/processed/activations/meta-llama/Llama-3.1-8B/cities_cities_conj --layer 15
```

## Evaluate probe

```
uv run python src/probing/evaluate.py --activations_dir data/processed/activations/meta-llama/Llama-3.1-8B/cities_cities_conj --layer 15
```

# License
This codebase is licensed under Apache 2.0 (see LICENSE). Unless otherwise specified, all datasets created by the authors of this repository are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Some datasets include third-party components and are subject to additional licensing terms (see CONTEXT file).

