# Data Generation and Evaluation for ThinkTime

## Installation
1. Install `python>=3.11` and `pip3 install -r requirements.txt`
2. Replace all `[LOCAL_LLM_PATH]`, `[OPENAI_BASE_URL]` and `[OPENAI_API_KEY]` with your own local LLM models, your own API KEY for data generation.
3. Download base model (ChatTS-14B) from [ChatTS Repo](https://github.com/NetmanAIOps/ChatTS). We did not modify the model architecture in CHatTS, but the code used for inference differs from ChatTS. Please follow the steps below to fully reproduce ThinkTime.

## Steps to Reproduce
### Step 1: Data Generation
1. **Warm-Up SFT Data:** Run `bash scripts/generate_warmup_sft_datasets.sh`.
2. **RL Data:** Run `bash scripts/generate_rl_datasets.sh`.

### Step 2: Warm-Up SFT Training
Please follow the steps in the `../SFT-Training` folder.

### Step 3: DAPO (RL) Training
Please follow the steps in the `../RL-Training` folder.

### Step 4: Evaluation
1. **Configure Checkpoint:** Please ln or move your trained checkpoint to `ckpt` folder.
2. **Model Inference:** Run `python3 -m thinktime.utils.inference_tool_vllm`
3. **Model Evaluation:** Run `python3 -m evaluation.evaluate_tsmllm_models`
4. **(Optional) Baseline Evaluation:** We also provide the source code for baseline evaluation. Please refer to the code in the `evaluation/` folder for more information.

## Reference
This code is built on [ChatTS](https://github.com/NetmanAIOps/ChatTS). We will comply with the relevant license requirements and open-source the code after acceptance of this paper.

## Notes
The current code is provided for review only. Please do not use it for any other purpose.

