
## Experiment log

### Preliminary results 

Resources: [wandb](https://wandb.ai/anonymous/project)


The preliminary experiment is conducted only on natural question (NQ) dataset (+ PPO) with a small number of training steps.


### v0.1

Resources: [wandb](https://wandb.ai/anonymous/project), [docs](https://github.com/anonymous/repo/tree/main/scripts/nq_hotpotqa), [scripts](https://github.com/anonymous/repo/tree/main/scripts/nq_hotpotqa/v0.1)


We extend the experiments from NQ to seven datasets with both PPO and GRPO methods. The studies are still on a small number of training steps with a big learning rate warm up ratio.


### v0.2

Resources: [wandb](https://wandb.ai/anonymous/project), [docs](https://github.com/anonymous/repo/tree/main/scripts/nq_hotpotqa), [scripts](https://github.com/anonymous/repo/tree/main/scripts/nq_hotpotqa/v0.2), [paper](https://arxiv.org/abs/2503.09516)


We fix several bugs including [retrieved token masking](https://github.com/anonymous/repo/pull/21) and [GRPO sample indexing](https://github.com/anonymous/repo/commit/9ec2fa9892fbf0315d0c67b4dc08ae8f6cf5f378). 
The former can largely improve the stablity of RL training. 
Then we adjust the training scripts, increasing the number of training steps and decreasing the learning rate warm up ratio, to obtain a better performance, and conduct experiments on different scale of LLMs (3B, 7B, 14B).


### v0.3

Resources: [wandb](https://wandb.ai/anonymous/project), [docs](https://github.com/anonymous/repo/tree/main/scripts/nq_hotpotqa), [scripts](https://github.com/anonymous/repo/tree/main/scripts/nq_hotpotqa/v0.3), [paper](https://arxiv.org/abs/2505.15117)

We conduct studies on (1) reward design; (2) LLM backbone; and (3) search engine.

- Reward design
  - Format reward
  - Intermediate retrieval reward
- LLM backbone
  - LLM type (e.g., general LLM or reasoning LLM)
  - LLM scale (3B/7B/14B/32B)
- Search engine
  - RL training dynamics
  - generalization during inference
- Data scaling

Details can be found in the [paper](https://arxiv.org/abs/2505.15117).
