## Description

Describe your changes in detail.

## Motivation and Context

Why is this change required? What problem does it solve?
If it fixes an open issue, please link to the issue here.
You can use the syntax `close #15213` if this solves the issue #15213

- [ ] I have raised an issue to propose this change ([required](https://github.com/PKU-Alignment/safe-rlhf/issues) for new features and bug fixes)

## Types of changes

What types of changes does your code introduce? Put an `x` in all the boxes that apply:

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds core functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Documentation (update in the documentation)

## Checklist

Go over all the following points, and put an `x` in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

- [ ] I have read the [CONTRIBUTION](https://safe-rlhf.readthedocs.io/en/latest/developer/contributing.html) guide. (**required**)
- [ ] My change requires a change to the documentation.
- [ ] I have updated the tests accordingly. (*required for a bug fix or a new feature*)
- [ ] I have updated the documentation accordingly.
- [ ] I have reformatted the code using `make format`. (**required**)
- [ ] I have checked the code using `make lint`. (**required**)
- [ ] I have ensured `make test` pass. (**required**)
