# FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collabration Pipeline

This is the official code (simplified version, just for paper submission) for the paper "FRUINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collabration Pipeline".

We develop FURINA-Builder, a multi-agent collaboration pipeline that enables automatic construction of fully customizable role-playing benchmarks at arbitrary scales. We utilize CustomRPBuilder to build FRUINA-Bench, a comprehensive benchmark featuring both established and synthesized characters with dimension-specific evaluation criteria.

Here we open source the `FURINA-Builder` and `FRUINA-Bench`, as well as the `Character synthesizing pipeline` demo.

All codes and datasets are for academic purposes only.

## Setup

1. Create conda virtual conda environment

```bash
conda create -n your_project_name python=3.10
```

2. Activate the created conda environment

```bash
conda activate your_project_name
```

3. Install necessary dependencies

```bash
pip install -r requirements.txt
```

## FURINA-Builder

![FURINA-Builder](./Figs/FURINA-Builder.png)

To start the role-playing dialogue simulation, please first configure the necessary config files in 

`./FURINA-Builder/configs/` for all simulation settings 

and 

`./FURINA-Builder/.env.example` for api_key and base_url for LLMs

Finally, one click start with well-constructed character-scene pool (`./FURINA-Builder/character_scene_pool`) by running the command 

```bash
bash ./FURINA-Builder/rp_simulation_en.sh or ./FURINA-Builder/rp_simulation_zh.sh
```

The simulation results and summary are saved in `./FURINA-Builder/results`

## FURINA-Bench

![FURINA-Bench](./Figs/FURINA-Bench.png)

`FURINA-Bench` dataset is in `./FURINA-Bench/datasets`

The model source of the dataset is as follow:

![English part](./Figs/Model_sources_english.png)

![Chinese part](./Figs/Model_sources_chinese.png)

To start the evaluation, similar with FURINA-Builder, first set up all necessary config files in 

`./FURINA-Bench/configs/`

Then start the evaluation main code

```bash
bash ./FURINA-Bench/rp_simulation_en.sh or ./FURINA-Bench/rp_simulation_zh.sh
```

## Character Synthesizing Pipeline

An end-to-end synthesis pipeline for generating role-playing scenarios that automatically creates worldviews, synthetic character profiles, scene settings, and initial dialogues at any scale.

## 🌟 Overview

This synthesis pipeline generates high-quality role-playing scenarios through five core steps:

1. **Topic Generation** - Create original novel worldview themes
2. **Worldview Construction** - Generate detailed worldview settings based on themes
3. **Character Generation** - Create character profiles that fit the worldview
4. **Scene Generation** - Design specific character interaction scenarios
5. **Dialogue Generation** - Produce initial character dialogues

## 📁 Project Structure

```
Synthesizing/
├── synthesizing_character_scene_pool_pipeline.py # Main pipeline script
├── utils.py                                     # Common utility functions
├── prompt_template/                             # Prompt templates
│   ├── dataset.json                            # Dataset template
│   ├── dialogue.json                           # Dialogue template
│   ├── initializing_topics.json                # Initial topic template
│   ├── prompt.py                               # Prompt template collection
│   └── scoring_topics.json                     # Topic scoring template
├── frontend_preview/                           # Web preview interface
│   ├── app.py                                  # Flask application
│   └── templates/
│       └── preview.html                        # Preview page
├── cache/                                      # Cache directory
│   ├── best_topic/                            # Best topics
│   ├── character_profile/                     # Character profiles
│   ├── raw_topics/                            # Raw topics
│   ├── scenes/                                # Scene files
│   ├── scored_topics/                         # Scored topics
│   ├── worldview/                             # Worldview files
│   └── tracking.json.example                  # Tracking file example
├── synthetic_character_scene_pool/             # Output result
└── Module scripts:
    ├── topic_generation.py                    # Topic generation
    ├── worldview_generation.py                # Worldview generation
    ├── character_generation.py                # Character generation
    ├── scene_generation.py                    # Scene generation
    ├── initial_dialogue_generation.py         # Initial dialogue generation
    └── character_selector.py                  # Character selector
```

## 🚀 Quick Start

### Environment Setup

1. **Install Dependencies**:
```bash
pip install -r requirements.txt
```

2. **Configure API Keys**:
Create a `.env` file and add the corresponding API keys.
```env
OPENAI_API_KEY=
OPENAI_BASE_URL=

ANTHROPIC_API_KEY=
ANTHROPIC_BASE_URL=

GEMINI_API_KEY=
GEMINI_BASE_URL=

DEEPSEEK_API_KEY=
DEEPSEEK_BASE_URL=

QWEN_API_KEY=
QWEN_BASE_URL=
```

### Basic Usage

#### Run Complete Pipeline from scratch

```bash
python synthesizing_character_scene_pool_pipeline.py
```

#### Custom Parameters

Specify available model and parameters:
```bash
python synthesizing_character_scene_pool_pipeline.py \
    --model claude-3-7-sonnet-20250219 \
    --temperature 1.0 \
    --num-characters 3
```

Enable selector by activate interactive frontend preview:
```bash
python synthesizing_character_scene_pool_pipeline.py --use-selector
```

An example of the Synthesized Character-Scene Pool Preview is shown below.

![](./Figs/frontend.png)

You can also get direct access to the frontend interface for:

- Previewing generated characters and scenes
- Browsing historical generation results
- Managing character files

Start the preview server:

```bash
cd frontend_preview
python app.py
```

Then visit `http://localhost:1210` to view the interface.



## 📝 Pipeline Description

![](./Figs/pipeline_new.png)The entire synthesizing pipeline is mainly based on LLM. Several mainstream LLMs are first employed to sample diverse thematic seeds, from which a coherent fictional worldview is constructed. Based on this worldview, multiple original character profiles are generated. A narrative scene is then created involving the selected characters, culminating in a dialogue that reflects their roles and relationships. Each intermediate step undergoes iterative self-review and refinement to ensure logical consistency, narrative coherence, and linguistic fluency.

---

**Enjoy your role-playing journey!** 🎭✨
