# V*Bench

**V*Bench** is a visual question & answer benchmark that evaluates MLLMs in their ability to process high-resolution and visually crowded images to find and focus on small details. This benchmark is in contrast with common multimodal benchmarks that focus on large, prominent visual elements.

V*Bench was introduced in [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://arxiv.org/abs/2312.14135). Penghao Wu, Saining Xie, 2023.

This implementation evaluates performance of MLLMs on the benchmark and does not implement V* visual search algorithm introduced in the paper.

## Dataset and Tasks

The dataset ([craigwu/vstar_bench](https://huggingface.co/datasets/craigwu/vstar_bench)) consist of 191 multiple choice questions in two categories:

**vstar_bench_attribute_recognition** - Attribute Recognition task has 115 samples and requires the model to recognize a certain type of attribute (e.g. color, material) of an object. This task has 4 multiple choice answers.

**vstar_bench_spatial_relationship_reasoning** - Spatial Relationship Reasoning task has 76 samples and asks the model to determine the relative spatial relationship between two objects. This task has two multiple choice answers.

## Evaluation

First, install the `inspect_evals` Python package with:

```bash
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Then, evaluate Attribute Recognition and Spatial Relationship Reasoning tasks:

```bash
inspect eval inspect_evals/vstar_bench_attribute_recognition --model openai/gpt-4o
inspect eval inspect_evals/vstar_bench_spatial_relationship_reasoning --model openai/gpt-4o
```
