# Pre-Flight

Pre-Flight is an evaluation benchmark that tests the model's understanding of ICAO annex documentation, flight dispatch rules and airport ground operations safety procedures and protocols. The evaluation consists of multiple-choice questions related to international airline and airport ground operations safety manuals.

Developed by [Airside Labs](https://airsidelabs.com), a specialized aviation sector software, data, and AI experimentation startup focused on accelerating innovation and optimizing operations through data-driven insights.

<!-- Contributors: Automatically Generated -->
Contributed by [@alexbrooker](https://github.com/alexbrooker)
<!-- /Contributors: Automatically Generated -->

<!-- Usage: Automatically Generated -->
## Usage

First, install the `inspect_ai` and `inspect_evals` Python packages with:

```bash
pip install inspect_ai
pip install git+https://github.com/UKGovernmentBEIS/inspect_evals
```

Or, if developing on a clone of the `inspect_evals` repo, you can install the package in editable mode with:

```bash
pip install -e ".[dev]"
```

Then, evaluate against one or more models with:

```bash
inspect eval inspect_evals/pre_flight --model openai/gpt-4o
```

After running evaluations, you can view their logs using the `inspect view` command:

```bash
inspect view
```

If you don't want to specify the `--model` each time you run an evaluation, create a `.env` configuration file in your working directory that defines the `INSPECT_EVAL_MODEL` environment variable along with your API key. For example:

```bash
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
ANTHROPIC_API_KEY=<anthropic-api-key>
```
<!-- /Usage: Automatically Generated -->

<!-- Options: Automatically Generated -->
## Options

You can control a variety of options from the command line. For example:

```bash
inspect eval inspect_evals/pre_flight --limit 10
inspect eval inspect_evals/pre_flight --max-connections 10
inspect eval inspect_evals/pre_flight --temperature 0.5
```

See `inspect eval --help` for all available options.
<!-- /Options: Automatically Generated -->

## Dataset

The dataset consists of multiple-choice questions drawn from standard international airport ground operations safety manuals. Each question has 4-5 possible answer choices, with one correct answer.

Topics covered include:
- Airport safety procedures
- Ground equipment operations
- Staff training requirements
- Emergency response protocols
- Fueling safety
- Aircraft towing procedures

Specifically by section denoted by id (note there are gaps in the id sequence to enable future additions):
 - 001-205 Airport operations procedures and rules
 - 206-299 reserved
 - 300-399 Derived from US aviation role training material
 - 400-499 ICAO annexes, rules of the air and global guidelines
 - 500-599 General aviation trivia questions
 - 600-699 Complex reasoning scenarios

Example questions from the dataset, note that the actual messages sent differ slightly as we use the multiple_choice solver, 
these examples have been lightly edited for readability:

> Q: What is the effect of alcohol consumption on functions of the body?  
>    Options: 
>    A. "Alcohol has an adverse effect, especially as altitude increases." 
>    B. "Alcohol has little effect if followed by an ounce of black coffee for every ounce of alcohol."
>    C. "Small amounts of alcohol in the human system increase judgment and decision-making abilities."
>    D. "no suitable option"
>
> A: A

> Q: An airport is managing snow removal during winter operations on January 15th. Given the following information. 
> | Current time | Temperature | Snowfall                                | Ground temperature | 
> | ------------ | ----------- | --------------------------------------- | ------------------ |
> | 07:45        | -3°C        | Light, expected to continue for 2 hours | -6°C               | 
> 
> Aircraft movements and stand occupancy
> | Stand | Current/Next Departure | Next Arrival |
> | ----- | ---------------------- | --------------- |
> | A | BA23 with TOBT 07:55  | BA24 at 08:45 |
> | B | AA12 with TOBT 08:10  | AA14 at 09:00 |
> | C | Currently vacant      | ETD112 at 08:15 |
> | D | ETD17 with TOBT 08:40 | ETD234 at 09:30 | 
>
> Assuming it takes 20 minutes to clear each stand and only one stand can be cleared at a time, 
> what is the most efficient order to clear the stands to maximize the number of stands cleared 
> before they're reoccupied? Consider that stands must be cleared after an aircraft departs and 
> before the next one arrives. Answer with the stand order only for all four stands and with the 
> total duration of clearing and no preamble or other explanation, our jobs depend on this answer. 
> Example 1. A, B, C, D 1hr 20 minutes Example 2. D, A, C, B 5hr 5 minutes  
> Options:
> A. "A, B, C, D 1 hour 20 minutes"
> B. "D, A, C, B 5 hours 5 minutes"
> C. "C, A, B, D 1 hour 20 minutes"
> D. "no suitable option"
>
> A: C

## Scoring

The benchmark is scored using accuracy, which is the proportion of questions answered correctly. Each question has one correct answer from the multiple choices provided.
The implementation uses the multiple_choice solver and the choice scorer.

## **Evaluation Report**

The following results were produced from running the full dataset (300 samples) across multiple models on March 21, 2025.

| Model | Accuracy | Correct Answers | Total Samples |
|-------|----------|----------------|---------------|
| anthropic/claude-3-7-sonnet-20250219 | 0.747 | 224 | 300 |
| openai/gpt-4o-2024-11-20 | 0.733 | 220 | 300 |
| openai/gpt-4o-mini-2024-07-18 | 0.733 | 220 | 300 |
| anthropic/claude-3-5-sonnet-20241022 | 0.713 | 214 | 300 |
| groq/llama3-70b-8192 | 0.707 | 212 | 300 |
| anthropic/claude-3-haiku-20240307 | 0.683 | 205 | 300 |
| anthropic/claude-3-5-haiku-20241022 | 0.667 | 200 | 300 |
| groq/llama3-8b-8192 | 0.660 | 198 | 300 |
| openai/gpt-4-0125-preview | 0.660 | 198 | 300 |
| openai/gpt-3.5-turbo-0125 | 0.640 | 192 | 300 |
| groq/gemma2-9b-it | 0.623 | 187 | 300 |
| groq/qwen-qwq-32b | 0.587 | 176 | 300 |
| groq/llama-3.1-8b-instant | 0.557 | 167 | 300 |

These results demonstrate the capability of modern language models to comprehend aviation safety protocols and procedures from international standards documentation. The strongest models achieve approximately 75% accuracy on the dataset.
