Hide navigation sidebar
Hide table of contents sidebar
Back to top
Toggle table of contents sidebar

# Benchmarking Inference

## Prerequisites

Before you begin, ensure you have done the following:

### Install A2Perf

For detailed instructions, please refer to our Installation Guide.

### Benchmarking Training Tutorial

Please refer to the Benchmarking Training Tutorial for instructions on how to train your agent. We will use the artifacts generated from the training tutorial for this inference tutorial.

### Update the `a2perf_benchmark_submission` Submodule

If you have not already done so for the training tutorial, update the `a2perf_benchmark_submission` submodule to the `baselines-local` branch:

    cd a2perf/a2perf_benchmark_submission
    git fetch origin
    git checkout baselines-local
    git pull origin baselines-local
    cd ../..

## Running the Inference Benchmark

After running the training benchmark, you will have a directory with the trained agent and other artifacts. We will use these for the inference benchmark.

### Running locally with XManager (Docker)

#### Running the Benchmark

    xmanager launch xm_launch.py -- \
      --experiment-name=test_inference \
      --root-dir=~/gcs/a2perf/experiments/ \
      --experiment-id=<experiment-id> \
      --domain=QuadrupedLocomotion-DogPace-v0  \
      --submission-gin-config-path=a2perf/submission/configs/quadruped_locomotion/dog_pace/inference.gin \
      --user=$USER \
      --participant-module-path=a2perf/a2perf_benchmark_submission \
      --participant-args="root_dir=/experiment_dir,policy_name=greedy_policy"

#### Command line arguments

- `root-dir`: Specifies the directory where experiment logs and artifacts will be saved.

- `experiment-id`: The ID of the training experiment from which to load the trained agent.

- `submission-gin-config-path`: Points to the Gin configuration file for inference in the Dog Pace environment.

- `participant-module-path`: Indicates the path to the directory containing the submission code.

- `participant-args`: Provides additional arguments for the participant’s code, including the path to the trained agent and the policy name to use.

XManager will automatically launch a Docker container with the necessary dependencies installed. It will create a new experiment directory for the inference results.

### Running Locally Without Docker

If you prefer to run the benchmark locally without using Docker, follow these steps:

#### Installing Dependencies

If you have not already done so for the training tutorial, install the required Python dependencies:

    pip install -r A2Perf/a2perf/a2perf_benchmark_submission/requirements.txt

#### Running the Benchmark

Once the dependencies are installed, you can run the inference benchmark with the following command:

    cd A2Perf
    export A2PERF_ROOT=$(pwd)
    a2perf $A2PERF_ROOT/a2perf/a2perf_benchmark_submission \
      --root-dir=~/gcs/a2perf/experiments/<experiment-id>/test/1 \
      --submission-gin-config-path=$A2PERF_ROOT/a2perf/submission/configs/quadruped_locomotion/dog_pace/inference.gin \
      --participant-args="root_dir=~/gcs/a2perf/experiments/<experiment-id>/test/1,policy_name=greedy_policy"

Note: Replace `<experiment-id>` with the actual ID of your training experiment. This ID is unique for each run and can be found in the output of your training command or in the experiment directory structure.

#### Command line arguments

The command line arguments are similar to those used in the Docker version, but adapted for local execution:

- `root-dir`: Specifies the directory where the training artifacts are located and where inference results will be saved.

- `submission-gin-config-path`: Points to the Gin configuration file for inference in the Dog Pace environment.

- `participant-module-path`: Indicates the path to the directory containing the submission code.

- `participant-args`: Provides additional arguments for the participant’s code, including the path to the trained agent and the policy name to use.

Make sure to adjust the paths according to your setup if they differ from the example provided.

After running the inference benchmark, you will find the results in the specified root directory. These results will include metrics on the agent’s performance during inference, such as average returns, inference time, and resource usage.