The MiniWoB++ benchmark is an extension of the OpenAI MiniWoB benchmark. It adds a number of tasks, Javascript interfaces, as well as Python code for interacting with the environment via Selenium. The benchmark was introduced in our paper: Reinforcement Learning on Web Interfaces using Workflow-Guided Exploration.
For usage, refer to the GitHub repository.
Original tasks. We use a subset of the original MiniWoB tasks that only involve (1) clicking and (2) typing text from the prompt.
Clicking and typing on off-screen elements are allowed since they are performed by Javascript calls.
choose-list | Choose an item from a drop down list. |
click-button | Click on a specific button in a generated form. |
click-button-sequence | Click on buttons in a certain order. |
click-checkboxes | Click desired checkboxes. |
click-color | Click the specified color. |
click-dialog | Click the button to close the dialog box. |
click-dialog-2 | Click a specific button in a dialog box. |
click-link | Click on a specified link in text. |
click-option | Click option boxes. |
click-shades | Click the shades that match a specified color. |
click-shape | Click on a specific shape. |
click-tab | Click on a tab element. |
click-tab-2 | Click a link inside a specific tab element. |
click-test | Click on a single button. |
click-test-2 | Click on one of two buttons. |
click-widget | Click on a specific widget in a generated form. |
count-shape | Count number of shapes. |
email-inbox | Navigate through an email inbox and perform some actions. |
enter-date | Use the date input to pick the correct date. |
enter-password | Enter the password into the form. |
enter-text | Enter given text to a textfield. |
enter-text-dynamic | Enter dynamically generated text to a textfield. |
enter-time | Enter the specified time into the input. |
focus-text | Focus into a text input. |
focus-text-2 | Focus on a specific text input. |
grid-coordinate | Find the Cartesian coordinates on a grid. |
guess-number | Guess the number. |
identify-shape | Identify a randomly generated shape. |
login-user | Enter user login details into the form. |
navigate-tree | Navigate a file tree to find a specified file or folder. |
search-engine | Search through a bunch of results to find a specified link. |
social-media | Interact with a social media feed. |
tic-tac-toe | Win a game of tic-tac-toe. |
use-spinner | Use a spinner to select given number. |
The UI elements in some tasks have animation delays or change the state when the browser is defocused. We provide the "nodelay" version without these issues.
book-flight | Search for flight results. |
choose-date | Learn to operate a date picker tool. |
click-collapsible | Click a collapsible element to expand it. |
click-collapsible-2 | Find and click on a specified link, from collapsible elements. |
click-pie | Click items on a pie menu. |
use-autocomplete | Use autocomplete element efficiently. |
book-flight-nodelay | [book-flight] |
choose-date-nodelay | [choose-date] |
click-collapsible-2-nodelay | [click-collapsible-2] |
click-collapsible-nodelay | [click-collapsible] |
click-pie-nodelay | [click-pie] |
use-autocomplete-nodelay | [use-autocomplete] |
Additional tasks. Some are harder versions of the existing tasks, while some are completely new.
click-checkboxes-large | [click-checkboxes] Click at least 5 out of up to 12 checkboxes |
click-checkboxes-soft | [click-checkboxes] Paraphrased entries |
click-checkboxes-transfer | [click-checkboxes] Train and test on different number of targets |
click-tab-2-hard | [click-tab-2] Varying number of tabs from 2 to 6 |
login-user-popup | [login-user] Random popup |
multi-layouts | Fill in forms of varying layouts. |
multi-orderings | Fill in forms with shuffled field orderings. |
social-media-all | [social-media] Do some action on all matching entries |
social-media-some | [social-media] Do some action on some matching entries |
email-inbox-forward-nl | [email-inbox-forward] NL instruction (30 templates) |
email-inbox-forward-nl-turk | [email-inbox-forward] NL instruction (100 templates) |
email-inbox-nl-turk | [email-inbox] NL instruction (100 templates for each subtask) |
Flight search tasks. These are server-free ports of the FormWoB tasks in the original World of Bits paper.
flight.Alaska | port of Alaska FormWoB |
flight.Alaska-auto | port of Alaska FormWoB but harder |
flight.AA | port of American Airlines FormWoB (unused) |
Debug tasks. These are easier versions of existing tasks. They are used for debugging.
choose-date-easy | [choose-date] December only |
choose-date-medium | [choose-date] December or November only |
click-tab-2-easy | [click-tab-2] One 1 tab |
click-tab-2-medium | [click-tab-2] Choose between a link or "no match" |
click-test-transfer | [click-test] Different buttons during train and test |
email-inbox-delete | [email-inbox] No scrolling + 1 subtask |
email-inbox-forward | [email-inbox] No scrolling + 1 subtask |
email-inbox-important | [email-inbox] No scrolling + 1 subtask |
email-inbox-noscroll | [email-inbox] No scrolling |
email-inbox-reply | [email-inbox] No scrolling + 1 subtask |
email-inbox-star-reply | [email-inbox] No scrolling + 2 subtasks |
Clicking non-elements. These tasks involve clicking at a specific point inside a canvas-like element.
bisect-angle | Find the line that bisects an angle evenly in two. |
circle-center | Find the center of a circle. |
count-sides | Count the number of sides on a shape. |
find-midpoint | Find the shortest mid-point of two points. |
number-checkboxes | Draw a given number using checkboxes. |
right-angle | Given two points, add a third point to create a right angle. |
use-colorwheel | Use a color wheel. |
use-colorwheel-2 | Use a color wheel given specific random color. |
Hovering. These tasks require hovering and moving the mouse cursor.
click-menu | Click menu items. |
click-menu-2 | Find a specific item from a menu. |
Dragging. These tasks involve dragging.
click-scroll-list | Click multiple items from a scroll list. (also require Shift + click) |
drag-box | Drag the smaller box into the larger box. |
drag-circle | Drag an item in a specified direction. |
drag-cube | Drag a 3D cube to show a specific face. |
drag-items | Drag items in a list, in a specified direction |
drag-items-grid | Drag items in a 2D grid around. |
drag-shapes | Drag shapes into a box. |
drag-sort-numbers | Drag numbers into sorted ascending order. |
highlight-text | Highlight all the text. |
highlight-text-2 | Highlight the specified paragraph. |
resize-textarea | Resize a textarea in a given direction. |
scroll-text | Scroll through a text area element and enter last word into text area. |
scroll-text-2 | Scroll through a text area in a given direction. |
text-editor | Modify a text"s style in a text-editor. |
use-slider | Use a slider to select a particular value. |
use-slider-2 | Use sliders to create a given combination. |
Typing free text. These tasks involve typing texts that are not substrings of the prompt. Some of these also require advanced reasoning (e.g., solving math problems).
copy-paste | Copy text and paste it into an input. |
copy-paste-2 | Copy text from a specific textarea and paste it into an input. |
enter-text-2 | Convert given text to upper or lower case. |
find-word | Find nth word in a block of text. |
read-table | Read information out from a table. |
read-table-2 | Read multiple pieces of information out from a table. |
simple-algebra | Solve for X. |
simple-arithmetic | Perform some arithmetic math operations. |
terminal | Use the terminal to delete a file. |
text-transform | Enter slightly transformed text into a text box. |
visual-addition | Count the total number of blocks. |
Timing. These tasks require the agent to wait for events to happen before acting, and a "nodelay" version is impossible to make.
chase-circle | Keep your mouse inside a moving circle. |
moving-items | Click moving items before they disappear. |
simon-says | Push the buttons in the order shown. |
Missing. These tasks are listed in the original paper but were missing from the OpenAI website.
ascending-numbers | Click on the numbers in ascending order. |
button-delay | Wait a certain period of time before clicking the second button. |
buy-ticket | Buy a ticket that matches the requested criteria. |
daily-calendar | Create an event on a daily calendar. |
drag-shape | Drag a randomly generated shape in a specified direction. |
drag-shapes-2 | Drag shapes into boxes, categorized by type. |
draw-circle | Draw a circle around a marked point. |
draw-line | Draw a line through a marked point. |
find-greatest | Find the card with the greatest number. |
form-sequence | Perform a series of instructions on a form. |
form-sequence-2 | Perform a series of instructions on a form. |
form-sequence-3 | Perform a series of instructions on a form. |
generate-number | Generate a random number that meets certain criteria. |
hot-cold | Find and click on the hot area. |
hover-shape | Hover over the colored shape. |
odd-or-even | Mark each number as odd or even. |
order-food | Order food items from a menu. |
phone-book | Find a contact in a phone book. |
sign-agreement | Sign a user agreement. |
stock-market | Buy from the stock market below a specified price. |