Domain independence is one of the main features of automated planning. Planners, in the context of cost-optimal classical planning, are developed with the intent of solving any type of problem that can be formulated in PDDL. We then compare planners by the number of problem instances they solve on a set of benchmarks, one point for each problem solved. However, does solving the most problems automatically result in having the best domain-independent planner?

In this paper, we compare the best performing, non-portfolio planners from the cost-optimal classical track of the International Planning Competition (IPC) 2018 on the complete set of benchmarks from the previous two competitions (2011 and 2014) and on a subset of the competitions from before 2011. Results show that, as the number of problems for each domain varies, current way of comparing planners (total coverage) can be biased towards the planners that perform the best in the domains with the most instances, but once we normalize those results, we can get a better picture for which technique is the most domain independent.

Benchmarks Old and New: How to compare domain independence for cost-optimal classical planning?

The International Planning Competition 2018 consisted of several tracks on classical, temporal and probabilistic planning. In this paper, we focus on the discrete MDP track of the probabilistic portion of the competition.

We discuss the changes to the input language RDDL, which give rise to new challenges for planning systems, and analyze each of the eight competition domains separately and highlight unique properties. We demonstrate flaws of the used evaluation criterion, the IPC score, and discuss the need for optimal upper bounds. An evaluation of the three top-performers, including their post-competition versions, and a brief analysis of their performance highlights the strengths and weaknesses of the individual approaches.

An Analysis of the Probabilistic Track of the IPC 2018

We argue that planner evaluation metrics should satisfy the independence of irrelevant alternatives criterion, i.e., the decision whether planner A is ranked higher or lower than planner B should be independent of planner C. We show that three metrics used in classical planning competitions do not necessarily satisfy this criterion and highlight alternative metrics that do so.

Planner Metrics Should Satisfy Independence of Irrelevant Alternatives

Over the last year, the amount of research in hierarchical planning has increased, leading to significant improvements in the performance of planners.
However, the research is diverging and planners are somewhat hard to compare against each other.
This is mostly caused by the fact that there is no standard set of benchmark domains, nor even a common description language for hierarchical planning problems.
As a consequence, the available planners support a widely varying set of features and (almost) none of them can solve (or even parse) any problem developed for another planner.
With this paper, we propose to create a new track for the IPC in which hierarchical planners will compete.
This competition will result in a standardised description language, broader support for core features of that language among planners, a set of benchmark problems, a means to fairly and objectively compare HTN planners, and for new challenges for planners.


Hierarchical Planning in the IPC

Designing an experiment, specifically choosing a baseline for comparison as well as the benchmark set, that adequately evaluates the performance of a new suggested technique is a challenging task. Arguably, planner performance in a competition may affect the choice of a baseline. Further, the benchmark set choice may be biased towards the more
recent domains, or, sometimes, the other way around, the most recent domains might be ignored.

We observed that the collection of domains introduced in a competition often aims at fostering research into a specific issue (e.g., general costs, conditional effects, large number of parameters) and thus the best performers in a competition, while excelling on these domains, might exhibit worse performance on other domains. Thus, the results of IPCs might sometimes be misinterpreted and misleading in general.

We suggest to extend the experimental evaluation performed at future IPCs to include as many available domains as possible. For fans of the current setting that evaluates planners exclusively on new, unseen domains, this extended evaluation could be an \emph{additional} evaluation. Furthermore, we suggest to create various suites according to task features and present additional performance information for existing relevant planners on each such suite. That might require to maintain such features, as well as label planning tasks according to these features, preferably in an automated manner. Such information can help, among other, in identifying the current state-of-the-art for specific fragments of planning, as well as the relevant
benchmark set, making designing an experiment easier, mitigating possible biases.

The Role of IPC in Setting Standards for Experimental Evaluation in Planning Research

Planning research has produced a large amount of tools for various formalisms. As planning is a computationally challenging task, it is important to come up with a variety of ideas and approaches to tackle the various sources of planning tasks complexity. However, as a by-product, it is unclear even to experienced planning researcher what tool will work well on a new planning task. The challenge is even harder for a layman. Most planning tools are not easily accessible and those that are might have non adequate performance on some tasks. The problem was partially addressed by the most recent International Planning Competition, with the competing planners being made publicly available in Singularity containers, allowing for easily building and running the planners. This, however, does not solve the problem of choosing the right planner for a given task. An impatient user might forgo the option of using domain-independent planners altogether as a result of an inadequate performance of one randomly chosen planner.

Online algorithm selection using machine learning techniques was shown to be able to produce planners that show good performance on previously unseen domains. For optimal planning, previous success stories were mostly exploiting the fact that most tasks, if solved, are solved quickly. Thus, being able to accurately predict planner performance opens new perspectives for better exploiting various existing planners in practice. The research in this field does, however, currently require a rather deep familiarity with the field of planning.

In order to allow researchers from outside of planning community to tackle the problem of planner performance prediction, we should alleviate the dependence on special knowledge in planning. One step towards achieving this goal is to provide data, complying with the assumptions made on data in the field of machine learning. The data consists of data points, some of which are labelled. Each data point represents a planning task, while labels represent planner performance on that task. The most important assumption about the data is independent and identical distribution (iid). Such an assumption is unrealistic
when planning domains are created manually. Another assumption is that the data is representative of the entire population. In domain-independent planning, where the population consists of all tasks representable in the language of choice (e.g., PDDL), creating planning domains manually cannot produce representable data. For both, it is required to produce data automatically, and in a way that will cover a variety of possible planning tasks.

We propose a new track at IPC to help achieving the mentioned goals. The track will provide an easy access to existing planners, as well as to the data -- a variety of hand-crafted, as well as automatically generated planning tasks, with an additional information that relates to the performance of these planners on the existing tasks. If providing this information seems to be unrealistic (e.g., due to computational load, as all planners would need to be run under the same conditions), an alternative could be to provide the instructions of how to obtain the performance information by running the planners (provided as containers as in the most recent IPC). Furthermore, instead of only providing a fixed set of benchmarks, one could also provide generators to automatically generate more data.

In some sense, this new track would be similar to the learning track of previous IPCs, but with planners being provided alongside benchmarks. The goal of this new track would be to come up with a practical system that can perform well across planning domains. The uniqueness of this track should be in alleviating the need for special knowledge in planning. The goal is to both achieve better exposure and to ease the use of planning tools outside of planning community.


Democratizing Usage of Planning Systems by Facilitating Research in Algorithm Selection for Planning

For  automated  planning  robot  behaviour  in  a  rather  unex-
plored world, often there are exogenous events introducing
new objects, triggered by the actions, that were unforeseen in
the initial state of the planning domain model. This suggests
extending PDDL, on every of its language levels, by a feature
for dynamic object creation. Syntactically, this is dealt with
by adding a keyword new to PDDL. Semantically, the core
challenge in dealing with dynamically created objects is that
they leads to a growing state representation during the plan
space exploration. As almost all existing automated planners
are not designed to deal with varying state vectors —mainly
due to relying on grounding of the lifted PDDL input in their
static analysis stage— in this short paper we present a first
solution to solve the dynamic object creation problem via a
translation of PDDL model into a term rewriting tool, where
plans can be found.

Introducing Dynamic Object Creation to PDDL Planning

The International Planning Competition (IPC) empirically evaluates state-of-the-art planning systems on a set of benchmark problems. The selection of this benchmarks plays an important role in the competition, since they can significantly affect competition results.

In this paper we analyze the diversity of the benchmarks employed in the last IPC through extracting some features from the domains and problems of the optimal track. Finally, we provide some insights from the collected data and propose to use a similar method to select the benchmarks of future competitions.

Insights from the 2018 IPC Benchmarks

The optimal track is one the most exiting events in the International
Planning Competition (IPC).

In this position paper we argue that —despite of not winning
the competition— symbolic search and pattern databases
were likely the most influential planning approaches in the
latest IPC in 2018, and, in continuation to the precursor IPC
in 2014, should be considered as candidates for the current
state-of-the-art.

Five of the Top 6 planners in the 2018 competition, namely
Complementary (1 and 2), Planning-PDBs, Symbolic-Bidirectional,
and Scorpion are based on these technologies.
These planners use the same technology across all domains
and plan in one state space.

The winner of IPC 2018 with an ~ 1% lead in problems being
solved, however, is a so-called portfolio planner, consisting of
a selection of many different planners, one of which is chosen
in a classifier that was trained on a manually selected set of
benchmark instances. In about half of its successful runs, it
called the winner of the previous IPC, which in turn is based
on symbolic search. We argue on whether or not to exclude
portfolios from the IPC is possible and wanted.

Cost-Optimal Planning in IPC 2018: Symbolic Search and Pattern Databases vs. Portfolio Planning

Solver competitions have been used in many areas of AI to assess the current state of the art, and to guide future research and real-world applications. AI planning is no exception, and the International Planning Competition (IPC) has been frequently run for nearly two decades. Due to the organizational and computational burden involved with running these competitions, solvers are generally compared using a single homogeneous software environment for all competitors.

In this work, we use the competing planners and benchmark instance sets from the 2014 IPC Agile and Optimal tracks to investigate two questions. First, how is planner performance affected by the specific choice of software environment?
Second, is it a good strategy to run planners with more recent versions of their software dependencies, in order to maximise performance? By running these competition tracks on eight distinct software environments, we show that planner performance varies significantly based on the chosen software environment, that the magnitude of this variation differs considerably between planners, and that using more recent software versions is not always beneficial.

ICAPS 2019 Workshop WIPC Submissions