
Every python script is idempotent like GNU Make by default, i.e.,
if the output file exists, it skips the experiment.
Nevertheless, each script has options for overriding the behavior (e.g., --force).
Run each python script with -h option to see the detail.

Shell scripts do not have these fancy mechanisms.

* Data generation

+ generate_data.sh --- Invokes generate_datum.py over all necessary combination of problem generation parameters.

+ generate_datum.sh --- Given a parameter for PDDL problem generator,
  + Generates the problem
    + Problem names are assigned uniquely (up to md5 hash collision) and deterministically
  + Solves it optimally with 5min, 8gb using A*+LMCut using fast downward
    + Trivial instances (which satisfy goals at the initial states) are ignored
  + for each intermediate state,
    + Computes h* from the remaining path length
    + Computes several heuristics (lmcut, ff, hmax)
    + Computes a satisficing solution cost (upper bound of h*) using lama-first
  + Saves metadata into json file
    + For X.pddl, the file name is X.json


* Training and accuracy evaluation experiments

+ train.py --- Trains a heuristic function and compute various test losses.
  Saves a json file containing evaluation metrics (e.g. best validation loss) and metadata/hyperparameters.

+ train_all.sh --- Reproduces the entire training experiments.
  It generates the entire set of command lines invoking train.py and submits them to a job scheduler.
  Each job trains a network, evaluates it, and saves the metrics to a JSON file.

+ test_all.sh --- Reproduces the entire evaluation results (JSON) from existing checkpoints.
  It generates the entire set of command lines invoking train.py and submits them to a job scheduler.
  Each job loads a checkpoint, evaluates it, and saves the metrics to a JSON file.

+ test_all_large.sh --- Generates evaluation results using a dataset that contains more challenging PDDL files.


* Planning experiments

+ plan.py --- Evaluates a heuristic using GBFS in our modified pyperplan.
  + To use a learned heuristic, provide a pathname of the JSON file generated by train.py as the name of the heuristic
  + Otherwise, use ff, lmcut, etc. that is supported by pyperplan.
  + It stores the search results+metrics in a JSON file with the same name/location as the PDDL file.

+ plan_all_{ff,hgn,nlm,rr}.sh --- Reproduces the entire planning experiment.


* Plan validation

+ validate.py --- Validates the plan stored in a JSON file produced by plan.py.
  + Uses VAL validator internally.

+ validate_all.sh --- Validates all JSON files under the data/*/plan directories.
