--------------------------------------------------
Instructions to Generate Paper Tables and Plots
--------------------------------------------------

I. cd into 'vowpal_wabbit/' directory and run 'make' to build the vw executable. All standard dependencies for vw must be installed first (https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Dependencies).

II. cd into the 'cb_bakeoff/' directory and complete the following steps:
- 1. Download and preprocess datasets from OpenML (saved in multiclass/):
--  'python oml_to_vw.py 3 1590'

- 2. Create additional shuffles for datasets used in plots:
-- 'python shuffle.py'

- 3. Run all of the experiments for the tables:
-- 3a. Edit run_vw_job.py so that the 'VW' variable points to your vw executable.
-- 3b. Execute './all_vw_jobs.sh'. This can take many hours to complete. You can check progress using top/htop. You may want to change the number of processes (default 32) based on your machine).
-- 3c. Merge output files by executing './merge_output.sh'

- 4.  Generate tables:
-- 4a. Run 'python paper_tables.py --opt_01 --base_name allbinary' to generate the left table.
-- 4b. Run 'python paper_tables.py --granular_name --name 01 --base_name allbinary' to generate the right table.

- 5. Run additional experiments and create plots:
-- 5a. Edit run_and_plot.py so that the 'VW_BINARY' variable points to  vw executable.
-- 5b. Execute 'python run_and_plot.py' (this should take < 3 hours on a modest consumer machine)
-- 5c. Check 'plots/' to see the output.
