# Ask Me Anything: A simple strategy for prompting open-source language models

## Data
All data lives in `/home/data`. Please follow the instructions below to download all necessary data for experiments. 
 
1. Download the PromptSource (P3) dataset from HuggingFace: https://huggingface.co/datasets/bigscience/P3 and run [download_p3.py](./download_p3.py)

2. A subset of the tasks are not included in P3 and require manual downloads. 
    * AGNews: https://huggingface.co/datasets/ag_news
    * Amazon Product: https://github.com/allenai/flex/blob/75d6d1cea66df2c8a7e3d429c6af5008ccf1544b/fewshot/hf_datasets_scripts/amazon/amazon.py
    * DROP: https://huggingface.co/datasets/drop
    * NaturalQuestions (NQs): https://huggingface.co/datasets/natural_questions      
    * RealTimeQA: https://github.com/realtimeqa/realtimeqa_public/tree/main/past/2022 (GCS files from June 17th - July 22, 2022)
    * ReCoRD: https://sheng-z.github.io/ReCoRD-explorer/
    * SST2: https://huggingface.co/datasets/sst2
    * StoryCloze: https://huggingface.co/datasets/story_cloze
    * WebQuestions (WebQ): https://huggingface.co/datasets/web_questions
    * WSC: https://huggingface.co/datasets/super_glue (wsc.fixed)

## Running Models
We run inference on models using the [Manifest](./manifest/) package. Navigate to the `./manifest` folder for install instructions and model hosting details.

## Running AMA Experiments

To run full suite of benchmarks:
1. follow the instructions in README in [Manifest](./manifest/README.md) to host a Neo, BLOOM, OPT, or GPT3 model
2. run the following command:

    ```python
    python3 tasks/<task_name>_final.py - --run_zeroshot 1 --run_fewshot 1 --run_decomp 1 --num_boost 3 --k_shot 3 --output_metrics_file ./metrics.json
    ```
3. running the command from (2) will save the zero_shot, few_shot and AMA results in `/home/final_runs/<task_name>` and output all performance metrics to `./metrics.json`

To run WS algorithm:

1. In our WS algorithm, we utilize various functions in the [MetalAMA](./metal-ama/), a slighlty modified version of the open source [Metal](https://github.com/HazyResearch/metal) package. Please install the package to your python environment by navigating to `./metal-ama` folder and running `pip install -e .`.

2. In order to run the WS algorithm on the predicted outputs of the AMA runs, use the following command:

    ```python
    python3 boosting/run_ws.py --task_name <task_name> --data_dir /home/final_runs
    ```




