# Problem Description

## Background
The paper *COGS: A Compositional Generalization Challenge Based on Semantic Interpretation* (Kim and Linzen 2020) showed that Transformer encoder-decoder models generalize poorly on compositional generalization tasks they proposed. 

However, in a follow-up paper *The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers* (Csordás et al. 2021), the authors claimed that turning off early stopping based on development set performance improves generalization performance of the same Transformer models substantially.

The papers will be available inside the provided repository in both PDF format as `kim_and_linzen_2020.pdf` and `csordas_et_al_2021.pdf` and markdown format as `kim_and_linzen_2020.md` and `csordas_et_al_2021.md` if you need to refer to them.

## Extension to be implemented
Your task is to modify the codebase provided (the original codebase of Kim and Linzen 2020) to implement the aforementioned version of the Transformer experiment from Csordás et al. 2021 that does not use early stopping to show that this modification indeed leads to improved generalization. Instead of using early stopping, the model should be trained for a large fixed number of pre-specified steps. Use the small version of the Transformer model mentioned in the paper and codebase and use 30,000 as the maximum number of training steps. No other hyperparameter should change.

Please make the experiment runnable by implementing a single script called `run_final.sh` in the root of the repository. This script should call all necessary commands with all parameters specified and should not have any command line arguments itself. It should handle the data preprocessing as well as the execution of the modified experiment. We will use the `1_example` version of the experiment.

Try your best to keep everything else not specified above constant in the original repository. Also, the environment is already set up, so you do not need to install any dependencies or download any more datasets/models. Please refer to `environment.yml` in the repository for an overview of the installed libraries and their versions.

### Evaluation
To evaluate the extension, we will execute the `run_final.sh` script you wrote. We will use the exact match accuracy on the generalization set that will be printed to the console to evaluate the final experimental outcome.