% DO NOT EDIT - automatically generated from metadata.yaml

\def \codeURL{https://github.com/skandermoalla/bet-reproduction}
\def \codeDOI{10.5281/zenodo.7937169}
\def \codeSWH{swh:1:dir:4a562f75c0fd44672b806498e18b67690a5baabd}
\def \dataURL{https://github.com/skandermoalla/bet-reproduction}
\def \dataDOI{10.5281/zenodo.7937169}
\def \editorNAME{}
\def \editorORCID{}
\def \reviewerINAME{}
\def \reviewerIORCID{}
\def \reviewerIINAME{}
\def \reviewerIIORCID{}
\def \dateRECEIVED{}
\def \dateACCEPTED{}
\def \datePUBLISHED{}
\def \articleTITLE{[Re] Reproducibility Study of Behavior Transformers}
\def \articleTYPE{Replication}
\def \articleDOMAIN{ML Reproducibility Challenge 2022}
\def \articleBIBLIOGRAPHY{bibliography.bib}
\def \articleYEAR{2023}
\def \reviewURL{https://openreview.net/forum?id=E0qO5dI5aEn}
\def \articleABSTRACT{Scope of Reproducibility - In this work, we analyze the reproducibility of 'Behavior Transformers: Cloning $k$ modes with one stone'. In assessing the Behavior Transformer (BeT) model, we analyze its ability to generate performant and diverse rollouts when trained on data containing multi-modal behaviors, the relevance of each of its components, and its sensitivity to critical hyperparameters.
Methodology - We use the open-source PyTorch implementation released by the authors to train and sample rollouts for BeT. However, the implementation does not include all the environments, evaluation metrics, or ablations studied in the paper. Consequently, we extend it by following the details in the paper and filling in the missing parts to have a complete pipeline and support all the experiments performed in this report. We conducted our experiments on an NVIDIA GeForce GTX 780 GPU, requiring 276 GPU hours to train our models.
Results - Running the code released by the authors does not produce an evaluation of BeT according to the metrics reported in the paper. After extending the implementation with the proper evaluation metrics, we obtain results that support the main claims of the paper in a significant subset of the experiments but that also diverge in many of the actual values obtained. Therefore, we conclude that the paper is largely replicable but not readily reproducible.
What was easy - It was easy to identify the main claims of the paper and the experiments supporting them. Moreover, thanks to the open-source implementation released by the authors, training the model and sampling rollouts were straightforward tasks.
What was difficult - Setting up the development environment was hard due to dependencies not being pinned. Not having the code for evaluation metrics available hindered our efforts to achieve similar numbers. Assessing the sources of discrepancies in our numbers was also difficult, as training curves and model weights were not accessible.
Communication with original authors - We communicated via email with the authors throughout the project. They provided clarifications and resources that helped us with our study. However, the communication was insufficient to reach a complete reproduction.}
\def \replicationCITE{Shafiullah, Nur Muhammad, et al. "Behavior transformers: Cloning \$ k \$ modes with one stone." Advances in neural information processing systems 35 (2022): 22955-22968.}
\def \replicationBIB{shafiullah2022behavior}
\def \replicationURL{https://proceedings.neurips.cc/paper_files/paper/2022/file/90d17e882adbdda42349db6f50123817-Paper-Conference.pdf}
\def \replicationDOI{10.48550/arXiv.2206.11251}
\def \contactNAME{Skander Moalla}
\def \contactEMAIL{skander.moalla@epfl.ch}
\def \articleKEYWORDS{rescience c, machine learning, reinforcement learning, imitation learning, behavior cloning, transformer, pytorch, python}
\def \journalNAME{ReScience C}
\def \journalVOLUME{9}
\def \journalISSUE{2}
\def \articleNUMBER{1}
\def \articleDOI{}
\def \authorsFULL{Skander Moalla et al.}
\def \authorsABBRV{S. Moalla et al.}
\def \authorsSHORT{Moalla et al.}
\title{\articleTITLE}
\date{}
\author[1,3,\orcid{0000-0002-8494-8071}]{Skander Moalla}
\author[1,3,\orcid{0000-0002-8205-404X}]{Manuel Madeira}
\author[1,3,\orcid{0000-0002-7999-7282}]{Lorenzo Riccio}
\author[2,3,\orcid{0000-0002-2853-9433}]{Joonhyung Lee}
\affil[1]{EPFL, Switzerland}
\affil[2]{Independent Researcher}
\affil[3]{Equal contributions}
