Transforming Genomic Interpretability: A DNABERT Case Study

Micaela Elisa Consens, Nicolas Papernot, Alan M Moses, BO WANG

Abstract:

While deep learning algorithms, particularly transformers, have recently shown significant promise
in making predictions from biological sequences,
their interpretability in the context of biology has
not been deeply explored. This paper focuses
on the recently proposed DNABERT model and
explores interpreting it’s decisions using modified Layer-wise Relevance Propagation (LRP)
methods to determine what the model is learning. This score is then compared to several other
interpretability methods commonly applied to
transformers, including the attention-score based
method proposed by the DNABERT authors. Results of mutagenesis experiments targeting regions identified by different methods show the
modified LRP interpretability scores can outperform others at 20 mutations, and also show attention cannot reliably outperform random scores.