Dream Diary: Case Study on Diffusion LLM’s Arithmetic Behavior

Chiung-Yi Tseng, Maisha thasin, Blessing Effiong, Somshubhra Roy, Danyang Zhang

Keywords: Diffusion, LLMs, mechanistic interpretability, math AI, Dream, causal inference, machine learning, neural network, large language models

Abstract:

Mechanistic interpretability studies of autoregressive (AR) models are abundant, while studies on diffusion models
(DLLM) remain less explored. In this study, we investigate the arithmetic behaviors of Dream-v0-Instruct-7B
(Dream). Future work includes causal study of DLLM to isolate the arithmetic neurons, particularly approximation
operations, extending the evaluation to larger benchmarks to gain statistical significance and providing mechanistic
interpretability study tools to the community.

Submission Number: 216