Capturing natural position relationships: A neural differential equation approach

Open Webpage

Changqing Ji, Liyong Wang, Jing Qin, Xin Kang, Zumin Wang

Published: 2024, Last Modified: 07 Nov 2025Pattern Recognit. Lett. 2024EveryoneRevisionsBibTeXCC BY-SA 4.0
Abstract: Highlights•Novel ODE method for implicit local position encoding in Transformers.•Capture natural sequence position without extra embeddings.•Recurrent attention network captures longer dependencies.•Our method is adaptable, applicable in long-term language modeling.