CoPE: Enhanced Transformer with Complex Positional Encoding

Avinash Amballa

Keywords: Positional encoding, ROPE, Transformer, GLUE

TL;DR: We propose CoPE, a lightweight complex positional encoding combining semantic (real) and positional (imag) info. With phase-aware attention, CoPE avoids long-term decay and surpasses RoPE, sinusoidal, and learned encodings on GLUE at lower cost."

Abstract: Recent studies have demonstrated the effectiveness of position encoding in transformer architectures. By incorporating positional information, this approach provides essential guidance for modeling dependencies between elements across different sequence positions. We introduce CoPE (Complex Positional Encoding), a novel architecture that leverages complex-valued encoding to encode both content and positional information. Our approach replaces traditional positional encodings with complex embeddings where the real part captures semantic content and the imaginary part encodes positional information. We introduce phase-aware attention in the first layer of the transformer model to capture position-dependent patterns, followed by standard attention layers for higher-levels. We show that CoPE doesn't exhibit long term decay and is compatible with linear attention. Experimental evaluation on the GLUE benchmark suggest that our approach achieves competitive performance with less computational complexity, compared to RoPE, Sinusoidal and Learned positional encodings. Code available at https://github.com/AmballaAvinash/cope

Submission Number: 39