Hydra: A Modular Architecture for Efficient Long-Context Reasoning

Siddharth Chaudhary, Dev Patel, Maheep Chaudhary, Bennett Browning

Keywords: reasoning, transformers, SGA, SSM, PKM, MoE

TL;DR: Hydra is a modular architecture combining an SSM backbone with sparse attention, MoE, and memory , outperforming a standard Transformer in both speed (3x) and reasoning accuracy (10x) on long-context tasks

Abstract: The quadratic complexity of transformers fundamentally limits reasoning system deployment in resource-constrained and long-context settings. We introduce Hydra, a modular architecture based upon a state-space backbone which adaptively routes between complementary efficiency mechanisms: sparse global attention, mixture-of-experts, and dual memories comprising a reasoning workspace and product key memory. We evaluate a 29M parameter model measuring logical chaining accuracy and throughput on synthetic sequences, plus throughput on WikiText. Ablation studies use component-specific synthetic datasets to isolate individual mechanisms. Hydra achieves $3.01\times$ and $3.0\times$ throughput gains at 8K tokens for synthetic and WikiText datasets, respectively, and $10\times$ accuracy improvements on multi-step logical composition compared to equal-sized transformers. Ablations confirm each component's contribution: sparse attention captures long-range dependencies, experts specialize to input domains, and product key memory enables selective retrieval.

Submission Number: 242