Don't Rewrite My Text: Sidetrack Decoding for Faithful Document Annotation

ACL ARR 2026 January Submission10206 Authors

Keywords: constrained decoding, text preservation, span tagging, structured output, verifiable generation

Abstract:

When prompted to annotate some text, Large Language Models (LLMs) may introduce subtle modifications that break the alignment between generated and source text, and cannot be reliably used for structured generation tasks.
We propose sidetrack decoding, a constrained decoding algorithm that guarantees verbatim text preservation during text annotation.
By inspecting the model's next-token probabilities over the original text, the algorithm determines when to "sidetrack" into generation mode to insert markup tokens, and when to merge back to the main track to copy the original text.
We evaluate on three extraction benchmarks and find that sidetrack decoding achieves 100% text preservation while maintaining competitive extraction accuracy on tasks suited for span tagging.
A speculative execution variant further improves efficiency, achieving 3.3x throughput compared to standard autoregressive decoding.

Paper Type: Short

Research Area: Natural Language Generation

Research Area Keywords: inference methods, text-to-text generation, efficient models

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 10206