SAFT: Structure-Aware Fine-Tuning of Large Language Models for AMR-to-Text Generation

Abstract: Large Language Models (LLMs) are increasingly applied to tasks involving structured inputs such as semantic graphs, yet adapting them to such inputs remains non-trivial. Common approaches either linearize graphs, discarding structural information, or rely on specialized architectures that are not directly compatible with standard pretrained LLMs. We present SAFT, a structure-aware fine-tuning method that augments LLMs with graph positional encodings derived from the magnetic Laplacian of the input graph. These encodings are projected into the LLM embedding space, introducing relational inductive bias without modifying the model architecture. While SAFT is conceptually applicable to any task involving directed graph inputs with node–token alignment, we focus on the task of generating natural language text from an input AMR (Abstract Meaning Representation) graph, a directed graph encoding predicate-argument semantics of natural language sentences. AMR-to-text generation requires models to integrate both linguistic fluency and structural faithfulness, making it a demanding evaluation setting. We show that SAFT consistently improves or matches standard fine-tuning across all tested model families and scales, with gains that increase with graph structural complexity, both on sentence-level graphs of increasing depth and on document-level graphs of increasing size, demonstrating that structural encoding provides a reliable and scalable inductive bias for LLM fine-tuning.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Greg_Durrett1

Submission Number: 7866