Go to ACL ARR 2025 February homepageFollow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents
Flowcharts are a critical tool for visualizing decision-making processes, yet their nonlinear structure and complex visual-textual relationships make them challenging to interpret automatically. Vision language models frequently hallucinate non-existent connections and decision paths when analyzing these diagrams, compromising the reliability of automated flowchart processing. We introduce the task of Fine-grained Flowchart Attribution, a post-hoc strategy to mitigate visual flowchart hallucination by tracing specific components grounding a flowchart referring statement. Flowchart Attribution ensures the verifiability of model predictions and enhances explainability by linking generated responses to the flowchart’s structure. We introduce FlowExplainBench, a novel benchmark for evaluating flowchart attribution, encompassing diverse styles, domains, and question types while incorporating high-quality attribution annotations. We introduce FlowPathAgent, a neurosymbolic agent that performs fine-grained attribution through graph-based reasoning. It first segments the flowchart, then converts it into a structured symbolic graph, and then employs an agentic approach to dynamically interact with the graph to generate attribution paths. Experimental results show significant performance improvements, with FlowPathAgent outperforming existing methods by 6% to 65% on FlowExplainBench.