Abstract: Highlights•Proposed Detect-Order-Construct for hierarchical document structure analysis.•Devised end-to-end solution by casting tasks as relation prediction problems.•Designed multi-modal relation models with structure-aware improvements.•Established Comp-HRDoc for hierarchical document structure analysis evaluation.•Achieved state-of-the-art on PubLayNet, DocLayNet, HRDoc, and Comp-HRDoc.