Heritage Building Information Management by Multimodal LLM and Knowledge Graph: Self-documentation, Database Generation and Intelligent Querying

JIAYING ZHANG, Chak Fu CHAN, Ziyu ZHAO, Jack C.P. Cheng

Abstract:

Heritage buildings face challenges in documentation due to inconsistent records and
complex data from historical documents, archaeological surveys, and materials.
Traditionally, converting unstructured data into structured formats required significant
expert effort. The advent of large language models (LLMs) has transformed heritage
research by enabling the creation and maintenance of knowledge graphs. These graphs
integrate diverse data sources, facilitating the preservation and study of heritage
buildings. LLMs help extract and organize unstructured data, improving knowledge
graph accuracy and consistency. This research proposes a comprehensive framework
that integrates multimodal data, including text, images, and videos, into a unified
knowledge graph. The framework employs LLMs for extracting information from
textual data, the CLIP model for aligning images with corresponding text, and keyword
searches for processing video content. The resulting knowledge graph is stored in a
Neo4j graph database, providing an interactive platform for users to query and explore
detailed information about heritage buildings. This approach not only supports
academic research but also contributes to practical applications in cultural heritage
conservation, enabling more efficient access to valuable information and enhancing
preservation efforts. The proposed method was validated in European 'Gothic' and
'Gothic Revival' architecture by comparing the relationships between components.