Hypergraph-based Zero-shot Multi-modal Product Attribute Value Extraction

Jiazhen Hu, Jiaying Gong, Hongda Shen, Hoda Eldardiry

Track: Semantics and knowledge

Keywords: attribute value extraction, multi-modal learning, zero-shot learning, heterogeneous hypergraph

TL;DR: We propose a hypergraph-based method for multi-modal, multi-label product attribute value extraction in the zero-shot setting.

Abstract: It is essential for e-commerce platforms to provide accurate, complete, and timely product attribute values, in order to improve the search and recommendation experience for both customers and sellers. In the real-world scenario, it is difficult for these platforms to identify attribute values for the newly introduced products given no similar product history records for training or retrieval. Besides, how to jointly learn the product representation given various product information in multiple modalities, such as textual modality (e.g., product titles and descriptions) and visual modality (e.g., product images), is also a challenging task. To address these limitations, we propose a novel method for extracting multi-label product attribute-value pairs from multiple modalities in the zero-shot scenario, where labeled data is absent during training. Specifically, our method constructs heterogeneous hypergraphs, where product information from different modalities is represented by different types of nodes, and the text and image nodes are embedded and learned through CLIP encoders to effectively capture and integrate multimodal product information. Then, the complex interrelations among these nodes are modeled through the hyperedges. By learning informative node representations, our method can accurately predict links between unseen product nodes and attribute-value nodes, enabling zero-shot attribute value extraction. We conduct extensive experiments and ablation studies on several categories of the public MAVE dataset and the results demonstrate that our proposed method significantly outperforms several state-of-the-art generative model baselines in multi-label, multi-modal product attribute value extraction in the zero-shot setting.

Submission Number: 2223