Resource-Efficient Learning for the Web

Chuxu Zhang, Kaize Ding, Jundong Li, Dongkuan Xu, Haoyu Wang, Derek Zhiyuan Cheng, Huan Liu

Abstract: Deep learning techniques have demonstrated impressive effectiveness across a wide array of web applications. Notably, graph neural networks (GNNs) and large language models (LLMs) have become essential tools for modeling the extensive graph-structured data and text/language data that populate the web. Despite their success, the advancement of these methods is frequently hampered by resource constraints. Key challenges include the scarcity of labeled data (data-level constraints) and the demand for smaller model sizes suitable for real-world computing environments (model-level constraints). Addressing these issues is crucial for the effective and efficient deployment of models across various real-world web systems and applications, such as social networks, search engines, recommender systems, question answering, and content analysis. Therefore, there is an urgent need to develop innovative and efficient learning techniques that can overcome these resource limitations from both data and model perspectives.In this lecture-style tutorial, we will focus on state-of-the-art approaches in resource-efficient learning, specifically exploring a range of data- and model-efficient methods for GNNs and LLMs, along with their practical applications in web contexts. Our objectives for this tutorial are threefold: (1)to categorize challenges in resource-efficient learning and discuss data and model constraints; (2) to provide a comprehensive review of existing methods and recent advances in resource-efficient learning, particularly concerning GNNs and LLMs; and (3) to highlight open questions and potential future research directions in this rapidly evolving field. Together, these objectives will provide participants with a comprehensive understanding of resource-efficient learning for GNNs and LLMs, its challenges, and its potential for future advancements.