LALO - A Virtual Data Lake Zone for Composing Tailor-Made Data Products on Demand

Christoph Stach, Yunxuan Li, Laura Schuiki, Bernhard Mitschang

Abstract: The emerging paradigm of data products, which has become increasingly popular recently due to the rise of data meshes and data marketplaces, also poses unprecedented challenges for data management. Current data architectures, namely data warehouses and data lakes, are not able to meet these challenges adequately. In particular, these architectures are not designed for a just-in-time provision of highly customized data products tailored perfectly to the needs of customers. In this paper, we therefore present a virtual data lake zone for composing tailor-made data products on demand, called LALO. LALO uses data streaming technologies to enable just-in-time composing of data products without allocating storage space in the data architecture permanently. In order to enable customers to tailor data products to their needs, LALO uses a novel mechanism that enables live adaptation of data streams. Evaluation results show that the overhead for such an adaptation is negligible. Therefore, LALO represents an efficient solution for the appropriate handling of data products, both in terms of storage space and runtime.