TableVault: Managing Dynamic Data Collections for LLM-Augmented Workflows

Jinjin Zhao, Sanjay Krishnan

Keywords: data lineage, Systems for LLM, AI agents, Systems for RAG

Abstract: Large Language Models (LLMs) have emerged as powerful tools for automating and executing complex data tasks. However, their integration into more complex data workflows introduces significant management challenges. In response, we present TableVault - a data management system designed to handle dynamic data collections in LLM-augmented environments. TableVault meets the demands of these workflows by supporting concurrent execution, ensuring reproducibility, maintaining robust data versioning, and enabling composable workflow design. By merging established database methodologies with emerging LLM-driven requirements, TableVault offers a transparent platform that efficiently manages both structured data and associated data artifacts.

Submission Number: 6