Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models

Daoyuan Chen, Yilun Huang, Xuchen Pan, Jiang Nana, Haibin Wang, Yilei Zhang, Ce Ge, Yushuo Chen, Wenhao Zhang, Zhijian Ma, Jun Huang, Wei Lin, Yaliang Li, Bolin Ding, Jingren Zhou

Keywords: data processing, foundation model, scalable infrastructure

TL;DR: A scalable system for foundation model data processing, offering 150+ multimodal OPs, cloud-native efficiency (TB-scale on 10k+ cores), and diverse interfaces (Python/APIs/chat), widely adopted in research and industry (e.g., Alibaba Cloud).

Abstract:

Foundation models demand advanced data processing for their vast, multimodal datasets.
However, traditional frameworks struggle with the unique complexities of multimodal data.
In response, we present Data-Juicer 2.0, a data processing system backed by 100+ data processing operators spanning text, image, video, and audio modalities, supporting more critical tasks including data analysis, synthesis, annotation, and foundation model post-training.
With seamless compatibility and dedicated optimization for popular dataset hubs like Hugging Face and computing engines like Ray, it improves upon its predecessor in terms of usability, efficiency, and programmability.
It features an easily accessible user interface layer that supports decoupled Python interactions, RESTful APIs, and conversational commands.
Its new runtime layer offers adaptive execution across diverse scales and environments, abstracting away system complexities.
Extensive empirical evaluations demonstrate Data-Juicer 2.0's remarkable performance and scalability, highlighting its capability to efficiently process TB-level data with 10k+ CPU cores.
The system is publicly available and has been widely adopted in diverse research fields and real-world products such as Alibaba Cloud PAI.
We actively maintain the system and share practical insights to foster research and applications of next-generation foundation models.

Code URL: https://github.com/modelscope/data-juicer

Primary Area: AL/ML data processing and benchmarking Infrastructure (e.g., metrics libraries, visualization libraries, data exploration libraries, distributed data processing solutions, scalable data analysis)

Submission Number: 2624