datajuicer/data-juicer

Project overview

data-juicer treats data processing as composable infrastructure that scales from a laptop to thousand-node clusters without requiring custom scaling code. Its YAML-driven pipelines are versionable, reproducible, and cover the full range of foundation-model and RAG data preparation tasks.

Project type

Workflow · Data Processing

Use cases

Knowledge Q&A · Data Analysis

Deployment

Refer to project documentation

License

Apache-2.0

Best for

• AI engineers and developers who need a composable, YAML-driven system for curating large-scale multimodal datasets across pre-training, fine-tuning, RL, evaluation, and RAG workloads.

Key capabilities

• More than 200 operators for processing text, image, audio, video, and multimodal data.
• Users can version, share, fork, and compose reproducible YAML processing pipelines.
• Supports data curation for pre-training, fine-tuning, reinforcement learning, and evaluation workloads.
• Provides extraction, normalization, semantic chunking, deduplication, and profiling for RAG data workflows.
• Supports large-scale processing on Ray nodes and includes Ray repartitioning functionality.
• Operators can be iterated on without restarting pipelines.

Getting started

• Install the package using uv pip install py-data-juicer, then run a demo processing job with the command dj-process --config demos/process_simple/process.yaml.

Evidence and sources

• README: from data_juicer.core.data import NestedDataset
• README: Data-Juicer (DJ) transforms raw data chaos into AI-ready intelligence. It treats data processing as *composable infrastructure*—providing modular building blocks to clean, synthes…
• README: - **200+ operators** spanning text, image, audio, video, and multimodal data
• README: - **Scale**: Process 70B samples in 2h on 50 Ray nodes (6400 cores)
• README: - **Recipe-first**: Reproducible YAML pipelines you can version, share, and fork like code

Reviewed at: 2026-07-16T13:07:42Z