RayOrch: Programming and Executing Lineage-Controlled Multi-Grain Dataflows for Foundation-Model Data Preparation

Abstract

Preparing high-quality training data for foundation models requires scalable data pipelines that transform heterogeneous documents and videos into structured records. RayOrch introduces a programming model and distributed execution engine for lineage-controlled, multi-grain dataflows, preserving parent relationships, order, completion state, and result routing across input-dependent expansions. Its compiler validates dataflow programs, while the runtime tracks child membership, immediate parents, ordinals, and terminal states, with per-call FIFO ready queues and parent-scoped failure handling. On H20 GPUs, RayOrch achieves up to 15.14x speedup for MinerU workloads from 4 to 64 GPUs and 7.82x for video workloads from 8 to 64 GPUs, reducing end-to-end time compared with Ray Data, Daft, and Docling.

Xiaochen Ma

Ph.D. Student in Computer Science and Engineering

I work on data infrastructure and high-performance distributed systems for large-scale LLM data preparation, with a focus on Ray and Apache Spark.