Not video-optimized: frames were stored independently with no inter-frame compression, inheriting the same cost as that JPEG baseline.

No native visualization: inspecting an episode meant converting it to a format some viewer could directly read.

H.264 encoding with larger-GOP: group-of-pictures provides a typical tradeoff between compression and random access, because seeking to an arbitrary frame requires decoding forward from the preceding keyframe. VLA training samples short, sparse windows, so it pays that seek penalty on every sample. World-action models read long contiguous sequences, which amortize one keyframe over many frames, so we can afford larger GOPs.

Topic-group chunking: MCAP's default writer gives each topic its own chunks, so assembling one sample costs a read per topic. We instead group topics that share a read pattern and write each group time-major: cameras interleaved with each other in one stream, proprioception and actions in another. The two never share a chunk, because a sample takes a few decoded frames but a long dense window of state. A fetch then costs one read per group rather than one per topic, so adding a camera or a state topic no longer adds a round trip.

Separate scalability: Each DAG step is dynamically allocated resources tailored to its specific requirements (CPU, memory, or GPU), maximizing physical resource utilization, without inflating the whole worker pool to the worst-case step's footprint.

Better decoupling and observability: steps are tagged critical or non-critical, so a non-critical failure no longer cancels the whole DAG run. The status of each step is clearly observed from the DAG and resumed from there.

Dynamic orchestration: Processing needs change constantly, especially once we began ingesting from external vendors, each with its own data types, formats, and quality quirks. Any subset of steps can be toggled on or off per run. Quality checks, for example, run as their own pre/post gate around transformation rather than being folded into it. This is controlled at runtime, not through a code change or a new script.

We first staggered the start times of different batches so identical steps no longer finished in lockstep, smoothing the write bursts that overwhelmed the scheduler DB.

We also introduced a joint optimizer that uses bin-packing to split input batches into near-equal bytes. These splits are further broken down into chunks, processed in parallel by an optimal number of airflow workers. This approach ensures we stay within physical constraints, such as network and I/O throughput, node, storage, and database capacity.

Production DB — the transactional-heavy system of record, enforcing data integrity and completeness.

Data warehouse — the analytical backend, which enables horizontal scalability. A curation reads a few columns across the whole table rather than every row end to end. The episodes table has since passed 50 million rows with no change to how manifests are built.

Memory: a GPU node typically has around 2 TB of CPU RAM, and loading a one-million-hour manifest across all ranks exceeds the RAM ceiling and crashes the training job without finishing a single step.

Load time: the manifest sits on a network-backed mount where each read becomes its own request, and a columnar format's footer-first, scattered-seek access pattern is close to worst case for that, so a cold read costs far more than pulling the same bytes sequentially.

Download once, not per rank: a single rank pulls the manifest with a direct, parallel transfer from the object storage API onto local node disk, bypassing the network mount entirely.

Map instead of read: every rank memory-maps that local copy as a columnar table, so the file stays on disk and only the pages actually touched become resident.

Shard on load: each rank takes a zero-copy slice covering its own 1/N of the rows, where N is the number of ranks. That only means anything on a mapped table. On an in-memory read you have already materialized the whole table before you can cut it, which is why the two ship as a single switch rather than independent options.

The corpus is not static. Ingestion keeps extending it and corrections land after the fact, so every copy is one more thing to keep in sync.

Every copy costs. Its own storage bill, plus the egress to fill it, multiplied by each cluster we run on.

Not every vendor sells object storage. Most offer a parallel file system like Weka or VAST, a different product at a different price tier.

It is natively distributed. Compared to traditional distributed cache solutions, Alluxio removes the single point of failure on the control plane, and it fits multinode large-scale training very well.

It caches pages, not whole files. A worker holds only the parts of an episode that were actually read, so the same NVMe covers far more distinct episodes, and eviction drops a page at a time instead of a whole multi-gigabyte file. This is where the chunking work earlier pays off again: grouping topics so a sample touches few chunks means few pages go resident per sample.

Ownership is spread by hash. Each file's pages live on one worker, chosen by a consistent hash of its path, so with tens of millions of files the read traffic spreads evenly across the cluster instead of piling onto whichever node happens to hold a popular episode.

Shard inside the node. Ranks split the work over NVLink using per-node NCCL sub-groups, so broadcast traffic never leaves the node.

Let nodes duplicate the work. Each node recomputes the same updates instead of exchanging them: more arithmetic, but no fabric traffic, and volume stops depending on node count.

Pick the strategy from the scale. At small node counts world-sharding is still faster, so the optimizer chooses by node count rather than committing to one.

Degraded GPUs that report healthy: the node passes every obvious check and jobs keep dying on it anyway. One of ours had logged over 250,000 corrected ECC errors before we connected the crashes to the hardware.

Drained nodes nobody resumes: one flaky driver query is enough to drain a healthy machine, and nothing pages you, because a drained node looks like a normal state. One sat idle about ten hours before we noticed we were short.

Stale GPU reservations: when a controller reboots mid-job, nodes come back idle but still holding their old reservations. Everything reports healthy, but the scheduler won't put work on them.

Not enough GPUs: a million-hour pre-training run would happily take the whole fleet for weeks, but everyone else still needs to train and evaluate while it runs. Supply is tight enough across the industry that no single vendor covers both, so we take capacity wherever it is free, which only works if we can stand a cluster up quickly.

Preflight checks: Before a job lands on a node, Slurm checks its GPU inventory, error counters, kernel log, local disk, and container runtime, and drains anything that fails. The tricky part was choosing which error counter to trust. Lifetime totals can't be cleared on some GPUs, so one bad stretch long ago would condemn a healthy node forever. We gate on the counters that reset at boot instead.

Auto-restarts: The job restarts its workers in place, and if the whole job dies Slurm requeues it and training resumes from the last checkpoint. Diagnostic traces are keyed to the attempt number so earlier ones survive, because otherwise you lose the evidence for why it died. Nodes drained on a transient come back on their own, and reservations stranded by a crashed controller are swept up on a timer.

One-click deployments: Cluster setup is a single Ansible playbook, with each cluster's differences kept in its own inventory. The clusters aren't alike: some vendors hand over machines with the scheduler and container stack already installed, others give us an image and nothing else. The same roles cover both, with the install steps switched off where the vendor already did the work. Standing up a new cluster is a config change now, and it takes days.