Workflow

Designing Graphs That Split Across Local and Cloud Nodes

ยท RenderBob team

A workflow that can only run as one monolithic job on one machine wastes the biggest advantage of a hybrid pipeline: the ability to spread work across many nodes at once.

A node graph sends light stages through local hardware and heavy stages across cloud nodes before merging.

A workflow that can only run as one monolithic job on one machine wastes the biggest advantage of a hybrid pipeline: the ability to spread work across many nodes at once. Design graphs to be splittable on purpose.

Separate the stages that spike memory

The single most useful split is between the sampling pass and the upscale pass. These two spike VRAM at different times, and forcing them to coexist is a common OOM cause. Save latents after sampling; run upscale as its own step. Now each stage can run on hardware sized for it, and the upscale, often the heaviest, can go to a bigger node.

Make frame ranges independent

For video, the biggest parallelism win is splitting a clip into frame ranges that render independently and stitch afterward. A nine-hour sequential render becomes forty minutes across several nodes, as covered in the hybrid case study. The workflow has to be authored so that frame ranges don't depend on shared mutable state: deterministic seeds and pinned settings per range.

Isolate the deterministic parts

Anything that must be byte-identical across nodes (model versions, precision, seeds) must be pinned and identical on every node the job touches. A split job whose slices run different custom node versions will produce visible seams. This is where the shared registry and environment matching (posts #8 and #18) stop being optional and become correctness requirements.

Keep heavy inputs local to the node

Large model files and assets should be staged on each node ahead of time, not streamed mid-job. Pre-syncing the registry to cloud nodes before a burst avoids a deadline-time scramble for a missing checkpoint.

Design the merge step

However you split, stages or frame ranges, plan the recombination: where slices land, how they stitch, and where the final output is verified. A clean merge step is what makes parallelism invisible to the artist, who submitted one job and gets back one result.

Stop authoring workflows as a single linear pipe. Author them as a set of stages and ranges that a scheduler can distribute. You do not have to solve the scheduling yourself; that is the control plane's job. You do have to write graphs it can actually split. A workflow designed for distribution turns "we need this faster" from a hardware purchase into a scheduling decision.

More from the blog

All posts