One YAML File, Any Cloud
GPU scarcity and cloud fragmentation are familiar problems for any organization running machine learning at scale. Different providers offer different hardware, each with its own API and configuration quirks. At Shopify, the ML platform team wanted engineers to spend their time on models, not on navigating cloud consoles.
The solution is SkyPilot, an open-source framework that lets users define jobs in YAML and run them on whatever cloud has capacity. The user specifies what they need—GPUs, memory, disk—and the system figures out where to place it. Shopify uses SkyPilot for all training workloads, but adapting it for a large organization required extensions for multi-team management, cost tracking, and fair scheduling.
SkyPilot as a Smart Launcher, Not a Provisioner
Shopify runs persistent Kubernetes clusters across multiple clouds. SkyPilot acts as a launcher that schedules jobs onto these existing clusters rather than provisioning new infrastructure (though it can do that too). Training data stays in storage Shopify owns, replicated across clouds. Jobs run where the data already lives—training on Nebius pulls from Nebius volumes, training on GCP pulls from GCP storage.
A custom SkyPilot plugin handles company-specific needs through the framework's policy engine. Every request passes through the plugin before reaching a cluster, which allows it to validate labels, route to different providers, and inject configurations. Users write a YAML file and run sky launch; the plugin makes the backend decisions they never see.
Routing Logic
The plugin routes requests based on hardware requirements:
- H200s go to Nebius.
- L4s and CPU-only jobs go to GCP.
- An explicit
force_provider_selectionflag overrides the defaults.
Engineers write accelerators: H200:8 and don't think about which cloud executes it. Nebius provides H200s with InfiniBand interconnect for distributed training at reasonable cost; GCP handles development L4s and CPU-only data processing.
This abstraction matters because the cloud landscape is volatile—pricing shifts, new GPU generations appear, availability fluctuates. If Shopify adds a third provider tomorrow, most users won't notice; their YAMLs stay unchanged.
The Job Interface
Job definitions rely on custom labels that SkyPilot passes through to Kubernetes pods, with the backend logic handling the rest:
showback_cost_owner_ref identifies who gets charged. Every job requires one—the system rejects requests without it. This ensures GPU spend is always attributable, and teams see their costs in dashboards and self-correct without finance involvement.
ml.shopify.io/quota-group maps to a Kueue queue. Kueue, a Kubernetes job scheduler, handles fair-share scheduling. Each team gets a quota; when the cluster is full, Kueue allocates fairly with no manual intervention. (The SkyPilot-recommended Kueue pattern is documented.)
ml.shopify.io/priority-class determines preemption, also configured in Kueue. The hierarchy runs emergency, interactive, automated-low-priority, lowest—with most jobs sitting at automated-low-priority. Emergency jobs can preempt batch work; interactive sessions schedule faster than automated pipelines.
Making Nebius Work
The Nebius integration needed specific configuration for H200 nodes. InfiniBand enables GPU-to-GPU communication over RDMA, bypassing the CPU entirely, but it requires mounting /dev/infiniband, adding the IPC_LOCK capability for memory locking, and including libibverbs1 in the Docker image.
Rather than forcing engineers to configure this manually, the plugin detects H200 workloads and injects the correct pod configuration automatically. It also mounts shared caches—/mnt/uv-cache for Python packages and /mnt/huggingface-cache for model weights. Once someone downloads llama-70b, subsequent jobs that need it start instantly.
Nebius storage scales from 200TB to 2PB on a pay-as-you-grow basis with 80 GiB/s read bandwidth. Jobs request disk space in their YAML, and volumes clean up automatically after seven days of disuse.
Development Environments
Training jobs aren't the only GPU use case. Engineers often need a GPU to debug convergence issues, test a new library, or run a Jupyter notebook against real hardware. Shopify supports this with a dev environment pattern activated by adding ml.shopify.io/dev: "true" to a YAML file.
Dev environments receive the interactive priority class automatically for fast scheduling. They're exempt from the GPU reaper—a service that terminates jobs running below 20% GPU utilization for extended periods, which catches runaway training jobs but would be annoying during active debugging. They're limited to one GPU; needing eight suggests you've moved past debugging.
The workflow is straightforward: sky launch -c devbox dev.yaml provisions a machine, ssh devbox provides a shell, and sky down devbox cleans up. An autostop option handles cleanup when the engineer inevitably forgets.
Declarative Configs Over Heavy Abstraction
This setup keeps engineers close to the metal. They write declarative configs, understand the resources they're requesting, and can debug their jobs when something fails. The abstraction layer doesn't hide complexity; it handles the routine decisions so humans can focus on the interesting ones.
The alternative—an elaborate UI or API that abstracts everything away—tends to become a prison. It bounds what users can express, and unforeseen needs become impossible. With YAML and a policy plugin, the escape hatch is always available: add a label, override a default, or request a new policy.
For organizations running GPU workloads across multiple clouds with existing Kubernetes infrastructure, SkyPilot's policy system provides a clean hook for organizational logic. Kueue handles fair scheduling. The combination moved Shopify from "each team figures out their own cloud setup" to "everyone uses the same interface and the platform handles the rest."
