Cloudflare R2 and MosaicML enable training LLMs on any compute, anywhere in the world, with zero switching costs

Training LLMs Without the Lock-In: R2 and MosaicML Take on Egress Fees

Training large language models and diffusion models demands more than just racks of GPUs. The datasets feeding those models can run into the petabytes, and checkpoints—snapshots of model state saved throughout a run—regularly hit hundreds of gigabytes each. Getting that data in and out of compute clusters quickly and reliably is a storage infrastructure problem as much as a compute problem.

Object storage has become the default home for these workloads, but most providers tie users to their platforms with egress fees. Moving data out to take advantage of cheaper or more available GPUs elsewhere becomes prohibitively expensive. With GPU scarcity driving teams toward multi-cloud strategies, that pricing model is a growing obstacle.

MosaicML and Cloudflare have teamed up to remove both the financial and technical barriers. MosaicML's open-source StreamingDataset and Composer libraries handle the heavy lifting of streaming data and checkpoints, while Cloudflare R2's zero-egress pricing makes moving between compute providers cost-neutral. The result: train on any cloud, pause, and resume elsewhere without paying data transfer penalties.

“With the MosaicML training platform, customers can efficiently use R2 as the durable storage backend for training LLMs on any compute provider with zero egress fees. AI companies are facing outrageous cloud costs, and they are on the hunt for the tools that can provide them with the speed and flexibility to train their best model at the best price.” – Naveen Rao, CEO and co-founder, MosaicML

Streaming Data Straight from R2

MosaicML's StreamingDataset library is built to pull training data from object storage without bottlenecking the GPU pipeline. The workflow starts by converting raw training data—images, text, video, or anything else—into .mds shard files via a provided Python API:

Once the dataset is sharded, upload it to an R2 bucket. The awscli tool works, as do other S3-compatible clients; direct cloud writes to R2 from StreamingDataset are also on the roadmap.

import numpy as np
from PIL import Image
from streaming import MDSWriter

# Local or remote directory in which to store the compressed output files
data_dir = 'path-to-dataset'

# A dictionary mapping input fields to their data types
columns = {
    'image': 'jpeg',
    'class': 'int'
}

# Shard compression, if any
compression = 'zstd'

# Save the samples as shards using MDSWriter
with MDSWriter(out=data_dir, columns=columns, compression=compression) as out:
    for i in range(10000):
        sample = {
            'image': Image.fromarray(np.random.randint(0, 256, (32, 32, 3), np.uint8)),
            'class': np.random.randint(10),
        }
        out.write(sample)

Any device with read access to the bucket can then consume the data—fetching individual samples, iterating over the full dataset, or feeding a standard PyTorch dataloader.

$ aws s3 cp --recursive path-to-dataset s3://my-bucket/folder --endpoint-url $S3_ENDPOINT_URL

The library ships with features essential for large-scale training: high throughput, elastic determinism, fast resumption after interruptions, and multi-worker support. It also implements smart shuffling and data distribution to minimize download bandwidth. Across workloads ranging from LLMs to diffusion models, MosaicML reports no training throughput degradation—no dataloader bottleneck—when training from object stores like R2.

Checkpoints Without a Shared Filesystem

Getting data into training solves half the problem. Saving model checkpoints back to durable storage is the other half, and Composer handles it by simply accepting an R2 path.

from torch.utils.data import DataLoader
from streaming import StreamingDataset

# Make sure that R2 credentials and $S3_ENDPOINT_URL are set in your environment    
# e.g. export S3_ENDPOINT_URL="https://[uid].r2.cloudflarestorage.com"

# Remote path where full dataset is persistently stored
remote = 's3://my-bucket/folder'

# Local working dir where dataset is cached during operation
local = '/tmp/path-to-dataset'

# Create streaming dataset
dataset = StreamingDataset(local=local, remote=remote, shuffle=True)

# Let's see what is in sample #1337...
sample = dataset[1337]
img = sample['image']
cls = sample['class']

# Create PyTorch DataLoader
dataloader = DataLoader(dataset)

Checkpoint uploads are asynchronous, minimizing training stalls, and the library works out of the box with multi-GPU and multi-node setups. Notably, it does not require a shared file system—no expensive EFS or NFS setup for the compute cluster. An Internet connection and credentials are all that's needed to land checkpoints safely in R2, which can save thousands of dollars per month in public cloud storage overhead.

Training Across Three Clouds, One Run

Combining these tools with R2's pricing model enables a workflow that was previously impractical: running a single training job across multiple cloud providers, with no data movement costs. The MosaicML training platform orchestrates the run—managing compute clusters, secrets, and job submissions via the MCLI command-line tool.

from composer import Trainer
...

# Make sure that R2 credentials and $S3_ENDPOINT_URL are set in your environment
# e.g. export S3_ENDPOINT_URL="https://[uid].r2.cloudflarestorage.com"

trainer = Trainer(
        run_name='mpt-7b',
        model=model,
        train_dataloader=train_loader,
        ...
        save_folder=s3://my-bucket/mpt-7b/checkpoints,
        save_interval='1000ba',
        # load_path=s3://my-bucket/mpt-7b-prev/checkpoints/ep0-ba100-rank0.pt,
    )

In one demonstration, an LLM training job starts on Oracle Cloud Infrastructure with data streaming from R2. Partway through, the job is paused and resumed on different GPUs in AWS. Composer loads model weights from the last checkpoint in R2, the streaming dataloader picks up at the correct batch, and training continues deterministically. The job then moves again to Google Cloud to finish. Throughout the entire multi-cloud run, the only costs are GPU compute and storage—no egress fees, no lock-in.

For teams navigating today's volatile GPU market, the practical benefits are clear. R2 eliminates the data-transfer cost of switching providers, and MosaicML's tooling makes the technical switch seamless. Training runs can adapt to GPU availability and pricing in real time, and resizing jobs to different cluster sizes is no longer a storage logistics problem.