Constellation expands: bigger models, caching, and XGBoost

Cloudflare announced Constellation during 2023 Developer Week as a set of APIs for running low-latency inference with pre-trained ML/AI models directly on its network. Roughly a month into the private beta, the platform is getting three upgrades based on customer feedback.

Model size limit raised to 50 MB. The previous 10 MB cap was restrictive for many pre-trained and optimized models. The new limit opens up a wider range of options while remaining conservative for a beta.

Input tensor caching. Repeated inference tasks send the same tensor objects over the wire even when only minimal parts of the input change, creating unnecessary network and parsing overhead. The client API now includes support for caching input tensors to cut latency and speed up inference.

XGBoost runtime. Constellation launched with the ONNX runtime, but the plan has always been to support multiple runtimes under one API. XGBoost, an optimized distributed gradient boosting library known for strong performance on structured and tabular data, is now available. XGBoost models can be uploaded and used today.

Updated documentation covers the new features and includes an example of using the XGBoost runtime with Constellation.

Machine learning at the edge

Constellation represents Cloudflare's first public release of globally distributed machine learning, but running ML across the network isn't new internally. Cloudflare has long used machine learning to handle core functions across its network in over 300 cities — classifying traffic spikes, routing requests, distinguishing bots from humans, and identifying zero-day threats without human intervention. Constellation opens that capability to developers.

The platform is still expanding: hundreds of new users join the beta daily, and Cloudflare continues to deploy new models and hardware to support AI workloads on the network.

Where low-latency inference matters

Speed is only part of the equation for web experiences. Personalization — knowing what a user wants before they ask — also requires accuracy. For e-commerce already running on Workers, Pages, or D1, Constellation can add tasks like customer categorization and product recommendations directly into the existing application stack.

Latency is even more critical in physical-world applications. For a face recognition doorbell deciding whether to unlock, every millisecond counts. Inference on Cloudflare's network puts compute within 50ms of 95% of the world's population, whereas centralized cloud regions can be hundreds of milliseconds away from users on other continents.

On-device inference sounds like an alternative, but it has real drawbacks:

  • Insufficient compute: Most devices run on microcontrollers, not high-end GPUs or server CPUs. Complex, high-volume workloads can turn milliseconds into seconds on local hardware.
  • Battery drain: Longer compute means worse battery life, even with ASICs or TPUs. That's a maintenance problem for a doorbell battery, and a costly one for fleets of devices at scale.
  • Hard to update: Upgrading hardware or pushing software across thousands of distributed devices is painful compared to cloud deployments that ship updates multiple times a day.

Adding AI to Workers with a few commands

The same developer experience that made Workers popular for shipping code now extends to machine learning. Constellation brings AI to Cloudflare's developer platform, making it possible to attach models to existing applications quickly. Access is available through the new Constellation tab in the Cloudflare dashboard, with a tutorial covering the first Constellation Worker. Cloudflare's roadmap already includes further expansion of the platform's AI capabilities.