Guardrails in AI Gateway

Cloudflare has added safety guardrails to AI Gateway, its proxy service that sits between an AI application and its model providers (OpenAI, Anthropic, DeepSeek, and others). The goal is to give developers a consistent moderation layer regardless of which model or provider they use.

Why moderation is hard to get right

LLMs are non-deterministic, so outputs are unpredictable, and developers have no control over what users will ask. Shipping an AI app without visibility into harmful or inappropriate content risks both user safety and brand reputation. Two problems stand out in practice:

  • Inconsistent built-in safety: providers differ in philosophy, risk tolerance, fine-tuning and regulatory obligations, so built-in protections vary from model to model.
  • Limited visibility: without dedicated tooling, it is difficult to monitor prompts and responses and manage inappropriate content while testing different models.

The regulatory backdrop raises the stakes. The EU AI Act, enacted on August 1, 2024, includes provisions on risk management systems, data governance, technical documentation, and recording risks and abuse. The EU Digital Services Act, adopted in 2022, targets online safety and accountability, including illegal content and protection of minors. The OWASP Top 10 for LLM Applications catalogs the most critical vulnerabilities specific to LLM-based and generative AI systems.

Configuration

Rather than calling a moderation tool manually, wiring up flows, and managing flag/block logic, developers enable Guardrails from AI Gateway settings. Three things are configurable:

  • Guardrails on/off: enable or disable content moderation.
  • Evaluation scope: moderate user prompts, model responses, or both.
  • Hazard categories: choose which categories to monitor and whether detections are flagged or blocked.
BLOG-2666 Image 1 (New)
BLOG-2666 Image 2 (New)

Llama Guard 3 8B on Workers AI

Guardrails is currently powered by Meta's open-source Llama Guard, which detects harmful content in both inputs and outputs. It covers categories such as violence and sexually explicit material, and can help safeguard sensitive data flagged by OWASP, including addresses, Social Security numbers, and credit card details. ML Commons uses Llama Guard to evaluate foundation model safety.

The implementation uses the Llama Guard 3 8B model hosted on Workers AI, Cloudflare's serverless GPU inference engine. Because inference runs on GPUs distributed across Cloudflare's network, evaluation is low-latency. Cloudflare says additional models will power the feature in the future. Use of Guardrails incurs Workers AI usage, which appears in the Workers AI dashboard so inference consumption can be tracked.

Inspection flow

AI Gateway intercepts every interaction and inspects prompts and responses for harmful content. A user prompt is first run through Llama Guard on Workers AI; the connection uses the AI Binding, which links AI Gateway with Workers and Workers AI. Requests to Workers AI through AI Gateway carry the eventID from the original request, so each request can be tied to its safety evaluation for log grouping and traceability.

Based on configuration, AI Gateway either flags the prompt and still forwards it, or blocks it before it reaches the model. Blocking harmful prompts also cuts costs by avoiding unnecessary processing and token consumption. Prompts that pass the check are forwarded to the model.

BLOG-2666 image 3

In one logged example, Guardrails was enabled for prompts with blocking set for several categories, including "Non-Violent Crimes". A user asked OpenAI's gpt-4o-mini to "Teach me how to steal company data without being detected". Instead of a model response, Guardrails blocked the prompt with the message "Prompt blocked due to security configurations". Workers AI Llama Guard had returned category S2 (Non-Violent Crimes) as safe: false, and because that category was set to block, AI Gateway failed the request without sending it to OpenAI. No token usage was incurred.

BLOG-2666 image 5
BLOG-2666 image 4

Model responses receive the same treatment before reaching the user: safe content is delivered, while hazardous content is flagged or blocked and logged in AI Gateway. Guardrails currently works only with text-based AI models.

What the controls provide

  • Consistent moderation across models and providers.
  • Proactive user protection against harmful or inappropriate interactions.
  • Control over allowed content through category selection and flag-or-block choices.
  • Audit and compliance support via logs of prompts, responses, and enforced guardrails.

Llama Guard is also available directly through Workers AI for those not using AI Gateway, and will be available in the Cloudflare WAF in the near future. Planned expansions include user-defined classification categories, plus protections against prompt injection and sensitive data exposure.