Roast: A framework for putting AI agents on rails

Shopify's Augmented Engineering DX team has spent the past year exploring how AI agents can tackle developer productivity problems like flaky tests and low test coverage. What they found is that unstructured AI agents—left to roam freely across millions of lines of code—are unreliable. Non-determinism is the enemy of reliability. The solution, now open-sourced as Roast, is a workflow orchestration framework that interleaves non-deterministic AI behavior with normal, deterministic code execution.

Roast is implemented in Ruby but works as a command-line tool with any programming language. It's convention-oriented, following the same philosophy as Ruby on Rails: create a workflow.yml file and corresponding prompt files, and you're ready to go.

Workflows are version-controllable, testable, and can be integrated into existing development pipelines. The name comes from the team's initial use case—"roasting" your tests to find areas for improvement—and the project has already attracted contributions from a dozen engineers on GitHub.

Step formats: One syntax, many styles

Roast interprets step formats based on their structure, giving you flexibility without configuration overhead.

For directory-based steps—the most common type—create a directory named after the step containing a prompt.md file:

Prompt files support ERB templating, giving you access to workflow context inside the prompt:

Shell commands wrapped in $() execute and capture output as command execution steps:

Any string containing spaces becomes an inline prompt to the step's AI model. Prefix with a ^ character to direct the prompt at Roast's built-in Coding Agent tool, powered by Claude Code:

For complex logic that doesn't fit a prompt, create custom Ruby step classes inheriting from BaseStep:

Nested arrays run their steps concurrently, enabling parallel execution within a workflow:

Built-in tools

Roast ships with a toolkit covering common AI workflow needs:

  • ReadFile: Reads file contents with line numbers
  • WriteFile: Writes content to files with security restrictions
  • UpdateFiles: Applies diffs/patches to multiple files
  • Grep: Searches file contents using regex patterns
  • SearchFile: Advanced file search with glob patterns
  • Cmd: Executes shell commands with configurable restrictions
  • Bash: Unix command execution with error handling
  • CodingAgent: Integrates Claude Code directly into workflows

The CodingAgent tool is the differentiator. It provides full Claude Code integration, bringing adaptive, agentic capabilities into structured workflows. This hybrid approach gives you deterministic guardrails with autonomous iteration inside those boundaries. The CodingAgent handles tasks requiring multiple attempts: complex code modifications, bug fixing with exploration, iterative performance optimization, and test generation that adapts based on coverage feedback.

In Shopify's workflow for adding Sorbet types (dubbed "Boba"), deterministic steps clean up code and run Sorbet's autocorrect first. Remaining issues are handed to the CodingAgent, which iteratively fixes type errors, runs tests, and ensures everything passes—a task nearly impossible with pure automation.

Shared context and control flow

Steps in a workflow share their conversation transcript, so later steps can reference and build on earlier discoveries without explicit data wiring from the workflow author:

Beyond sequential execution, Roast supports iteration over collections, conditional execution, and case statements for multi-branch logic:

Session replay and the developer experience

Every workflow execution is automatically saved, enabling replay from any step:

This is a significant productivity feature for workflow development—no more rerunning expensive AI operations repeatedly while debugging a pipeline.

Example: Grading a Ruby unit test

In practice, Roast workflows resemble a senior engineer reviewing tests. The team's test grading workflow demonstrates the full flow:

Executing the workflow:

Produces a comprehensive assessment report:

Getting started

Roast is distributed as a Ruby gem:

Prerequisites are Ruby 3.0+, an OpenAI API key (or OpenRouter for other models), and optionally shadowenv and ripgrep for enhanced functionality.

Built on Raix

Under the hood, Roast leverages Raix, a Ruby AI extensions library providing an abstraction layer across AI providers. Raix adds retry logic, response caching, and structured output handling. Behavior is customizable through initializers placed in your .roast/initializers directory:

This opens up fine-grained controls including custom authentication schemes, retry strategies, response logging, provider-specific configuration, and token usage tracking.

Real-world adoption at Shopify

Internal deployment has produced several notable use cases. Engineers have analyzed thousands of test files, automatically identifying antipatterns and increasing coverage. The Boba workflow adds Sorbet type annotations to test files—performing cleanup with sed, bumping tests to strict typing, running autocorrect, and delegating residual issues to the coding agent. Running Boba fully typed a test file that previously lacked annotations while ensuring both tests and type checking passed.

The SRE team built a workflow that periodically scans internal Slack channels for early indicators of emerging issues, analyzing conversation patterns and alerting appropriate teams. Another workflow aggregates competitive intelligence from news sources, migration data, and CRM-derived market trends into actionable reports.

A "Chesterton's Fence" research tool addresses context loss in codebases. When developers encounter puzzling code, the workflow investigates commit history and associated PRs, then explains why specific lines exist—preventing removal of code that looks unnecessary but is actually critical.

Structured execution is the next step for AI in development

Sam Schmidt, an early Roast user, put his finger on what makes the tool different from typical LLM integrations: you can sketch a step you do not fully understand yet with an AI approximation that mostly works, and later swap it out for a deterministic implementation once the problem space is clearer. That inversion of the normal development cycle matters. Instead of requiring complete understanding before automation, Roast lets engineers use AI as a placeholder that keeps the pipeline moving, then replace it with reliable code as knowledge grows.

I can handwave a step I don't quite know how to do yet with an AI approximation that mostly works. As I understand the problem space better, it's very easy to drop the AI step for a deterministic one that always works. But the AI step accelerates my understanding by letting me edge around harder problems until I have a better sense of what's needed.

That workflow acts like a junior developer absorbing the parts you have not solved, letting you prototype an entire automation before committing to a concrete implementation. It is a faster path to complex systems because the expensive part—figuring out exactly what each step must do—happens while the workflow is already running.

What structured workflows enable

Roast is built around the idea that AI workflows belong in the same category as any other code artifact. That means version control, testing and review apply to them, rather than treating AI as an ad-hoc chat interaction. The practical consequences stack up:

  • AI workflows become first-class citizens in the development process, version-controlled and tested like any other code
  • Hybrid workflows seamlessly blend deterministic steps with AI-powered analysis
  • Workflow marketplaces emerge where teams share battle-tested workflows for common tasks
  • AI becomes predictable and reliable when it needs to be through structured execution rather than free-form generation

The ambition is that Roast workflows become as standard as GitHub Actions or Jenkins pipelines in a team's toolkit. Schmidt's week-long trial suggests that is plausible:

On an overall note, I've spent the last week building workflows with Roast, and I am very impressed. This is an excellent workflow automation tool that also makes it easy to integrate LLM steps. Being able to trivially combine a simple deterministic sed with a high powered "do something like this" agent, and jump back and forth between the tools to ensure each one is crafting the input to the next step to give that tool the best chance of success is absurdly powerful. I feel like I'm only just scratching the surface of what I might be able to build with this.

The key detail in that feedback is the back-and-forth between tools: a deterministic step can shape input for an AI step, which in turn produces something a deterministic step can refine. Each segment gives the next the best possible chance of success. Roast is positioned as more than a library or a prompt wrapper—it is a different way to structure AI-assisted engineering, where the workflow itself is the product.