Automating the Speed Page
The public YJIT benchmarks page is the visible part of a fully automated operation pipeline. Benchmarks run twice a day on a controlled AWS instance, results are processed and committed back to a GitHub Pages site, and the only time a human hears about it is when something fails.
The setup combines yjit-metrics, Google Sheets for tracking experiment configurations, and custom Ruby code that does the heavy lifting of running benchmarks, generating reports, and checking results against performance thresholds.
The cold numbers at the top are computed from a handful of public benchmarks including liquid-render, railsbench, and erubi-rails among others, all run against a baseline CRuby build. The current results, as seen in the pushed commit data, show YJIT delivering roughly 35-50% speedups on the headline benchmarks, with somewhat smaller gains on the broader suite.
Choosing the Toolchain
Running benchmarks in a continuous integration service like GitHub Actions is a bad idea. The shared hardware means noisy results, and reproducing microarchitectural differences becomes nearly impossible. Instead, the scheduler runs on GitHub Actions, but the actual benchmark execution happens over SSH on a fixed EC2 instance.
That split brings its own friction. The runner needs an SSH key, which has to end with a newline when stored as a GitHub secret or it will silently fail to parse. It needs -oStrictHostKeyChecking=no because AWS recycles IPs and GitHub Actions keeps a shared known_hosts file, so the host key from a previous session may no longer match. Environment variables don't carry over SSH by default either, so secrets are interpolated directly into the ssh command line.
The team's scheduled workflow shows the full pattern: checkout the repo, set up whatever dependencies are needed, generate a fresh benchmark cycle on the remote instance, pull the reports back, then commit and push the updated site.
Jekyll as a Reporting Engine
Publishing via GitHub Pages normally means using Jekyll, which is convenient until you want fine-grained control over the output. To customize the theme's CSS, such as replacing the markdown processor's <b> handling with <strong>, you copy the theme's layout files into the repo and modify them directly.
The trickier part is handling benchmarks as data objects. Jekyll collections are the tool for outputting each benchmark run as its own page URL, so both the front page timeline and an individual detail page can reference the same sets of graphs. The project's continuous_reporting code, called from a scheduled action, orchestrates the whole pipeline. That script knows which Jekyll collection directory to copy the new benchmark JSON files into before triggering a build. In that same script, it can compare current numbers with historical medians cached from the prior commit, and file a GitHub issue if a benchmark's performance drops below a threshold value.
Graphs Twice a Day
The reporting layer is pure Ruby. For the timeline graph on the front page, one of the report templates generates a static page with embedded D3 and JSON data. On a tool like D3, laying out axes for a small graph of large numbers works better than manually positioning ticks and labels in raw SVG. D3 also allows adding features that are simple to implement, such as showing a tooltip when the mouse passes over a time slot.
For the individual reports, each benchmark run stores pre-rendered SVG. Creating these inside a template is more verbose and doesn't allow for dynamic JavaScript without embedding an iframe. However, it buys portability and stability when someone wants to hotlink a particular graph into an email or an external writeup. Raw SVG requires styling all elements inline, since it will be included in an unknown page context with unknown CSS.
The current benchmark suite runs twice daily. The scheduled workflow uses cron syntax to define the trigger times. Each run creates a new report directory, copies fresh JSON performance measurements into the Jekyll collection data folder, regenerates the SVG report images, and re-renders the headline summary from an ERB template. That template produces the large summary table at the top of the front page.
Reports are not limited to graphs. The headline text paragraph is itself a generated report, an ERB template that summarizes the same underlying benchmark results. The drop detector reuses the per-benchmark data as a tripwire report, so on every run the same code that fabricates the human-oriented summaries checks the performance numbers and decides whether to automatically file a regression report as a GitHub issue on the repository.
For what the fixed weekly runs report, the callback path in the benchmark_and_update script handles the threshold checking after committing the collected data, so no manual intervention is required unless a benchmark regresses outside the tolerable range.
What This Setup Buys You
GitHub Actions and GitHub Pages cover most of the operational overhead for a batch-updated dynamic site. There are a few quirks to work around — copying code that already works helps more than usual here — but the combination handles scheduling, builds, and hosting without you running a server.
On the performance side, YJIT delivers. It’s worth keeping an eye on for anything Ruby throughput matters to, and the benchmark harness built for this experiment is a practical way to measure it.
The charts do most of the talking. Performance work without visualization is guesswork, and the ability to render results next to each commit makes regressions obvious before they ship.
