Why Slack finally moved off Jenkins
Slack’s CI infrastructure spent years running on Jenkins, but developer frustration kept mounting: security issues, downtime, and poor UX all contributed. That pressure eventually pushed the team to migrate the majority of its CI jobs to GitHub Actions. The migration presented an obvious automation opportunity, and the result was a conversion tool that is expected to cut migration effort in half and save more than 1,300 hours.
Before diving into the tool itself, a few terms are worth defining:
- Pipeline: A set of Jenkins plugins used to organize Jenkins jobs; in Slack’s case, a pipeline is one or more ordered Jenkins jobs. At the project’s start, 242 pipelines needed migration.
- Instance: A Jenkins host; Slack was migrating off three instances.
- Workflow: A GitHub Actions CI job.
- GHA: GitHub Actions.
Defining success and choosing the toolchain
The project had three concrete goals: build tooling that automates the hardest parts of a manual Jenkins-to-GHA conversion, run that tooling against a large subset of existing Jenkins jobs, and produce documentation that captures the conversion process.
Early research surfaced one primary tool plus a fallback strategy:
- GitHub Actions Importer: GitHub’s official tool can audit a Jenkins instance and convert, partially convert, or fail to convert each pipeline. Most Jenkins pipelines convert cleanly, but not all do, so additional tooling is needed for the failures.
- Python scripting: Many Importer errors are trivial and can be handled with regex-based fixes. More intricate errors were earmarked for large language models (LLMs).
Why Only Some Conversions Were Successful
When we ran GitHub's Importer across several Jenkins instances, the audit summaries it produced held the first real surprises. Roughly half of the Jenkins pipelines converted to GitHub Actions workflows without any errors. Another 5% failed entirely, and the remaining jobs came through as partial conversions — the Importer generated a workflow YAML file but couldn't populate it completely.
The audit summaries also revealed which Jenkins build steps and environments the Importer couldn't handle. At first glance, hundreds of unconverted pipelines looked like a serious problem. But digging into the audit data showed something far more manageable: just eight unsupported Jenkins build steps and environments accounted for over 90% of the partial conversions.
This narrowed the scope considerably. Instead of trying to address dozens of unsupported steps, we needed to answer two questions for only eight items:
- Is this actually unsupported by GHA, or do working equivalents exist?
- If equivalents exist, can we write scripts to update the Importer's outputs?
Four Categories of Corrections
Reviewing dozens of the Importer's YAML outputs revealed that nearly every workflow needed some form of correction, even the ones that "fully converted." Four types of issues recurred:
- Rate-limited actions needed replacing with internal mirrors
- Some actions needed replacing with different, preferred actions
- Useful comments for end users needed to be added
- Unnecessary comments needed to be removed
Handling Rate-Limited Actions
Actions pulled from the GitHub Marketplace, like actions/github-script, make API requests that hit github.com's rate limits. Actions that target our self-hosted GitHub Enterprise instance, however, have substantially higher limits. Replacing rate-limited marketplace actions with internal mirrors of those actions was straightforward and prevented end users from hitting frustrating job failures due to rate limiting. That gave this correction a high return on investment.
Swapping Actions for Preferred Alternatives
In some cases, the Importer chose an action we wouldn't have picked ourselves. For instance, it frequently selected rtCamp/action-slack-notify for sending Slack messages when Slack's own internal messaging action would be better. Converting between two actions with different syntax is difficult using Python string methods alone — which made this an appealing candidate for LLM-based correction. Initial testing showed the LLM performed perfectly on these conversions, and the value to end users was clear: they'd otherwise need to know both actions well to make such a swap themselves.
Adding Directional Comments
Some workflow items simply can't be automated. A good example: the Importer's generated workflows assumed secrets would be stored on GitHub, but our organization doesn't store secrets there — and moving them would violate security policies and take too long. End users are the only ones who know where their secrets live and how to access them. For these cases, the best the correction tool could do was leave a comment telling users which lines to change and pointing them to relevant documentation in a Slack channel. Useful, yes, but still requiring manual work from the end user. Medium return on investment.
Removing Clutter Comments
When the Importer partially converted a pipeline, it left comments like "X Jenkins item was unsupported." But examining these failures showed that many of those Jenkins features were either already built into GHA or simply unnecessary. That made most of the unsupported-item comments worthless noise. Removing them only declutters the workflow files — a small benefit — but since writing a string-deletion script is trivial, the effort-to-value ratio was still decent. Low-medium returns.
The Corrections Tool
The corrections tool that automated all of this work was mostly non-AI. Of the four correction categories, only the action replacement task needed LLM assistance. Building the tool took about three weeks.
The architecture was simple. The tool took a path to a directory containing workflow YAML files, then:
- Wrote the path of each YAML file in that directory to an array
- For each path, read the file contents, applied every relevant non-AI correction, wrote the changes back, then applied every relevant AI correction and wrote those changes back
The result: every YAML file in the input directory was edited in place.
Non-AI: String Replacements
Most of what needed correcting came down to adding a comment after a line containing a certain string, or replacing one string with another. Python's String.replace() method handled these cases extensively.
AI: Prompt Engineering for Action Swaps
Prompt engineering turned out to be both science and art. The science part is a repeatable formula:
- Structure the prompt with three sections: context setting, specific steps, and output instructions
- In the context-setting section, include relevant syntax and documentation
The steps should be granular — one instruction per specific task. A prompt that's too long is far safer than one that's too vague. The art comes from iteration: start with a basic prompt, observe the LLM's response, then add detail and context to fix any errors it made.
A representative prompt for the Slack action swap looked like this:
I'm going to provide you with a GitHub Actions workflow yaml file. In the file an action called rtCamp/[email protected] is being used. Your job is to replace every rtCamp/[email protected] action with the slack/[email protected] action, and output the resulting workflow yaml file. For reference, I will provide you with the syntax for the slack/[email protected] action. I'll put it between <syntax></syntax> XML tags:
Now, perform the following steps:
- Find every instance of the rtCamp/[email protected] action being used.
- Replace each instance with the slack/[email protected] action with the aforementioned syntax.
- Replace channel-name with the value of the SLACK_CHANNEL field in the rtCamp/[email protected] action.
- Replace text-message with the value of the SLACK_MESSAGE field in the rtCamp/[email protected] action.
Also, there are a few things you should absolutely never do:
- Do not remove any comments from the file
- Do not alter any other action in the file besides rtCamp/[email protected]
I will now provide you with the workflow file you will be making edits to between <workflow></workflow> XML tags. When providing your output, only provide the contents of a YAML file. Do not describe what you did. Do not provide any justifications for your decisions. Only provide the corrected GitHub Actions yaml file, with nothing else before or after the yaml file in your output. Now, please perform the instructions I gave you:
Checking roughly two dozen of the LLM's outputs showed 100% accuracy on these replacement tasks. Hallucinations and unintended side effects were the main worry going in, but the LLM performed every bit as well as a human on this narrow "replace A with B" work. For a category of errors that Python's string methods alone couldn't handle, that was exactly the gap the LLM needed to fill.
What the Migration Saves
Before the conversion tool existed, moving a pipeline from Jenkins to GitHub Actions meant four steps, in descending order of effort:
- Write the GHA workflow YAML from scratch, using the Jenkins pipeline as reference.
- Test and debug the workflow.
- Document the workflow.
- Delete the pipeline from Jenkins.
The tool removes step 1 entirely and makes step 2 far less painful, since users start with workflow files that are largely correct.
The team attempted conversions on roughly 242 pipelines. Manual conversion time varies by experience level, with estimates as follows:
- Experienced with Jenkins and GHA: 2 hours per pipeline.
- Experienced with Jenkins only: 5 hours, since GHA syntax must be learned.
- New to both: 10 hours, including time spent getting help from others.
Assuming 10% of users are in the first group, 40% in the second, and 50% in the third, a fully manual migration would take roughly 1,700 hours. With the conversion tool, half the pipelines run without any edits — a realistic outcome given that half convert completely — and the remaining half require about 3 hours of debugging each. That puts the total at roughly 360 hours.
The projected savings come to over 1,300 hours, or about 80% of the time a full manual migration would have consumed.
Feedback and Lessons Learned
Two weeks after the tool's outputs were published, feedback from developers at Slack shows they find it useful, though some generated workflows do contain errors. That was expected from the start; the concern was how to triage issues efficiently.
That worry proved unfounded. A Canvas doc was set up for the conversion effort, and every problem end users reported was logged there with details and solutions. As more developers used the tool, the doc grew more comprehensive, and triage stayed straightforward.
The standout takeaway from this project is how much easy communication and collaboration matter. Slack's tooling made it simple to share updates, find the right person for help, and keep organized notes — all of which made the internship more productive and enjoyable.
Special thanks go to Jerry Shen, Buddhadev Veeramallu, Ellen Wong, Shane Gearon, Sergii Gorbachov, Catherine Li, and Nick Matute for their contributions.
