Reliability is the last manual phase of software delivery
Over the last decade, engineering-led practices have pushed manual IT operations out of the software development lifecycle. Build, test, and deployment pipelines are automated; code review and CI/CD are standard. But one area remains stubbornly manual: operating software in production.
When a monitoring system fires an alert in even the most advanced organizations, what follows is still largely a by-hand process: triaging the alert, correlating it with other signals, diagnosing the root cause, remediating, and then coordinating with stakeholders throughout. This is true despite the fact that, for every other phase of the lifecycle, a mostly manual process has already been replaced with repeatable, code-driven automation.
The missing piece is the ability to define operational runbooks as code—enrichment steps, diagnostic logic, remediation procedures—so they can be managed with the same rigor as application code.
Operational processes with buildable logic
StackPulse treats the response to an alert as a software artifact. Rather than a checklist, each step of an operational process is an executable function: it receives arguments and performs a focused action. The logic connecting those steps is built into the code itself, not left to the judgment of the on-call engineer in the middle of an incident.
Consider the common case of an alert for a sluggish database server. The resulting process might follow these steps:
- Identify which database server generated the alert.
- Determine whether all requests are slow or only certain types, grouping by source, data operation, or other attributes.
- Flag which dependent services and business processes violate their response expectations as a result.
Such steps become a concrete, versionable expression of the process:
apiVersion: stackpulse.io/v1
kind: Playbook
metadata:
name: enrich-and-diagnose-database-alerts
description: This playbook enriches alerts related to database services and tries to identify most common sources of possible problems
parameters:
- name: database_server
type: var
description: Address of the database server
steps:
# Retrieve general information about the database server
- name: stackpulse/general_db_info
id: get_db_info
env:
DATABASE: '{{ $.params.database_server }}'
AUTH: '{{ secret "database_access_key" }}'
# Get a breakdown of database requests latency for the past 30 minutes
- name: stackpulse/db_requests_latency_analysis
id: get_db_latency
env:
DATABASE: '{{ $.params.database_server }}'
AUTH: '{{ secret "database_access_key" }}'
TIME: "30m
* * *That sample represents an enrichment phase, but the same principle extends to diagnosis and remediation. The strict requirements during the design of the platform are that authoring the logic must be straightforward, that conditional branching between steps is possible (so real multi-step workflows are representable), and that the resulting playbook is portable across environments.
Designing to those constraints yields a useful property: operational processes are now a special kind of software. And like all software, they can be modular, generic, and equipped with clear interfaces between units.
What happens when runbooks live in a repo
Once a runbook is code, the tooling a team already relies on for application delivery can manage it. Each GitHub repository containing a service's business logic can hold the enrichment, diagnostic, and remediation logic for that service as well. Those playbooks can lean on shared infrastructure or be entirely service-specific.
Standard Git workflow rules then apply to reliability logic:
- Proposed changes happen on branches.
- Promoting a playbook to production occurs through pull requests and GitHub Actions pipelines.
- Version history of the process is tracked by the repository.
- Change requests arrive as issues; the pace of change is measurable via Insights.
These processes do not have to be locked inside a single organization. Playbooks for generic or open-source components can live in public repositories, letting operational expertise spread across the industry, not just along a single team's wiki page. Teams can take the “you build it, you run it” principle one step further: incorporating reliability conditions and SLAs into a GitOps flow rather than treating incidents as a break in that flow.
An example configuration with token demonstrates the syntax: id, on if conditions, and a series of steps:. The process is defined in the GitHub Action’s yml file, offering reusable workflows that support both conditional and parallel execution. The context is explicitly set in the [StackPulse public repository](https://github.com/stackpulse/playbooks), the GitHub Action is published on the [Actions Marketplace](https://github.com/marketplace/actions/stackpulse-apply-playbook), and testing an implementation begins with the platform's [free edition](https://stackpulse.com/get-started/).
Putting operational behavior under the same code review, environment promotion, and auditing that governs the software it supports turns reliability into a CI/CD partner rather than a manual phase that follows deployment.
