From GUI Clicking to Reviewable Pull Requests

Every large application eventually accumulates a mix of code the team wrote internally and third-party services that must be configured externally. Services like AWS or Docker come to mind; for Shopify's Conversations team, the third-party piece was Twilio TaskRouter, a multi-channel, skill-based routing API that assigns tasks such as voice or chat to the appropriate agent based on routing rules and skills.

The problem with such services is not that they change frequently, but what happens when they do. There is no test suite, no rollback path, and no revision history when someone updates a configuration through a web GUI. For developers accustomed to reading commit messages, reviewing pull requests, and reverting bad changes, this is a gap in the engineering workflow. The Conversations team built a contact center on top of TaskRouter, and as the business grew, so did the routing complexity. The console became too difficult for any single person to fully understand, and configuration errors could not be traced or undone.

This is where Terraform came in. Terraform is an open-source infrastructure-as-code tool that acts as a state machine for your infrastructure. Instead of manual GUI changes, it lets you apply the same engineering practices you use for application code: version control, code review, and controlled rollbacks. It requires three components to function for any given service:

  • A reliable API: Terraform drives updates through the API, so anything that cannot be changed via API cannot be managed with Terraform.
  • A Go client library: Since Terraform is written in Go, it needs a Go library to make the HTTP(S) calls to the target application.
  • A Terraform provider: The provider, also written in Go using the Terraform Plugin SDK, tells core Terraform how to interact with the target API.

With those pieces in place, you can manage nearly any application as code.

Building a Custom Provider

When the Conversations team started the project, Twilio did not yet have a Terraform provider, so they built their own. (Twilio has since released an official provider.) A provider adds a set of resources that Terraform can manage via CRUD operations; it is not part of Terraform core, but a separate plug-in created for a specific application.

Image from: https://learn.hashicorp.com/img/terraform/providers/core-plugins-api.png<

The provider folder contains Go dependencies, a Makefile for common commands, a local development example, and a subdirectory for the provider itself. Within that directory, there is a resource file for every resource type, and each file contains the CRUD instructions plus an Importer function. The importer is especially useful when you already have running infrastructure and want to start managing it with Terraform; it enables Terraform to adopt existing resources.

Each resource also defines a schema, the set of parameters the API accepts for CRUD operations. For a TaskRouter activity, those parameters are friendly_name, available, and workspace_sid. The create function in a resource file is largely boilerplate: it receives context, a schema resource, and an empty interface; instantiates the Twilio API client; locates the workspace that activities belong to; formats the parameters according to the schema; and calls the library's create method. It then sets the resource id to the sid returned by the API, the unique identifier Twilio uses. Once that setID call runs, Terraform knows about the new resource and can subsequently manage it. You do not need to be a Go expert to write a provider; the SDK and documentation carry most of the weight.

Defining and Applying Resources

Once a provider exists, the user-facing side of Terraform is a straightforward DSL, simple enough that non-developers can safely modify infrastructure with some guidance. Defining a new TaskRouter activity, for example, is just a resource block that names the resource type and passes in the schema variables. Because activities live inside a workspace, the block also declares that dependency; Terraform will only attempt to create the activity after the workspace resource exists.

From there, the workflow is command-driven:

  1. terraform plan prints a text representation of every change Terraform will make. This diff exposes unintended modifications before they go live. If a same-named activity already exists, the plan output changes from a create to an update, requiring you to rename your resource block and re-run the plan.
  2. terraform apply executes the planned changes. Terraform then records the resource's generated ID and manages it exclusively from that point on.

The power of this workflow comes from integrating it with a code review tool. Shopify paired Terraform with Atlantis, a GitHub integration, so that new pull requests trigger a terraform plan and display the results as a comment on the PR. Collaborators review the proposed changes just as they would a code diff; once approved, the comment atlantis apply -p terraform makes the change and the PR is merged. Configurations now have a clear history that can be traced with git blame and rolled back with a revert.

Empowering the Whole Team

The most positive feedback came not from developers but from other teams. Business and support teams used to submit requests and wait for developers to adjust TaskRouter workflows; Terraform let them make those changes themselves, and some non-developers ended up authoring their first-ever pull requests. This freed up engineering time and gave the business teams more independence.

Managing third-party infrastructure is an inevitable part of building software. While a GUI works fine in the beginning, the lack of accountability, review, and testing becomes a burden as complexity grows. Terraform brings those missing pieces back, making infrastructure changes reviewable, predictable, and reversible – and opening a path for people who don't write application code to contribute safe changes on their own.