Replicated’s Cloud-Hosted Development Environment

Replicated, a five-year-old infrastructure software company focused on Kubernetes Off-The-Shelf (KOTS) software delivery, has moved its entire development environment into the cloud. The 22-person engineering team now runs full-stack development instances on Google Cloud, secured and accessed through Cloudflare’s Argo Tunnel and Access service. The result: developers no longer wrestle with local Kubernetes clusters, and code no longer lives on laptops.

Why Local Development Stopped Working

Replicated’s product, KOTS, runs inside Kubernetes and manages the lifecycle of third-party applications within a cluster. Building and validating the software requires access to a cluster. Early on, the team standardized on Docker for Mac with its built-in Kubernetes distribution, running the entire stack on powerful MacBook Pros with up to 64 GB of RAM.

That approach created recurring problems. Battery life suffered from constant CPU usage, and Docker for Mac’s bundled Kubernetes didn’t always behave like real clusters. Worse, the local K8s distribution would occasionally stop working entirely, forcing a full uninstall and reinstall. Engineers lost hours each week troubleshooting, and when frontend developers—who weren’t expected to be Kubernetes experts—ran into issues, they had to pair with backend engineers, costing two people their time simultaneously.

Moving the Stack to IaaS

Replicated’s solution replaces local Docker with per-developer Google Cloud instances. These instances are created from a machine image with all prerequisites installed, including a Kubernetes distribution local to the server and a Docker registry running as a cluster add-on. Significantly, the servers have no public IP addresses and no open ports in the Google Load Balancer—not even for SSH.

The key component is cloudflared, which runs on each server and powers Argo Tunnel. When it starts, cloudflared establishes four secure HTTP/2 tunnels to two Cloudflare data centers. Incoming requests for a development machine are routed through Cloudflare directly to the instance running that environment. This works across any cloud provider and in any region, with no configuration changes needed when spinning up a new environment. A developer’s hostname, like marc.repl.dev, is reachable securely from anywhere in the world.

Access is controlled through Cloudflare Access, which provides BeyondCorp-style authentication. Policies define which machines each user can reach, enforced uniformly across the infrastructure. Developers log in with the same Google credentials used elsewhere in the company, and access is revoked instantly if someone leaves—eliminating the need to manage long-lived SSH certificates or worry about lingering public keys.

The Developer Experience

Developers connect to their remote instances using the Remote SSH extension for VS Code. This extension allows VS Code to read and write files on the remote filesystem, making the experience feel local. Authentication is handled by Access, and all traffic is proxied through Argo over SSH, so the environment remains accessible and fast from anywhere with an internet connection.

Locally, each developer maintains an .ssh/config file containing port-forwarding rules. When a developer opens the embedded VS Code terminal, it connects to the remote server. A skaffold dev command starts the full stack on the cloud instance. The developer can then browse to http://localhost:8080 locally, with traffic forwarded via SSH to port 30080 on the remote server—a static NodePort exposing the web server running in the cluster. All APIs and web servers use static NodePorts for local development environments.

New hires now go from zero to a working environment in under an hour, and a Terraform script can replace any developer’s machine in seconds.

Benefits After Nine Months

Replicated has used this setup for nine months. Problems with Kubernetes behavior and Docker disk usage still arise, but much less frequently than with Docker for Mac. The cloud-based architecture has also unlocked capabilities that weren’t practical with local environments:

  • Collaborative debugging: Backend engineers can SSH directly into a colleague’s instance through the Argo Tunnel to help troubleshoot, even when they’re not in the same room.
  • Disposable environments: If a server misbehaves for unknown reasons, a developer can delete it and start fresh rather than spending hours investigating.
  • Multiple environments: Developers can easily spin up additional instances to test against different Kubernetes versions.
  • Improved hardware needs: Laptop battery life is restored, and the team no longer requires top-of-the-line machines—even Chromebooks and tablets work, since SSH is the only local dependency.
  • OS flexibility: Developers can work on macOS, Windows, or Linux.
  • Security: Code never resides on a laptop, so losing a device no longer risks exposing the codebase.

Reproducing the Setup

Replicated shared the basic steps to replicate this architecture, assuming a domain already configured with Cloudflare. The company also published a detailed gist for team-ready configuration.

  1. Create an instance in your cloud of choice to act as the development environment.
  2. Configure cloudflared to run at instance startup with a hostname such as dev.mysite.com.
  3. Write an Access policy to limit access to yourself.
  4. Configure your local machine to route SSH through Cloudflare.
  5. Install VS Code and the Remote Development extension pack.
  6. In VS Code, select “Remote-SSH: Connect to Host…” and enter the hostname, e.g., [email protected]. You’ll be prompted to authenticate in a browser with your configured identity provider.
  7. You’re done: files and terminal sessions operate on the remote machine.

Cloudflare has made its Access and Teams products free for small businesses through September 1, 2020, in response to pandemic-driven work-from-home policies. More information is available at cloudflare.com/smallbusiness.