Day 21: wrangling systemd & setting up git deploys to a VM

A Rails VM with Push-to-Deploy: Notes From the Trenches

After hitting friction with GCP's IAM, I pivoted to DigitalOcean for an incidents-as-a-service Rails project. The goal was a simple VM deployment without sacrificing the conveniences of a PaaS like Heroku. Here's what I learned while getting that set up.

DigitalOcean's App Platform Is Heroku-Compatible—with Limits

DigitalOcean's app platform is remarkably easy to get started with and can deploy Heroku apps almost seamlessly. However, it ultimately wouldn't work for my use case because it appears to run apps inside containers behind a restrictive firewall—you can't SSH out to other instances from within an app. That forced me to look at a plain VM instead.

Useful Pre-Built VM Images

DigitalOcean's marketplace includes a pre-configured Ruby on Rails image. Creating a droplet from that gave me a VM with nginx, Rails, and Postgres all installed, plus systemd services wired up for each. Having Postgres already running on the machine is a significant cost saver relative to managed database options, and it can always be swapped for a managed service later.

Push-to-Deploy via a Git Post-Receive Hook

To recover the "push to deploy" flow I was used to, I implemented a git post-receive hook on the VM. The setup is more of a hacked-together bash script than a polished deployment system, but after some wrestling with systemd it worked reliably. There's something appealing about this simpler architecture where git push targets the machine running your code directly.

Running Setup Tasks Before Rails Starts

Once new code arrives via git push, I need to install new gems, run Rails migrations, and perform similar maintenance before restarting the app. Instead of orchestrating that externally, I set an ExecStartPre directive in systemd. Now, running service rails restart triggers a custom restart.sh script first, handling any pending gem installs or other setup steps:

[Service]
Type=simple
User=rails
Group=rails
WorkingDirectory=/my/rails/directory
ExecStart=/bin/bash -lc 'bundle exec puma'
Environment=RAILS_ENV=production
ExecStartPre=/bin/bash -x /path/to/my/restart/script
TimeoutSec=300s
RestartSec=300s
Restart=always

Note that TimeoutSec needed to be increased because bundle install inside that pre-start script can occasionally take a while.

Debugging systemd-journald

The most frustrating part of the setup was when systemd-journald didn't capture my Rails app's logs, despite the service running normally. Restarting journald eventually fixed the problem, though the root cause remains unclear—systemd intricacies remain a learning curve to tackle another day.

Final Thoughts

There was a certain enjoyment in standing up a VM with a decidedly non-reproducible build and deploy process. If this project ever matures into something I need to maintain, containers or a more structured deployment approach would be worth revisiting. For now, the hacked-together pipeline is working fine.