Bringing a Server Under NixOS Control

After years of ad-hoc server administration, one engineer decided to experiment with NixOS as a solution to the problem of undocumented, unreproducible server state. The author's previous Ansible setup worked, but manual changes made outside of it meant the server's actual state was unknown. NixOS offered the possibility of a fully declarative system — one that could be rebuilt from scratch at any time. Here’s a concise walkthrough of the setup process.

A Different Philosophy: NixOS vs. Ansible

The fundamental difference is that NixOS is the operating system. It manages users, services, and packages, giving it full control to enforce a desired state. With Ansible, only the explicitly managed components are declared; anything else can drift. In contrast, any manual tinkering on a NixOS machine would be forcefully reverted on the next nixos-rebuild run. This eliminates the "chaotic ad-hoc" problem that motivated the change.

Migration Path: From Ubuntu to NixOS on Hetzner

The installation didn't involve booting from a NixOS ISO. Instead, the author created a fresh Hetzner server running Ubuntu and used the nixos-infect script to convert the running operating system in place. The command was:

curl https://raw.githubusercontent.com/elitak/nixos-infect/master/nixos-infect | PROVIDER=hetznercloud NIX_CHANNEL=nixos-23.11 bash 2>&1 | tee /tmp/infect.log

Attempting the same procedure on DigitalOcean failed for unknown reasons. While the NixOS wiki suggests other hosters and methods, this one worked. The author notes that using an official ISO is probably the more robust path, avoiding the risk of unintended issues from the "transmogrification" process. The README strongly advises reading the nixos-infect script first; this step was skipped, but the risk was deemed acceptable since the server was disposable.

After the conversion, the generated system configuration was pulled down to a local git repository:

scp root@SERVER_IP:/etc/nixos/* .

This action copied three key files: configuration.nix (the main configuration), hardware-configuration.nix, and networking.nix. The author left the latter two files untouched.

Managing Configuration with Flakes

The next step was to wrap configuration.nix in a Nix flake. The author admits the reasons for using flakes aren't fully clear to them, but it proved to be a working setup. Here is the flake.nix:

{ inputs.nixpkgs.url = "github:NixOS/nixpkgs/23.11";

  outputs = { nixpkgs, ... }: {
    nixosConfigurations.default = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";

      modules = [ ./configuration.nix ];
    };
  };
}

Working with flakes introduced a key gotcha: every .nix file must be staged with git add before Nix will acknowledge its existence. The rules felt counterintuitive:

  • git add is mandatory.
  • Committing changes is not required.
  • Staged files can have unstaged changes.

This behavior likely stems from an optimization where Nix only copies staged files into its store, but the author finds the logic odd and points to a GitHub issue where it’s being tracked.

Deploying with nixos-rebuild

Rather than adopting a third-party deployment tool, the author found that the built-in nixos-rebuild command has --target-host and --build-host options, allowing for remote deployments. The deployment was handled by a simple bash script:

nixos-rebuild switch --fast --flake .#default --target-host my-server --build-host my-server --option eval-cache false

This particular script sets both --target-host and --build-host to the same (single) server, which works fine for a low-stakes personal setup. The --option eval-cache false flag was a crucial addition. Without it, Nix would frequently offer a generic error: cached failure of attribute 'nixosConfigurations.default.config.system.build.toplevel' instead of the useful, underlying error message.

To build Go repositories directly on the server, SSH agent forwarding was configured in the local ~/.ssh/config:

Host my-server
   Hostname MY_IP_HERE
   User root
   Port 22
   ForwardAgent yes

AddKeysToAgent yes

This allowed the server to authenticate with private Git repositories during the build process.

Defining a Go Service in a Single File

The most challenging task was figuring out how to compile and run a Go web service with all its configuration in a single, self-contained Nix file. The author resisted the common practice of separating the package definition from the service definition. The resulting my-service.nix (name anonymized) is shown below:

{ pkgs ? (import <nixpkgs> { }), lib, stdenv, ... }: 
let myservice = pkgs.callPackage pkgs.buildGoModule {
  name = "my-service";
  src = fetchGit {
    url = "[email protected]:jvns/my-service.git";
    rev = "efcc67c6b0abd90fb2bd92ef888e4bd9c5c50835"; # put the right git sha here
  };
  vendorHash = "sha256-b+mHu+7Fge4tPmBsp/D/p9SUQKKecijOLjfy9x5HyEE"; # nix will complain about this and tell you the right value
}; in { 
  services.caddy.virtualHosts."my-service.example.com".extraConfig = ''
    reverse_proxy localhost:8333
  '';

  systemd.services.my-service = {
    enable = true;
    description = "my-service";
    after = ["network.target"];
    wantedBy = ["multi-user.target"];
    script = "${myservice}/bin/my-service";
    environment = {
      DB_FILENAME = "/var/lib/my-service/db.sqlite";
    };
    serviceConfig = {
      DynamicUser = true;
      StateDirectory = "my-service"; # /var/lib/my-service
    };
  };
}

A few crucial things were noted about this configuration:

  1. It uses extraConfig for Caddy, which allows for standard Caddy syntax instead of learning Nix's specific format.
  2. The DynamicUser directive is used to create a system user for running the service. This simplifies the process of creating a unique user per service without managing UIDs/GIDs manually.
  3. The StateDirectory directive manages a persistent location for a SQLite database, placing it at /var/lib/my-service/.

The service was brought online by adding ./my-service.nix to the configuration.nix imports and enabling Caddy with services.caddy.enable = true;.

Why Caddy? The author switched from nginx for hobby projects because Caddy was able to automate Let's Encrypt certificate issuance. Its configuration language is also viewed as less complex.

Troubleshooting a Cryptic Error

One significant hurdle was a fetchTree requires a locked input error:

error: in pure evaluation mode, 'fetchTree' requires a locked input, at «none»:0

The message was baffling due to its obtuse phrasing. Debugging, with help from the Nix community, revealed several time-saving facts:

  1. The fetchGit function internally aliases to fetchTree, causing such confusing errors.
  2. Nix can truncate long stack traces, so more context might be available with the --show-trace flag.
  3. Error messages often lack line numbers for the calling code, even with trace flags.
  4. The --option eval-cache false workaround remains the most effective way to get the true error message instead of the cached failure notice.

Ultimately, the root cause was a simple oversight: the fetchGit call was missing its rev = "efcc67c6b0abd90fb2bd92ef888e4bd9c5c50835"; parameter. This kind of Nix syntax complexity remains a pain point for the author, who prefers to copy-paste working configurations over learning the language in depth.

Outstanding Questions and First Impressions

Even after a successful setup, some questions linger. When running nixos-rebuild, Nix verifies that systemd services are functioning, but the author is unsure of the scope—does it only check for a successful start, or does it monitor for continuous uptime? Also, the deployment workflow currently requires manually copying the Git SHA of each new revision, which likely isn't the most efficient solution.

The declarative single-file approach is a clear win over the previous Ansible drift. At this early stage in the author's week-long experiment, it looks promising. The tool remains difficult to debug, but the consolidation of server knowledge makes the struggle seem worthwhile.